November 30, 2008 - December 6, 2008


"Outliers" (Including One About Outliers)

Topics: Outliers Feature

The Minneapolis Star Tribune lets you do your own Franken-Coleman recount (via Fairbanks). Seriously. This is well worth the click.

Gary Langer reviews the ABC News results on the automaker bailout.

Andrew Gellman explains the economic models that predict presidential elections (and then adds more).

Christopher Beam looks for evidence that we are a "center-right nation."

Carl Bialik reports on yet another John Ziegler survey.

Brietbart TV's B-Cast interviews Ziegler and his "fiercest critic" (via Silver).

PPP wraps up the polling on the Georgia runoff.

Mark Melman says watch wealthy Americans, voters under 30 and Latinos.

David Hill summarizes what the Republicans need and makes an RNC chair endorsement.

Chris Bowers gets his polling fix and counts himself among those disapproving Obama's performance.

David Pogue questions the ubiquitous Consumer Reports ratings graphic.

And finally, "Outliers!" The version that, unlike this one, fetches its author a six figure (or larger?) book deal.   

More on the "Convergence Mystery"

Topics: Convergence Mystery , David Moore

My colleague David Moore has posted here and on AAPOR's Survey Practice about the intriguing convergence of the national polls at the very end of this year's fall campaign (I made a similar observation in assessing the final round of national polls last month). I wondered whether the same phenomenon occurred at the state level. What follows is some preliminary data and my own answer to Moore's question about why this happens.

First, let's review Moore's data. His Survey Practice posting includes the following table, which presents both the average Obama lead and the variance in that lead on national surveys broken out across six different time periods on October.


As Moore points out, "Obama's average lead each week varies only slightly over the whole month of October," yet the "substantial" variability of the polls drops off significantly in the final week. "Why," Moore asks, "do different polls show such variability over the month of October, and then suddenly converge in the last week of the campaign?" He continues:

Of course, it's true that opinions "crystallize" in the final weeks, but why should that make polls so relatively unreliable during the campaign? Shouldn't polls conducted at the same time produce the same results, even if many people are still mulling over their decisions? Shouldn't different polls find the same proportion of indecisive people?

Moore adds that the editors of Survey Practice will be asking media pollsters for their explanation and analysis of the "convergence phenomenon," reactions that they plan to publish.

I wondered whether we would see the same pattern in statewide surveys. So I replicated Moore's calculation within the twelve battleground states in which we logged 20 or more polls during October and November (Colorado, Florida, Georgia, Minnesota, Missouri, Nevada, New Hampshire, North Carolina, Ohio, Pennsylvania, Virginia and Wisconsin). Then it calculated the mean and median variance across all 12 states, along with the average Obama margin over McCain in each.   

As the table below shows, the same general pattern occurs: Even though Obama's average lead remains roughly constant throughout October, the variance in that lead is even larger at the state level before a marked convergence over the last week of the campaign [I have also copied a table showing the values for each state individually, including counts of polls in each, after the jump].


So what's the explanation for this convergence phenomenon? I too look forward to reading the response of the media pollsters in Survey Practice, but I have my own theory. I know that many will wonder, as reader DTM does, "if there was some deliberate convergence." A deliberate "thumb on the scale" may have happened in some cases. But I want to suggest a more benign explanation.

Just about every political pollster that I know cares deeply about the accuracy of that last poll before the election. It is, as the cliche goes, the pollster's "final exam" every two years. Our clients use that final poll as a test of our accuracy, and our credibility and business are at stake every election.

So now put yourself in the position of a pollster who is producing results in any given race that are out of line with other public polls. Trust me, everyone notices, and your paying clients will ask you why your survey is so different from the rest. Every pollster in that position will check everything they can about their own survey and methods -- the sample, the interviewing procedures, the weighting, the screen questions and so on. Keep in mind that even the most by-the-book media pollsters leave themselves a fair amount of room for subjective judgment calls about their likely voter screen questions, models and weighting procedures.

Now consider this basic observation: Pollsters that look hard enough for something amiss will likely find and "correct" it. Those that don't -- those that feel comfortable that their estimates are in line with other pollsters -- will not look as hard for "problems" and, thus, are less likely to find anything in need of "fixing." Take that underlying tendency and apply it to the actions of the many different pollsters over the final weeks in October and I think you have an explanation for why results tend to converge around the mean.

This is just a theory, of course, but whatever the explanation, it should leave us questioning the value of trying to sort out "good" pollsters from "bad" using accuracy measurements based only on "the last poll."

Continue reading "More on the "Convergence Mystery""

US: Obama Confidence (Gallup-12/1-3)

Gallup Poll
12/1-3/08; 1,564 adults, 2% margin of error
Mode: Live Telephone Interviews


Are you confident or not confident in Barack Obama's ability to be a good president?

    65% Yes
    25% No

"While observers have noted that Obama's picks of Clinton, Gates, and retired Marine Gen. James Jones as White House national security adviser represent more centrist and conservative positions than his own, Gallup Poll Daily tracking data find party loyalists neither rattled nor reassured. Republicans, independents, and Democrats remain as confident in Obama as they were before the announcements."

Full results available here.

How Do I Become a Pollster?

Topics: AAPOR , CASRO , CMOR , JPSM , MRA , Pollsters , Training

A few weeks ago, I received this email message from a reader who describes herself as a seventh grade teacher in Los Angeles.

Yesterday, one of my students expressed an interest in having a career as a pollster. I looked online to find out how one becomes a pollster, but I found nothing other than the requirement of a college degree. How does one start a career as a pollster? Any advice you could give me for my student would be helpful.

Great question. First of all, congratulations to your student for their interest. Unfortunately, while polling and survey research demand specific kinds of training, our profession lacks some of the characteristics of professions like medicine, law or professional accountancy. We do have professional organizations, such as the American Association for Public Opinion Research (AAPOR), The Marketing Research Association (MRA), the Council for Marketing and Opinion Research (CMOR),  and the Council of American Survey Research Organizations (CASRO), and most of those groups have ethical codes that bind their respective members. However, the profession lacks a set of formal qualifications, a proscribed course of post-graduate study and any formal licensing or required examinations for competence.

That said, I can offer four specific suggestions to students interested in becoming a pollster or survey researcher some day.

1) Take a lot of Math and Statistics. Statistical science is the building block upon which everything else in the polling world is based, so the more "numerate" and statistically literate you can become, the better. Fortunately, as I understand it, many high schools now offer some exposure to Statistics, so make sure to take those classes. And it doesn't hurt to have a knack for computer programming, as so much of data analysis is now facilitated by statistical packages like R, SAS and SPSS that really sing when driven by command line programming.

2) But be sure to study the other subjects that interest you. Survey researchers are usually more than just survey mechanics. They are usually experts in their chosen field, be it politics, business, medicine  or something else (keep in mind that political polling is just one small piece of survey research -- surveys are also used in government and business to measure everything from customer satisfaction to the prevalence of poverty and disease). Most practicing political pollsters earned degrees in fields like political science, sociology, history or social psychology, but if you are interested in specializing in other kinds of surveys, make sure you learn as much as possible about those subjects.

3) Consider a graduate program in survey methods. Many of the individuals that currently work as pollsters or survey researchers had academic training in political science, sociology or a similar specialization that included significant coursework on statistics and survey methods. In recent years, however, a growing number of universities are offering degree programs in survey methods, including the University of Michigan, the University of Maryland's Joint Program in Survey Methodology (JPSM), the University of Connecticut and the University of Nebraska-Lincoln. A number of other schools offer graduate certificate programs (AAPOR maintains a complete list).

4) Become an apprentice pollster. This was my path into the profession, and it is a more common experience than you might assume. I first went to work for a polling firm when I was 23 years old. My prior experience was mostly as a campaign field worker who had gained some exposure to surveys and data analysis as an undergraduate political science major at the University of Michigan. Most of my real training in survey methods came on the job working for two Democratic campaign polling firms during the 1980s.  I later went back for course work at JPSM at Maryland, but the reasons I'm a pollster today is that I went to work for one and kept at it. 

I know that many professional pollsters and survey researchers read this site, and I hope they will offer their suggestions or thoughts in the comments.

US: Recession/Depression (CNN-12/1-2)

12/1-2/08; 1,096 adults, 3% margin of error
Mode: Live Telephone Interviews


Do you think the economy is in a recession, or not?

    89% Yes
    10% No

As you may know, the U.S. went through a depression in the 1930s in which roughly one out of four workers were unemployed, banks failed across the country, and millions of ordinary Americans were temporarily homeless or unable to feed their families. Do you think it is very likely, somewhat likely, not very likely, or not likely at all that another depression like that will occur in the U.S. within the next 12 months?

    10% Very likely
    28% Somewhat likely
    41% Not very likely
    20% Not likely at all

PA-Sen: Specter 46, Matthews 43 (Rasmussen-12/2)

Rasmussen Reports
12/2/08; 500 likely voters, 4.5% margin of error
Mode: IVR

Sen: Specter (R-i) 46, Matthews (D) 43

Turnout: Who Predicted 140 Million?

Topics: Turnout

Was turnout really less than the expert's expected? Perhaps, but not as much as implied by Bloomberg's Heidi Przybyla under the headline, "Obama Won Without Voter-Turnout Surge Experts Had Predicted" (emphasis added):

President-elect Barack Obama bet on an unprecedented surge of new voters to carry him to victory last month. He won without the record turnout.

About 130 million Americans voted, up from 122 million four years ago. Still, turnout fell short of the 140 million voters many experts had forecast. With a little more than 61 percent of eligible voters casting ballots, the 2008 results also didn't match the record 63.8 percent turnout rate that helped propel President John F. Kennedy to victory in 1960.

When I read this yesterday, I wondered: Who are the "many experts" that forecast a turnout of 140 million? I turned to Google and Nexis. So far, I can't find any.

Google the words "140 million" and turnout and you will find links to a number of stories claiming, in various formulations, that unnamed "experts" or "officials" were expecting a turnout "could approach a record 130 million to 140 million." Of course, if that is what the "experts" claimed, they were right. The final, certified count of the national popular vote looks to be at least 130 million, as Przybyla reports, and more likely closer to the 131 million that George Mason University's Michael McDonald now estimates (based on a combination of "final or certified county level election results within states that have not recently updated their state level reporting").

But I still can't find the "many experts" who "forecast" a turnout of a 140 million. The two true experts who get quoted most often on the subject are McDonald (who, full disclosure, is also a friend and an occasional commenter on Pollster.com) and Curtis Gans, director of the Committee for the Study of the American Electorate.

In an op-ed published by Politico in late September, McDonald speculated about turnout:

The impending presidential election may be the election of a century. Record primary voting, floods of new registrations, more small campaign donors and highly rated political conventions show that people are intensely interested.

These indicators augur a high turnout. Undoubtedly, more people will vote than the 60 percent who turned out four years ago, which was the highest rate since 1968. The question is, how many more? If participation tops the 1960 level of 64 percent, then we must go all the way back to 1908 -- literally a century of American politics -- to find the next highest rate: 66 percent.

All McDonald confidently "predicts" here is a turnout of at least 60 percent. He was right about that.

Just before the election, however, in an article by the the Boston Globe's Brian Mooney, McDonald ups the ante a bit:

McDonald projects that 64 percent of eligible voters will cast ballots this year, exceeding the most recent high of 63.8 percent in 1960, the John F. Kennedy-Richard M. Nixon contest. Four years ago, turnout was 60.1 percent of eligible voters. Prior to 1960, the previous high was 65.7 percent in 1908, McDonald said. In that election, before women could vote, William Howard Taft defeated William Jennings Bryan.

With about 213 million Americans eligible to vote this year, a 64 percent turnout would mean about 136 million votes cast in the presidential race, compared with a 60 percent turnout of eligible voters in 2004 and 123 million votes cast that year, according to McDonald's United States Elections Project website.

So yes, McDonald's final prediction, at least as reported by Mooney, was a bit high [but see McDonald's comments in the update below]. Still, 136 million, though certainly a bigger number than 131 million, is not quite 140 million.

Curtis Gans may have been the one who first floated "140 million" figure, although not because he was predicting that level of turnout. An early October post on the Chicago Tribune Swamp blog reports Gans skepticism of a looming record turnout:

So will the general election turnout reach the historic volume that has been foreshadowed?

Unlikely, says Gans.

1960's turnout -- 67 percent of eligible voters -- is the figure to beat, and would require nearly 140 million Americans to cast a ballot, he says in the report. To put that in perspective, 122 million people headed to the polls, meaning that turnout would have to increase by 18 million voters to break the record.

Obviously, Gans did not expect a turnout of 140 million. A Roll Call editorial that appeared just before the election (found via Nexis) was more specific about Gans' prediction:

It would take turnout of 140 million to match the all-time record of 65 percent in 1960, according to Curtis Gans, director of the American University Center for the Study of the American Electorate, whose guess is that 2008 will come in at 61 percent to 63 percent.

It is worth noting that Gans and McDonald compute turnout percentages differently, and may have come to different conclusions about what number of voters it would have taken to match the turnout from 1960. Gans calculates turnout using an estimate of the voting age population (VAP) in the denominator. McDonald prefers to use his the voting eligible population (VEP) his own estimate on the number of adults eligible to vote (more details here).

Either way, the bottom line is that the predictions from both Gans and McDonald were reasonably close to each other and to the actual result. More important for this topic, neither predicted a turnout of 140 million.

So who did? Is there any basis for the claim of a "forecast" by "many experts" of a 140 million turnout?

Update - Michael McDonald emails with more:

I tried to place caveats in all my discussions with reporters that a 63.8% turnout rate was a possibility, not a certainty. I spoke with hundreds of reporters, so I fault myself for not being clearer to Brian Mooney from the Boston Globe that the word "will" should properly be "might." Indeed, as the election neared, I speculated to many reporters that the increasingly likely Obama victory might actually depress turnout from what we might otherwise see if the election outcome was uncertain.  We will have to look into this more deeply in the months ahead, but it stands to reason that less enthusiastic Republicans would be disproportionately affected by an likely impending Obama victory, leading to fewer Republicans willing to volunteer for McCain, a less effective ground game for his campaign, and ultimately lower turnout among Republicans that would lower the overall national turnout rate.

One final word: not only will we most likely exceed 131 million votes for president, the total number of votes cast including blank and spoiled ballots will exceed 132 million. An aspect of the election that we will have to ponder is the aftermath is the 500,000 to 750,000 mail-in ballots that were rejected.

US: Auto Industry Bailout (CNN-12/1-2)

12/1-2/08; 1,096 adults, 3% margin of error
Mode: Live Telephone Interviews


The major U.S. auto companies have asked the government for a program that would provide them with several billion dollars in assistance. The auto companies say they may go into bankruptcy without that assistance. Based on what you have read or heard, do you favor or oppose this program?

    36% Favor
    61% Oppose

If the government provides several billion dollars in assistance to the auto companies, do you think that program would treat U.S. taxpayers fairly or unfairly?

    29% Fairly
    70% Unfairly

US: 2012 GOP Nom (Zogby-11/7-18)

Topics: PHome

Zogby Interactive
11/7-18/08 (12/1 release); registered Republicans (out of 24,964 voters)
Mode: Internet

National Republican Nomination
Palin 24%, Romney 18%, Jindal 16%, Huckabee 10%

Thanksgiving Leftover "Outliers"

Topics: Outliers Feature

ABC News reports the lowest level of consumer confidence since they started tracking in 1985.

Michael McDonald updates his estimates of 2008 turnout

David Moore asks pollsters to comment on the 2008 pre-election poll "convergence mystery."

PPP recaps their summer tests of matchups of Democrats against Mel Martinez.

ABC's Peyton Craighill examines exit poll data on the 2008 "flip states" that went for Bush in 2004 but Obama in 2008.

Jennifer Agiesta looks at poll and focus group data on how voters view the Obama transition.

Sheri and Allan Rivlin offer advice to Democrats on how to hold their majority.

Brookings posts a 91-page PDF transcript of their political scientist panel discussion of the 2008 election (via Nyhan).

Andrew Gellman shares Ben Lauderdale's maps and charts of estimated Obama-McCain vote among non-blacks by county.

Ruy Teixeira digs into the 2008 exit polls.

Nate Silver says Obama's popular vote margin is the largest ever for a non-incumbent.

John Sides warns aganst using the exit polls to examine turnout.

Brenden Nyhan flags Kathleen Parker for confusing correlation with causation.

Patrick Ruffini makes a good point about what the Georgia runoff result may tell us about turnout on 11/4.

Barry Ritholtz warns of misreading of survey-based reports on Black Friday sales (via Sullivan).

Carl Bialik notes the failure of the "big state primary theory."

Nielsen adds up the ad buy for Obama and McCain during 2008.

Compared to What?

Topics: Obama , Transition

One important question to ask whenever looking at poll results is, "compared to what?" In other words, since individual pollsters sometimes have "house effects" and since it can be hard interpret results based on vague answer categories, the best approach is to look at how any given result compares to a similar question asked by the same pollster previously.

Consider the result from the USA Today/Gallup poll showing 78% of Americans expressing approval for "the way Barack Obama is handling his presidential transition." That's obviously a big number, but Obama is certainly not the first newly elected president to experience a "honeymoon" of public support. How does it compare to past transitions? Fortunately, the Gallup summary includes this especially helpful paragraph:

That overall 78% approval rating compares favorably to the reaction Americans had to George W. Bush's transition, for which an average of 63% approved in January 2001, and Bill Clinton's transition, when an average of 66% approved from November 1992 through January 1993.

So the Obama transition approval number isn't just big, it's bigger than the last two transitions during which Gallup asked the question.

US: Obama Transition (Gallup-12/1)

USA Today / Gallup
12/1/08; 1,010 adults, 3% margin of error
Mode: Live Telephone Interviews
(story, results)


Do you approve or disapprove of the way Barack Obama is handling his presidential transition?

    78% Approve
    13% Disapprove

Do you approve or disapprove of Barack Obama's choice of Hillary Clinton as his secretary of state?

    69% Approve
    25% Disapprove

As you may know, Barack Obama has asked Robert Gates, the secretary of defense for President Bush, to stay on the job for at least a year to continue dealing with the Iraq war and other matters of national defense. Do you approve or disapprove of having Robert Gates stay on as secretary of defense?

    80% Approve
    14% Disapprove

Barack Obama says that once he takes office he will push Congress to pass a large spending package, estimated by others at $500 billion to $700 billion, to spur economic growth. Would you favor or oppose such a measure?

    58% Favor
    33% Oppose

GA-Sen: Chambliss 50, Martin 46 (IAdvantage-11/30)

11/30/08; 744 likely voters, margin of error 3.5%
Mode: IVR

Sen: Chambliss (R-i) 50, Martin (D) 46

View our Georgia Senate Runoff chart here.

CT-Sen: Lieberman Ratings (R2K-11/11-13)

DailyKos.com (D) / Research 2000
11/11-13/08*; 600 likely voters, 4% margin of error
Mode: Live Telephone Interviews

Favorable 35%, Unfavorable 63%
Job Approval 36%, Disapproval 61%

If the 2012 election for U.S. Senate were held today would you to reelect Joe Lieberman would you consider voting for another candidate or would you vote to replace Lieberman?

    35% Reelect
    14% Consider
    48% Replace

If you could vote again for U.S. Senate would you vote for Ned Lamont theDemocrat Alan Schlesinger the Republican or Joe Lieberman an Independent?

    59% Lamont
    34% Lieberman
    3% Schlesinger

Full results are available here. *Our apologies to readers for neglecting to post this on 11/14.

Sarah Went Down to Georgia

With today's release of the latest PPP poll in Georgia, we now have three surveys conducted in the last week showing incumbent Republican Saxby Chambliss with leads ranging from 3 to 7 percentage points over Democratic challenger Jim Martin in the Georgia Senate Runoff election that will be held tomorrow (all of the surveys are included in our Georgia Runoff chart). As PPP's Tom Jensen explains, the result in these surveys is highly sensitive to the African American composition either measured or assumed by each pollster, although the three public pollsters active in the last week are in rough agreement on the African American composition of tomorrow's electorate: PPP puts it at 28%, DailyKos/Research2000 at 27%, Insider Advantage at 23%. According to the network exit poll, Georgia's African American composition was 28% in the November 4 election.

What are the campaign's internal polls telling them? A fouth survey conducted a week ago by Democratic pollster Mark Mellman for the Democratic Senatorial Campaign Comittee also had Chambliss ahead, but by only two percentage points (48% to 46%), although the DSCC release included no information about that survey's racial composition.

For what it's worth, NBC's Chuck Todd had this to say this morning on MSNBC in a discussion about why President-elect Obama has not campaigned in Georgia for Martin (my transcript, no link):

Had Obama thought he could win this thing for Martin, had Obama thought he could drag Martin over the finish line, he would have gone down, he would have done an event [in Georgia]. But I talked to some Obama folks who said, "that's when you'll know whether we really believe we can win this thing is if we go." And the fact that they didn't go, tells you they think they're going to come four, five points short.

And speaking of inferring what internal polls are saying based on the actions of campaigns, consider the decision by the Chambliss campaign to bring Sarah Palin to Georgia to speak at four rallies today. For all the popular derision of Sarah Palin's future in Republican politics (see Doonesbury). the move by Chambliss to bring her to Georgia implies their internal surveys say Palin has continuing appeal to the conservative base in Georgia.

A handful of national surveys show that while Palin's popularity fell considerably among Democrats and independents since her debut at the Republican convention, Palin remains very popular among conservative Republicans. Specifically,

  • 86% of Republicans interviewed in the 11/6-9 CNN/ORC survey rate Palin favorably.
  • 67% of Republicans, and 73% of conservative Republicans, told Gallup (11/5-16) they would like to see Palin run for President in 2012.
  • 77% of Republicans told CNN/ORC just before the election (10/30-11/1) that the would support Palin for president in 2012 if McCain was not elected this year; 46% said they would "strongly" support Palin in four years.

Those numbers tell us that Palin will likely remain a force to be reckoned with in Republican party over the next four years. It is worth remembering that Barack Obama's campaign demonstrated a new model of how to run for president. His biggest asset, especially during 2007, was his ability to draw large crowds to campaign rallies. The Obama campaign effectively harnessed that energy, using rallies to collect small contributors, email addresses and incoming text messages and ultimately to build the grassroots campaign that enabled him win the Democratic nomination. Obviously, Obama brought far more to the race than celebrity, but his campaign's ability to use that appeal to raise funds and recruit volunteers made the rest of his campaign possible.

Poll Performances: Crazy October

The final presidential contest predictions of the major media polls all came pretty close to the actual results, predicting Obama to win by anywhere from 5 to 11 percentage points (he actually won by 6.7 points).


However, the polls showed a great deal of variability even during the last four weeks of October leading into the election, raising questions about how to measure poll "accuracy" during the election campaign itself.


Shown below is a graph of the results of 10 polls that publicized results for at least the final three weeks of October.

2008 Oct weekly poll trends.png

An examination of the daily tracking polls provides no better picture of poll accuracy.  

2008 Oct daily poll trends.png 

The differences in the overall trends are quite substantial, as are individual points.


On October 12, IBD/TIPP shows an Obama lead of two points, while DailyKos says it's 12. The next day, IBD/TIPP produces a 3-point lead, while GWU has a 13-point lead. On October 25, GWU's lead is just three points, while DailyKos has it at 12 points. Even right before the election, IBD/TIPP shows just a 2-point lead, while ABC/WP says the lead is 11 points.


More important are the many different pictures of the dynamics of the race. If we single out, say, DailyKos from GWU, one would never know they are measuring the same contest - except that they both converge in their final predictions. Another example: Rasmussen shows only a little variability in the race, between an Obama lead of 3 to 8 points, ending at 6, while GWU goes from 13 points down to 1, finally ending at 5. Likewise, Zogby's description of the campaign dynamics shows a relatively stable race for the first half of the month, followed by a major surge, a big decline, and then a last minute surge. DailyKos usually had the most optimistic Obama leads, mostly double digits, except for the middle of the month, and then at the end when the lead declined to just five points.


What can we say about poll performances when there are such different stories about the October dynamics? The notion that the polls were mostly "accurate" must be modified to reflect how divergent they were during the campaign.


GA-Sen: Chambliss 53, Martin 46 (PPP-11/29-30)

Public Policy Poling (D)
11/29-30/08; 1,276 likely voters, 2.7% margin of error
Mode: IVR

Sen: Chambliss (R-i) 53, Martin (D) 46

View our Georgia Senate Runoff chart here.