Pollster.com

Articles and Analysis

 

WSJ's Bialik on Pollster Performance


The Wall Street Journal's Numbers Guy Carl Bialik has a must-read post-election review on how the polls did and on the future of polling. Here is his bottom line:

[A]s Americans watched the news networks call states one by one for Mr. Obama and his Republican opponent, Sen. John McCain, Tuesday night, pollsters could breathe a sigh of relief. There wasn't a single big miss in the presidential race. Most polls showed virtual ties in Indiana, Missouri and North Carolina, and none of those states was decided by more than a point. "Pollsters generally did very well," says Mark Blumenthal, a former Democratic pollster and co-founder of Pollster.com.

Zogby International polled in eight states in the last week, including six of the closest races, and missed the final margin by an average of less than two points -- as accurate as the poll aggregators such as Pollster.com.

The lopsided nature of the race helped pollsters. Just seven states were decided by fewer than five percentage points, and just 15 by fewer than 10. That contrasts with 10 and 21, respectively, in 2004, and 12 and 22 in 2000. With that many close races in the past, it was more than likely you'd have at least one polling gaffe simply because of the error introduced by random sampling. Pollsters generally underestimated Mr. Obama's support in Nevada this year, and overestimated it in Iowa. But those misfires won't be judged harshly because he won both states by comfortable margins.

Bialik also considered the performance of polling aggregation sites like ours:

The biggest winners may have been poll aggregators, who were combining disparate polls as far back as 2002, but gained new members and reached a new level of national prominence this time around. Their advantage is twofold: Their composite results may dilute the effect of any error in one poll, and their results are more expansive, including regions that no one pollster can typically afford to cover. A dozen or so Web sites combined polls to forecast the election, and just about all of them put Mr. Obama's electoral-vote total at between 338 and 393; he likely will finish with 364 or 375. (Those that also forecast congressional races generally foresaw Democratic gains.)

At least two sites -- Pollster.com and fivethirtyeight.com -- also estimated the winning margin for each state, using poll data and their own formulas. They typically missed the margin by just 2.4 and 2.3 percentage points, respectively. Each site beat each of the 10 pollsters active in at least eight states, head to head, except for Zogby.

Thanks, Carl.

Bialik's companion blog item also has some useful links to ongoing discussions of voter turnout:

Perhaps the most interesting variable in the contest was among the hardest to predict: Voter turnout. While the popular-vote total so far has barely cleared the 2004 turnout of 122 million, several forecasters estimate that, once all ballots are counted, turnout will be between 125 million and 136.6 million. [Franklin & Marshall College statistician Brian] Adams and Sam Wang, a Princeton University neuroscientist and poll watcher, forecast turnout of 133 million and 135 million, respectively. Allan Keiter, of the forecasting site 270 to Win, may also have come close.

And he saves me the trouble of an "outliers" post for these "preliminary analyses of pollsters’ performance... Pollster.com, Huffington Post, the Atlantic’s Marc Ambinder and a Fordham University political scientist."

 

Comments
Arkie_In_CT:

Questions:

1. Any thoughts on what happened in Alaska? Why so small a turnout relative to 2004? Are there a lot of votes yet to be counted?

2. Which were better - expanded, present-intention-based "likely voter" models (such as Gallup's newer model), or the more traditional, more restrictive approaches? In the same vein, can you offer any idea as to how we got from Gallup's prediction of 64% of the universe - about 138.2M votes - and Larry Sabato's of 135-140M down to the actual turnout, which appears to be 125-130M, an improvement over 2004, but not the earth-shattering result many were expecting? Do we chalk it up to Democratic voters' complacency, even in view of the ferocity and thoroughness of the Obama turnout machine, or was there some irrational exhuberance in projecting turnout, a bit of overhype?

3. It's interesting that South Dakota and North Dakota voted by almost identical margins in the Presidential race. Any idea why SD reflected so much poorer than did ND in polls the three previous weeks or so to E Day? My own suspicion, if I may say, was that if one of those states was in play, they both would be, given the great degree of similarity between the two. Did Obama-Biden spend as much money on field and tv in SD as in ND? If not, they may have picked the wrong Dakota!

4. Any thoughts on why Zogby's polls fluctuated so wildly - probably moreso than did any other major pollster - yet he comes in, evidently, with the best final state-specific results (though he did apparently miss the national presidential popular vote by 4 points or more). Might there be any cause for suspicions of gamesmanship being reflected in some of his polling results, including the 1 point pro-McCain margin from his single day sample of last late Thursday/early Friday?

Thank you as always, Mark and team. You've become mainstays for me and countless others.

-Chris Stratton

____________________

cinnamonape:

My understanding is that there are a substantial number of ballots still uncounted in Alaska, including many absentee ballots.

http://ltgov.state.ak.us/elections/

38,000 of 55,000 votes come from District 1 and 3 with very few from heavily Republican District 2 (Mat-Su). Given that the Stevens-Begich and Berkovich races are so close, it's likely that these uncounted votes are going to be decisive.

It may also explain the very odd situation where an election with their Governor in a Presidential campaign, and with both their Senaote and House races were actually competitive would have LOWER turnout...in an election in which national turnout was very much higher...would be low. The SOS simply hasn't counted the votes.

____________________

cinnamonape:

There were 224057 votes counted in Alaska with 55663 absentees not yet counted...plus about 5600 "questioned" ballots (mainly from strongly Democratic areas ~amazing that there were ZERO challenged ballots from Mat-Su). That's about 20% of the total potential votes.

Many of these, judging from the District votes for State Representatives in the counted votes are from areas that went 75% or more for Democrats (Anchorage and Juneau). I'd suspect that this will bring the votes back toward the late polling averages.

____________________

Mike In Maryland:

Re: Chris' point 4:

I've been wondering if any pollster tried to 'game' the system - trying to influence the final result by showing the race much closer or much greater difference than it actually was, until the end when they brought their polling results into (general) alignment with the rest of the pollsters, and thus came close to the final result.

I know it's impossible to assess the accuracy of any poll except by having the actual election. But it seems a bit suspicious when one or more pollster is consistently below or above all others, but in the last week the numbers converge to the consensus, which turns out to be near the actual final result.

____________________

mac_1103:

There were 224057 votes counted in Alaska with 55663 absentees not yet counted...plus about 5600 "questioned" ballots

So that's 285,320 votes. There were 308,593 presidential ballots in Alaska in 2004. So, "The SOS simply hasn't counted the votes." does not appear to be the answer to the turnout question.

____________________

Isher:

"Any thoughts on why Zogby's polls fluctuated so wildly - probably moreso than did any other major pollster - yet he comes in, evidently, with the best final state-specific results (though he did apparently miss the national presidential popular vote by 4 points or more). Might there be any cause for suspicions of gamesmanship being reflected in some of his polling results, including the 1 point pro-McCain margin from his single day sample of last late Thursday/early Friday?"

Zogby spent the entire year getting everything wrong and then adjusted the democrat/republican expected turnout ratio to match the aggregates of pollster and fivethirtyeight. He wants to be known as the most accurate pollster because of the ego/power it affords.

____________________

thoughtful:

I found it curious that Carl Bialik chose to ignore RCP in his round up.

I never quite worked out whether the value of RCP was as a simple cut & paste with intrade featured.

The two leading sites were 538 and pollster.com on the one hand with this great site you had raw polls featured and each one in turn dissected by various commentators. So whilst the site might have been aggregating and Charles and Mark publishing a plethora of polling erudition. The majority of the site was in the comment section where comments varied from the delusional to the incisive in observation on each and every poll.

At 538 Nate & Sean added colour to the numbers. Sean Quinn deserves an award for his "On the Road" feature. I thought that it was huge. Nate Silver's gift is that he is not a pollster,he has an extraordinary understanding as to how present applied maths in a simple way and use it pragmatically, and at the same time preserve the mystique round it. Blind some of us with a bit of science. It helps if you're not a member of the pollster union, you can pan them!

I for one await the final turnout numbers and maybe an idea for pollster is to publish State by State with the exit numbers for us lot to dissect and comment on?

Some of us are missing our polling drug!

____________________

cinnamonape:

Maybe the 2004 polls were the ones that were artificially "inflated" ?

In 2000 there were 469,866 registered voters and 281,812 voted.

Though admittedly it's odd that this year 485,331 registered, and 285,320 voted. That's 16,000 more registrants and only about 3,500 more voters.

Maybe a lot of Alaskans decided NOT to vote for Stevens or Palin, disgusted at the situation. Maybe they lost enthusiasm for Republican candidates, but didn't want to support the Democrats either.

Until those final 20% of the votes are counted we won't actually know if the fix was in to get Stevens and Edwards elected. If the final result shows that Stevens and Edwardshave significantly lower support than Bush then it's a good chance that there was conservative voter fatigue.

Remember that Stevens won his last election by 70%.

____________________

cinnamonape:

It's a pity that we only have one real-world data point to "test" the accuracy of pollsters. If we are to believe Zogby there were huge shifts of electorates in the last few days. If you follow the other pollsters the political landscape was set weeks ago, with small shifts often within the MOE's of the polls.

And as has been noted here there seemed to be two sets of polling results...with the "aggregates" being most accurate. Similarly the polls that established different RV,LV, modified LV models showed that these converged to a single result as early voting continued...which was congruent with the aggregate poll.

Where there was the most "error" it was in poorly polled states where one or the other candidate had a winning margin. Statewide polls tended to underestimate the size of these margins. How much these underestimates in statewide polls may have been detected in the national polls might be an interesting study. Bigger than expected margins in California, New York, Texas, etc. may have suggested bigger national margins than the battleground states suggested.

It'll be interesting to see a set of post-mortems for various pollsters on this site...along with commentaries as to whether their final results required major shifts to reach their ultimate outcome...and whether those shifts were related to earlier events (convention surges, the financial meltdown, debates).

____________________

Gary Kilbride:

Agreed, it's kind of pathetic that the pollsters who tried to pretend there was a tidal shift in the electorate over the final week or two ended up with accolades. PEW fits also. Their final margin was very close while the previous one may have been representative of some other country, certainly not this one.

Zogby essentially provided himself an either/or but not both situation. His state margins made no sense with his national margin. That was glaring. So he could claim accuracy on the high end if Obama won huge but blew out his state margins, while a lower end Obama victory had potential to ring true with his state projections. He was very fortunate the second scenario unfolded.

Alaska polling always stinks. That's a major variable that no one wants to accept. I can't count how many times I have taken advantage. There's a quote from Young the House member in one of the Alaska papers, shouting, "The pollsters don't understand Alaska, they never have," or something very close to that. I don't know how anyone can disagree. I was prepared to pounce on Stevens at bargain odds if the verdict had gone the other way. You think that wouldn't have been a living cinch? This cycle there obviously was pathetic Democratic turnout in Alaska, violating the brilliance of the Obama GOTV machine, but even if turnout had been equal to targeted states I'm confident the polling would have been typically inept by Alaska standards. I suggest the pollsters ignore Anchorage in the future. Or cut the Anchorage influence significantly. The wilderness is being undervalued.

I've started to study the 2008 exit polls. One thing jumped out immediately, the ideological shift in certain newly emphasized states. My theory every cycle is to cluster states based on liberal/conservative percentage. It seldom lets me down. This year the early Excel indication was Obama could not win any state above 37 or 38% conservatives. That turned out to be correct, but I was narrowly burned when two states altered considerably in their liberal/conservative breakdown.

Indiana made a remarkable transformation, from a 14/42 liberal/conservative percentage in 2004 to 20/36 this time. Wow. When I saw that exit poll number it stunned me, but also made sense why my model had been off by more than 2 points in Indiana projection in relation to the national margin. I had pegged Indiana at likely 18/38. New found emphasis on a state by one of the parties is always difficult to project, how much of a nudge it will create. Normally my estimates are somewhat low the first cycle, then the following cycle the state has a natural recoil that I anticipate while the pundits prefer to anticipate further movement in the same direction, not unlike sports where many similar situations evolve.

North Carolina was a much more predictable move than Indiana, from 17/40 in 2004 to 19/37 this time. That's exactly how I estimated it. It took a 5.5% national margin to pull North Carolina and Obama managed somewhat more.

Frankly, it's pretty funny when pundits and message board pollsters are suddenly describing states with 20/36 and 19/37 ideological percentages as "blue states." They are anything but blue states. They happened to tip ever so slightly blue in a national landscape with a 6+ national margin. As James Carville emphasized on Tuesday night, "There is no way you can separate the national number from the state number." In 2012, Obama will have to win nationally by more than his 2008 margin to drag Indiana and North Carolina along. That is a given. Those states will be fortified by the GOP, particularly in the next two years when Democrats rest. When a party loses a dependable state it is a system shock and steps are taken. That's what happened between 2000 and 2004 in Florida, Republicans understanding how fortunate they were to get away with taking Florida for granted in 2000.

The national ideological bent did not change appreciably, from 21/34 in 2004 to 22/34 this year. But certain states that Obama carried did move, including Nevada (18/34 to 22/34), and New Mexico to lesser extent, from 22/36 to 22/34. Perhaps most predictable of all, Virginia moved from conservative breakdown in 2004, 17/38, to classic swing state numbers in 2008, 21/33. By 2012, that state should reveal a blue state margin.

Colorado was an absolute stunner, a state where I'd like to see confirming data. The state exit poll suggests a move from 22/35 in 2004 to 17/36 this time. Does not compute. If accurate, I suggest Democrats not take Colorado for granted. You can't count on a 17/36 state voting 63-35 for the Democrat among self-identified moderates.

Sorry for the marathon. :)

____________________

Gary Kilbride:

Whoops, I meant message board posters, not pollsters.

Not exactly surprising I typed pollsters on this site. :

____________________

Basil:

I felt that as the race progressed, some mainstream outlets slowly realized that the methods employed by Pollster and 538 were probably providing a higher order of information than single polls.

In the future I would expect more sensible news sources to recruit the creators of this site, 538, and RCP as more or less official poll-crunchers, rather than occasional guests.

On the one hand, such a move would make election coverage more accurate, which I suppose is a good thing. On the other hand, it may remove some of the sensational roller coaster thrills that come from focusing on high and low outliers--or "outriders" as Pat Buchanan, trying to be hip, called them.

Campaigners and more partisan media may still try to emphasize favorable outliers in the future, but thanks to sites like this one, fewer voters will believe them.

____________________

SC:

Could weather have been a factor in the Alaska turnout? I don't know what the weather was around election day up there, but given the remoteness of a lot of voters to polling locations a weather system moving through could account for the drop in turnout.

____________________

RS:

How does Bialik calculate his "missed the final margin" numbers? Because that appears to be how he says Zogby was the most accurate - "less than two points."
How much lesser? SUSA and PPP were apparently off by just 2.1%.
And if you go by PPP's numbers and calculation here:
http://publicpolicypolling.blogspot.com/2008/11/ppps-bias.html
PPP was off by just 0.125% - across 16 polls! The only state they missed was MT; for the rest, they called the winner correctly, especially NC, IN and FL.

Of course, now Zogby will again claim he is the most accurate pollster... But unless I see the methodology used a la PPP, I won't buy it. And for a supposedly partisan pollster, PPP did a fine job, and I will keep an eye on their data rather than on Zogby's.

____________________

thoughtful:

@Gary

Really interesting post - so the US is still a centre right country?

Have you got any where with your Georgia analysis?

____________________

cinnamonape:

Once again, if you look at the distribution of those 20% uncounted Alaskan voters - they are in heavily Democratic districts in Juneau and Anchorage. Mat-Su has been completely counted.


Take the State Assembly/Senate districts and look at the partisan winners of those races in the general election. Then contrast that with the district distribution of the uncounted absentee/early ballots. It will be clear that the vast majority of these uncounted votes are going to be Democrats, or support Democratic candidates.

It will also bring up the proportionate distribution of Democrats to what would be expected. There was actually no " pathetic Democratic turnout". In fact, my guess is that neither Stevens nor Don Edwards will, when all is said and done, will return to Congress.

____________________

thoughtful:

@RS

I am in agreement, I was shocked when the final PPP States polls were published because of further tightening in their polls. But you could see the pollster had done their work, I am not as hot with the internals as most observers because the smaller the number the more prone to errors - but I want to see them.

Our little discussion re Selzer's Indiana final poll of 88% AA for Obama is easily fixed/explained with a swap of 3 AA voters from McCain to Obama, or is it 2?

With Zogby you just don't know what is driving the numbers, you have more of indication with Rasmussen, but Ras does produce consistency in his numbers if that is indeed what is happening.

____________________

Arkie_In_CT:

Gary K:

You have reasonable points about certain "red" states which moved toward the blue, but I'd like to see how many voters in these states are calling themselves "moderate." My understanding is that Pres.-Elect Obama won the moderates by a wide margin.

As to a recoil in Indiana and some other states, I think it is also possible for a state, such as Indiana, to decide they like a particular charismatic leader and to follow him or her like a Pied Piper into new territory. I'd say the jury is still out on red recoil versus further blue advancement, and it should be particularly noted that for four years Pres.-elect Obama will have the powers and perquisites and prestige of the presidency, and I am sure he will not be shy about lavishing the love on all corners of the Hoosier State, as well as purplish states like Colorado, Virginia, the Dakotas, Montana, Nevada, New Mexico, Florida, North Carolina, Georgia, Arizona, etc.

Ambitious and competitive sort that he is, I am sure that Mr. Obama would have loved to have outscored Bill Clinton's first victory in the EC, just as he did, in fact, in the popular vote, and I have no doubt be will be gunning with gusto for 400 EVs and northward in 2012. So you can expect states like South Carolina, West Virginia, Kansas, Nebraska, Texas, Lousiana, Arkansas and Mississippi to get favorable attention where political opportunities present themselves.

I'm not predicting here and now that Obama *will* succeed in leading further electoral, political and ideological advances in Democrats' favor, but I am suggesting that this year's results could just as easily - perhaps moreso - turn out to be a beachhead as a high water mark.

Best,

Chris Stratton

____________________

RS:

@thoughtful:
Yes, I did indeed think at one point that Selzer has perhaps found the five Blacks in IN that supported McCain ;-)

@Gary Kilbride:
That's the politics of the past! I have heard people say "socially I am liberal, but a fiscal conservative" - they'd call themselves moderates, perhaps, in this 3-choice world of yours. But they'd vote Democratic. Even for liberal candidates. Further, the term "liberal" has been used so much as a form of abuse in this country (see McCain calling Obama a Liberal, but proudly proclaiming himself a Conservative), I'd suggest many moderates are actually closet liberals - a corollary of the "shy Tory" factor.

Obama himself - surprise - is not all that liberal, socially. I'd love to see him endorse gay marriage, or (unlikely) have government confer only civil unions to everybody, gay or straight. But in that sense, he's still "moderate."

As for Dems possibly losing CO - well, both Obama and "Boulder liberal" Mark Udall won by 10% or more...

Again, these labels are outdated... Gotta come up with something different for the millenials, who will make up more and more of the voter base. And as for NC, migration of educated folks there will only make things harder for the present-day GOP, methinks.

Just my $0.02.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR