Pollster.com

Articles and Analysis

 

Where Was the Error Bigger: NH or SC?

Topics: 2008 , Barack Obama , Bradley/Wilder , Hillary Clinton , Mason-Dixon , Pollster , SurveyUSA , The 2008 Race

So in which state did pollsters have a tougher time, New Hampshire or South Carolina? The answer, based on an objective measure of survey error, may surprise you: South Carolina.

SurveyUSA has put together a report card that calculates one measure of poll accuracy for all of the final surveys conducted by every pollster during the presidential primaries so far (there are other measures, but I will leave that subject for another day). Here's their bottom line:

  • Average Pollster Error in South Carolina’s Democratic Primary was 16.6 points.
  • Average Pollster Error in New Hampshire Democratic Primary was 9.6 points.

Funny how the psychology of poll numbers works. Two weeks ago, when the final polls showed Barack Obama leading by an average of eight points, but he lost to Clinton by three. Since the winner was a surprise, the election night coverage obsessed over polling problems. Tonight, the polls were off to an even larger degree -- polls showed Obama leading by nine points but he won by 28 -- but since they missed only the magnitude of Obama's margin, the difference goes largely unnoticed.

What happened? In this case, given the differences between live-interview and automated polls, we have decent evidence of what Noam Scheiber termed a "reverse Bradley/Wilder effect," something I wrote about at more length yesterday. Also, at least three of the final polls reported an African American share of the vote in the 42% to 46% range, while the exit polls report that percentage at 55% (though as Mason-Dixon pollster Brad Coker pointed out yesterday, the hard numbers from the South Carolina Secretary of State may look different). So the enormous turnout -- nearly double that of 2004 -- may have thrown off the likely voter models.

The bottom line is that polling is always going to be perilous for primary elections, especially when turnout exceeds all expectations and large numbers of "likely" voters express uncertainty in the final hours before casting their ballots Yet despite the surge in participation, yesterday's Democratic turnout amounted to just 16.5% of South Carolina's eligible adults. Pollsters still had a challenge in terms of selecting likely voters.

The primary season is far from over, and we may be in store for more surprises.

 

Comments
Chris S.:

In any case, that's an enormous polling error in both states, and in different directions (pro-Clinton in one case and pro-Obama in the other). Given that, is it even possible to track this Democratic nomination contest with polls? Should we even be paying attention to the polls of the Feb. 5th states? And if the Feb. 5th state polls are going to be off by that much, is there any way to guess whether they'll all be off in the same direction, or will some be wildly off in the Clinton direction, and others wildly off in the Obama direction?

____________________

carbog:

It seems like the polls can't handle the record turnouts that we are seeing where voters are coming from all over the place, places not covered by the likely voter models.

____________________

Henry:

Yeah I think the above says it. Must be extremely hard to judge the number of first time voters who turned out.

With nearly twice the turnout in SC as in 2004, and most of those first timers apparently going to Obama, that seems to have skewed it way further to Obama than any of the polls could predict.

It's worth noting, however, that the PPP poll of two days ago showing 44, 24, 19, came actually relatively close to the final result of 55 27 18. The key part is it showed 14% undecided. So you could interpret it as Edwards losing 1 point at the polls, and 11% of those undecided voters going to Obama, and only 3% of them to Clinton.

Realistically it's probably somewhere between those two explanations (undecided going mostly to Obama and 'underpolling' of first timers who also went mostly for Obama).

The next nine days through Super Tuesday are going to be a blast.

____________________

JohnG:

Polling is based on projecting a sample to the broader electorate. If there is a major change in the composition of the electorate who votes (young people, more minorities, etc.), then the projection will be off.

So, in South Carolina we probably had the "reverse Bradley/Wilder" effect and a large number of new voters, who predominently went for Obama. This may be a year when no model is safe!

____________________

Nick Panagakis:

Chris S. The magnitude of these poll errors are shocking and may have been to others.

However, my colleague Jay Leve does himself and others an enormous disservice by using a method which says if Pollster A says Smith beats Jones by 8 points, and Smith in fact beats Jones by 4 points, then Pollster A has an error of 4 (8 minus 4). A 4 would be recorded in the table at their site for that contest. The Survey USA site says there are limitations to this and other measures of pollster accuracy. You bet there are.

This is because it is the poll estimate that is subject to sample error, not the margins; e.g., 48% voting for A and 52% for B would be a 4 point margin. If the election turns out to be 46%/54%, an 8-point margin, the margin error is 4 points. The error of the estimates in this case is 2 points, half the margin error. Differences between poll and election margins in statistical analysis, or error of the margin, should not be used.

In summary´┐Ż

1. Elections are zero-sum games. This means that two points high for one candidate MEANS two points low for the other. So estimate error is the more valid measure. Estimate errors are not additive which is the effect of using the difference between election and poll margins.

2. This is also the only error measure that can be compared with sample margin of error always included in poll reports. Whatever method is used should be comparable to stated statistical margin error. Only the error of the estimates does that.

Nick Panagakis

____________________

Roy:

1) Mr. Panagoukis, when was the last time you saw a poll that was 52/48, i.e. with no undecided voters? If a poll in your scenario had 'predicted' a 48/40 result with 12% undecided, it would be absurd to say it was six points off; everyone will interpret that poll as showing an 8-point lead, and thus being essentially correct. People use margin because that is the only thing people are interested in. You can divide the margin error by 2 if it makes you happy.

2) The basic point remains that most of the pollsters in SC would have done better by pulling numbers out of their derriere than they did with their sophisticated screening methods. This should make members of any profession do some soul-searching.

____________________

Nick Panagakis:


52/48 after allocating undecideds

____________________

G. G.:

Several comments here state or suggest that the high turnouts are due to first-time voters for Obama. I must have missed a source on that?

Is it possible that high turnout also is owing to women turning out, and also for Clinton? After all, polls showed a rather astonishing 60% turnout of women in SC -- well above, as I recall, the 54% of women voters in recent general elections, and above their 51% share of the general population.

If so, is it also possible that a question could be why the other candidates aren't bringing out more men?

____________________

Chris S.:

Is 61% women really that strange for a Democratic primary? After all, women are more likely to be Democrats than Republicans. In SC, it was 61% women in the Dem. primary and 49% women in the GOP primary, which averages out to 55% in the two combined.

____________________

Nick Panagakis:

Also, if a poll showed 48/40 that would be an 8-point margin.

If the election turned out to be 46%/54% that is an 8-point margin, as in my example.

8% poll margin minus 8% election margin equals 0, no error.

Glad to help.


____________________

Shane:

You're, of course, technically right. The polls in SC were further off-mark than those in NH.

However, in spirit, the polls in NH were far more troublesome.

The utility of the polls is to tell us who is winning and who is not.

In NH the margin of error was smaller, but it made a huge difference. In SC, the margin was larger, but more forgivable, since the polls did appear to echo the sentiment of the voters: That Obama had more support in the state.

Perhaps this is a joke for you in the business. I understand that the polls in SC could just as easily been skewed in the opposite direction. But from us political junkies, all I really care about is 1, 2, & 3. Hell, you can just keep the percentages to yourself :)

____________________

Kevin Houston:

Thanks for pointing out to me another entire class of bias in polling: Turnout bias.

I hadn't even thought about that one...

I have to ask again: why, oh why, do we let the media get away with presenting poll results they way they do, and using it as justification for shutting certain candidates out of the process?

Case in point: Giuliani Vs Ron Paul.

For the past year, we have heard about how Giuliani was this unassailable juggernaut of a candidate that poll after poll showed had 30% support, while pitiful Ron Paul was "mired" in the single digits...

Even though no primary or caucus came anywhere near 30% for Giuliani, he was treated to a lot more attention in the press (and positive attention at that) than Ron Paul. To such an extent, that certain news editors thought Ron Paul supporters were somehow cheating the on-line polls (yeah yeah, not scientific... I know) because of the discrepency between polling results.

____________________

jerry:

people are not going to put a black man in a high office. why isnt that factored in the polls. as much as we hate to admit it, it still exsist. i hear many people, who are registered to vote say the say thing. is that possibly the "undecided' vote.

____________________

Joe From the Beach:

The obvious answer to the problem of why the polls are not consistent with the election results is that the votes are not being tallied correctly.

Surely, someone may have noticed that the pro war candidates are getting the benefit of the counting process. See http://www.commondreams.org/headlines03/1013-01.htm

The stakes for democracy are very high now. Somebody's poll said that 70% of American do not support the war, yet all the presidential candidates, except Paul, support the war. Politicans can get away with that stance only if assured that the vote count can be controlled.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR