Pollster.com

Articles and Analysis

 

Polling Errors in New Hampshire

Topics: 2008 , New Hampshire , The 2008 Race

1NHPollErrorDem19.png

Hillary Clinton's stunning win over Barack Obama in New Hampshire is not only sure to be a legendary comeback but equally sure to become a standard example of polls picking the wrong winner. By a lot.

There is a ton of commentary already out on this, and much more to come. Here I simply want to illustrate the nature of the poll errors. These show the nature of the problem and help clarify the issues. I'll be back later with some analysis of these errors, but for now let's just see the data.

In the chart, the "cross-hairs" mark the outcome of the race, 39.1% Clinton, 36.4% Obama. This is the "target" the pollsters were shooting for.

The "rings" mark 5%, 10% and 15% errors. Normal sampling error would put a scatter of points inside the "5-ring", if everything else were perfect.

In fact, most polling shoots low and to the left, though often within or near the 5-ring. The reason is undecided voters in the survey. Unless the survey organization "allocates" these voters by estimating a vote for them, some 3-10% in a typical election survey are left out of the final vote estimate. Some measures of survey accuracy divide the undecided, either evenly across candidates or proportionately across them. There is good reason to do that in another post. But what the pollsters publish are the unallocated numbers (almost always) and so it seems fair to plot here the percent of the vote the pollster published, not one with undecided reallocated.

What we see for the Democrats is quite stunning. The polls actually spread very evenly around the actual Obama vote. Whatever went wrong, it was NOT an overestimate of Obama's support. The standard trend estimate for Obama was 36.7%, the sensitive estimate was 39.0% and the last five poll average was 38.4%, all reasonably close to his actual 36.4%.

It is the Clinton vote that was massively underestimated. Every New Hampshire poll was outside the 5-Ring. Clinton's trend estimate was 30.4%, with the sensitive estimate even worse at 29.9% and the 5 poll average at 31.0% compared to her actual vote of 39.1%.

So the clear puzzle that needs to be addressed is whether Clinton won on turnout (or Obama's was low) or whether last minute decisions broke overwhelmingly for Clinton. Or whether the pollster's likely voter screens mis-estimated the make up of the electorate. Or if the weekend hype led to a feeding frenzy of media coverage that was very favorable to Obama and very negative towards Clinton, which depressed her support in the polls but oddly did not lower her actual vote.

On the Republican side we see a more typical pattern, and with better overall results. About half of the post-Iowa polls were within the 5-ring for the Republicans, and most of the rest within the 10-ring.

2NHPollErrorRep19.png

As expected, errors tend to be low and left, but the overall accuracy is not bad. This fact adds to the puzzle in an important way:

If the polls were systematically flawed methodologically, then we'd expect similar errors with both parties. Almost all the pollsters did simultaneous Democratic and Republican polls, with the same interviewers using the same questions with the only difference being screening for which primary a voter would participate in. So if the turnout model was bad for the Democrats, why wasn't it also bad for the Republicans? If the demographics were "off" for the Dems, why not for the Reps?

This is the best reason to think that the failure of polling in New Hampshire was tied to swiftly changing politics rather than to failures of methodology. However, we can't know until much more analysis is done, and more data about the polls themselves become available.

A good starting point would be for each New Hampshire pollster to release their demographic and cross tab data. This would allow sample composition to be compared and for voter preferences within demographic groups to be compared. Another valuable bit of information would be voter preference by day of interview.

In 1948 the polling industry suffered its worst failure when confidently predicting Truman's defeat. In the wake of that polling disaster, the profession responded positively by appointing a review committee which produced a book-length report on what went wrong, how it could have been avoided and what "best practices" should be adopted. The polling profession was much the better for that examination and report.

The New Hampshire results are not on the same level of embarrassment as 1948, but they do represent a moment when the profession could respond positively by releasing the kind of data that will allow an open assessment of methods. Such an assessment may reveal that in fact the polls were pretty good, but the politics just changed dramatically on election day. Or the facts could show that pollsters need to improve some of their practices and methods. Pollsters have legitimate proprietary interests to protect, but big mistakes like New Hampshire mean there are times when some openness can buy back lost credibility.

Cross-posted at Political Arithmetik.

 

Comments
Anonymous:

is it possible the clinton's stole it?

____________________

hwc:

I wonder if the media will "open its books" for a similar analysis of what went wrong?

I'm not holding my breath.

____________________

Jackson Browne:

I don't think the Clintons stole it. Furthermore, when are people going to stop speculating on "stolen election" (i.e. Bush 2000 and 2004) based on polling (exit or otherwise) error. If the 2008 election is teaching us anything so far it should be that its time to pull the plug on overheated and vitrolic partisan rhetoric.

____________________

Lester:

Here's a simple cross-tab comparison of the Jan 7 ARG poll with the exit poll:

. . . . . . ARG . . Exit Poll
. . . . HRC BO . HRC BO . Diff
Men . . .25 44 29 40 -19->-11
Women 35 37 46 34 -2 -> 12
Dem . . .34 37 45 34 -3 -> 12
Indep . . 25 44 31 41 -19->-10

If we trust these numbers, the big jump here for Clinton is among women and Democrats. As was pointed out above, although Obama dropped a bit, Clinton rose much more, suggesting that her big boost came from women and Democrats switching from non-Obama candidates or turning out more heavily. On the other hand, ARG's likely voters were broken down as 63% Dem, 37% Indep, whereas the actual voter breakdown was 54% - 44%. So although Obama didn't do as well with independents as predicted, there were more independents to ameliorate that.

____________________

Anonymous:

Just a guess: The unusually high turnout makes me wonder if the "likely voter" filters were off. I don't know, though, why that would favor Clinton over Obama.

____________________

I feel like I'm becoming a crank about this, but must persist til I get an answer: What about absentee ballots?

It appears from the exits that HRC's big advantage is among people who made up their minds more than a month ago. She would logically have an equally large advantage among absentee voters who checked the box before Iowa. A large pro-HRC absentee vote would also help explain why the early, "nonadjusted" exit polls pointed to an Obama victory.

If there's been any discussion of this issue other than on my site and at OpenLeft, I've missed it. We're counting on you, folks. Help us understand.

Ed Kilgore
www.thedemocraticstrategist.org

____________________

Michael:

Well you are right about one thing it is not as bad as 1948. Mind you, the polling industry has had 60 years to eliminate that sort of error. Alas, those pesky voters keep making the polls look....

____________________

Patrick Oliver:

The Clintons certainly didn't steal it. It's that a lot (a few thousand, at least) of undecided voters in NH, mostly women, looked deep inside themselves and voted their gut feeling about who would make a better president (not who gives better speeches and uses the work "change" the most), as well as their personal connection to an extremely capable woman who is actually electable for the first time in over 200 years of US history in which half our population has been shut out of virtually all major positions of power, most especially the Presidency. The real problems are our completely unfair and arbitrary voting system and our ridiculous, irresponsbible media. Right up until the media frenzy around the Iowa caucuses (in which 12% of Democrats participated in a small state that is not at all representative of the country as a whole, is the ONLY state to never have elected a single woman to any national or statewide office, hasn't picked the eventual nominee in many years, and neighbors Obama's home state), Hillary had a comfortable lead, both in NH and nationally. Then suddenly (with an 8% win of a few thousand Iowa votes), the media 'annointed' Obama the new front-runner, he got all the positive press, she got only negative press, suddenly Clinton's lead vanished overnight, and all we hear is that Obama is ahead by double digit margins in NH. This shows that people's votes really are changed by the media. People want to feel like they're supporting a "winner". Never mind that almost no votes in the entire nation had actually been cast yet. No one disputes that if we had a national primary (the only truly fair thing to all American voters; aren't we electing a national leader?), Hillary would easily have won. Instead, the nomination was almost stolen from her (and may still be) by a few voters in a few states and the media, who basically had pronounced her entire 35 year political career dead as of yesterday morning. Then a few thousand women (mostly older) in another small state saved her - and now she's the "comeback kid" with all the press. This is a great example of how arbitrary and pathetic our system of voting is. It's rotten from the first vote cast to the last -- which may or may not actually matter because it's 540 political appointees in the Electoral Collage and not the 300 million American people who decide the Presidency (remember 2000?). Pathetic.

____________________

cmmcd:

Charles, Could you and Mark address the exit polling discrepancies, too? Apparently the exit polling indicated an Obama win (by about 8 points, I thought I heard on one cable news network). If this is true, that would suggest that the "Bradley Effect" at least partially explains the errors in all the pre-primary polls.

Could you also address the variance in survey response rates between men and women, especially when age and education are taken in account, and discuss whether this might have meant an underestimation of Clinton's support? When I worked in survey research (long ago in the dark ages of the 1980s), it was well-known that while women answered the phone about 70% of the time (making it more challenging to survey men) women were more likely to answer survey questions with "Undecided/Don't know/Not sure". This was particularly noticeable for older and less educated women, even though the voting rates in the 1980s for all women were already higher than those of men and were getting higher all the time.

I don't know if women still resist telling pollsters what they really think but if they do, does that help explain why the pre-primary polling was off? Women might have also disproportionately contributed to the "Bradley Effect" in the exit polling yesterday, if indeed the exits showed Obama ahead. Survey research seems to have consistently found that Clinton does best with older and less-educated/lower income women, exactly the kind of women I remember were so difficult to get answers from

Finally, like Ed Kilgore, I wondered about the absentee ballots and women, especially the older women who form Clinton's base, although absentee voters are not necessarily less likely to respond to polls unless New Hampshire has a large population of snowbirds who left for points South right after the holidays. What do we know about who tends to use absentee ballots and when they tend to be turned in?

____________________

Emily Shore:

Ed Kilgore: I don't know what the results were for absentee ballots in New Hampshire. I do know, however, that the Clinton campaign was pushing very hard for people to get them and to fill them out. They were calling up as many of their supporters as they could manage (this was early December) and they were sending out absentee ballot forms, including one to my own household. So if the Obama campaign wasn't doing this (I don't know if they were or not), then it certainly could have been a factor.

____________________

cmmdc: "I don't know if women still resist telling pollsters what they really think but if they do, does that help explain why the pre-primary polling was off? Women might have also disproportionately contributed to the "Bradley Effect"..."

Wouldn't this be more of a "Hillary Effect," in which women voters resist saying they are voting for the woman because of perceived social pressure not to (excessive nasty negative media / lots of talk about how its "wrong" for women to vote for a woman "just because she's female") and then do what they want in the voting booth?

Lots of women are socialized to not disagree publicly or be publicly "disagreeable" - and the Obama people + the media + Edwards at the last debate certainly helped make it seem "unacceptable" to be pro-Hillary. Pro-Hillary voices were completely crowded out of the public discourse - it wouldn't surprise me at all if some women decided to not advertise their preference. Talking to some of these voters, it almost seemed like they had been "closeted" until election day - and their reaction on election night seemed cathartic.

I personally heard a lot of women voters say variations on the second sentence of what Patrick Oliver posted above. I wouldn't have thought it possible if I hadn't seen it myself.

____________________

cmmcd: the exit polls were not particularly far off, I'm not sure where you heard that. They indicated a very close race and could not make a call, which they certainly would have if they had Obama up by 8.

As far as women being unwilling to tell pollsters the truth -- they couldn't tell the truth to a machine? The ARG and Rasmussen polls are done by computers, not real people, so that would negate that theory. As for the "Bradley/Wilder" effect, that has been disproven more than it has been proven, most recently in the Deval Patrick gubernatorial race in MA, where the polling data was very accurate.
The CNN/WMUR poll actually slightly overestimated the number of women who would vote, so if they were unwilling to state their opinion, it didn't show up in the numbers. Additionally, there was a higher percentage of women who said they had definitely decided, also not indicative of any additional softness.
Unless we assume all of these polls were in error -- an unreasonable mathematical assumption -- something happened between Sunday and the vote. Sen. Clinton herself has stated that her showing emotion helped her connection with the voters and played a part in her comeback. What we do know is that whatever it was that happened happened almost exclusively with women. Had it been turnout, or experience, or a collapse of Obama support, one would have expected a more even split in the shift.
We also know that the Clinton organization itself was just as surprised, as multiple sources, including Andrea Mitchell on NBC, report that they were hoping to lose by only 4 or 5 points. They were so displeased with their organization that they have already brought on several veteran hands, including Maggie Williams, to fix the "problems" (which resulted in a victory, apparently).
If there is a change pollsters need to make, it is to keep polling until the day of the election.

____________________

p_lukasiak:

To me, it looks like a sampling error, combined with "volatility"

The Suffolk poll internals showed that the race was almost dead even among "unlikely to change" voters. But it included only 53% of women in their sample, when they comprised 57% of the electorate. (Can't tell if the 53% was predictive or "forced" so the results reflected expectations based on prior elections.) Given the wide discrepancy in how the genders were predicted to vote (women +15 for clinton, men +26 for Obama) adjusting for sampling error goes a long way toward explaining why the "39 Obama 34 Clinton" was incorrect.

The other issue is volativity. Among "committed voters" Suffolk had Obama up by only 1.5 points because Clinton voters were more sure who they were going to vote for. And, contrary to conventional wisdom, men were overall significantly less "committed" than women. Suffolk hit a near bullseye in terms of the distribution of the vote of women -- exactly the same (46%) for Clinton, off by one for Obama (33% predicted, 34% exit poll). In other words, it doesn't look like women changed their minds at all!

With the men (who were more like to not be fully committed) Clinton got 9 points more than predicted (to 29%) and Obama lost four points (to 42%).

So it looks like there were two surges... a surge of women in general to vote (which benefitted women), and a last minute surge by men toward clinton, and away from Obama.

____________________

mike0625:

p_lukasiak- you've got to be wrong about sampling error- sampling error will be very small over an average of 15 polls.

Does anybody know how long it takes for Gallup or CBS to deposit their data at the Roper center? It would be interesting to see what the LV models actually look like. And when does the raw exit poll data become available?

____________________

Ron B.,:

I don't think the polls were wrong at all. I do think that in the last 24-36 hours the media went into a frenzy over the "tear jerking" and many of the male commentators were blatantly skeptical or scornful of Sen. Clinton's "performance". I think it was a case of the media being able to give it to Clinton in response to her having treated the media badly during the Iowa campaign.

From my perspective it was seen as a case of "piling on". It was compounded by the debate question about Clinton being disliked by the public.

I am a strong Obama supporter and was in Iowa 4-5 times, including caucus night. And my read during that week-end was that the TV media was truly obsessed by the "event".

Finally, I think that a large number of women in N.H. said, "Enough is enough" and voted in massive numbers for Clinton.

And just to express their scorn and contempt for the media they thumbed their collective noses at exit polls.

Just a theory, but it makes sense to me.

____________________

kimbiaje:

The real losers were white men.

The N.H. democratic primary result is probably not an example of the �Bradley Effect� or covert racism. Pollster.com's standard estimates leading up to the vote were 36.7% for Obama, 30.4% for Clinton, 18.4% for Edwards, 5.6% for Richardson, 2.5% for Biden, and apparently 6.4% undecided or voting for minor candidates. The actual results were 36.4% for Obama (almost exactly correct), 39.1% for Clinton, 17% for Edwards, 5% for Richardson, and 1% for Kucinich.

The only candidates that actually lost ground on Clinton were white males (Edwards and Biden). More than likely, the massive media attention focused on Clinton's heart felt explanation for why she is running and her debate performance helped her capture a slight lead among undecided voters as well as taking some votes (perhaps women) away from other candidates in the race. There is really no evidence of a �Bradley Effect� since the only candidates that actually lost percentage points to Clinton were white.

Another possible contributing factor that other people have noted is what may be come known as the "Clinton Effect" - that people were embarrassed to admit they were supporting her since she was unpopular in the national media. That theory works well with the actual results where she took votes away from all of the candidates and not just Obama.

____________________

Hangups, screeners, and rabid supporters....

The number of people simply refusing to answer polls is huge. Who would stay on the line? Rabid supporters of their candidate. Obama probably has more rabid supporters than Clinton. There was another post here about soft support, and a lot of Clinton's was soft.

Also note that, in both IA and NH, the two outfits that were most accurate were LOCAL. I'd bet they also got a higher participation rate.

____________________

Ed Kilgore asks above about absentee voters in New Hampshire. Here is a first cut answer. (And no, Ed, I don't think you are a crank at all.)

New Hampshire requires you to certify you will be out of town or are physically unable to vote in person, or have a religious reason for not voting in person on election day. It is a misdemeanor to vote absentee unless these conditions apply.

So there is no early voting and absentee ballots are the traditionally restrictive type, not the more recent "easy" absentee ballots.

The result, in 2004, was that 4.99% of ballots statewide were cast by absentee vote for the Democratic primary.

That is not enough to account for the differences we see between polls and outcome, even granting (and I do) Ed's point that absentee votes would have been cast before Iowa. With only 5% absentee, the absentee vote would have to be 20 points different from the election day vote to move the total vote by just ONE Percentage point. For example, 30%*.95+50%*.05=31%.

The proportion of absentee votes have not yet been released by the New Hampshire Secretary of State's Office. Because the absentee rules have not changed, I don't expect a dramatic increase in absentee participation in 2008. Even if absentee votes doubled, the proportion seems still too small to account for the 8 point underestimate of the Clinton vote.

Thanks for raising this. In many states it might be much more important. Just not in this one.

Charles

____________________

Joel:

This is absolutely fascinating. Did the polls get Edwards and Richardson right? If it overstated their support, could that be what the polls were missing in the late break for Clinton? If it did not overstate their support, is this a good argument against the methodological explanations?

____________________

Michael:

I know what happened. Clinton supporters were turned off by the media's Obama coronation and refused to participate in polls. Therefore only the support for Clinton was underestimated.

This wasn't a shift in opinion, but a shift in sampling. Pollsters need to look at how likely supporters of a candidate are to participate in polls during overwhelmingly negative media coverage of their candidate.

____________________

Anonymous:

p_lukasiak- you've got to be wrong about sampling error- sampling error will be very small over an average of 15 polls.

I'm not a statistician, and I think I used the wrong word.

Methodological error perhaps? Anyway, the point is that polls tend to adjust their results to reflect a chosen demographic profile -- and that includes the "right" gender balance. What you don't see in these polls is projected turnout -- and unless you are projecting turnout, whenever the assumed demographic mix turns out to be different from the actual demographic mix, you have problems.

And what I suspect happened that accounts for a lot of these polls is that they assumed that the Democratic vote would be 53-54% female... and it turned out to be 57%. So even if they project correctly the right percentage of the male and female votes for each candidate, the projections as a whole turn out to be wrong.

Add to that the fact that the discrepancies in gender turnout are not taken into consideration when calculating the margin of error, and you get completely unreliable polling results.

I think pollsters are going to have to start being a lot more careful about making adjustments based on assumed demographic charateristics of the people who will vote, especially since there are "open" primaries, and both parties have competitive races.

____________________

Matt:

YEA BUT I WAS AT A PRECINCT, AND BARACK GOT 800+ VOTES, HILLARY 500.

RON PAUL ALSO GOT MORE THAN 7%.

Of course, this was just one voting location, but still, I don't believe the results.

Clear Evidence Of Widespread Vote Fraud In New Hampshire: http://www.truthnews.us/?p=1603

____________________

Harald K:

"We also know that the Clinton organization itself was just as surprised, as multiple sources, including Andrea Mitchell on NBC, report that they were hoping to lose by only 4 or 5 points."

This is supported by another source: prediction markets. They were very confident that Obama would win until the results started ticking in.

Prediction markets naturally rely a lot on polls, but when they do better, many think it's because they take insider information into account.

____________________

Randy Brooks:

I was in New Hampshire working for John McCain, waving signs, working phones and going to polling places.

My day job is market research - 35 years - President of a firm I founded 20 yrs ago.

The press from about sunday on focused on "what will clinton do now that she has lost Iowa and New Hampshire. Chatter all day long was Begala and Carville were coming on board. It was over for Clinton - Obama the winner. The people of New Hampshire pay a huge amount of attention to this stuff. I think Obama voters failed to show up (young people often do) and that his staff and volunteers may not have worked the "get out the vote" stuff as hard as they should have.

Rookie mistake

____________________

e. Flaherty:

I think (not that it makes any difference) that we should take another poll to see what went wrong with the first poll. Then we have the election to see what poll was right.

____________________

beezy:

Polling companies did not capture Hillary's coffee shop meeting.....and the public did not take it in until after the 5p Monday.

The greatest impact of the coffee shop event was on women. Look at the post caucus/primary data difference between IA and NH. They may be different states.....but do women fell that differently?

The cause.....coffee shop.....the effect.....moving women to come to HIllary's aid.

____________________

Paul B:

Perhaps the pollsters should factor in a new technology. Cell phones. Yeah, it sounds stupid but think it through. What demographics have already turned off the land lines and gone high tech. thereby eliminating themselves from pestering calls? My totally imperfect polling of friends and family shows that Dem's in the family have tuned out at a much higher rate as have those who make enough money to afford the switch.
Almost everyone I know who supports Hil has gone exclusively mobile and/or signed the do not call lists. Most of my Obama friends have left a phone in the house.
Chances I am right with this are almost nil. But thinking outside the box will provide more answers than poring over old data to make it fit new realities. Shop floor statistics 101.

____________________

bruce:

Since polls of primaries are a mix of estimates on WHO will vote, and HOW these voters lean, isn't it at least possible that the Republican figures so touted as accurate were simply a case of errors in turnout canceling errors in the lean?

In other words, it is possible that the GOP models were just as flawed as the Democrat, but that on the GOP side the errors canceled each other out.

____________________

Kevin Houston:

from the article:


In fact, most polling shoots low and to the left,

Really? Pollsters shoot to the left? Do tell. };>

On a non-snarky note, what about the feedback effect? The effect that poll results have on how much media attention is given to each candidate, and the effect that has on voters forming their opinions on who to vote for?

If polling can be so far off, is it a good idea to use polling thresholds to determine debate inclusion, etc?

On the one hand, it might tend to herd people into a certain self-fulfilling result. On the other hand, the fact that it can be wrong is strangely comforting; it shows that people are capable of forming their own opinion despite any effect the polls might have.

____________________

John Anderson:

How sad that even those within the polling industry are so quick to denigrate their own work. There are two possibilities here - either the polling results were incorrect or the tabulation of the election results was incorrect. When the votes are counted by machines that are known to make errors, it would make sense to rule out that possibility first, yet it is not even mentioned in most media. No need for conspiracy theories, no need to attack the Clinton campaign, no need to attack the polling methods, just machine error. Fortunately, Dennis Kucinich is willing to pay for the hand recount. If this was a machine issue, then once again there are two possibilities, accidental machine error or human manipulation...

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR