Pollster.com

Articles and Analysis

 

But What About an "Unbounce?"

Topics: 2008 , New Hampshire , The 2008 Race

One of the theories raised to explain the problematic New Hampshire primary polls is a late shift, perhaps on the heels of the Hillary Clinton "tears" story, that polls missed either because they stopped interviewing or completed the bulk of their calls on Sunday or earlier. Two prominent pollsters say such as shift is unlikely given one result in the exit polls.

For example, Andrew Kohut, writing in his must-read op-ed in yesterday's New York Times, concludes:

Yes, according to exit polls the 17 percent of voters who said they made their decision on Election Day chose Mrs. Clinton a little more than those who decided in the past two or three weeks. But the margin was very small — 39 percent of the late deciders went for Mrs. Clinton and 36 percent went for Mr. Obama. This gap is obviously too narrow to explain the wide lead for Mr. Obama that kept showing up in pre-election polls.

Gary Langer, director of Polling for ABC News, agrees in a lengthy review posted this morning:

[T]he exit poll asked voters the time of their decision. Seventeen percent said they decided on Election Day; they voted for Clinton over Obama by a 3-point margin, 39 to 36 percent – hardly a significant swing from the overall result (Clinton +2). Those who said they decided in the previous three days, 21 percent, favored Obama over Clinton by 3 points, 37-34 percent – further deflating the late-decider argument. Those who decided previously, 61 percent of voters, favored Clinton over Obama by 41-37 percent.

For reference, here is the exit poll data that Kohut and Langer cite:

01-11 2008 time of decision.png

But wait. Putting aside the issue of whether respondents can accurately recall when they reached a decision, why does the absence of a big Hillary bump among the late deciders rule out the possibility of some late shift away from Obama? Keeping in mind where Clinton stood in polls before Iowa, we would not be looking for a Clinton bump as much as an Obama "unbump" (to paraphrase the comment left yesterday by my friend reader Mark Lindeman).

Let me explain. Start by looking at our chart of the polls conducted in New Hampshire during 2007. Our trend estimate shows Clinton winning between 35% and 40% of voters between June and early November. As always some individual polls were a little higher, some lower. Her support declined slightly in December, to an average of about 32%, with the usual variation slightly higher and lower. In December, the undecided category averaged 11% and the support for Richardson, , Biden, Dodd and Kucinich was roughly twice what they received on Election Day. Give Clinton a proportional share of the undecideds and those that moved away from the single digit candidates, and she was headed to roughly 37% of the vote. Thus, putting aside whatever happened in the weekend after Iowa, Clinton would not have needed any massive last minute gains to get to the 39% she received in the final count.

 NHTopzDems400m0111.png

Now on the other hand, if Obama had surged after Iowa to a double digit win as the polls seemed to predict, we certainly would have expected to see late deciders favoring him heavily. But if that weekend bump collapsed (or if it never existed in the first place)? In that case, we would not expect to see much of a difference between early and late deciders. Clinton needed only a modest increase in support to reach 39% of the vote.

Consider a historical example. In 1984, Gary Hart surged to victory along a similar trajectory as what polls seemed to forecast for Obama last week. According to the ABC News/Washington Post polls compiled by Samuel Popkin in his book, The Reasoning Voter, Hart trailed Mondale 13% to 37% just before the Iowa Caucuses (with John Glenn running second (at 20%). The Iowa Caucuses were held on a Monday in 1984, a full week before the New Hampshire primary. On interviews conducted Wednesday through Friday after Iowa, Mondale's numbers held constant (38%) while Hart moved up (to 24%). Over the course of the week, Mondale steadly lost support while Hart continued to rise until moving ahead on the Monday before the primary by an eight point margin (35% to 27% -- although ABC reported a three-day rolling average at the time showing the candidates tied). The next day, Hart defeated Mondale, 37% to 28%.

The ABC exit poll also asked voters when they made up their mind, and the pattern is what we would expect. Among those who made up their minds over the final weekend or the two days just before the primary, Hart led Mondale by a four-to-one margin (56% to 14%).

01-11 1984 time of decision.png

So let's consider a hypothetical question: What would have happened if late deciders had broken for Barack Obama by the same margins as they preferred Hart to Mondale in 1984? If I go to my spreadsheet, play "what if" and imagine that late deciders -- those who made up their minds over the last three days -- had preferred Obama 53% to 20% (while all other preferences held constant), Obama would have defeated Hillary Clinton by 10 points (43% to 33%).

Of course, none of this explains exactly what happened last week, and the time-of-decision exit poll data is just one small piece of the puzzle. Opinion polls may have accurately measured a post-Iowa bounce for Barack Obama that "unbounced" over the last 24 hours, or the apparent surge may have been an artifact of some survey error (or perhaps some combination of both). But either way, the lack of a difference between late and early deciders does not tell us much. It certainly does not preclude -- by itself -- the possibility of shift of voters to Obama on Saturday and Sunday that shifted back to Clinton on Monday.

 

Comments
p_lukasiak:

what happened last week is easy to explain...

A whole lot of people who had been "Hillary leaners" prior to Iowa looked at Obama, and liked what they saw, and told pollsters that they were now "leaning to Obama." Then they realized that they didn't really know anything about Obama, and that while his rhetoric was red-hot, there was no record to determine if he had the chops to govern the way he said he would. And enough of them switched back to their "comfort zone" of Hillary to give her the win.

The dynamic on the GOP side was different. There was a huge pool of voters who had supported McCain in 2004, and who were supporting him at the beginning of 2007. They knew him, and liked him, but when the media declared his candidacy doomed, they started looking elsewhere, and leaning toward "other candidates". When McCain suddenly became viable again, those voters "leaned back" to McCain, and because they knew him and were comfortable with him, they wound up voting for him.

(Indeed, I would posit that there were more than a few people who switched from Obama to McCain in the last 2-3 days....)

Thus, the GOP polls proved to be "right", and the Democratic polls didn't. But it was all about "comfort zones" and familiarity and previous support of a candidate. If Obama had had another week in NH, he might have won (then again, Hillary could have made the necessary adjustments in her campaign, and come out even more ahead...)

____________________

Gary Kilbride:

I almost always prefer the explanation requiring the least number of variables. In this case it's very obvious where that leads me: The huge Obama bump never existed in the first place.

Chris Bowers posted a thread on OpenLeft yesterday, saying, (paraphrased) "Maybe Primary Polling Just Sucks." IMO, that's more of a valid explanation than the vast majority of the multipronged guesswork.

I know I've never fully trusted primary polling, and one wagering website didn't in the late '90s, when political betting on the internet started to explode. There is a very popular option called, "Action Points," where you are penalized or rewarded based on margin from base expectation. In other words, taking New England -13.5 on Saturday and they win by 28 you would be rewarded accordingly, instead of the same 10/11 payoff regardless if they won by 14 or 114.

Anyway, that sharp website accepted Action Points on political wagers during general elections, but not on primaries. I emailed them about it, and point blank they told me they weren't confident enough in primary polling, and therefore race status, to rely on it with Action Point liability. I always thought that was rather strange and wimpy, but you've got to remember at that point polling was much more scarce, particularly in primary season.

I don't mean to be questioning polling legitimacy on a poll-oriented website. But it's simply reality that not all polling is equally reliable. That's borne out here when different methodologies are scrutinized. Just like I know darn well some of my betting systems and angles are superior to others. Give me general election polling over a primary version, almost every time.

____________________

David s:

I think a very large part of this was under-polling of poorer (white) voters.

As a group, they don't like to talk to pollsters. It's extremely difficult for pollsters to get a true grip on a demographic that largely refuses to participate in polls.

It was an unseasonably warm day. A lot of these poorer voters came out to vote, and the exit polls show us that Hillary received a strong majority of the poor vote.

Is this another way of saying it was the Bradley effect? In a small way, yes. But not because voters lied to the pollsters. Not because voters were embarrassed to say there weren't voting for a black candidate. Only because the pollsters were unable to reach a demographic that largely didn't vote for a black candidate.

In hindsight, un-reachability by pollsters, not voters lying to pollsters, may be the true explanation for the Bradley effect.

____________________

Neil:

I'm surprised more people aren't considering the Medved/Garner theory. 43% of New Hampshire voters are independent. Their first choice was Obama. Their 2nd choice was JOHN McCAIN. Once the Obama bounce polling data was released, he had a huge lead, while McCain was running neck and neck with Romney. The Independents who had polled for Obama figured with Obama enjoying a comfortable lead, their votes would be better served for McCain. Consequently Obama was the surprise loser and McCain the surprising large winner.

____________________

David s:

Neil,

I agree that if Obama voters crossed over to the Republican race, they probably would have voted for McCain. And before the final numbers were in, I also had that theory in mind.

But the evidence just doesn't seem to agree with the theory. McCain finished right in line with a most of the pre-election polls. Most of the polls had him getting between 34% and 35% of the vote.

In the actual election, McCain received 37%. That's only 2 or 3 points off. And the spread between he and Romney was also almost exactly what the polls predicted. Although both McCain and Romney received about 3 points more than the polls predicted.

While Obama finished only 3 or 4 points off of his predicted percentage, the predicted spread between he and Clinton was between 11 and 17 points off!

Had any significant number of "Obama voters" crossed over to McCain, McCain 'should' have finished with a far larger percentage of the votes. And if McCain's extra 3% was Obama's loss, where did Romney's extra 3% come from?

That said, I'm certainly not ruling out the possibility. For all we know, one demographic of voters abandoned McCain while at the same time, Obama voters came to McCains rescue. Or maybe Obama voters went to Romney as well. Either possibility would have made the Republican race's numbers only appear correct by sheer luck.

Professional pollsters with the raw exit poll numbers might be able to tease out the truth. But I'm not a professional pollster and I don't have access to the raw numbers.

But taking the numbers we have at face value, and setting aside any wild, self canceling possibilities like those above, I don't see the evidence support the theory that any significant numbers of Obama voters crossed over to vote for McCain.

I actually think it's more likely that the pollsters simply didn't talk to enough poor white voters. People who planned to vote for Clinton the entire time. Not the Bradley effect as such, but definitely a large demographic of voters who are reluctant to vote for a black candidate.

____________________

Mark Lindeman:

Gary, I think Mark B. would be the first to acknowledge that polling in New Hampshire in a four-day window after the Iowa caucuses, including a weekend, was an iffy proposition.

But I don't think it's good enough simply to say that the bump "never existed." You can't use a constant ("Primary Polling Just Sucks") to explain a variable (that orange line shooting up).

John Zaller, or some crackerjack political psychologist, should stop by and blog this. Seems to me that most NH Democrats are pretty positive about both Clinton and Obama (and Edwards, for that matter). Think of them carrying little mental bags with two kinds of marbles (aka "considerations"): reasons to vote for Obama and reasons to vote for Clinton. Over the weekend, a bunch of Obama marbles got dumped on top, and people who responded to surveys tended to grab one of those when they were asked who they would vote for. By Tuesday, some folks had some new Clinton marbles, but also, people have more time to think about their vote decisions than their survey responses, so lots of people in effect remixed their marbles, and found that they still had more reasons to vote for Clinton.

I think something along those lines (also accommodating late decisions about whether to vote) might reconcile your impression that nothing 'really' happened with our impression that something did. (How one prefers to describe it depends on one's theory of mind. I assume the process was more emotional or affective, at least for some voters, than the image of marbles in a bag suggests.)

____________________

The results differential between Clinton and Obama is enticing everyone to look at the wrong campaign dynamic. We should be looking at Edwards, too, to understand what happened in Hew Hampshire.

As I argue in my Brooking's commentary, Clinton and Obama are drawing support from different Democratic voters. To state it crudely, Obama's support comes primarily from those under 40 while Clinton's comes from those over 40. Importantly, Clinton is competing for the older, "establishment" voters with Edwards. The New Hampshire "surprise" is thus a function of Edwards voters shifting to Clinton.

Obama's support between Iowa and New Hampshire is very consistent. He won almost the same share of delegates as he did vote, and the exit polls show that his support among those under age 40 remained steady.

Clinton's overall support between Iowa and New Hampshire increased by almost the same amount that Edwards' declined. This also reveals itself in the exit polling. Clinton's support among those over age 40 increased by almost the same amount as Edwards' declined. (There were turnout differentials at play, too.)

Applying this story to the pre-election polls, the polls conducted on the 5th and 6th seem to show a bounce for Edwards, likely arising from his Iowa second place finish. Clinton's support decreases. Those conducted through the 7th seem to suggest a shift from Edwards back to Clinton.

If I were to guess at an explanation, Edwards looked to be piling on Clinton coming out of Iowa, attacking her rather than going after the front runner, Obama. Since Edwards and Clinton have the same support base, this made sense from a tactical standpoint. The voters, however, did not see this as fair. Edwards looked mean, kicking an opponent when they were down. Clinton's emotional moment on Monday crystallized this feeling among these last minute wavering voters.

____________________

Mark Lindeman:

Michael, maybe so. But what I see in the final vote counts by municipality is a high correlation between Obama and Clinton vote shares, and much less variable Edwards support. That's a different answer to a different question, and I have no way of knowing what the correlation matrix "would" have looked like.

____________________

Mark, I'm not really sure I understand your posting. Can you elaborate how the correlation matrix provides evidence that refutes my theory?

According to the entrance/exit polls, Edwards' support was flatter across nearly all age demographic categories in New Hampshire, compared to Iowa, where he did better among older voters. This is consistent with your observation of the low variability of Edwards' New Hampshire votes across localities. Clinton did much better among older voters (and they turned out at higher rates) in New Hampshire than she did in Iowa. Thus, reason tells me that Clinton's victory in New Hampshire is a result of these voters shifting from Edwards to Clinton. Understanding why the pre-election polling failed to pick up this shift up will illuminate why these polls did so poorly at forecasting the New Hampshire election outcome.

____________________

Bob T:

Wow. There goes Mark again.

First, he praises the screwed-up Des Moines Register Poll which was plain lucky to get the right results with a very bad sample and now he is using the screwed up exit poll results from NH to make a case against late deciding voters.

As Chris Matthews has pointed out repeatedly, the NH exit poll was off by a mile and had to be drastically weighted to match the final vote totals. Of course you will not see any strong late shift to Clinton because the original exit poll had Obama winning outside of the margin of error.

All of the exit poll results are not trustworthy because of this and the same can be said of the Des Moines Register poll because its sample was so messed up too.

A quality poll is not one that is just lucky to get the right results or one that is weighted to the final vote totals, although the more I read what Mark has to say here, the more it seems he believes these things are true.

____________________

David s:

Bob,

Referencing your point about the very inaccurate exit polls. Matthews isn't the only member of the media to make mention on this. In fact, it seems all the major media were using these exact same exit polls. The raw, un-weighted numbers should be available to any members of the media participating in the pool.

I've seen a LOT of media pundits twisting themselves into knots in order to deny any possibility that this was a reemergence of the Bradley effect. Yet not one of these pundits has referenced any "hard" data other than the heavily weighted final numbers. I suspect many of these pundits don't even realize that the exit poll numbers reported by CNN, MSNBC, Fox and the others were heavily weighted to match the final results.

Exit poll numbers weighted to match the outcome should (I think) make it completely impossible to prove or disprove the Bradley effect. Is anyone aware of any media source or professional pollster performing analysis on the pre-weighed, raw exit numbers to determine the cause of the error?

I suspect a thesis or two (or ten) and any number of published research papers will eventually be written on this exact tally. But it's such a major story, I have to wonder if anyone in the press is doing (or has commissioned) immediate analysis?

If not, why not? The results could be one hell of a story.

____________________

jan:

the day she cried she won Newhampsher but lost the general election. in the exit poll people actully said that obama was the most electble, more people said that then before.
look for that the electibility factor. she became a victim and lost the street cred.
she is now seen as "the girl" instead of a tough woman. if you are a pollster this should be a conclusive proof of sympathy vote. if she winns but people think of her less electible then obama then that proves my asirtion

____________________

Michael, I am confused by one point you made above. In what sense is Edwards competing with Clinton for "establishment" voters? The rhetoric and stated policy positions of his campaign seems anything but "establishment." (Unfortunately, your Brookings link would not work when I tried it before posting this comment.)

____________________

Just a brief note re: the issue of exit poll results 'matching' the voting results: Ideally, the two should agree within a margin of sampling error, allowing for the fact that the vote is supposed to represent the TRUTH, vis-a-vis the 'population' of relevance, while the sample is just that, a sample. When exit polls don't agree (considering the m.o.e.) these polls are usually regarded as faulty. But all modern-day polling experts recognize that characteristics of those polled have to be taken into account (adjusted)...and so do the characteristics of those who are NOT polled (more adjusting)! Two interesting possibilities remain, neither of which seems to have been mentioned: 1. that inadequate account was taken in the exit polls of the 'relevant' characteristics of those who were polled AND those who were not, Or 2. That the population 'data' do not represent the population itself. The latter would correspond to errors in voting machines (hacking?) or in the counting, somehow. (Both could be operative.)
Just such a possibility is reflected in Kucinich's call for recounts, based on the apparent mismatch (details needed!) between the counts based on paper ballots and those based on machines. Why is the latter not getting more attention?
Final note: Joseph Stalin once stated: "The people who cast the votes decide nothing. The people who count the votes decide everything."

____________________

Mark Lindeman:

Bob P, the possibility of massive miscount isn't getting more visible attention because there isn't much to support it. My comments here may be helpful.

Michael, I don't think the correlations refute your argument. They couldn't. But if Clinton and Edwards were in some particular sense competing head-to-head for votes in New Hampshire, I would have expected some covariation across place. More a hunch than a hypothesis.

I'm also having a hard time spotting the Edwards bumplet.

I would have to think more about the Iowa/New Hampshire comparison, but I am resisting. I see that indeed Obama's share in Iowa is very similar to his share in New Hampshire, but he got there by such different means that it doesn't seem very consequential to me. I am somewhat struck that Edwards did especially well in Iowa among voters who prioritized "cares about people," and Clinton beat him among that group in New Hampshire. But even there, Obama's share drops almost as far. (I also see the age tabs; I just don't have much to say about them right now.)

I'll keep thinking about it. I'm just not convinced that Edwards was very important in the last few days.

____________________

Fran:

Will the raw exit polls and the weights that were applied to them ever be released? I saw a lot more confidence in the early broadcasts that Obama had won than I saw in the early broadcasts in November 2004 that Kerry had won. We need to either get our exit polling straight or stop doing it - other countries use it to validate the honesty of their voting. We can't do that.

____________________

MSS:

I wonder if anyone can help with this question: Why would the NH GOP exit poll results reported by the Los Angeles Times and by MSNBC be so different? For example, according to the LAT, McCain beat Romney among independents 40-27, but among Republicans, it was 34-35, respectively. However in the poll reported at MSNBC, McCain led in both groups (38-30 among independents, 37-33 among Republicans).

That is quite a substantial difference!

In the source notes, the LAT says it was the National Election Pool Exit Poll, by Edison/Mitofsky, and NBC is listed as part of that pool. So, it appears to be the same poll, until one looks at the breakouts.

What is going on here, or what am I missing?

____________________

On the issue of the exit polling confirming the pre-election polling, I suspect that the media misinterpreted the exit poll data (Disclaimer: I consult with Edison Media Research on the exit poll decision desk, though I was not present on Tuesday).

The evidence: At 8:01pm, FOX News released a projection that the "exit poll estimate" was Obama 39, Clinton 34. Matthews relays a similar story of being told at 5:30pm that the exit polls showed Obama would win big. Yet, later Tuesday evening, I heard Tim Russert say that the exit polling provided the first hints that Clinton would do better than expected.

How can both viewpoints be right?

My experience on the decision desk of the exit poll organization tells me these poor projections were most likely based on what is known as the "composite estimate", one of several forecasting models that are available in the exit poll computer system to predict election outcomes. The composite estimate incorporates several pieces of information, including what is known as "the prior", the best guess of the election results based on pre-election polling. I don't think I have to say anything about the accuracy of those pre-election polls.

There are other forecast models that do not incorporate the prior into their estimates. I strongly suspect that these other forecasting models were pointing to a different outcome than the composite estimate. Whoever was providing information to Russert was clued in to this fact.

This is a lesson for all consumers of the exit polls, particularly those in the media who filter the information to the public. Be sure that all evidence points in the same direction before trusting early projections.

We will be able to confirm my suspicions once the exit poll data is publicly released. For now, the media consortium that commissions the exit poll keeps the information to themselves so they can benefit from what they paid for (it's a multi-million dollar survey, and one of the few private surveys that is publicly released).

P.S. for those who could not access my Brooking's commentary, it is available at this URL:

http://www.brookings.edu/opinions/2008/0109_elections_mcdonald.aspx

____________________

The pollster.com posting engine cut the URL. A link to my commentary is still on the Brooking's front page:

http://www.brookings.edu/

____________________

Mark Lindeman:

Fran, any country that uses exit polls to validate the honesty of their voting ought to stop, immediately, and find a better method.

It's true that the U.S. government sometimes funds exit polls in other countries. Make of that what you will. But it's no way to build a reliable system.

____________________

Greg:

Per comments by Sunday's Gust Pollster, I'm curious if anyone ever asks in an exit poll "when did you decide you were going to vote in the election?" Or is that the kind of question that just doesn't get a useful answer?

____________________

cmmcd:

The ABC-Washington Post poll story tonight says that nationally Obama leads Clinton among all men (42% to 33%) and among black women (59% to 35%) but that Clinton has a 20 point advantage over Obama among white women and a 53% to 30% advantage among married women, resulting in an overall Clinton lead of 42% to Obama's 37%.

Aren't these the kind of demographic differences that could have lead to the emergence of the Bradley effect in New Hampshire? Older white married women might have underreported their support for Clinton in pre-primary polling (partly by failing to respond to surveys) and possibly by overreporting Obama support to young interviewers in the exits.

____________________

Why are we looking for racism in a Democratic primary in NH when there was no evidence of it in a general election in neighboring MA involving Deval Patrick? Especially when the "Bradley/Wilder Effect" didn't happen in Iowa. Somehow the Iowans are more open to voting for a black man than people in NH? Going back 20 years to justify a theory is dubious at best.
I think the bounce happened, I think the odds against all those polls being so wrong are immense, especially when they got the GOP race right. But I also think the bounce was temporary, that when voters, especially women, got up Monday to read newspaper articles about how the race would be over if Obama won decisively (something which NH voters have often rebelled against), and that there were Clinton advisers who might tell her to get out of the race to avoid further embarrassment, and then saw Hillary's emotional response to the question in the diner (and saw it repeatedly), many voters who had shifted to Obama in light of Hillary's third place finish and his stirring speech "returned home". When asked in the exit poll when they "finally decided", they chose to ignore their temporary dalliance with Obama and considered themselves months-old Clintonites, which most of them had been.

____________________

Greg:

Um, Barry, in Iowa there wasn't a secret ballot. The "Bradley/Wilder Effect" assumes that people behave differently in private than in "public". Iowa would therefore be more likely to have a reverse "Bradley/Wilder Effect", since voting in a caucus is far more public than just talking with a pollster.

____________________

Greg

Good point. I still feel there isn't any evidence that it existed in NH (given the shift being mostly among women), but I guess it can't be ruled out in Iowa. I'm not exactly sure why people in an almost entirely white state would have to worry about being seen voting against a black candidate, but if the B-W effect exists, I guess the example of Iowa could be used.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR