Eric Dienstfrey | February 2, 2008
McCain 39, Romney 32 (release)
Clinton 44, Obama 42... McCain 41, Romney 35
Obama 44, Clinton 42... Huckabee 31, McCain 29, Romney 27
McCain 39, Romney 32 (release)
Clinton 44, Obama 42... McCain 41, Romney 35
Obama 44, Clinton 42... Huckabee 31, McCain 29, Romney 27
New York State
Clinton 54, Obama 38... McCain 61, Romney 24
WSMV-TV/Crawford Johnson & Northcott
I'm blogging from an airport with just enough time to like and block quote two helpful explanations of what the missing "cell phone only" households mean to the vast majority of surveys that do not attempt to interview Americans on their cell phones.
First, Wall Street Journal "Numbers Guy" Carl Bialik has a very clear, concise review of the central issues. He draws on conversations with both Jeff Jones of Gallup and Scott Keeter of the Pew Research Center, the two organizations that have done the most significant political surveys of Americans over their cell phones. It is well worth reading in full if you are new to this issue. Bialik stresses that the evidence so far shows that the missing "cell onlys" have little impact on political survey results:
The impact of losing cellphone-only respondents, however, may be exaggerated. Their numbers aren't big enough to budge most poll results by more than a point or two, Gallup has found.
People who use only cellphones, on average, are younger, more likely to rent their homes and have lower incomes than their tethered-telephone peers. But once you adjust for age, cellphone-only users have similar political viewpoints. Although he thinks cellphones should be included, Jeffrey M. Jones, managing editor of the Gallup Poll, asks, "It's still a lot of cost and effort, and what's the payoff?"
Meanwhile the Pew Research Center yesterday released another mega-study of the cell-phone-only issue. It's a must read for those who want all the nitty-gritty details. Here's the bottom line:
On key political measures such as presidential approval, Iraq policy, presidential primary voter preference, and party affiliation, respondents reached on cell phones hold attitudes that are very similar to those reached on landline telephones. Analysis of two separate nationwide studies shows that including interviews conducted by cell phone does not substantially change any key survey findings.
Update: Carl Bialik has much more on his blog, and also reminds me to link to the recent Public Opinion Quarterly special issue on cell phone surveys and my own series from last summer on the cell phones and political surveys (Part I and Part II).
Dems: Clinton 47, Obama 37
Reps: McCain 48, Romney 20
Dems: Obama 31, Clinton 24
Reps: McCain 20, Huckabee 19, Romney 12
Clinton 48, Obama 35... McCain 43, Romney 25
Obama 51, Clinton 40... McCain 48, Romney 34
Greenberg Quinlan Rosner Research (D)
Clinton 44, Obama 38
ABC News and The Washington Post reported results of a new poll today, which asks Americans which major event they are more excited about, the Super Bowl or the Super Tuesday primaries. It got me thinking about a very good point made yesterday by reader "fourth" in our comments section:
I remember seeing something about weekend polling here and how Sunday afternoon/evening was actually a great day to catch people at home. This sunday is the Super Bowl! How are any of the final polls going to be able to account for this? Half the nation is going to be somewhere else watching the game. Basically we won't get to know the real impact of the debate tonight, as we have to question any movement over the weekend.
What fourth remembers is an observation I made in discussing the potential pitfalls of weekend interviewing:
When pollsters like me worry about "weekends" we mean Friday night and Saturday, not Sunday. Actually, late Sunday afternoons and evenings are among the best times to catch people at home, especially in the winter. And I see much less to fear from a survey that begins calling on Friday and finishes on Sunday, so long long as all of the "no answer" numbers from Friday and Saturday get dialed again on Sunday night.
Sunday night is typically a great time to call, but obviously, as fourth and other readers point out, the Super Bowl presents a big barrier. Ordinarily, pollsters would not call during the game itself, as reader bdevil02 points out, as "the response rate would be dismal." In this case, frankly, I am not sure what the media pollsters will do.
Depending on the procedures that pollsters use, the Super Bowl does create some chance of a bias due to people who will simply be unavailable on Sunday. We saw evidence that strongly suggested a skew that worked against Obama over the Thanksgiving and and Christmas holiday weekends. Could Sunday night provide a similar effect?
My hunch is that it probably will not, for two reasons. First, as the ABC/Post poll shows, voters who are more excited about the Superbowl than Super Tuesday tend to be male, younger, single, less educated and from the Northeast (not surprising given the teams involved). Those subgroups cut both ways in terms of the Obama, Clinton contest: Younger voters and men tend to support Obama, while Clinton does better among downscale white voters and in the northeast. So the net impact -- assuming any sort of Super Bowl "bias" -- may cancel out.
Of course, that particular ABC/Post question is not completely on-point. The demographic break-outs we would really like to see involve the questions they asked about plans to watch the game and identify the "big" fans of professional football (those who would not respond to a poll call during the big game).
Second, polls that call back unavailable respondents over successive evenings -- such as the Gallup Daily -- will minimize the impact of any potential Super Bowl skew. Consider: If you are heading to an evening party, the pollster still has a good chance of catching you at home and willing to participate on Saturday afternoon or Monday or perhaps earlier in the day (or later in the evening on the West Coast) on Sunday. As long as the pollster is persistent about calling back, the Super Bowl itself should not provide the same sort of barrier as travel over a holiday weekend.
Of course, the Super Bowl presents a much bigger potential problem for surveys that make little or no effort to re-contact unavailable respondents (such as the Rasmussen Reports nightly tracking or any survey completed in just one night). And remember, just because a pollster calls over multiple nights does not mean they go to the extra trouble and expense of doing "call backs." That particular wonky detail is, unfortunately, something that pollsters often fail to disclose in their public reports.
Update - I emalled Gallup to ask about their plans. Gallup's Jeff Jones responds:
We won’t be calling during the Super Bowl. Even though its not an official holiday, it might as well be in terms of being able to conduct interviews. We’ll call during the day Sunday and finish it before the Super Bowl begins.
Update: The Washington Post's Jon Cohen sends the demographic breakdown of the question on Super Bowl viewing
Title fixed. Thanks to all for the edit, we do appreciate it. Bear with us folks, the next few days will be pretty hectic, and we're only human.
Capital Survey Research Center/Alabama Education Association
My National Journal column, which discusses the fundamental unanswered question about the Obama campaign, is now online.
Dems: Clinton 43, Obama 39...
Reps: McCain 37, Romney 22...
Minnesota Public Radio/University of Minnesota
Clinton 40, Obama 33... McCain 41, Huckabee 22, Romney 17
Georgia: Romney 32,McCain 21,Huckabee 24...
Obama 51, Clinton 41
Western New England College
Massachusetts: Clinton 43, Obama 15
(Note: We will constantly be updating these charts to reflect the most recent poll data.)
No time for commentary this morning, so here is an explanation.
It is ironic and annoying that the most important date on the primary schedule is also the date with the fewest polls per state. Just as the campaigns are struggling to run 22 simultaneous campaigns, so pollsters and the media have invested little in comprehensive polling of the Super Tuesday states. Even large states such as New York and California have fewer than 10 polls since January 1, far fewer than we saw last week in Florida for example. As a result, we have many states with no data at all, preventing a comprehensive overview of the prospects for Tuesday. Even where we do have polls, we lack enough to consistently estimate the trend with data taken since Iowa. Where we can estimate trends, we've done so on the "regular" state pages at Pollster.com. You should go there for the best trend estimates we can manage with so little data.
The charts here are a way of seeing the entire set of Super Tuesday states (where we have polling) at a glance.
Rather than plot the usual trends with so few data points, each poll is a point and the darker the point the more recent the poll. The points are also scaled in size to be proportional to the number of delegates at stake in the state.
Instead of a trend estimate, this plot highlights the median of all post-Iowa polling in the state. The shading of points will then let your eye tell you whether there is a visible trend around that median. Be your own data analyst!
When more states become available, they'll be added to updated charts. If a state is missing, we don't have polling for it. (If you think we've missed a state with polling data, let us know!)
Polling had a pretty good night in Florida on Tuesday. While early polls understated both McCain and Romney's vote, the polls got better as election day closed in, reflecting a (measured) upward trend in both candidates support. By the last day or two of polling most polls were within the ten-ring, and a number were close to the five-ring.
There was more disagreement among the polls as to who was ahead, and this included some late polls that put Romney ahead of McCain. However, most of these polls reflected a race "too-close-to-call", rather than egregious errors about the leader, with one large exception towards the lower right in the plot.
As we've seen all year, the polls got the vote for third and fourth place finishers very accurately, almost all within the five-ring.
New Rasmussen Reports automated surveys show:
In Massachusetts: Clinton 43, Obama 37
In Illinois: McCain 34, Romney 26
Although it hasn't stopped folks from trying, it's a little too early to tell how people will vote in the November general election. But primary turnout so far suggests that the gender gap is poised to increase.
The gender gap, which is the difference across gender in the vote for the winner, has existed in every presidential race since 1980. It was a high of 11 points in 1996, and a low of 4 points in 1992 (when Ross Perot was a viable 3rd party candidate). A good one-pager on the gender gap is here.
So far, in every single primary, women made up a much higher percentage of Democratic primary voters than Republican primary voters. As the table below shows, in South Carolina, 61% of Democratic primary voters were women. In the early Democratic contests, women were 57%.
By comparison, only in South Carolina (where apparently women love to vote!) did women make up about half of the Republican primary electorate. In most contests, women were clearly a smaller part of their process.
Further, I don't think this simply reflects Clinton encouraging new women voters (although that may be happening). For example, Florida, which is a closed primary state, showed one of the biggest dropoffs on the Republican side. Also, South Carolina, a state that Obama won decisively, had the highest female turnout of all the Democratic contests.
What should really concern Republicans is that in nearly every contest, the percentage of women participating in the primary dropped from 2000, the last time no incumbent was running. We don't have as many 2000 figures for the Democratic contests, but a trend seems to be emerging on the Republican side. Women are becoming even less likely to vote in Republican primary contests.
Does this mean that women will be even more likely to vote Democratic this November? Perhaps too early to say, but certainly turning out in a Democratic primary, or sitting out the Republican contest, are good first steps. We'll keep track of this metric, and report back if things change.
The latest from the Gallup Daily tracking survey of Democratic and Democratic leaning voters nationwide:
Barack Obama has now cut the gap with Hillary Clinton to 6 percentage points among Democrats nationally in the Gallup Poll Daily tracking three-day average, and interviewing conducted Tuesday night shows the gap between the two candidates is within a few points.
Go read it all, but in this case, the picture tells the story:
See my post from Monday for more details on this new Gallup tracking survey.
Update: Andrew points out in the comments that the Rasmussen automated daily tracking survey shows no gain for Obama over the last week. However, the Gallup trend brings the two surveys into closer agreement on the Democratic race now than ten days ago. On January 20, they reported Clinton leading by just four points (38% to 34%), while Gallup had Clinton ahead by 20 (48% to 28%). Clinton now leads on the Rasmussen survey by nine points (41% to 32%).
Back in April 2007, when Rasmussen's surveys were showing a closer race than other national polls, we looked closely at the potential reasons for the difference. One important issue is that Rasmussen's methodology effectively samples a narrower segment of the population. Whether that difference makes it better or worse in this context is a point of debate.
For those who may have missed it (I tucked a mini-announcement in last night's exit poll thread), we are now reporting poll results for
twelve thirteen of the February 5 primary states, and will be adding another half dozen or so in the next 24 hours for which only a handful of polls are currently available. The links below also appear in the right column throughout Pollster.com:
Needless to say, we will be hard at work updating these pages as new polls become available over the next week. If you know of a poll in the public domain, or if you spot a typographical error that we have missed in our haste to get these data posted, please email us (at questions at pollster dot com).
One important note: We only plot our regression trend lines on the charts when eight or more polls are available. When you do not see trend lines plotted, the estimate for each candidate that appears in the chart legend is the median result among all available polls.
In several cases, that median result is probably a less accurate estimate of the current state of the race than the most recent poll released. Consider Colorado. The median result shows Clinton leading by eleven points (34% to 23%), but four of the five polls were conducted more than four months ago. The most recent Denver Post/Mason Dixon survey, however, shows a close race, with 34% for Obama and 32% for Clinton.
So in looking at polls for February 5, if our trend estimate does not plot, we recommend focusing more on the most recent polls conducted in January 2008 than the median value we report in the chart legend.
Readers have also inquired about sites that tally delegate counts and provide more information about the number of delegates up for grabs in each February 5 state and the rules for their allotment. Tracking that information is well beyond the scope of Pollster.com, but we are happy to recommend two excellent resources put up by our partners:
We know that many other news organizations are tracking delegates, but we would appreciate your support for our partners at the National Journal and Slate.
I'm doing updates blog style -- Find links to the exit poll tabulations at the bottom of this post. All times Eastern.
8:40: I'm off to help get my two pre-schoolers to bed. I leave you in the capable hands of Mark Lindeman (in the comments section). But please keep in mind that in the exit poll tabulations, margins of a few percentage points are statistically meaningless. So a 2-4 point "lead" is not really a lead. The Republican race will be called using actual returns.
8:24 p.m. More from Lindeman:
They did a quick reweight, perhaps with early votes and/or turnout. McCain up 32.6-31.4 or 33.0-31.2 depending on how I look at it.
Mark likes to be precise, but let's keep in mind that the decimals are meaningless, and the Romney-McCain margin in the sample estimate falls well, well within the statistical "margin of error" for an exit poll. The networks will almost certainly need to wait until all (or nearly all) voters are counted before calling a winner.
8:09 pm - Regular pollster commenter Mark Lindeman has posted the current overall estimates extrapolated from the current tabulations in our comments section:
Right now it's Romney by a nose, maybe 32.2-31.6 depending on how I estimate. Clinton 50-32, Obama with 76% of the black vote.
More explanation here on how these early estimates improve over the course of the evening.
8:05 pm - Wow. The exit poll tabulations are up on MSNBC (see links below). Among Republicans who made up their minds in the last three days: Romney 37, McCain 36. Among those who made up their minds in the last week: Romney 38, McCain 38.
7:47 pm - While I have your attention, good news for regular readers: Earlier today, we added poll pages for the Democratic and Republican contests in nine new February 5 states in addition to the the three we had put up earlier. We will be adding charts in any state with at least 8 polls conducted over the last year, and we will add a page for any state for which we can find public polling. So if you know of a poll that is not included here please email us.
The links to each of the pages are in the sidebar column to the right.
7:22 p.m. - Demographic tabulations from the network exit polls should be available at these links when the all the polls have closed in Florida at 8:00 p.m. eastern time:
Note: Florida's polls close at different times. Quoting reader Daniel T, who reported this succinctly in a comment earlier, polls close at "7pm in the south and 8pm in the north, essentially after 8pm results should start coming in." Since the tabulations have not yet appeared as of this writing (7:15 p.m.) we will presumably see them shortly after all the polls close at 8.
The good news is that since the polls are closing in South Florida, many of the exit poll interviewers are able to report back the actual results for their sampled precincts. The exit pollsters can use that data to check the precinct level poll results against the precinct level count and use the resulting statistics to correct for any observed bias. That means that when we do se tabulations at eight, the estimates will be more reliable than the what the networks analysts were looking at 20 or 30 minutes ago.
The Associated Press (via The Page) has also reported some initial findings, limited for now to the questions on issues and demographics. Also, for those wondering about how the exit polls have handled early and absentee voting, AP reports:
From partial samples of 970 Republican primary voters and 989 Democratic primary voters conducted by Edison Media Research and Mitofsky International in 40 precincts across Florida on Tuesday. The samples include 235 Republican voters and 294 Democratic voters who voted early or absentee and were surveyed in the past week by telephone.
One note on early voting and the Florida Republican primary. Charles Franklin’s excellent “endgame” summary shows a roughly eight point drop in Rudy Giuliani’s support since December. But the Giuliani campaign sees some hope in early voting. As Newsday reports:
Giuliani’s campaign made a case that it could win here on the back of its get-out-the-vote efforts aimed at early and absentee voters, who are expected to top 450,000 and to account for a third of the turnout.
Are polls showing Guiliani running ten to fifteen percentage points behind frontrunners Mitt Romney and John McCain missing the impact of early voting? Not likely. Of the eight organizations, only three reported the number of Republican primary voters who said they had “already voted” at the time they were interviewed: 27% by SurveyUSA, 25% by PPP and 19% by the Suffolk University poll. All three also provided tabulations comparing early voters to those yet to cast a ballot:
The SurveyUSA and PPP results show Giuliani running a few percentage points higher among early voters, although neither difference is large enough to be statistically significant given the relatively small sample sizes involved. The Suffolk survey finds few Giuliani supporters in its even smaller subgroup of early voters, but even if you ignore the Suffolk result and treat the differences measured by SurveyUSA and PPP as statistically meaningful, they offer Giuliani little hope. Those results are also consistent with Giuliani campaign manager Mike DuHaime’s acknowledgement to Newsday that early voting “could make a difference of only a few percentage points.”
Rudy Giuliani just appeared on Morning Joe on MSNBC and reiterated his confidence in the early vote:
I think the early voting in unprecedented in number. I believe we’ll do very well because we campaigned all during that period bringing out that early vote.
One handicap the Giuliani faced is that the period of early voting occurred over the last fifteen days. However, as Charles Franklin’s endgame graphic shows, significant erosion in Giuliani’s support had already occurred by January 1, so the potential gain from early voting is limited:
What about the five or six other polls whose public releases said little or nothing about how they handled early voting? As poll consumers, unfortunately, we are once again in the dark. I assume that most handled early voting as the three cited above did: Their screen questions presumably gave respondents the option to say they had already voted, and hopefully rephrased their trial heat questions to ask early voters about their preferences in the past tense. But we do not know that for sure. Given that early voters may cast a third of Florida’s ballots, disclosure of the way media polls handle early voting ought to be a no-brainer. Why don’t their media sponsors demand it?
PS: Tonight’s exit polls in Florida will include sample of interviews conducted by telephone among early voters, and those results will be weighted with the interviews conducted at polling places today to the best estimate available of absentee ballots as a percentage of all votes cast.
Clarification: Daniel is right to point out a distinction I had missed that Florida makes between “early” and “absentee” voting:
There is an important point of clairification. Early voting *at the polls* has only been going on for the last 15 days. However, it’s my understanding that abstentee voting by mail has been going on since early December. I have no data on what percentage of the 400K votes were at the polls vs by mail, nor do I know anything about the exact wording of the voter screens. My point is that “early voting” and “abstentee voting” are not the same concept in Florida.
Unfortunately, the Florida Secretary of State's absentee voting page does not indicate when "absentee" voting begins. Either way, the available survey data suggest that the preferences of early and absentee voters are, at best, a few percentage points more favorable to Guiliani.
Either way, it is extremely unlikely that pollsters systematically excluded or screening out early and absentee voters. My assumption is that most accounted for early/absentee voters as SurveyUSA and PPP did but reported nothing about their procedures. The worst case is that a pollster that made no modification to their screen and trial heat questions to accommodate early or absentee voters. Under that scenario, I would imagine that those who had already voted would choose the "very likely" to vote option (or the interviewers would choose it for them), and that voters would report their actual choice as the candidate they would support "if the election were held today." So one way or another, the choices of those early voters are probably included in the surveys we have before us.
The polls in Florida point to a very close contest between Mitt Romney and John McCain. As of Monday's polling used here (there will be new final polls available Tuesday morning) Romney has a small lead based on our trend estimators (both standard and sensitive). But that lead is so small that a "dead heat" is probably still a good characterization of the race.
As we've seen in several previous states, the final vote has often broken strongly in favor of the winner with little going to the second place finisher. But who will that winner be? Other than our trend estimates, the data have little more to tell us on that score. With Ron Paul polling consistently at between 4 and 5 points, there are at least 10% of voters yet to make up their minds. A strong surge could boost one of these front-runners from about 30% to near 40% and a very significant win. An even split of undecideds will make for a close finish with both around 33-35%.
Both Romney and McCain have been gaining ground in Florida, but Romney's rise has been consistently sharper than McCain's. Moreover, there is little evidence that McCain has enjoyed a post-South Carolina spurt in Florida. Nor does Romney's Nevada win seem to have helped him beyond his already considerable upward trend.
As the Florida race narrowed to a two way contest for first place, Giuliani and Huckabee have subsided into the 12-15 percent range. They too could go either way for 3rd and 4th, though at the moment Giuliani has a small advantage for third place. As mentioned above, Ron Paul trails with 4-5% for fifth place.
The chart below shows how much the polls in Florida have overlapped since the South Carolina primary and Nevada caucus 10 days ago. While the trend estimates are slightly distinct, the huge overlap of polls for Romney and McCain show that we should remain quite uncertain as to who is "really" ahead. Sometime Tuesday night, we'll find out.
Cross-posted at Political Arithmetik.
A new CNN/Los Angeles Times/Politico California survey (1/23 through 1/27) finds:
690 Democratic LVs (± 4%)
427 Republican LVs (± 4.5%)
Earlier today, I participated in a blogger conference call with Gallup's Frank Newport, in which he talked about their latest surveys in California and New York, as well as their intriguing new project, the Gallup Daily. Starting in early January, Gallup started completing 1,000 interviews each night. Starting last week, they are now reporting a three-day rolling average of the presidential primary trial-heat questions. They will release new results each day at approximately 2:00 p.m eastern time -- today's results are here, each new release can be found here along with a daily report on consumer confidence.
True polling junkies will know about another similarly structured poll tracking program: Rasmussen Reports has put out daily updates on the President Bush's job approval rating for many years, and has been reporting three-day rolling average presidential primary tracking since October. Of course, the Rasmussen data is collected with an automated methodology known as interactive voice response (IVR), in which respondents hear a recorded message and answer using the keys on their touch-tone phones.
Whatever you think of IVR polls -- and the methodology certainly remains controversial in the survey research world -- the Gallup program is distinct from Rasmussen in other ways than the use of live interviewers. According to Newport, the Gallup Daily uses the "same robust methodology" as all other Gallup polls: live interviewers, a random-digit-dial sample, as many as 5 "call-backs" to those not home when they call, cell-phone sampling to reach those in cell-phone only households (something Gallup also introduced to their standard methodology this month) and Spanish speaking interviewers available for when they reach a household in which only Spanish is spoken.
That "call back" procedure may sound excruciatingly wonky, but it is important and a key distinction from the Rasmussen tracking. So far at least, Gallup has used a procedure that dialed each sampled number as many as five times over successive nights if the initial attempts were unsuccessful (that is, if the number was busy or if no one answered the telephone). They structure their calling procedure so that the sample on any given day is equivalent to a sample dialed for many days. Each day has the same mix of attempts (first, second, third and so on).
The Gallup Daily survey design has two critical benefits, according to Newport. First, obviously, it allows us to attempt to monitor the impact of major campaign events on a daily basis, such as the Barack Obama's victory in South Carolina. Newport reported that their sample last night had Clinton ahead by 10 percentage points (he did not provide specific percentages), while the result over the last three nights had Clinton leading Obama by 11 points (43% to 32%). So Newport's sense is that the South Carolina results has not yet dramatically changed the Democratic standings.
However, a quick glance at their chart shows something else that Newport confirmed. The Democratic race has narrowed significantly over the last week. Clinton lead by 20 points (48% to 28%) on their January 20 release but, again, by only 11 points today (43% to 32%). Given the large samples involved (approximately 1,200 leaned Democrats for each release), both the Clinton decline and Obama increase over the last week are statistically significant.
The second "tremendous virtue," as Newport put it, is the ability to pool very large data sets for subgroup analysis. Consider that Gallup will be interviewing 7,000 adults each week and over 30,000 adults a month. As such, they can pool data and examine demographic subgroups to a degree impossible with once-every-two-week samples of 1,000. As an example of this "unique and powerful ability," Newport cited his recent analysis of the Democratic primary contest within socio-economic differences of African Americans. Ordinarily, in a sample of 1,000 adults, the total number of black Democrats would be well under 100 respondents. But with over 20,000 interviews to work with, Newport was able to compare the presidential preference of African-Americans earning less than $24,000 a year to those making $90,000 or more. That's extraordinary.
This program has one other virtue that Newport did not mention, and that involves the vast analytical resources at Gallup. Given their ability to do something with all that data, it is the sheer number of interviews that will make the contribution to our understanding of politics and the presidential campaign during 2008, especially given Gallup's commitment to methodological transparency and disclosure. Hat's off to Frank Newport and Gallup. This is truly a big deal.
600 Republican LVs (± 4%)
600 Democratic LVs (± 4%)
481 Democratic LVs (± 4.5%)
585 Republican LVs (± 4.1%)
New York: 750 Republican LVs (± 4%)
5 None/No opinion
New York: 767 Democratic LVs (± 4%)
5 None/No opinion
California: 755 Republican LVs (± 4%)
7 None/No opinion
California: 779 Democratic LVs (± 4%)
6 None/No opinion
The polls had a bad day on Saturday in grossly underestimating the support for Barack Obama, though they nailed the Clinton and Edwards votes quite well. Not one poll came within the ten-ring, and the final poll of the primary understated Obama's vote by nearly 15 points. Several polls flirted just inside the 20-ring, and one hapless example of the consequences of poor question wording, the Clemson University poll, understated Obama support by nearly 30 points. The Clemson poll allowed 36% to remain "undecided", hopelessly biasing downward their estimates of candidate support, but especially so for Obama.
My colleague Mark Blumenthal has posted a nice comparison of these South Carolina results with those "terrible" polls from New Hampshire. Below is the same poll error chart for New Hampshire, but scaled the same as the one for South Carolina above.
The New Hampshire results were mostly inside the 10 ring and all were inside the 15 ring. And a couple even touched the 5 ring. Judged by distance from the bullseye, New Hampshire doesn't look that bad, certainly not compared to South Carolina.
But there is another difference, and this is where New Hampshire was terribly wrong and South Carolina not so bad: All but one of the New Hampshire polls had the wrong leader. None of the South Carolina polls, not even Clemson's, got the leader wrong.
So while the distance from the bullseye was quite a bit worse in South Carolina, the creation of confounded expectations was not. It was the expectations that were created and then confounded that make New Hampshire a polling disaster, while there has been little said about the polling errors in South Carolina. (Except here, where we care about such things all the time!)
The other interesting comparison is the parallel that the number 2 finisher in both South Carolina and New Hampshire was quite well estimated. The SC polls got Clinton within normal margin of error. And the New Hampshire polls also got the 2nd place finisher there, Obama, within reasonable error.
The problem in both cases is in the substantial underestimate of the first place finishers vote. The final choices of late deciding voters is a challenge for all polling, and perhaps especially so in primaries where there is no "party identification" to come home to if you can't make up your mind. In New Hampshire the Clinton win rested on significantly more voters supporting her than expected. In South Carolina is was the magnitude of the victory, rather than first place itself, that confounded the polling.
Increases in voter turnout in this cycle may be part of the story (a 75% increase in South Carolina), but here we see those late deciders breaking for different candidates, and yet in both cases for the ultimate winner. Second place results may on average be slightly low compared to the polls, but the first place "bonus" seems quite strong. At least for the Democrats. In the Republican South Carolina primary, both first and second place finishers were a bit underestimated, so there was not the same asymmetric error for first place. The New Hampshire Republican race also about equally understated the votes for first and second. The relatively lightly polled Michigan Republican race shows somewhat greater underestimate of first place (Romney) and second place (McCain). And in Nevada, with only 3 late polls, Romney was dramatically underestimated, while Ron Paul finished second but was only moderately underestimated.
So perhaps these reflect pollsters' difficulty in discerning the likely behavior of undecided voters, or perhaps these are last minute decisions to vote by "not-so-likely" voters who are screened out of the sample but who turn out for the ultimate winner in larger than expected numbers.
Turning to 2nd and 3rd place, the chart below shows that the polls had a pretty good day predicting the Clinton and Edwards votes. Despite some chatter about a late Edwards surge and a Clinton fall (including some evidence in our sensitive trend estimates that such a movement was occurring) most of the late polls were within the five-ring for 2nd and 3rd place, and all got the order of finish right.
Cross-posted at Political Arithmetik.
So in which state did pollsters have a tougher time, New Hampshire or South Carolina? The answer, based on an objective measure of survey error, may surprise you: South Carolina.
SurveyUSA has put together a report card that calculates one measure of poll accuracy for all of the final surveys conducted by every pollster during the presidential primaries so far (there are other measures, but I will leave that subject for another day). Here's their bottom line:
Funny how the psychology of poll numbers works. Two weeks ago, when the final polls showed Barack Obama leading by an average of eight points, but he lost to Clinton by three. Since the winner was a surprise, the election night coverage obsessed over polling problems. Tonight, the polls were off to an even larger degree -- polls showed Obama leading by nine points but he won by 28 -- but since they missed only the magnitude of Obama's margin, the difference goes largely unnoticed.
What happened? In this case, given the differences between live-interview and automated polls, we have decent evidence of what Noam Scheiber termed a "reverse Bradley/Wilder effect," something I wrote about at more length yesterday. Also, at least three of the final polls reported an African American share of the vote in the 42% to 46% range, while the exit polls report that percentage at 55% (though as Mason-Dixon pollster Brad Coker pointed out yesterday, the hard numbers from the South Carolina Secretary of State may look different). So the enormous turnout -- nearly double that of 2004 -- may have thrown off the likely voter models.
The bottom line is that polling is always going to be perilous for primary elections, especially when turnout exceeds all expectations and large numbers of "likely" voters express uncertainty in the final hours before casting their ballots Yet despite the surge in participation, yesterday's Democratic turnout amounted to just 16.5% of South Carolina's eligible adults. Pollsters still had a challenge in terms of selecting likely voters.
The primary season is far from over, and we may be in store for more surprises.