Mark Blumenthal | June 2, 2008
Topics: 2008 , ABC , Barack Obama , CNN , Exit Polls , Hillary Clinton , John Edwards
The exit poll conducted for Michigan's Democratic primary, and more specifically, the way it was used to help allocate Michigan's delegates this past weekend, has been the source of controversy over the last 48 hours. I will let others debate whether it is every appropriate to use any survey -- much less an exit poll -- to award delegates. However, amidst all the spin, some pertinent facts about exit polls and their performance this primary season have been confused.
The best example comes from two brief clips from CNN's coverage of the Puerto Rico results yesterday, the first from an interview of Clinton campaign chairman Terry McCauliffe and the second a response by CNN's Bill Schneider that followed soon thereafter:
Here is the gist of McCauliffe's complaint (my transcription):
The one thing I find amazing is, Wolf, is they say, actually, we are going to base some of this on exit poll data. There has not been an exit poll in this campaign -- I can remember standing in New Hampshire on election night saying, "hey Terry, you're going to lose by fifteen points." None of the exit polls have been right, and you're going to use that to take votes away from Hillary Clinton?
And Schneider's response and discussion a few minutes later with CNN's Wolf Blitzer:
Schneider: I can tell you that the exit polls have been pretty accurate in all the primaries so far, including New Hampshire. The exit polls have been very close to the actual result.
Everyone remembers the Waterloo of the polls in New Hampshire. That wasn't the exit poll. That was pre-election polls. The pre-election polls in New Hampshire, those taken before primary day back in January, many of them indicated that Barack Obama would win New Hampshire. In the event, on primary day, he actually lost New Hampshire. The exit poll got that right.
It was the pre-election polls that did not capture the final last minute swing of a lot of women, particularly older women, who had been undecided, they swung at the last minute to Hillary Clinton and that's what put her over the top.
Blitzer: And when we do these exit polls, we have three waves of exit polls, and sometimes the media gets told of only the first wave which may be distorted, may not be precise. It takes three waves to get an accurate assessment of what is actually going on. As a result, some confusion about the accuracy of these exit polls.
So either "none of the exit polls have been right" or the exit polls have been "pretty accurate in all the primaries." Not surprisingly, the whole truth lies somewhere in between.
Let's start with the comment from Wolf Blitzer at the end of the clip. He is right to point out that exit poll results get phoned in by interviewer/reporters in three waves, and that the biggest "errors" have involved early data based on the second wave or on the third wave called in just before the polls close (the networks now keep data "quarantined" and do not release it to network producers and decision desks until after 5:00 p.m.. Eastern time).
The second wave (late afternoon) estimates have often leaked this year, and those results have, more often than not, erred in Barack Obama's favor during the primaries (especially on 2/5 and 3/4 and in Pennsylvania). As I wrote back in March, "the early leaked results overestimated Obama's strength in 18 of 20 states, for an average error of 7 percentage points on the margin."
Assessing the accuracy of the third-wave, "at poll closing" estimates based only on the exit poll interviews is harder, because those numbers rarely leak. What we see more often, at least indirectly, are the estimates used to weight the official exit poll tabulations that appear on network web sites as the polls close. The estimates are usually a blend of the exit poll interview results and an average of pre-election polls (more details on this process here). These results, as extrapolated by our friend Mark Lindeman each primary night, have been far closer to the final results than the early leaked numbers. When I looked at the numbers from 2/5 and 3/4, I found that while big errors in Obama's favor persisted in six states, the errors in the remaining 11 states were small and canceled out (averaging to zero).
With respect to New Hampshire, I have heard rumors that mid-afternoon numbers showed Obama leading Clinton (Chris Matthews appears to say as much in this clip), but nothing as large as the "15 point lead" that McCauliffe claims. As the polls closed, our friend Mark Lindeman extrapolated a 38.3% to 36.9% margin in Obama's favor from the official exit poll tabulations appearing on network web sites. So if anyone told Terry McCaullife on "election night" that Clinton would lose by 15 points, it was not on the basis of an exit poll.
What is misleading about this entire discussion, however, is that the Michigan results relied up by the DNC on Saturday (and analyzed in more detail by Brian Schaffner) were not the second-wave or "at poll closing" estimates of the official count, but rather the results of this question after the tabulations had been weighted to match the official count:
To be more specific about the weighting: Once the Associated Press reported a final count for Michigan on the evening of January 15, the exit poll analysts reweighted their tabulations so that the size of each poll region (labeled as "Geo Stratum Code") and the candidate vote shares within each of those regions matched the actual result. Thus, the "vote estimate" at the top of this final tabulation produced by Edison/Mitofsky (and posted online by ABC News) shows 56% for Clinton, 4% for Kucinich, less than 1% each for Dodd and Gravel and 39% supporting uncommitted.
If these had been the candidates on the ballot today, for whom would you have voted in the Democratic presidential primary?
46% - Hillary Clinton
12% - John Edwards
2% - Dennis Kucinich
35% - Barack Obama
1% - Bill Richardson
Can we rely on the final exit poll data when weighted data "forced" to match actual results? That is the difficult-to-answer question many of us have been been pondering this year. Does weighting the result by the vote preference and turn out eliminate all possible bias with respect to demographics or other attitudes? Perhaps. In this case, however, the correction for statistical bias is right on point. We know that the results are weighted so that the percentage who chose Clinton matches the actual count.
Actually, if anything, the final weights may favor Clinton slightly, for two reasons. First, the truly final count (available after these tabulations were done on the evening of January 15), gives Clinton 55% (not 56%) and undeclared 40% (not 39%).
Second, as reported on Saturday, the official count did not include approximately 30,000 write-in votes that were never counted or included in the official totals because no candidates filed the necessary papers to request the counting of write-in votes. Most assume these write-in votes were cast for either Barack Obama or John Edwards. The voters who cast write-in votes presumably had no idea their write-in votes would not count as they left their polling place and, we can assume, would have been just as likely to participate in the exit poll as other voters.
The exit poll questionnaire had a response option for other ("Other: Who? _______") that, presumably, would have been chosen by write-in voters (though I am not sure how the exit pollsters handled any such responses in the final tabulations). Since no write-in votes were reported, however, the weighting of the final tabulations did not reflect votes that could have increased the total vote by as much as 5%. So the weighting of the exit poll -- like the official count -- may have overstated Clinton's vote by a few percentages points over what it would have shown had all write-in votes been counted.
Another potential source of error would be those voters who cast absentee or early. Michigan allowed for early voting, but in this case, the exit pollsters did not conduct a telephone survey to specifically capture the attitudes of absentee voters. I have not been able to find any report on the percentage that voted early or by absentee ballot. However, the final tallies used to weight the final tabulations included absentees.
Again, reasonable people may question whether it is ever appropriate to use any survey -- no matter how accurate -- to allocate delegates from a primary election. However, the case for labeling this particular application of this survey as inaccurate is weak. The final Michigan exit poll tabulations are best evidence we have on which candidate voters would have favored had the names of all candidates appeared on the Michigan ballot. The weighting procedure provides reassurance that, in this case at least, the percentage of Clinton voters was either right or erred slightly in her favor.