Mark Blumenthal | January 2, 2008
Topics: 2008 , The 2008 Race
I have to admit I experienced a bit of Iowa Caucus overload yesterday, and as I'm not in Des Moines with every political reporter in the Western World, I went off the grid for a bit and spent much of the day yesterday with family. As such, my mental "in-box" has filled to overflow with thoughts about what the last round of polls are telling about the Iowa Caucuses. What follows is an attempt to sum up and empty the in-box:
1) We Have No Idea Who Will Win. Yes, despite tens of thousand of interviews, polls of every shape, size and method and our own fancy charts featuring regression-derived trend lines of varying degrees of sensitivity, the only thing we can say with confidence is that the Democratic and Republican races are close. It is hard to know much more than that given the small margins and, more importantly, the huge variations in the kinds of likely caucus goers sampled.
2) Methodology Matters. That conclusion has been the theme of virtually everything I have written about the Iowa Caucuses this year. As we learned from our Disclosure Project, no two pollsters select and define the "likely caucus goers" exactly the same way, and the differences in their methods are directly related to the vote. On the Democratic side, it has long been clear that John Edwards does better in polls that include bigger percentages of past caucus goers, and more recent surveys show Hillary Clinton and especially Barack Obama doing better as the samples include more first-time caucus goers.
Polls always show some differences due to pollster "house effects," differences that are usually more about how hard interviewers push uncertain voters than about how they select likely voters. But in the case of the Iowa Caucuses, differences in method produce huge variation in the kinds of voters selected as likely caucus goers. And what is truly surprising is that those differences appear to be growing in the final round of surveys (and speaking of the Des Moines Register survey, I'll have much more below).
3) Our Regression Estimates (and Poll Averages) Don't Help. They don't help, that is, in providing a more precise estimate of who may be ahead or behind. What Charles Franklin's trend lines do, and do very well, is provide the best possible summary of the combined wisdom of all the public polls in the race. Since those polls are all over the place in terms of the kinds of people they are sampling, however, that combined wisdom may be off the mark.
Our colleagues Gary Langer and Jon Cohen, directors of polling for ABC News and The Washington Post respectively, wrote an op-ed over the weekend with "tips for decoding election polls." It is worth reading in full, despite their advice to "avoid being seduced by averages" (Langer also blogged a similar warning about "getting sucked into the horse race clutter"). They argue that "a collection of good and bad polls" will not provide a "better estimate" than one good poll. In theory that's true. In reality however, especially when we look at general election polls, the differences among polls are usually not much more random than the "margin of sampling error" would predict. That is why, our averages and those posted by RealClearPolitics, were more "accurate" in Senate and Governor races in 2006 than those from individual polls.
But in the case of the Iowa Caucuses, the warning about averages from Langer and Cohen is probably right. Polls are measuring different kinds of people, and those differences are producing results that vary far beyond the statistical margin of error. Our current regression estimates are incredibly close, with only a point or two separating Clinton, Obama and Edwards among Democrats and Huckabee and Romney among Republicans. However, those estimates essentially split the difference among the various methodologies. If the consensus guess about the best way to "model" the turnout is wrong, then the averages will be wrong too.
4) So What Do We Know About the Horserace? The Democratic race is now obviously a three-way contest between Clinton, Edwards and Obama and between Huckabee and Romney on the Republican side, with a (though that would be what one former colleague of mine calls the "duh" finding). But we know more than that.
On the Democratic side the variation in the composition of the likely electorate tells us the likely story of the race. We just don't know the ending yet. The Atlantic's Marc Ambinder put this as succinctly as I've seen anywhere:
If Obama is going to win the caucuses, he's probably going to win big, thanks to an enormous infusion of new caucusers. If Clinton wins, it's because hard corps Democrats turned out in larger numbers.
The Republican story is similar, although the relationship to poll methodology less clear. Most surveys have shown that Mike Huckabee does much better among a base of rural Republicans and fundamentalist christians, and one question may be whether polls are over or under-representing evangelicals. Polls use different kinds of questions to measure this population, so comparisons across polls are difficult. But it is worth remembering that one of the biggest misses in pre-Caucus polling was back in 1988 when Pat Roberston finished second (ahead of George H.W. Bush) with 25% of the caucus vote. Polls had shown Robertson running a distant third, receiving 17% in one poll, single digits in most others.
5) Does John Edwards really win second choice? Maybe, maybe not. One poll finding that has taken hold as conventional wisdom over the last week or so is that Edwards wins second choice. This conclusion, along with the generally accepted assumption that his ground organization, largely intact since 2004, will be in a better position to gain when candidates that fail to meet the 15% viability requirement within each precinct are forced to choose a second choice (Ambinder again has a concise review).
The c.w. on second choice comes from four recent polls -- Los Angeles Times/Bloomberg, CNN/ORC, Mason-Dixon/MSNBC/McClatchy -- that all showed Edwards "winning" second choice followed by Clinton and Obama (in that order) among all likely caucus-goers. The key issue is not second choice overall, but the second choice of the supporters of candidates (Richardson, Biden, Dodd and Kucinich) who look like they will fall short of "viability." Keep in mind that this reallocation process occurs separately within each precinct. so if there are regional patterns that push any of the bottom-four over 15%, or any of the top three below, a simple-statewide reallocation will be off.
Four recent polls have done the reallocation of Richardson, Biden, Dodd and Kucinich supporters statewide, and guess what? Once again, the Des Moines Register poll tells a different story. The other three all show the reallocation working in Edwards's favor -- and by a net six and eight points by respectively, Mason-Dixon and InsiderAdvantage. Zogby's first numbers from Sunday show small single digit benefit to Edwards and their report today implies the same. The Register reports, however, "that the results would change little if the votes for the lower-rated candidates were redistributed among the front-runners." So here is yet another unresolved conflict that only the actual results will resolve.
David Yepsen's column provides some specific numbers that appear to be the second choices of supporters or the candidates with single digit support, although he describes the results as a reallocated second choice (something noticed by commenter MSS). I emailed for a clarification and will update when I hear more.
6) Are Trends Our Friend? While Franklin and I have been cautious about what the trend estimates say about who would win or lose, we have seen value in the trend lines as a measure of, well...trends. Generally speaking, Franklin's blue "standard estimator" has been consistent with the trends reported by individual polls in Iowa (and elsewhere) over the course of 2007. Are the trend lines showing any particular pattern?
Here I recommend reading Franklin's graphical review of "endgame trends." He drew both his blue "standard" estimator and "ready red" sensitive estimator through the results since mid-December. The lines are based on all data, including earlier polls, and those show virtually no trend, save for a minor uptick for McCain and possibly Edwards. If you have doubts about how well the lines fit the data, he blows up the last two weeks so you can judge the "fit" of the line for yourself. Again, the major issue is that "the noise of the individual polls around these trends is large" (see point #2 above).
As long as we are on the subject, the Des Moines Register added a bit more fuel to the spin wars by posting a graphic (scroll to the end of the story, bottom right) showing "trends" in the Democratic race based on two-day rolling averages calculated over the course of their four day field period. Unfortunately, the sample sizes involved were so small that, as far as I can tell, none of the observed "trends" were strong enough to attain statistical significance. I agree with the commenters who question the Register's decision to characterize the trends as meaningful.
7) Holiday Polling Effect? - On the day after Christmas, I blogged a theory about the potential for a holiday polling effect that might temporarily depress Barack Obama's standing, creating the false sense of a downward trend. My sense is that we have seen just such effect centered around interviews conducted over the four-day Christmas weekend. I had planned to include that thought in this in-box clearing, but then reader Michael did me one better in the comments below. I'll promote the comment and let him explain:
I recently went on vacation. As a devoted reader of this site and Obama supporter, I recall checking the trends and smiling: Obama was trending steeply up, at ~29%, Clinton was declining/flat, at ~26-27%
Get back after a little over a week abroad, and you can imagine my surprise to find Obama trending down, at ~26%, Clinton trending up and at ~29%. What happened in 5-10 days that flipped the race on its head??? At first I thought maybe the news about Benazir Bhutto and the apparent campaign "gaffe" by David Axelrod.
But then something interesting happened.
Obama started polling back in the high 20's and low 30's. His sensitive trend estimate started to spike. His standard trend estimate went from declining to near flat.
What's more, Hillary's trend standard trend estimate has started to flatten. Her sensitive trend estimate has plummeted.
Which lead me to an interesting thought: Before I left for vacation, Obama was trending up at 29%. A few days after I got home from vacation, Obama's sensitive trend estimate shows him gaining ground and actually now at 30%, right in line with pre-vacation trends.
Only, in between, his numbers plummeted.
Before I left for vacation, Clinton was falling (though there were some signs of her stemming that drop before I left). She was in the high tenths of 26% of the low end of 27%. A few days after I return from vacation, and her sensitive estimate puts her at...27.5%
The sensitive estimate, basically, is right in line with where the race was at 10 days-2 weeks ago. But everything in between has been the complete reverse of that period.
The period where everything flipped on its head was, of course, a time when most everyone who can go on vacation does go on vacation. And the DMR poll's cross-tabs certainly suggest that among the block of voters I'd guess are more likely to leave home over the winter holiday, Obama cleans up: ~40% to 20% each for Hillary and Edwards amongst college educated and those making over 60K. Meanwhile, the less-educated and less-affluent constituencies that support Clinton and to a lesser extent Edwards strike me as precisely the type of voter who will be home more often over the winter holiday.
I couldn't have put that much better myself. As commenter DTM points out, Michael's hunch about the kinds of people who travel is exactly what the data in my earlier post suggests. Michael probably didn't see it because he was. . . on vacation.
You can see the down-and-up pattern in the black
last-5-poll average trend line connecting the individual polls in Charles Franklin's recent endgame trends charts. You do not see the same pattern in the regression trend lines (that Michael saw in our daily report of estimate end points), because the slope the lines adjusts every day as we add more polls to the charts to better fit the underlying data.
And so finally . . . back to the Des Moines Register/Selzer & Company poll that shows more independents and first-time caucus goers in the Democratic sample than other polls. Some things worth remembering:
8) The Register/Selzer poll did not change its 'turnout model" - One common misperception regarding the new Register poll is that it altered its "model." Clinton pollster and lead strategist Mark Penn no doubt contributed to the confusion with his rapid response claim that the poll "adopts an unprecedented new turnout model for the caucuses" (emphasis added).
The mix of independents and first time attendees in the new Register sample may be unprecedented, but it "adopts" nothing new in terms of methodology. They used the same sampling procedure and screen for the new survey as the one in October showing Clinton with a six-point lead that Penn was happy to cite. The difference is that the same methodology that yielded a sample of 23% first time caucus goers in October yielded 60% first-time caucus goers and 40% independents this week. That's not a change in the "model." It's a finding.
JUDY WOODRUFF: oday's Iowa poll is out. You're assuming that 60 percent of the voters in the Democratic caucuses will be first-time caucus attendees. How did you assume that? Why did you assume it?
ANN SELZER, Des Moines Register: Well, actually, I assumed nothing. That's what my data told me.
We put our method in place, and we let the voters speak to us. And we found that 60 percent of the people who told us they were definitely or probably going to the caucus indicated that it would be their first time at caucus.
It's not all that much bigger than 2004. It was 45 percent then. But this stands to be a historic caucus, in terms of turnout. [...]
JUDY WOODRUFF: You're also saying that 40 percent of the voters in the Democratic caucuses are going to be independents. How do you know that?
ANN SELZER: Well, again, that's what our data is telling us. And is that a surprise? Sure. It's something that would raise an eyebrow, because that's more independents than would have come to any previous caucus.
But as you know in this campaign, just keeping your ear to the ground, there are a lot of people who are independent who are planning to come to caucus. The campaigns are certainly courting independents.
And the trick with all of this is that Democrats, to people who proclaim that they are Democrats, are more for Hillary Clinton. And so, people who are more independent are more likely to vote for Barack Obama.
So you try playing with those numbers, and you can come up with any different scenarios. But, again, this is what our data is telling us, and we feel comfortable with it.
JUDY WOODRUFF: And, again, you were looking at voters who say they are either definitely or...
ANN SELZER: Or probably...
JUDY WOODRUFF: ... probably.
ANN SELZER: ... going to attend. And that gives us the advantage, Judy, also to take a look at just those definite attenders and say, "OK, well, what's the core of what's going to happen here?" And when you look only at those, it is still a Barack Obama win.
Yes, there are other aspects of Register surveys past and present that I wish they would disclose. They do not, for example, tell us the percentage of adults that the current sample represents, although given the increase in independents and first-timers, that percentage is undoubtedly higher than the 12% they disclosed to me in October. But Selzer could not be more clear about the consistency in her method. In all of her surveys this year, she assumed nothing about either the size of the turnout or about the characteristics of likely caucus goers. She let "the voters speak to us."
9) That Register/Selzer number is based on party identification, not party registration. In order to participate in the Democratic caucuses "non-Democrats" (i.e. those registered as Republicans or with "no party" registration) must fill out a form to register as Democrats on caucus night. A straight-shooting friend -- who is also a prominent, Clinton supporter -- called me several weeks ago to argue that even the 20-25% Republican/independent share reported by most of the public polls (including the 2004 entrance poll) is too high, because Caucus regulars say such switches have been far more rare.
What virtually every survey measures, however is some form of self-reported party identification. Here, for example, is the wording of the question used by the Register on all of its surveys -- the question used to produce the widely cited 40% independent/5% Republican finding:
In politics as of today, do you consider yourself a Republican, Democrat, or independent?
And the question that appeared on the network entrance poll (and on virtually all network exit polls) that showed 19% Democrat, 1% Republican in 2004.
No matter who you are supporting tonight, do you usually think of yourself as a [check one] strong Democrat, not strong Democrat, independent, Republican.
Given the difference I wondered how many registered Democrats in Iowa think of themselves as independent. I asked a few pollsters that have used a samples drawn from lists of registered voters how actual registered Democrats answered the party identification question. The answers (and keep in mind that question text and formats can affect the results):
- 12% of actual registered Democrats on the CBS/New York Times Iowa survey identified as independents.
- 17-18% of actual registered Democrats on the internal polls of one of the Democratic candidates (not Clinton or Obama) identified as independents.
What these results tell us is that a good chunk of the 20-25% that have reported being "independent" in the 2004 entrance polls and earlier polls in 2007 have been independent minded registered Democrats. The number of new registrants or party switchers may have been as low as 5-10%. Of course, the 40% Democratic and 5% Republican result is still a big change, but perhaps not as big as some may assume.
10) "I'm calling on behalf of the
Des Moines Register Iowa poll" - Commenter Mark suggests four different methodological reasons why the Register poll might differ from the others. All are perfectly reasonable and worth considering. But in so doing, we might want to think about aspects of the methodology might lead only the Register to pick up an influx of first time caucus goers in the last week of the campaign.
For example, Mark wonders if "registered Democrats are getting huge numbers of calls from campaigns, and are thus not answering their phones," while no-party voters are getting fewer and thus more likely to participate in the survey. That seems plausible, but leaves open the question as to why other polls are not seeing the same effect? Perhaps they are all setting quotas or weights based on party or past caucus history, but if so, they have shut out the possibility of picking up any increase in independents or new-comers.**
In that vein, let me add one possible distinction that others have overlooked: The Register poll has one huge advantage. Its interviewers are the only ones that can say in their first breath when a potential respondent picks that they are calling on behalf of the must trusted media brand in Iowa, especially among Democrats:
The Des Moines Register "Iowa Poll."
[Correction: I had this partly wrong. The interviewers describe the survey as "The Iowa Poll" in their introduction, but do not cite the name of the newspaper. They inform respondents of the Register's sponsorship at the end of the call, though interviewers will cite the name of the paper if pressed for more information by respondents.. Although the Register has always branded its survey as "The Iowa Poll," the association with the paper may be less than universal. Thus, this difference may be less important than I had initially assumed. Apologies for the error].
11) So is the Register/Selzer right? Honestly, I am unsure, but let me clarify (and amplify) a few points I made in my first post.
First, you should not accept the Register results just because their 2004 poll may have been slightly more accurate by some measures as compared to other polls. They did catch more of John Edwards' rise sooner than any other poll in 2004. However, commenter p_lukasiak is right to point out that the "definite" voter cut is the one that scores slightly better, while the current result is based on all likely voters (though see Ann Selzer's comments in #9 above about the "definite" result this time). I am among those who have probably made too much of the fact that only the Register got the order of the top four finishers right in 2004. Given how close the candidates were bunched on the Register and other polls, their success on that score may have been a matter of random chance.
Second, you should also not accept the Register results as accurate just because of it's stellar reputation. Reputation does not, in and of itself, translate into magical powers of accuracy. Even the most well respected pollsters are fallible. Their surveys are still subject to most of the potential errors as those with lesser reputations. The Register's methodology may have caught an imminent influx of new participants that other surveys have missed. Their methodology may also have a previously unseen flaw that allowed too many independents slip through their likely caucus-goer screen because of an interest and enthusiasm unrelated to a true intent to participate. And of course, both may be true. We will know soon enough.
However, as I wrote Tuesday morning, whether we believe the result or not, we need to understand the courage and integrity it took for Ann Selzer to put out the results she did. One commenter, badgerfan, wondered:
[I]sn't what the DMR did in this poll just setting themselves up to have it both ways? If Obama wins, it will most likely be because of an unconventional turnout model. If Obama does not win, then everyone can shrug and say I's and R's or new D's jsut didn't participate.
Badgerfan happens to be a personal friend, but he's flat wrong here. Everyone will "shrug" if the result is wrong? Fat chance. Have you noticed the intense criticism this poll has taken since it's release in virtually every news story about the race? If Ann Selzer had wanted to play it safe, she could have weighted her results by past caucus participation or party identification (or both) as many other pollsters do. Her results would have been in line with other polls, far less controversial and no one would have questioned her judgment. But she didn't do that. As an Iowa based survey researcher, she put her own reputation and that of her most important client on the line because she believes in her methods and trusts her results.
None of that proves the accuracy of the results, but that sort of integrity and courage is exactly the combination you want in a pollster. Those qualities are a big reason why her colleagues rate Ann Selzer's work on behalf of the Des Moines Register as favorably as they do.
12) "Unprecedented" - And finally . . . the campaigns that were unhappy with the Register poll held nothing back yesterday in their efforts to knock down what was obviously an unfavorable story for their candidates. But what no one seems to have noticed is that by spinning so strenuously Obama's opponents risk spinning themselves into a corner.
Consider what some of the campaign pollsters said yesterday about the Register result:
Clinton pollster Mark Penn: "An unprecedented new turnout model...an unprecedented departure from historically established turnout patterns in the caucus."
Biden pollster Celinda Lake: "I'm sure [the independent percentage] will be higher, but [40%] just seems impossible . . . That would be a revolution."
At very least, by elevating bit of polling wonkery -- the argument over independents -- into a two-day front page story, Obama's opponents have helped hand him more than a "momentum" story on the eve of the caucuses. His precinct captains now also have a strong electability argument to make tomorrow night: Obama attracts independents.
But more important, what if the Register is right? What if an influx of first-time caucus goers propels Obama to a modest victory margin? Given their spin yesterday, it will be quite a challenge for the other campaigns to shrug it off as an inconsequential result they saw coming all along. Now, if Obama wins with the help of a wave of caucus newcomers, it's not just a "win," it's an "unprecedented departure," a result "at odds with history," perhaps even a "revolution."