Pollster.com: How Accurate Were the Iowa Polls in 2004 and 2000?

December 29, 2007

How Accurate Were the Iowa Polls in 2004 and 2000?

Given the way we are scrutinizing the final Iowa Caucus polls, this seems like a good time to take a look a the final pre-caucus polls from 2004 and 2000. One of the questions I get most frequently is which pollsters were "most accurate" in previous years, and as the old data will show, that is a far more difficult question to answer than most people assume.

Consider the final polls for the Democratic Caucuses in 2004. Only five organizations released public polls conducted in the final week before the Caucuses, which were held on Monday, January 19 that year. Since both John Kerry and John Edwards experienced late surges in support, polls conducted before that would show considerably more "error," since they obviously missed the late surge. Also, those who continued to call through Sunday night might have some advantage in catching the late breaking trend (or, as more cynical pollsters will point out, those releasing late polls also had the benefit of seeing the results of the other earlier surveys).

The table below shows the results of the final week's polls, plus the results of the network "entrance poll," which asked participants their first preference as they entered their caucus location.

[Click on table to pop-up full size version]

The first issue, unique to the Democratic contest, is that even a perfectly accurate survey can tell you only about the initial preference of the caucus-goers. The Iowa Democratic Party does not report the head-count of initial preferences. Instead, the official "results" they report on caucus night will be based on the state delegates for each candidate chosen to the state convention. The delegate selections are based on a second round of voting after supporters of "non-viable" candidates (those who receive less than 15% on the first round) realign to their second choice.

So as should be obvious, the second round voting means that initial vote preference -- even as measured by an entrance poll -- does not directly measure the final results. So there is an important element of inaccuracy built into any Democratic preference poll. In 2004, both Kerry and Edwards did better in the reported results than the entrance poll. Most observers attribute much of the six-point gain for Edwards to a deal struck on caucus morning between the Kucinich and Edwards campaigns that sent most Kucinich supporters into the Edwards camp on the second round. Exit pollster Joe Lenski reports that most Kucinich supporters chose Edwards as their second choice in the entrance poll.

Putting aside the viability issue, which poll was "most accurate?" The answer depends on the yardstick applied, which is a tough call in a multiple candidate primary or caucus. The Des Moines Register poll has received much credit for being the only one to correctly "predict" the order of the top candidates, but notice that Edwards "led" Dean on their larger "likely caucus goer" sample by a not statistically significant three percentage points (23% to 20%). Among their narrower "definite voter" subgroup, the Edwards-Dean order was reversed (Dean had 21%, Edwards 19%). So the getting the order right may have been partly a matter of good fortune.

I will spare readers the minutiae of the various error scores, but if we measure accuracy in terms of how well the polls predicted Kerry's percentage the Des Moines Register's narrower "definite voter" subgroup does slightly better. The same Register sample also does best in terms of the average of the errors for all the candidates. Ironically, the smaller Register "definite voter" sample, the one that had Dean nominally (though not significantly) "ahead" of Edwards, was the most accurate on these criteria even though the larger Register sample has been credited with "predicting" the order of finish.

If, on the other hand, we focus on the Kerry-Edwards margin, the final Zogby poll comes slightly closer to the actual result. In any case, the differences between the pollsters are small enough on all of these criteria that random chance was certainly a factor in determining which did best. And notice that everyone was way off on the final margin between Edwards and Dean, whether we compare to the entrance poll head count (Edwards +6), or the post-realignment actual results (+14).

What about the 2000 Caucuses? The number of final week polls was again fairly limited. On the Republican side -- where the actual results are a simple head-count -- the LA Times and the Des Moines Register came closest to George Bush's ultimate share of the vote, and the Times had the narrowest (and thus most accurate) Bush-Forbes margin. But all of the polls underestimated the support received by Forbes and Keyes.

On the Democratic side, the Des Moines Register had the Gore-Bradley margin exactly right, but a University of Iowa poll, which overstated Gore's margin, had Gore's percentage of the vote exactly right.

So what's the point of these comparisons? Trying to score such a small number of polls solely on the basis of accuracy is a confusing, contentious and ultimately futile exercise. The 15% viability threshold on the Democratic side undermines the accuracy of all polls. The timing of the final poll is critical and the differences among the various pollsters in the final week have been relatively small. Moreover, different measures of accuracy can lead to different conclusions. So if you take away nothing else, remember that none of the Iowa polls have been a perfect crystal ball. All have missed significant aspects of the final results in 2004 and 2000.

Sources: I obtained the results above from the subscriber only archives of The Hotline and the Polling Report. The SurveyUSA results are still available online and the InsiderAdvantage survey from 2004 was released at the time via PRNewswires.

-- Mark Blumenthal

December 29, 2007 in Iowa, The 2008 Race

Comments

Chris:

Another source of distortion in the Democratic delegate count is the rule about rounding to whole numbers. That distortion especially affects smaller precincts that have relatively few delegates to apportion. Under the party rules, for example, if you live in a precinct that gets three delegates, and the vote at that precinct is split 59-24-18 between three candidates, each candidate would receive one delegate. That's some distortion!

Posted on December 29, 2007 3:14 PM

____________________

TPD:

The body text referring to them correctly reads "2000", but the final two tables are incorrectly titled "2004".

Posted on December 29, 2007 4:12 PM

____________________

katerina:

Am I misreading the table or did Insider Advantage also get the order correct of the top 4 finishers (though obviously the totals and margins off by quite a bit)? Many have been very dismissive of Insider Advantage this cycle, but now I wonder if that is a mistake.

Posted on December 29, 2007 4:47 PM

____________________

Mark Blumenthal:

TPD: Thanks - fixed

Katerina: It was (another) error, but thank you for flagging it. It too is now fixed.

But to address the substance of your observation, I'll count myself among those that was unaware that InsiderAdvantage did an Iowa poll at all in 2004. It was not reported by The Hotline or Polling Report, and I cannot find any reference to it anywhere online. Robert Novak did mention it online on CNN just prior to the Caucuses (I found a transcript).

Readers can draw their own conclusions about the accuracy of showing 23% "undecided" on the eve of the caucuses.

Posted on December 29, 2007 5:03 PM

____________________

Kevin Houston:

2004 GOP results? Or is it because George Bush was essentially unopposed?

If the latter, then is the "likely primary voter" screen inherently biased?

Thanks.

Posted on December 29, 2007 6:34 PM

____________________

katerina:

Do you think the high undecideds was due to not pressing respondents for a firm answer (I presume this is very uncommon right before voting)? If so, is that an indicator of how late some caucus voters went in with a "soft" choice or do you think these were people who probably didn't end up showing up to caucus altogether? Just curious...

Posted on December 29, 2007 7:06 PM

____________________

desmoinesdem:

Some Kucinich supporters went to Edwards as a second choice, but many went to Dean despite the deal with Edwards. Also, Kucinich's overall support was quite low.

The Gephardt supporters were what really pumped up Edwards' delegate counts. From everything I've heard, they overwhelmingly went to Edwards as a second choice, and they were a much larger group in most precincts than Kucinich was (even where Gephardt was not viable).

Posted on December 29, 2007 9:39 PM

____________________

Mike:

Chris, regarding delegate counts, you are -incorrect-. Viability thresholds below 4 delegates in a precinct differ:

1: Majority vote
2: 25% for viability
3: 1/6th for viability
4: 15% for viability

Your suggested 59-24-18 goes 2-1-0.

At 59-23-19, it goes 1-1-1.

Posted on December 30, 2007 3:17 AM

____________________

Chris:

Mike -- I'm taking that example straight from Iowa's 2008 Delegate Selection Review Plan, page 4. You can find it at

http://www.iowademocrats.org/ht/a/GetDocumentAction/i/1012213

Posted on December 30, 2007 10:57 AM

____________________

Anonymous:

This is off subject but earlier this month Mark posted sample characterisitcs of Iowa pollsters. Most polls showed first-time attendee percentages much lower than past entrance polls.

In 2000 and 2004 entrance polls showed 41% to 46% showing up for the first time at IA caucuses.

I am not surprised at this given low overall turnout for a two hour event debating their candidate virtues.
Most voters are simply not prepared to do this every four years.

2004 Dems only
First time caucus-goers. 45%.

2000 Dems.
46% first time caucus-goers.

2000 Reps.
41% first time caucus-goers.

Nick

SOURCE: CNN 2004
http://www.cnn.com/ELECTION/2004/primaries/pages/epolls/IA/

SOURCE: CNN 2000
http://www.cnn.com/ELECTION/2000/primaries/IA/poll.html

Posted on December 31, 2007 9:59 AM

____________________

nick panagakis:

This is off subject but a couple of weeks ago, Mark posted sample characterisitcs of Iowa polls. Most were way low on first-time causcus attendees compared to past entrance poll results.

In 2000 and 2004 entrance polls showed 41% to 46% showing up for the first time at IA caucuses.

I am not surprised at this given low overall turnout at two hour event debating their candidate virtues.
Most voters are simply not prepared to do this every four years.

2004 Dems only
First time caucus-goers. 45%.

2000 Dems.
46% first time caucus-goers.

2000 Reps.
41% first time caucus-goers.

SOURCE: CNN 2004
http://www.cnn.com/ELECTION/2004/primaries/pages/epolls/IA/

SOURCE: CNN 2000
http://www.cnn.com/ELECTION/2000/primaries/IA/poll.html

Posted on December 31, 2007 10:06 AM

____________________

Andrew:

Masterful breakdown of the long-awaited list of 2004 and 2000 pre-primary polls. Congrats.

Posted on December 31, 2007 10:18 PM

____________________

george :

with the demoine register endorsing hillary and there latest polls as of today jan 1 2008 showing Obama 7 points ahead it looks like the end for hillary thank god! so go the polls if the register has been the most accurate than good bye hillary good by bill!

Posted on January 1, 2008 9:35 AM

____________________

John Nienstedt:

Mark: Excellent review. However, I think the best way to compare the accuracy of pre-election polls to election results is by removing the undecideds from former. After all, there are no undecideds in the actual vote, so removing undecideds from the polling data allows you to compare apples to apples. Otherwise, the polling data, on average, is necessarily (as long as it contains undecideds) going to be off on the low side. That's not saying the Insider Advantage 23% unsure issue should be dismissed. I think that's a problem, but of a different sort. Best,

Posted on January 3, 2008 8:09 PM

____________________