Mark Blumenthal | December 21, 2007
Topics: 2008 , Iowa , Likely Voters , The 2008 Race
Notice the deluge of polls from Iowa and New Hampshire the last few days? It has been pretty hard to miss. We have seen six new Iowa polls in the last three days. Have we reached the point where, as one valued reader put it via email, do we now have "too many polls, too little meaning?" Is it time to stop watching polls altogether?
The big problem, particularly in Iowa, is the way a close race (especially for the Democrats) combines with wide variations in "likely caucus goer" methodology to thoroughly confuse everyone. And for good reason. Consider the screen shot from our Iowa Democrats chart (below) which shows the results for Obama (yellow), Clinton (purple) and Edwards (red) over the last two months (the light blue grid lines are 5 percentage points apart). Forget the lines, for the moment and look at the points. They are all over the place.
Put another way, consider the following results from the last six Iowa polls, all fielded over the last week. The support for the candidates ranges between:
- 24% and 30% for Clinton
- 25% and 33% for Obama
- 18% and 26% for Edwards
- 6% and 20% (on the Republican side) for McCain
Some of this variation is the purely random sort that comes with doing a survey (the part that the "margin of error" quantifies), and how hard each organization pushes those who are initially undecided, but a large portion also comes from how they define and select "likely caucus" goers. What makes Iowa different is that the last source of variability. It is bigger and more consequential than for other types of polls. So if we take into account both the closeness of the Democratic race and all sources of potential poll error, we really have no idea who is truly "ahead" at this point in the race. The polls are simply too blunt an instrument, especially given all the uncertainty about who will participate.
So what should we keep in mind when looking at the new polls?
1) The fact that results vary with methodology tells us something important: For the Democrats, the nature of the turnout -- what kinds of voters show up on January 3 -- will likely determine the outcome. We can see the same thing within individual polls. As I noted earlier in the week, the ABC News analysis puts this best.
Applying tighter turnout scenarios can produce anything from a 10-point Obama lead to a 6-point Clinton edge -- evidence of the still-unsettled nature of this contest, two weeks before Iowans gather and caucus. And not only do 33 percent say there's a chance they yet may change their minds, nearly one in five say there's a "good chance" they'll do so
2) Comparisons between apples-and-oranges can be very misleading. Different methodologies can produce different results, so it's a fools errand to directly compare say, yesterday's ARG result to the Post/ABC poll from earlier in the week. Averaging five or six polls at a time can help reduce the purely random variation, but in this instance (to torture the metaphor), it leaves us comparing a basket of apples, oranges and pears from this week to a basket of apples, bananas and grapefruit from the week before that. Put another way, notice the way the last 13 polls have been done by 11 different pollsters. An average of the last six polls has only two pollsters in common (Strategic Vision and Rasmussen) with the seven polls released the previous week.
2) Apples-to-apples comparisons are safer, but unless the race shifts dramatically, they don't tell us much about day-to-day or even week-to-week variation. The table below shows how results from five of this week's polls compare to results from the same organizations conducted in mid-to-late November. It shows a slight average gain (+3%) for Obama, with changes of less than a point for the other candidates. Interesting, but we had too look back nearly a month to get a decent, averaged comparison, and even then the direction of the change is inconsistent across individual polls.
3) Our charts illustrate most of the variation that we can "see." Some readers may be frustrated that the lines do not shift as much as a rolling average, but more often than not, the day-do-day variation is just the "noise" of pollster house effects. If we were looking at a few hundred interviews conducted every night using a constant methodology, we might be able to see more genuine day-to-day variation. Given the data available, however, our lines are showing us about as much of the real variation as we can truly "see" through the methodological clouds.
One caveat on the above, however, is that Professor Franklin can alter the sensitivity of those trend lines to check for any short term shifts that may better fit the data. Requests for another "sensitivity analysis" have filled my email in box over the last few weeks. We have heard you, and we will have another sensitivity analysis later today, and updates next week.
So what do we know? Among Republicans, Mike Huckabee has clearly seen a dramatic increase in support for the last month, and now leads nominally in eight of the last nine polls (the individual margins may not be statistically significant, but the mostly consistent direction tells us that that Huckabee's advantage is most likely real). Still Huckabee's support remains soft and the Republican ad war is turning negative. Things can still change a lot over the next two weeks.
For the Democrats, Obama has gained over the last month, but the latest round of surveys are neither consistent nor powerful enough to tell who would win if the Iowa Caucuses were held today. And obviously, with the race as close as it appears to be, changes over the next two weeks could also prove decisive.
And now the race goes "behind the dark side of the moon," as it were, given the challenges of polling between Christmas and New Year's. I will have more to say about that very soon.