Pollster.com

Articles and Analysis

 

Abramowitz: Tracking the Tracking Polls: Real Change or Random Noise?


Alan I. Abramowitz is the Alben W. Barkley Professor of Political Science at Emory University in Atlanta, Georgia. He is also a frequent contributer to Larry Sabato's Crystal Ball.

It may take some time for historians to decide whether the contest between Barack Obama and John McCain should be considered one of the most important presidential elections in American history. But we already know that it will be the most thoroughly polled presidential election in American history. Lately it seems that not a day goes by without dozens of new national and state polls being released. There are now no fewer than eight national tracking polls underway. These are polls that interview a random sample of voters every day and then combine the results over three or more days, adding one new day and dropping one earlier day, in order to measure changes in candidate preferences within the electorate.

In the past few days there has been a lot of speculation in the media about whether the presidential race has been tightening. Some pundits have argued that such tightening is inevitable in the final days of a presidential race. So what do the national tracking polls tell us about the state of the race between Obama and McCain with less than two weeks left until Election Day?

An analysis of the seven national tracking polls that have been up and running since at least October 12th (Rasmussen, Gallup, DailyKos/R2K, Diageo/Hotline, Battleground, IBD/TIPP, and Zogby) leads to several conclusions. Perhaps the most important one is that despite differences in sampling, interviewing, and weighting procedures, Barack Obama led John McCain on every day in every poll. Beyond that basic finding, however, there are some clear differences in the results of these seven tracking polls.

The results in Table 1 show that while all seven tracking polls have had Obama ahead over the past ten days, the size of that lead has varied considerably. During this time period Obama's average lead has ranged from a low of 4.4 points in the IBD/TIPP Poll to a high of 9.8 points in the DailyKos/R2K Poll. In addition, some of the tracking polls have shown much more volatility than others: the standard deviation of Obama's lead has ranged from a low of 0.8 points in the Rasmussen Poll to a high of 3.9 points in the Battleground Poll which has had both the smallest (1 point) and the largest (13 points) Obama lead in the past ten days. In contrast, Obama's lead in the Rasmussen Poll has varied only from a low of 4 points to a high of 6 points. In general, polls like Rasmussen that weight their results by party identification tend to produce more stable results than polls that do not weight by party identification.

Alan1022tab1.png

There is no evidence in these data of any tightening of the presidential race over this time period.
Figure 1 shows the trend in Obama's average lead in six tracking polls that provided results every day between October 12 and October 20 (the Battleground Poll does not report results on the weekend). While Obama's lead increased in some polls and decreased in others during this period, the results in Figure 1 show that the overall average changed very little. Obama led by an average of 6.5 points on October 12thand he led by an average of 7.0 points on October 20st, the final date included in this analysis.

alan1022fig1.png

If there is no overall trend, then what explains the day to day movement in the tracking polls? One possibility is that most if not all of the day to day movement was due to sampling variation-that it was nothing more than random noise. In order to test this hypothesis, I calculated the correlations among the day-to-day results of the seven tracking polls over these ten days. The correlations between individual pairs of polls varied considerably. Some were strongly positive, some were very weak, and some were strongly negative. Nothing much should be made of this, however, because of the very limited number of days on which these correlations were based. What is significant, however, is that the average correlation among the seven tracking polls over this ten day period was -.06. This means that there was basically no relationship in the day-to-day movement of these polls during this time period. Whether Obama's support was going up or down in one poll was unrelated to whether his support was going up or down in the other six polls.

The lesson that should be drawn from these findings is not that there is any fundamental flaw in the tracking polls. Random variation is unavoidable in public opinion polling. What these findings do indicate, however, is that poll-watchers should not pay too much attention to the day to day movements in these polls unless they see all or most of them moving consistently in one direction over a period of time. Similarly, polling organizations should avoid overemphasizing the significance of the day to day movements in their own polls and pay more attention to whether their results are consistent with those of other polls.

 

Comments
johncoz:

The one thing you will not get pollsters to do is "pay more attention to whether their results are consistent with those of other polls." If they did then we would not need sites like this, and many of us would not be maintaining our own models to try to sort signal from noise.

And one can expect only slightly more luck with them "overemphasizing the significance of the day to day movements in their own polls". It is only the aristocrats of polling, Rasmussen and Gallup, that regularly point out that small movements are simply fluctuations within a very stable trend. Self-poromoters such as Zogby will trumpet every tiny movement, often resulting in silly and contradictory statements.

Finally, though Obama's numbers have indeed been stable over the period looked at, it is also true that McCain's numbers improved last week by almost 2 points, peaking on Saturday's reports. This becomes immediately obvious if one weights the polls by sample size, and only aggregate polls which have the same survey period (eg 3-day trackers only). Both of these measure go some way to eliminating noise and statistical artifacts.

http://farm4.static.flickr.com/3194/2962718755_5e9cebde07_b.jpg

____________________

chrisk314:

The danger in having pollsters pay attention to whether their results are consistent with those of other polls is the loss of randomness. If pollsters start weighting/massaging their data with an eye on the whole set of polls, there is potential for "run with the pack" error, which can build on itself. The beauty of outliers is that they are expected, and that they generally confirm a healthy distribution of samples.

____________________

Alan Abramowitz:

My point here is not that polling organizations should massage their results to bring them into line with the overall trend but that they should not over-interpret day to day shifts in their own results.

____________________

chrisk314:

My apologies for interpreting your comments wrongly. It's certainly true that jumps like we saw today are dangerous because of how they are reported rather than because of what they represent.

____________________

johncoz:

Quite frankly, a number of pollsters simply produce nonsense.

Rather than pick on poor Zogby again, consider Tipp: that tracker started on 10/13 with a +2 spread which had reached +7.3 by 10/18, the very day on which everybody's aggregations showed the tightest spread in over a fortnight (5-6).

Any admission they may have been in error? Not on your Nelly! They just soldier on pretending that they haven't made a complete ass of themselves. And this from an outfit that self-promotes as the "most accurate".

This sort of rubbish will continue until pollsters are made to expose their methodology and publish their raw data. In the meantime expect commercial considerations to trump professionalism and honesty every time.

____________________

Allen:

I believe there was a small (~3 point) drop in the Obama minus McCain margin between Oct 6 and Oct 13. It can be seen in the Rasmussen and Tarrance tracking polls. It can also be seen in the electoral vote projections at http://election-projection.net (although as always, the state polls and therefore the electoral vote projections lag slightly behind the national tracking polls). If you only looked at national tracking polls from Oct 12 to today, then you would have missed this shift. The race does appear to have flattened out since then.

____________________

johncoz:

@allen

My weighted aggregates of the 3-dat trackers shows the largest spread on 10/11 as +8.1 and falling from there to +5.4, but with Obama's support in that period remaining within a tight band just above 50%.

Since then the race has flattened but improved for Obama (see graph linked I posted above). This movement is consistent with the reports of RCP, Pollster and 538 -- though of course everyone has slightly different numbers.

____________________

Interesting, same analysis and same results I came up with last night :) Guess I wasn't out in left field. The race seems remarkably stable - now the question becomes what happens with these stubborn indy's.

____________________

THX1138:

Fantastic article! Thanks Alan!

This felt like a visit to the Tracking Poll Doctor - you present it so clearly, and make sense of all the chaotic reporting we've heard in the past week about the polls. Thanks for your steady hand.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR