Pollster.com

Articles and Analysis

 

On Trackers

Topics: Barack Obama , Divergent Polls , Gallup , John McCain , National Journal , Pollsters , Rasmussen , USAToday Gallup

My Nationaljournal.com column, on the differences between rolling-average tracking polls and other "traditional" surveys, is now online.

Regular readers may be interested in the chart in the column created by our own Charles Franklin (see below) and spiffed up considerably by the National Journal's Reuben Dalke (see the column). I wondered how the tracking poll trends compare to standard trend estimates that you see on our national chart. The chart that Franklin created plots the trends on the Obama margin (Obama percentage minus McCain percentage) using a loess regression trend line based on the non-overlapping releases from Gallup Daily, Rasmussen Reports and all other national polls. To make for a fair comparison, all three lines are plotted with the same sensitivity.

I was also curious how the trends would look if we simply "connected the dots" between the non-overlapping tracking poll releases by Gallup and Rasmussen tracking surveys as well as the "traditional" USA Today/Gallup results (based on "likely voters") . You see that below.

The Gallup Daily line looks more variable than what you are used to seeing on Gallup's Daily release, partly because the time scale is more compressed, partly because we are plotting the Obama-McCain margin rather than separate lines for each candidate and partly because we are plotting only every third or fifth day which eliminates the "smoothing" effect of the overlapping intermediate samples.

What conclusions do you draw?

Update: In the comments, PatrickM asks:

As to the sampling process for the Gallup tracking survey: I thought the purpose of the tracking survey was to draw a discrete sample each night. Since completion quotas are set for each night, non-respondents must necessarily be "replaced" for that night's calls. Theoretically, all these replacements should balance out if non-response is random.

But Gallup seems to be taking a second bite at this apple by drawing an entirely new sample on the second night and supplementing it with non-respondents from the first night until the nightly completion quota is reached. So theoretically, the 3-day rolling results could include data from the originally drawn sample point AND its doppelganger replacement phone number.

I'm not a sampling expert, but is there anybody out there who can describe the rationale behind why this is OK?

The best explanation I have seen of "rolling cross section design" (a more technically correct term than "rolling average") is Kate Kenski's description of the National Annenberg Election Survey (NAES) in Chapter 4 in Romer, Kenski, Winneg et. al., Capturing Campaign Dynamics 2000 & 2004 .

The NAES, ongoing now for 2008 but mostly held back for academic analysis, uses the same general "tracking design" as Gallup only with far more rigor: In 2004, they protocol involved dialing non-contacts as many as 18 times over as many as 14 days.

I won't try to summarize the whole chapter, but this paragraph gets closest to answering Patrick's question:

What is important to note here is that there were strict procedures in place so that no telephone number was treated differently from any of the other numbers selected. Telephone numbers released on Tuesdays were not handled differently from telephone numbers released on Fridays. This protocol ensures that the probability of being interviewed is a random event. By stabilizing the proportion of respondents who completed an interview after having been called numerous times, the representativeness of the daily cross-sections is maximized.

Why is it important that the date of the interview be a random event? if the date of interview is random, then the characteristics of the sample on any given day will not vary systematically.

 

Comments
1magine:

Conclusion - Gallup is volatile.
Conclusion - Avg of other polls less volatile show trends well.
Conclusion - All polls taken together paint a absolutely clear picture of the race at this moment.

____________________

PatrickM:

What's most interesting to me is the Gallup v. Gallup differences, particularly for that mid-June data point.

As to the sampling process for the Gallup tracking survey: I thought the purpose of the tracking survey was to draw a discrete sample each night. Since completion quotas are set for each night, non-respondents must necessarily be "replaced" for that night's calls. Theoretically, all these replacements should balance out if non-response is random.

But Gallup seems to be taking a second bite at this apple by drawing an entirely new sample on the second night and supplementing it with non-respondents from the first night until the nightly completion quota is reached. So theoretically, the 3-day rolling results could include data from the originally drawn sample point AND its doppelganger replacement phone number.

I'm not a sampling expert, but is there anybody out there who can describe the rationale behind why this is OK?

____________________

Mike_in_CA:

Something that strikes me is the abrupt bump in Rasmussen once Obama captured the nomination. This was also around the beginning of June, and it was said once in one of these threads that Rasmussen weights their polls by Party ID, specifically by the previous month's party ID. Presumably then, on June 1, Rasmussen switched to different Party ID numbers. A gallup poll out today shows that the number of Dems has been steadily increasing since late last year.

In conclusion, I wonder if the abrupt jump in Obama's numbers at the beginning of June has less to do with a post-nomination bounce, as it does with a re-weighted of Party ID per Rasmussen. Prior to then, Ras was consistently showing Obama underperforming according to Gallup. Now they show him overperforming...

Just some food for thought. Any thoughts on this Mark?

____________________

Chris G:

The Rasmussen deviation from other polls is very interesting because there's actually a trend in the deviation itself. their Obama margin was consistently below average until June, now about on par. perhaps they changed sampling techniques or have a bias towards some demographic that itself has increased in support for Obama more than others (e.g. Clinton supporters)

____________________

Mike_in_CA:

@Chris G: See my post above. I think it may have a lot to do with Party ID numbers, because the big jump happened in early June, when Ras switched to new, May, ID numbers... Maybe Mark has input?

____________________

Chris G:

Mike_in_CA- I think Party ID has been pretty stable relative to the trends we see above:
http://www.rasmussenreports.com/public_content/politics/mood_of_america/party_affiliation/partisan_trends

The other thing is the jump, I see it too and thought maybe the nomination explains some of that. but also look at the slope of the increase since that trough in mid-March (seen in both Rasmussen and average of others). the slope in Rasmussen is steeper than the average, which is very difficult to explain using purely random sampling fluctuations. overall, the Rasmussen time series shape is similar to the average, but more exaggerated. could they be sampling more swing voters? might be useful to look at raw Obama and McCain numbers or undecideds

____________________

Mark Blumenthal:

@PatrickM: I just updated the main post with an answer.

@Mike: I'm also curious about that apparent change in the Rasmussen line in June, but have no idea what explains it. I hear that Rasmussen may publish some new analysis of their party ID trends over the weekend. Perhaps that will offer some clues.

____________________

PatrickM:

@Mark -- Thanks for the response. That kind of sounds like they?re releasing a different set of replicates each day.

However, I'm still left scratching my head about Gallup v. Gallup -- which theoretically should control for everything but how the sample is released. So, why does Obama consistently do better in the Gallup/USA Today polls? The McCain numbers show the kind of random, within-MOE variability one would expect, but not the Obama numbers (see below).

Obama
Date   Track   G/USA   diff
6/19       46     50      +4
6/1         46     49      +3
5/3         42     47      +5
4/20       45     49      +4
3/16       44     49      +5

McCain
Date   Track   G/USA   diff
6/19     44      44          0
6/1       46      44        -2
5/3       47      48       +1
4/20     45      44        -1
3/16     46      47       +1

An interesting side note: Starting on June 8th, Gallup daily tracking went from a 5 day to 3 day rolling average, but Gallup/USA went from a 3 day to a 5 day field period (but Obama would still do consistently better if they reported both polls for the same field periods).

____________________

PatrickM:

Not sure what happened to that table (it looked OK in the preview). Here it is again:

Obama
Date Track G/USA diff
6/19 46 50 +4
6/01 46 49 +3
5/03 42 47 +5
4/20 45 49 +4
3/16 44 49 +5

McCain
Date Track G/USA diff
6/19 44 44 0
6/01 46 44 -2
5/03 47 48 +1
4/20 45 44 -1
3/16 46 47 +1

____________________

PatrickM:

CORRECTION: Sorry about this. My prior comment compared Gallup tracking's RV base to USAToday's Likely voters. Here's the RV to RV comparison. This still raises a question as to whether the way sample is released is contributing to a systematic bias in Obama's support.

Obama
Date Track G/USA diff
6/19 46 48 +2
6/01 46 47 +1
5/03 42 45 +3
4/20 45 47 +2
3/16 44 48 +4

McCain
Date Track G/USA diff
6/19 44 42 -2
6/01 46 44 -2
5/03 47 47 0
4/20 45 44 -1
3/16 46 47 +1

____________________

brambster:

I recall reading that Rasmussen uses samples from the previous month to establish party ID weighting in the current month. I believe that this is why Rasmussen closed the gap with the other polls after showing a consistent 5 point deficit with the average.

It also suggests to me that if a bounce in party ID did occur in June (as Rasmussen's numbers do show change there), then even if Obama's surge slows in July, Rasmussen may well over-inflate the result.

____________________

So what effect do these tracking polls (including the state tracking polls) have on the beliefs of the state of the horse race if there are significant house effects (for whatever reason)? Won't these tracking polls be given too much weight due to their frequency?

____________________

jsh1120:

Mark,

Sorry if it appears this question is appearing on a number of different threads, but perhaps you can provide a clue.

As you note in your column, Rasmussen weights their polls by estimated PartyID while Gallup does not. I also believe (from what I've read) that Rasmussen uses the last three month partisan split to weight their daily samples. It's unclear to me whether that is a rolling or a discrete 3 month period.

My question is how Rasmussen weights their STATE level polls. I presume they don't use the same national partisan split. But as far as I can tell, Rasmussen is completely silent on the partisan weighting they do use for state level polls, either for President or for state offices.

____________________

jsh1120:

Mark,

Sorry if it appears this question is appearing on a number of different threads, but perhaps you can provide a clue.

As you note in your column, Rasmussen weights their polls by estimated PartyID while Gallup does not. I also believe (from what I've read) that Rasmussen uses the last three month partisan split to weight their daily samples. It's unclear to me whether that is a rolling or a discrete 3 month period.

My question is how Rasmussen weights their STATE level polls. I presume they don't use the same national partisan split. But as far as I can tell, Rasmussen is completely silent on the partisan weighting they do use for state level polls, either for President or for state offices.

____________________

BlueMerlin:

Can someone explain how 'partisan' poll weighting actually works? And I mean with the kind of detail required of a scientific paper. Or point me to a reference.

Why would anyone want to take seriously the results of a polling organization that does not publish its methods?

In particular there's something highly fishy about gallup's and rasmussen's daily presidential tracking polls. They're way too stable to be real, and I suspect something is being done numerically that overdamps them.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR