December 26, 2007

Polling on the Dark Side of the Moon

Those of us with memories of the Apollo moon missions will remember those moments when the orbiting command module went around the so-called "dark side of the moon."** At that point the moon itself physically blocked broadcast signals, and despite all the dazzling "space age" telecommunications on display, live transmissions from the astronauts came to a total halt. While Walter Cronkite stalled for time, we watched and waited for the astronauts to regain contact.

Right now, with interest peaking in Iowa and New Hampshire polls, we have entered into our own dark side of the moon period. Unlike the moon missions, however, we are not in a total blackout. At least one pollster has released an Iowa survey conducted over the weekend, and others will surely follow later this week. However, with so many Americans traveling away from home for holiday travel, those surveys will be of unknown reliability, at best. The worst-case scenario (for the pollsters) will be if these surveys indicate a false trend, a shift that is less an indicator of real change than an artifact of respondents missing due to holiday travel.

Unfortunately, the survey research profession has relatively little experience with surveys conducted in the week between Christmas and the New Year (and the weekends surrounding that week). All of the polling firms I worked with rarely fielded surveys in the second half of December and typically shut down altogether between Christmas and New Years. CBS News Polling Director Kathy Frankovic reports in her column this week that the Roper Center archives include no public polls conducted between Christmas and New Years for either 2006 or in 2003 (other than one continuously running financial monitoring survey).

The reason that pollsters typically avoid polling around the holidays is the assumption that a big chunk of the population is away from home and unavailable for survey calls. As with any sort of "non-response" problem, we risk getting skewed or biased results if the missing respondents are both numerous and different in terms of their political views from those at home when we call.

What kinds of voters might be missing right now? Three years ago, in a survey concluded a full week before the holidays (12/15-17/2004), the Gallup Organization asked a national sample of 1,002 adults whether they planned "to travel more than fifty miles from home this holiday season." Twenty-eight percent (28%) said "yes." More important, as the table below indicates (based on data drawn from the Roper Archives), those planning holiday travel had a very distinctive demographic profile. Holiday travelers were much more likely to be younger and better educated. Notice also that holiday travelers were not just college students. Adults between the ages of 30 and 44 are twice as likely to travel for the holidays than those over 65. (Also, while I do not show it here, the pattern in these results by age, education and income was nearly identical for Democratic and Republican identifiers).

So pollsters fielding surveys this week are going to have a harder time finding younger, better educated respondents. Why is that important? Because in the Democratic race, at least, there are huge differences in vote preference by age and education: In virtually every survey, including those in Iowa and New Hampshire, Barack Obama does best among younger, better educated voters while Hillary Clinton's base of support is older and less well educated. Consider the data from the Boston Globe/University of New Hampshire survey conducted last week: Obama wins the support of 47% of voters age 18 to 34, but only 22% of those over 65; 40% of those with graduate degrees but only 25% of those without a college degree.

Given the age and education pattern, a sample with fewer voters under 45 or fewer with a college education will skew in Clinton's favor. Pollsters who see a demographic shift may choose to "weight" their samples to match the demographics of pre-holiday polls. That approach may lessen the bias but not correct it entirely. Weighting by age and education in this instance essentially replaces the younger, college educated voters who are away from home with other younger, college educated voters who are available to do the survey. If travelers and non-travelers are still different politically, regardless of age or education, then some bias will remain. And even if not, an extreme "weighting up" always increases the statistical error, in effect reducing the sample size.

Do we have any evidence that holiday travel bias might affect vote preference? Maybe. Consider the data below reported by Rasmussen Reports just before and after the Thanksgiving holiday.

Rasmussen runs a daily, rolling-average national survey that tracks presidential primary vote preference. Rasmussen uses an automated interactive-voice-response (IVR) methodology in which respondents answer recorded questions by pressing keys on their touch-tone phones. Each night, they call roughly 175-200 likely Democratic primary voters and roughly 150-160 likely Republican voters, then roll together and report a rolling average of the last four nights of interviews. Thus, each of the points on the chart below represents 750-800 likely Democratic voters with a reported margin of sampling error of +/-4%. The trend line is a regression estimate that Charles Franklin created for me using the Rasmussen data.

Notice what happens to the Obama trend line just before and just after Rasmussen took a five-day break from interviewing, Wednesday through Sunday, over the Thanksgiving weekend (Thanksgiving was November 22). The regression trend line essentially splits the difference between the 17% low points immediately before and after the Thanksgiving break and the much higher 26% results that came just before and just after that. What one makes of the variation may be in the eye of the beholder. Either the Obama trend got unusually erratic in both directions during the last two weeks of November, or there was a very unusual and precipitous plunge from what should have been a plateau around 24-26 to 17% centered on the period of Sunday, Monday and Tuesday before Thanksgiving and the Monday just after. Either way, the Obama variation around Thanksgiving was highly statistically significant. It did not occur by chance alone. Either there was a see-saw in Obama's real world support that week, or something changed in the kinds of voters Rasmussen sampled.

Given what we know about the demographics of holiday travelers and Obama's supporters, I'd bet on the latter.

Now I should point out that the Rasmussen national tracking may be a special case. According to Scott Rasmussen, each daily sample is essentially "fresh." Unlike many other pollsters, they do not attempt to call back unavailable respondents on successive nights. If they sample your phone number on Monday and you are not home, they will not call you back again on Tuesday. As such, their surveys may be more prone to a holiday effect than others that do more callbacks. And while I believe that Rasmussen weights their samples by gender, age and race to force consistency for each four-day report, they may not weight by education.

All of this brings us to the survey that the American Research Group released on Monday fielded between Thursday December 20 through Sunday December 23, a survey that shows Clinton gaining and Obama falling. Some will read this post as an attempt to debunk that result, and the findings above certainly argue for considerable caution in reading results from any survey this week. But the problem in trying to assess the ARG poll is that we know so little about it. Does ARG make call-backs to unavailable respondents? What was the sample composition on any ARG Iowa survey this year in terms of age and education level, and was this one suddenly different? Did ARG weight the results by age or education this time, and if so, by how much? We are in the dark on all of these questions.

It is also worth remembering, as some commenters noted yesterday, that real changes may be occurring in vote preference this week even if surveys may be severely challenged in their ability to measure it. Clinton may be gaining and Obama falling. So it is quite a leap for anyone to say they know conclusively that the ARG result is either right or wrong.

The hard truth is that we are behind the dark side of the moon this week, and we may not know much with certainty until next Wednesday night.

**Technically, the moon does not have a "dark side." Although the more appropriate term is "far side" of the moon, I still prefer the Pink Floyd version.

-- Mark Blumenthal

December 26, 2007 in Divergent Polls, Iowa, The 2008 Race

Comments

Post a comment

URL:

Comments: (you may use HTML tags for style)

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button.

Please email us to report offensive comments.

See our comment policy here. Note that we require commenters to share their email address via Typekey. We will never share your email address with anyone without your explicit permission.