Articles and Analysis


Polling on the Dark Side of the Moon

Topics: 2008 , Divergent Polls , Iowa , The 2008 Race

Those of us with memories of the Apollo moon missions will remember those moments when the orbiting command module went around the so-called "dark side of the moon."** At that point the moon itself physically blocked broadcast signals, and despite all the dazzling "space age" telecommunications on display, live transmissions from the astronauts came to a total halt. While Walter Cronkite stalled for time, we watched and waited for the astronauts to regain contact.

Right now, with interest peaking in Iowa and New Hampshire polls, we have entered into our own dark side of the moon period. Unlike the moon missions, however, we are not in a total blackout. At least one pollster has released an Iowa survey conducted over the weekend, and others will surely follow later this week. However, with so many Americans traveling away from home for holiday travel, those surveys will be of unknown reliability, at best. The worst-case scenario (for the pollsters) will be if these surveys indicate a false trend, a shift that is less an indicator of real change than an artifact of respondents missing due to holiday travel.

Unfortunately, the survey research profession has relatively little experience with surveys conducted in the week between Christmas and the New Year (and the weekends surrounding that week). All of the polling firms I worked with rarely fielded surveys in the second half of December and typically shut down altogether between Christmas and New Years. CBS News Polling Director Kathy Frankovic reports in her column this week that the Roper Center archives include no public polls conducted between Christmas and New Years for either 2006 or in 2003 (other than one continuously running financial monitoring survey).

The reason that pollsters typically avoid polling around the holidays is the assumption that a big chunk of the population is away from home and unavailable for survey calls. As with any sort of "non-response" problem, we risk getting skewed or biased results if the missing respondents are both numerous and different in terms of their political views from those at home when we call.

What kinds of voters might be missing right now? Three years ago, in a survey concluded a full week before the holidays (12/15-17/2004), the Gallup Organization asked a national sample of 1,002 adults whether they planned "to travel more than fifty miles from home this holiday season." Twenty-eight percent (28%) said "yes." More important, as the table below indicates (based on data drawn from the Roper Archives), those planning holiday travel had a very distinctive demographic profile. Holiday travelers were much more likely to be younger and better educated. Notice also that holiday travelers were not just college students. Adults between the ages of 30 and 44 are twice as likely to travel for the holidays than those over 65. (Also, while I do not show it here, the pattern in these results by age, education and income was nearly identical for Democratic and Republican identifiers).


So pollsters fielding surveys this week are going to have a harder time finding younger, better educated respondents. Why is that important? Because in the Democratic race, at least, there are huge differences in vote preference by age and education: In virtually every survey, including those in Iowa and New Hampshire, Barack Obama does best among younger, better educated voters while Hillary Clinton's base of support is older and less well educated. Consider the data from the Boston Globe/University of New Hampshire survey conducted last week: Obama wins the support of 47% of voters age 18 to 34, but only 22% of those over 65; 40% of those with graduate degrees but only 25% of those without a college degree.


Given the age and education pattern, a sample with fewer voters under 45 or fewer with a college education will skew in Clinton's favor. Pollsters who see a demographic shift may choose to "weight" their samples to match the demographics of pre-holiday polls. That approach may lessen the bias but not correct it entirely. Weighting by age and education in this instance essentially replaces the younger, college educated voters who are away from home with other younger, college educated voters who are available to do the survey. If travelers and non-travelers are still different politically, regardless of age or education, then some bias will remain. And even if not, an extreme "weighting up" always increases the statistical error, in effect reducing the sample size.

Do we have any evidence that holiday travel bias might affect vote preference? Maybe. Consider the data below reported by Rasmussen Reports just before and after the Thanksgiving holiday.

Rasmussen runs a daily, rolling-average national survey that tracks presidential primary vote preference. Rasmussen uses an automated interactive-voice-response (IVR) methodology in which respondents answer recorded questions by pressing keys on their touch-tone phones. Each night, they call roughly 175-200 likely Democratic primary voters and roughly 150-160 likely Republican voters, then roll together and report a rolling average of the last four nights of interviews. Thus, each of the points on the chart below represents 750-800 likely Democratic voters with a reported margin of sampling error of +/-4%. The trend line is a regression estimate that Charles Franklin created for me using the Rasmussen data.


Notice what happens to the Obama trend line just before and just after Rasmussen took a five-day break from interviewing, Wednesday through Sunday, over the Thanksgiving weekend (Thanksgiving was November 22). The regression trend line essentially splits the difference between the 17% low points immediately before and after the Thanksgiving break and the much higher 26% results that came just before and just after that. What one makes of the variation may be in the eye of the beholder. Either the Obama trend got unusually erratic in both directions during the last two weeks of November, or there was a very unusual and precipitous plunge from what should have been a plateau around 24-26 to 17% centered on the period of Sunday, Monday and Tuesday before Thanksgiving and the Monday just after. Either way, the Obama variation around Thanksgiving was highly statistically significant. It did not occur by chance alone. Either there was a see-saw in Obama's real world support that week, or something changed in the kinds of voters Rasmussen sampled.

Given what we know about the demographics of holiday travelers and Obama's supporters, I'd bet on the latter.

Now I should point out that the Rasmussen national tracking may be a special case. According to Scott Rasmussen, each daily sample is essentially "fresh." Unlike many other pollsters, they do not attempt to call back unavailable respondents on successive nights. If they sample your phone number on Monday and you are not home, they will not call you back again on Tuesday. As such, their surveys may be more prone to a holiday effect than others that do more callbacks. And while I believe that Rasmussen weights their samples by gender, age and race to force consistency for each four-day report, they may not weight by education.

All of this brings us to the survey that the American Research Group released on Monday fielded between Thursday December 20 through Sunday December 23, a survey that shows Clinton gaining and Obama falling. Some will read this post as an attempt to debunk that result, and the findings above certainly argue for considerable caution in reading results from any survey this week. But the problem in trying to assess the ARG poll is that we know so little about it. Does ARG make call-backs to unavailable respondents? What was the sample composition on any ARG Iowa survey this year in terms of age and education level, and was this one suddenly different? Did ARG weight the results by age or education this time, and if so, by how much? We are in the dark on all of these questions.

It is also worth remembering, as some commenters noted yesterday, that real changes may be occurring in vote preference this week even if surveys may be severely challenged in their ability to measure it. Clinton may be gaining and Obama falling. So it is quite a leap for anyone to say they know conclusively that the ARG result is either right or wrong.

The hard truth is that we are behind the dark side of the moon this week, and we may not know much with certainty until next Wednesday night.

**Technically, the moon does not have a "dark side." Although the more appropriate term is "far side" of the moon, I still prefer the Pink Floyd version.



There may or may not be a trend, but ARG's numbers are also way off the mainstream in NH and SC. In NH, they have Clinton +14 at the same time as three other polls have it essentially tied (Globe has Obama +2, Gallup has it tied, and Rasmussen has it Clinton +3).

In SC, an ARG poll in late November has Clinton +24, whereas other polls at the time and since have it very close, with either Obama or Clinton leading.



Who is the media partner for ARG? Who is footing the bill for their polling?


Interesting, but your own chart shows Clinton rising and Obama falling, before the Chrismas holiday.



I still have a spreadsheet of 2003-04 democratic primary polls and I have no polls recorded for Iowa between the first week of December and the first week of January. There were a couple other interesting points though, as I look back at it.

One is that John Edwards' share of Iowa seems to have jumped a full 5 points between early December and the first batch of January polls. That increase seems to have come at Undecided's expense. All the other candidates were essentially flat across that period.

The other interesting is that both Kerry and Edwards were up another 7-8 points each the following week, which would roughly correspond to now in terms of days out from the caucuses. That week also marked the beginning of Dean and Gephardt's decline. Dean dropped about 6 points that week and Gep was down about 3. So it would not be out of line with 2004 to see some significant changes about now.

In New Hampshire in 2004, polling took a break from 12/19 to 12/26, during which time Howard Dean seems to have shed about 7 points. That turned out to be real. ARG's NH polls looked to be pretty good in 2004 BTW. I think they're based in NH. They did not poll IA in 2003-04.



Interesting. What I still don't understand is why no one seems to be talking about the limited demographic that pollsters actually do reach. The majority of my friends and acquaintances under 35--both married and single--do not have landlines; just cellphones. I have a landline as a weird archaic non-cellphone user, but I have caller ID, like most middle-class people under 70--and I only answer the phone when I recognize the caller due to telemarketers. If I don't, and the caller is someone I want to talk to, they'll start leaving a message and I pick up. But I don't know many who would answer random calls if they have caller ID.

On top of that, pollsters call at a time when a certain demographic is more unlikely to be home. That's 3 strikes against many, many people. So would it not be more accurate to say that polls reflect a demographic that 1) owns a landline, 2) either do not have caller ID or, if they do, actually answer calls from unknown callers, or 3) are home at the time pollsters call.

Looking at your figures, this would make a lot of sense--someone who fits the bill with these three tenuous requirements would be my 94 year old neighbor who does not have a high school education. She is still undecided, but according to your figures, it would seem that she fits into the category most likely to vote for Hillary. Again, I'm amazed that this seems to not be discussed much.


Mark Lindeman:

Lucy, people actually talk a lot about that point. The way you've put it, it sounds like it would be rare for anyone under 70 to talk to a pollster, but that isn't actually true. But you're right that the "raw" demographics are pretty skewed, and that is pretty scary.

In 2004, the best available evidence indicates that the age skew wasn't a big problem, because the relatively few younger voters who did talk to pollsters had similar preferences (and turnout) to those who didn't. So, by weighting their raw data to match national distributions, pollsters did OK.

However, it's fair to wonder whether polling as we know it (Random Digit Dialing to land lines) is going behind the "dark side of the moon" and headed for the Delta Quadrant -- and we just can't tell how fast. This concern predates the wide use of cell phones, and isn't limited to visible distortions in survey samples.


John M.:

Fascinating discussion. It's been my understanding that while many pollsters attempt to correct for cell phone users by weighting age, gender, etc., ARG does absolutely no reweighting. To paraphrase their president, young voters are unlikely to vote, so they're not worth accounting for. If so, it would explain why ARG polls have differed so markedly from the mean in NH, SC, and IA. Can you get hold of their raw numbers and methodology and confirm that this is the case?



I'm aware of at least one pollster (The Field Poll in California) which specifically states they poll cell phones as well as land lines. They use a state database of registered voters, and that database has phone numbers attached that the voters provided when they registered-and some of those numbers are cell phones.

The Field Poll is also known for being extremely accurate. Maybe this is one reason why. Too bad they are California-only.



I've never really given the Field poll very high marks in general. But I also thought it was illegal for call centers to make unsolicited calls to numbers that have to pay for incoming calls. And of course, since the majority of American households now have both wired and cell service, I don't know how you make sure you're not double-dipping if you call both.


Post a comment

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.