Articles and Analysis


Our Poll of Pollsters - Part I: Holiday Interviews

Topics: 2008 , Iowa , Pollsters , The 2008 Race

As we expected poll releases to be a little slower than usual this week, we decided to conduct one of our own. Last week, we invited about a hundred professional pollsters who work for the news media and political campaigns to participate in a brief survey about the public polls in Iowa and New Hampshire. We will be releasing results from our "Poll of Pollsters" over the next few days. Today I want to discuss the very last question we asked, which concerns the same topic I blogged on yesterday: The reliability of polls conducted this week, between Christmas and New Year's Day.

First, some details about the survey. We invited two categories of pollsters to participate, those that work for political campaigns and those that conduct surveys for news media outlets. We selected campaign pollsters that had been listed in a scorecard published by the National Journal's Hotline that had worked on behalf of candidates for Governor, Senate or U.S. Congress in 2006. The media pollsters included those who conduct the well-known national polls that we regularly track and the organizations that had released surveys in Iowa and New Hampshire.

We sent out email invitations to just over 100 individuals, and as of yesterday a total of 46 had completed the entire survey (21 media pollsters and 25 campaign pollsters; of the latter category 14 were Democrats and 11 Republicans). As with any other survey, we pledged to keep respondent identities confidential and report their answers only as aggregated data. The pollsters responded online, using an online survey form prepared by our sponsor/owner, Polimetrix.

We also want to be very clear on one point: This survey is not a "scientific" sampling of any population beyond the 46 pollsters that replied. Thus, there is no "margin of error." The results represent no more and no less than the opinions of 46 individuals that were willing to respond. Since we made it clear that we would ask about the public polls in Iowa and New Hampshire, I assume that many of the non-participants are simply less interested in the topic. As one prominent campaign pollster who emailed his regrets put it, "in this case I feel like an ill-informed respondent." So the 46 we heard from are probably among the most opinionated on this subject.

So let's start with the topic of polling between Christmas and the New Year. Inspired by the quotations from pollsters in this recent Associated Press article, we decided to ask pollsters choose between two statements about the challenge of polling this week. But rather than forcing them to agree entirely with one statement or another, we asked the pollsters to pick a point along a scale to indicate which statement they agreed with more. The scale they used is reproduced below. They could use their mouse to click on any point along the scale (you can click on it to see a full size version):

There has been a lot of talk about the difficulty of polling during final week or so before the Iowa caucuses that includes both Christmas and New Years Day. Some believe that polling in this period will be challenging but can be done reliably. Others believe that polls done in this period will be so unreliable that they should be ignored. What about you?

Please click or drag on the ruler below to indicate which statement you agree with more.


The pollsters' responses spread across the scale as shown in the chart below. Not one signaled total agreement with either statement by moving the slider all the way to the left or all the way to the right of the ruler. They divide almost evenly in terms of which statement they lean to, with 48% placing themselves closer to the notion that polls "can be done reliably" this week, and 50% placing themselves closer to the statement that polls this week are "so unreliable they should be ignored" (note: one pollster/respondent clicked "not sure," three skipped the question completely).


However, the skeptics hold stronger opinions than the believers. Compare the two bars at the far left of the chart above (representing 17% who have the most doubts about the reliability of Christmas week polling) to the two bars at the right (5% who have the last doubts). For purposes of tabulation, we have translated the hash-marks along the ruler into a 100-point-scale, where 1 is "should be ignored" and 100 is "can be done reliably." The mean score for the full sample is 48 and the median 45, indicating a modest skew toward "should be ignored" (for the truly wonky, each of the bars in the chart represents a decile, 1-10, 11-20, 21-30 and so on).

As the second chart (below) indicates, the media pollsters are more skeptical than the campaign pollsters. The average score for media pollsters is a 43 (with a median of 29). In other words, the media pollsters cluster near the "so unreliable they should be ignored" end of the scale, while the campaign pollsters divide more evenly, with an average score of 50 and a median of 48. [For those interested in more details, click here for a spreadsheet with the number of responses for each increment clicked on the 1-100 scale].


Think of it this way: Even professional pollsters are not sure what to make of the surveys done this week. While few advise ignoring these surveys altogether, most recommend treating the results with a lot more caution than usual. Their uncertainty probably reflects the overall lack of experience our profession has with surveys conducted during the latter half of December. Just about everyone sampled recommends treating the results from this week with a lot more caution than those available at other times, although few would advise that we ignore these results completely.

We will have more results from our "Poll of Pollsters" over the next few days.



Great posts (both of them)! I believe, as I am certain many other bloggers believe, the first caucuses and primaries are going to shape the nomination. And, the desire of many is to have information ready to go if not to make some type of prediction. I would also suspect that people who take vacations also have more disposable income (are in general better educated) and so have different issues they consider important when they vote. I don't see how pollsters can avoid the bias of not being able to contact this group of people.


Richard :

Campaign pollsters are paid by campaigns that are desperate for up to the minute information. If these pollsters were to concede that holiday week polls are unreliable, campaigns would be less inclined to hire them to conduct such polls.

On the other hand, media pollsters work for generally independent organizations that only want periodic snapshots of popular opinion; they are not inclined to pay for more frequent information. If media pollsters were to concede that holiday week polls can be reliable, they could not justify taking polls periodically. Their polls would be called into question because they admittedly did not contain up to date information.

This poll proves nothing. It is like asking the opinions of plaintiffs' lawyers and defense lawyers about caps on personal injury awards. If both types of pollsters agreed generally on one side or the other of the question, that would be meaningful. All that can be inferred from this poll is that the respondents answered in ways consistent with their business models.


Frank Weinberg:

All polling is a sampling of a population. During the holidays, the population that can be sampled declines because of travel or other activities that make it more difficult to reach them. Does this new--smaller--population have the same characteristics as the larger (non-holiday) population? It depends on the characteristic sampled. Blue eyed or left handed people should be the same in both populations. If one political candidate appeals to a different demographic than another, however, the polling will remain valid only if the differering demographics have an identical likelihood of remaining available for polling. If there is a positive correlation between travel and income and a positive correlation between income and political choice, the validity of the holiday polling is impossible to determine.


Post a comment

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.