December 27, 2007

Our Poll of Pollsters - Part I: Holiday Interviews

As we expected poll releases to be a little slower than usual this week, we decided to conduct one of our own. Last week, we invited about a hundred professional pollsters who work for the news media and political campaigns to participate in a brief survey about the public polls in Iowa and New Hampshire. We will be releasing results from our "Poll of Pollsters" over the next few days. Today I want to discuss the very last question we asked, which concerns the same topic I blogged on yesterday: The reliability of polls conducted this week, between Christmas and New Year's Day.

First, some details about the survey. We invited two categories of pollsters to participate, those that work for political campaigns and those that conduct surveys for news media outlets. We selected campaign pollsters that had been listed in a scorecard published by the National Journal's Hotline that had worked on behalf of candidates for Governor, Senate or U.S. Congress in 2006. The media pollsters included those who conduct the well-known national polls that we regularly track and the organizations that had released surveys in Iowa and New Hampshire.

We sent out email invitations to just over 100 individuals, and as of yesterday a total of 46 had completed the entire survey (21 media pollsters and 25 campaign pollsters; of the latter category 14 were Democrats and 11 Republicans). As with any other survey, we pledged to keep respondent identities confidential and report their answers only as aggregated data. The pollsters responded online, using an online survey form prepared by our sponsor/owner, Polimetrix.

We also want to be very clear on one point: This survey is not a "scientific" sampling of any population beyond the 46 pollsters that replied. Thus, there is no "margin of error." The results represent no more and no less than the opinions of 46 individuals that were willing to respond. Since we made it clear that we would ask about the public polls in Iowa and New Hampshire, I assume that many of the non-participants are simply less interested in the topic. As one prominent campaign pollster who emailed his regrets put it, "in this case I feel like an ill-informed respondent." So the 46 we heard from are probably among the most opinionated on this subject.

So let's start with the topic of polling between Christmas and the New Year. Inspired by the quotations from pollsters in this recent Associated Press article, we decided to ask pollsters choose between two statements about the challenge of polling this week. But rather than forcing them to agree entirely with one statement or another, we asked the pollsters to pick a point along a scale to indicate which statement they agreed with more. The scale they used is reproduced below. They could use their mouse to click on any point along the scale (you can click on it to see a full size version):

There has been a lot of talk about the difficulty of polling during final week or so before the Iowa caucuses that includes both Christmas and New Years Day. Some believe that polling in this period will be challenging but can be done reliably. Others believe that polls done in this period will be so unreliable that they should be ignored. What about you?

Please click or drag on the ruler below to indicate which statement you agree with more.

The pollsters' responses spread across the scale as shown in the chart below. Not one signaled total agreement with either statement by moving the slider all the way to the left or all the way to the right of the ruler. They divide almost evenly in terms of which statement they lean to, with 48% placing themselves closer to the notion that polls "can be done reliably" this week, and 50% placing themselves closer to the statement that polls this week are "so unreliable they should be ignored" (note: one pollster/respondent clicked "not sure," three skipped the question completely).

However, the skeptics hold stronger opinions than the believers. Compare the two bars at the far left of the chart above (representing 17% who have the most doubts about the reliability of Christmas week polling) to the two bars at the right (5% who have the last doubts). For purposes of tabulation, we have translated the hash-marks along the ruler into a 100-point-scale, where 1 is "should be ignored" and 100 is "can be done reliably." The mean score for the full sample is 48 and the median 45, indicating a modest skew toward "should be ignored" (for the truly wonky, each of the bars in the chart represents a decile, 1-10, 11-20, 21-30 and so on).

As the second chart (below) indicates, the media pollsters are more skeptical than the campaign pollsters. The average score for media pollsters is a 43 (with a median of 29). In other words, the media pollsters cluster near the "so unreliable they should be ignored" end of the scale, while the campaign pollsters divide more evenly, with an average score of 50 and a median of 48. [For those interested in more details, click here for a spreadsheet with the number of responses for each increment clicked on the 1-100 scale].

Think of it this way: Even professional pollsters are not sure what to make of the surveys done this week. While few advise ignoring these surveys altogether, most recommend treating the results with a lot more caution than usual. Their uncertainty probably reflects the overall lack of experience our profession has with surveys conducted during the latter half of December. Just about everyone sampled recommends treating the results from this week with a lot more caution than those available at other times, although few would advise that we ignore these results completely.

We will have more results from our "Poll of Pollsters" over the next few days.

-- Mark Blumenthal

December 27, 2007 in Iowa, Pollsters, The 2008 Race

Comments

Post a comment

URL:

Comments: (you may use HTML tags for style)

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button.

Please email us to report offensive comments.

See our comment policy here. Note that we require commenters to share their email address via Typekey. We will never share your email address with anyone without your explicit permission.