Mark Blumenthal | December 29, 2007
Topics: 2008 , Iowa , Pollsters , The 2008 Race
The main impetus for our "Poll of Pollsters" (from which I posted the first results on Thursday) is the desire to find better objective criteria to help sort out pollsters. As both Charles and I review today in complementary posts (here and here), the handful of past Iowa polls do not lend themselves to easy assessments of accuracy. The lack of transparency by many public pollsters makes it difficult to fairly assess differences in their methods, although I would propose the degree of their disclosure as a surrogate measure of quality. A third possibility is to measure pollster reputation, especially among their peers. We decided to start with a survey of pollsters about the public polls now in the spotlight in Iowa and New Hampshire.
I included all the details in the previous post, but here are the highlights: We sent out invitations to just over a hundred pollsters and had 46 complete the entire survey, although a few more (49) completed the questions about the reliability of the polls in Iowa. Of those, 22 are media pollsters and and 27 campaign pollsters (16 Democrats and 11 Republicans). There is no margin of error because the results represent nothing more or less than the views of the pollsters that participated. Like any survey respondents, we promised to keep their identities confidential.
We started with a simple question asked about each of the 16 pollsters that have released public polls in Iowa: "How reliable do you consider surveys of IOWA CAUCUS goers done by each of the following organizations, very reliable, somewhat reliable, not very reliable or not reliable at all?" We also provided an option to say they "do not know enough to rate" each organization.
We left "reliability" in the eye of the beholder, but it is fair to assume that few are in a position to evaluate the performance of each organization in past Iowa caucuses. As one pollster put it (in a space provided for comments the end of the survey) that there is "no way we can know who's most reliable until we can compare their final estimates with actual vote." Instead, it is safe to assume that most based their judgements on the reputation of each organization and its methods. As you will see, the pollsters had little trouble making such judgements.
As the following table shows, the Des Moines Register "Iowa Poll" conducted by Selzer and Company easily earns the highest marks, with virtually all rating it either very (36%) or somewhat (50%) reliable. The other pollsters with the highest scores are nationally known media surveys: ABC/Washington Post, the Pew Research Center and CBS/New York Times.
The pollsters receiving the lowest scores are Zogby International, the American Research Group and Rasmussen Reports. In the case of Zogby, four out of five pollsters rated their surveys as not very (28%) or not at all reliable (52%).
Not surprisingly, the media pollsters are generally more positive than their campaign consultant colleagues (keep in mind that we invited principals at all 13 organizations that are polling in Iowa to participate). Their rankings are generally similar, although the University of Iowa and Strategic Vision rank higher among the campaign than the media pollsters.
We also asked our pollster-respondents to select from the same list the "pollsters you consider MOST and LEAST reliable in Iowa." As the table below shows, the Des Moines Register/Selzer survey easily stands out as the favorite, especially among the campaign pollsters:
As for the least reliable pollster in Iowa, once choice easily led the pack. One third (33%) of the campaign pollsters and just less than half (45%) of the media pollsters picked Zogby International:
Again, reputation cannot tell us everything we need to know about the quality of the numbers a pollster produces. Pollsters with poor reputations may conduct quality polls and even the best pollsters are fallible. However, when a pollster earns the respect of their colleagues, it should tell us something.
Up next, similar ratings of the pollsters in New Hampshire. Meanwhile, Pollster.com readers, what do these ratings tell you?
Note: I accidentally omitted InsiderAdvantage which released their Iowa poll as we were in the process of drafting this survey. I apologize for the oversight.