Pollster.com: Poll of Pollsters: Rating the NH Pollsters

January 05, 2008

Poll of Pollsters: Rating the NH Pollsters

A week ago I posted results from our poll of pollsters on how they rated and ranked their colleagues in Iowa. Tonight, their ratings of the New Hampshire pollsters.

First, a bit of review. The results below give us a sense of the pollster's relative reputations among their colleagues. Needless to say, reputation does not necessarily correlate with accuracy. The surveys of all pollsters -- those with good reputations and bad -- are subject to survey errors, random and otherwise. Even the best pollsters are fallible.

Second a review of our own methodology: Just before Christmas, we sent email out invitations to just over a hundred pollsters. A little less than half (46) responded and completed the entire survey online. Of those, 21 are media pollsters and and 25 campaign pollsters (14 Democrats and 11 Republicans). There is no "margin of error" for these data because they represent nothing more or less than the views of the pollsters that participated. Like the respondents to any survey, we promised to keep their identities confidential.

The questions we asked were the same as for Iowa. We asked: "How reliable do you consider surveys of NEW HAMPSHIRE PRIMARY voters done by each of the following organizations, very reliable, somewhat reliable, not very reliable or not reliable at all?" We also provided an explicit "do not know enough to rate" option for each organization.

In New Hampshire, the pollsters rated most reliable are ABC News/Washington Post (72% reliable), CNN/WMUR/University of New Hampshire (65%), CBS News/New York Times (61%), the Pew Research Center (59%) and the Boston Globe/University of New Hampshire (56%). Keep in mind that the same University of New Hampshire Survey Center partners with both CNN/WMUR and the Boston Globe.

As in Iowa, the two lowest scoring pollsters are Zogby International and the American Research Group. The "not at all reliable" score is again by far the highest (54%) for Zogby.

The media pollsters are once again more positive about their colleagues work than the campaign pollsters, although both groups provide generally similar rankings.

We also asked our pollster-respondents to select the "pollsters you consider MOST and LEAST reliable in New Hampshire." Once again, a local survey organization stood out. The University of New Hampshire Survey Center (combining its partnerships with both CNN/WMUR and the Boston Globe) is the first choice of slightly less than a third (31%) of our respondents. Notice, however, that slightly more (34%) have no particular favorite in New Hampshire.

01-05 trust most.png

And finally, on the question of the least reliable pollster in New Hampshire, Zogby International once again ranked first. More than a quarter (28%) of the campaign pollsters and half (52%) of the media pollsters picked Zogby International.

01-05 trust least.png

Again, reputation is just one way to judge a survey organization. A good reputation is no guarantee of accuracy. However, as we learned in Iowa, reputation can tell us a lot.

-- Mark Blumenthal

January 05, 2008 in Pollsters, The 2008 Race

Comments

jhm:

Is there any speculation as to the reasons for such a disparity between the ranking that Pew gets from the media and the partisans?

Posted on January 6, 2008 1:41 PM

____________________

pm:

I was unaware that UNH uses different methodologies depending on its media partner. Does TV give Andy Smith more resources to do a better job than does his print media partner? Or are the 10 or so respondents who gave different ratings evaluating the media outlet rather than the pollster?

Posted on January 6, 2008 1:54 PM

____________________

csa:

It is interesting what pollsters think of each other, but wouldn't a comparison of their last polls before past elections and the actual election outcomes be a much more meaniful measure of their abilities?

Posted on January 6, 2008 1:58 PM

____________________

Mark Blumenthal:

jhm: Notice that just about everyone on the list gets lower ratings from the campaign pollsters. Keep in mind that the media pollsters are rating themselves.

pm: Sorry if I confused. As I understand it, UNH uses the same methodology with both partners and the trial-heat numbers are directly comparable between WMUR/CNN and Boston Globe polls. The questionnaires are different.

csa: We did just that (look at past accuracy) in Iowa(here and here), and will have more of the same, probably tomorrow, for NH. Charles Franklin was working on something but had flights cancel yesterday and thus has been out of pocket all weekend.

Posted on January 6, 2008 2:06 PM

____________________

Barry Rubinowitz:

Take a look at the difference between the two groups on Mason-Dixon: +47 from media, -8 from campaigns! Oddly, none of the media pollsters considered them the most reliable while 2 of the campaign respondents did. Given the relative importance of accurate polling data to the two groups, I would tend to respect the campaign viewpoint more here.

Posted on January 6, 2008 2:52 PM

____________________

NuisanceMan:

Zogby gets the lowest rating even though it very frequently outperforms the other polls (right on the money in Iowa, for example). The reason is that Zogby refuses to tow the bipartisan center-right Washington consensus line, not the accuracy of its polls. The above result is just one more indication of how far the inside-the-Beltway crowd are out touch with the country. Accurate polls just aren't acceptable if the pollster provides analyses that lack the false, mandatory tropes that spin everything to the right. Especially unacceptable is the fact that Zogby blasphemously dared to hint that there might have been something wrong with the 2004 vote count. Of course, there was, but Blumenthal, typical of the Democratic party establishment although not of rank-and-file Democrats, has earned lots of brownie points through his determined campaign to obfuscate this fact.

Posted on January 6, 2008 5:09 PM

____________________

Steve:

Where is Gallup in this list?? Am I missing something here.

Posted on January 6, 2008 7:46 PM

____________________

je:

Regarding least reliable, the text and the table are inconsistent. The text says "More than a quarter (28%) of the campaign pollsters and half (52%) of the media pollsters picked Zogby International" but the table says the opposite. The table is the correct version, isn't it?

Posted on January 7, 2008 10:55 AM

____________________

Mark Blumenthal:

je: No, the text was right, I mislabeled the table. Apologies for that. I is corrected now (though you may need to click Shift-reload so your browser displays the new version).

Steve: Yes, we left off Gallup. Unfortunately they did not field their first NH survey until the day after our survey went live, but I probably should have anticipated they would be active (since they have been in the past). That oversight was entirely mine - apologies for it.

This project has been (and continues to be) a learning experience. The critiques and comments have been helpful.

A question for those interested enough to have read this far: Would you like to see a similar survey about national and other statewide polls? If so, any thoughts on what we might do differently?

Posted on January 7, 2008 12:43 PM

____________________

Greg A:

Not surprised Rasmussen is in negative numbers. His polls this past year have been exceedingly suspicious....Through the summer his were the only polls that showed a lead for Fred Thompson. I'm thinking "cooked" here.

Posted on January 7, 2008 1:11 PM

____________________

Jeff Winchell:

Mark, to answer your question about whether to have more survey's about polling reliability on other states, or the national polls, here's what I'd like to see discussed

I'd like to see NO national polls until we get past at least SC and NV.

Frankly, I think the national polls have been misleading, or rather the national media have misled the nation by relying too much on them to projecting inevitability of the big brand names (i.e. Clinton, Giuliani, Thompson) when it should have been far more obvious that the driving factors were going to be the early contests.

Now, of course, since those early contests have much different dynamics than the up till now useless national polls, we have a different "event" - the suddent surprise of the rapid rise of Obama, the fall of Romney, rising of McCain and Huckabee, which in turn affects the early state polling. This event would never have occurred had the media and population had payed more attention to the early state polls or there had been no national polls at all.

Posted on January 7, 2008 3:29 PM

____________________

SteveM:

I would like to see similar data on national and statewide polls. One thing I would like to see in addition to what's above is some sort of context to help interpret the results. For example, in the above, Zogby is clearly singled out as unreliable, but why?
Perhaps an open-ended question regarding reasoning for extreme rankings could be included, and some of the responses reported in your results. This is the first election that I've paid close attention to, and I just feel that maybe I'm missing something obvious. Also maybe the above suggestions would help to catch other information useful even to poll-watching veterans.

Posted on January 9, 2008 10:50 AM

____________________

steve:

I am amazed at how out of date much of the US polling blogosphere is when a major campaign is underway. Even some of your sidebar's 'POLL BLOGS AND SITES' lists sites that are not currently updating regularly.

Also could someone explain for non US readers which are the polls you have found to be most accurate in past elections and which pollsters are tied to which campaign teams.

Posted on January 10, 2008 6:45 AM

____________________

steve:

John Zogby doesn't seem to be much help with his explanation of what went wrong with the NH polling.

http://www.huffingtonpost.com/john-zogby/polling-the-new-hampshire_b_80657.html

Posted on January 10, 2008 7:31 AM

____________________