Pollster.com

Articles and Analysis

 

AAPOR Announces Evaluation of NH Polls

Topics: 2008 , New Hampshire , Pollsters , The 2008 Race

My colleagues at AAPOR have just put out this release:

In the wake of the New Hampshire pre-election polls, the American Association for Public Opinion Research (AAPOR) today announced the formation of an ad hoc committee to evaluate pre-election primary poll methodology and the sponsorship of a public forum on the issue.

"Pre-election polls have a long-running record of being remarkably accurate," said AAPOR President Nancy Mathiowetz. "Sixty years ago the public opinion profession faced a crisis related to the poll predictions of the Truman-Dewey race. The way survey researchers reacted then – with a quick, public effort to identify the causes – played a key role in restoring public confidence and improving research methodology."

The work of the ad hoc committee will be twofold: (1) To review and assist in the dissemination of the evaluations currently being conducted by the individual polling organizations who were engaged in polling prior to the New Hampshire primary; and (2) to request and archive the data related to the New Hampshire primary for future scholarly research.

As has become obvious over the last few weeks, we have no shortage of theories of what happened in New Hampshire. What we lack, and what AAPOR is stepping forward to provide, is an effort to collect, archive and evaluate the relevant data and make it available through a public forum. Hopefully, this effort -- like the one in 1948 -- will provide that function.

Interests disclosed: I serve on AAPOR's Executive Council.

 

Comments
FlyOnTheWall:

Mark,

Congratulations. This can only be a result of your advocacy, and as a member of the voting public, I appreciate your stand on the issue. This sort of public inquiry is the only means for us - ordinary folk - to understand the magic that takes place inside the black boxes of most polling operations, and how in this critical case it went badly wrong. It will, I hope, go a long way to remedying problems and restoring credibility.

____________________

Gary Kilbride:

I hope they start with this baseline: It happened to be New Hampshire. A difficult state to poll. And apparently to exit poll, based on the '04 debacle.

That's my theory. In a different state the polling would have been wonderful and pollsters could have merrily pranced toward Nevada and South Carolina. Instead, you had an anticipated huge bounce that made no sense, based on Hillary's long term level of support and opposition. Here's a related and incredibly safe guess: If Hillary is the nominee and a debate or gaffe or outside event appears to shift preference dramatically toward or away from her in the fall, it will be a similar mirage. Think status quo and small shifts in '08, if Hillary is involved.

Another aspect that no one seems to be mentioning: NH is a transitional state, switching from slight red to slight blue. It's very logical for a state like that to prefer the more moderate Democrat. That would be Hillary. In states with similar shifts Democrats have been nominating and narrowly electing Salazar in Colorado and Webb in Virginia. Hardly ultra liberals.

I don't understand the dismay. Other states that historically spit out shaky polling numbers have continued to do so. Sarah Palin wasn't supposed to receive 50+% in the 3-way Alaska GOP gov primary in '06, but she did. Ralph Reed, bless his heart, wasn't expected to be crushed by double digits in the GA lieutenant gov primary in '06. Cynthia McKinney was projected to cruise in a GA '06 primary but it was close enough to require a runoff, which she eventually lost.

All I see is NH being NH, once again. High profile unpredictable still rounds out to unpredictable.

Maybe the '08 NH result will serve to point out what I've always maintained, that national polling is underrated. Bloggers and analysts are in a frenzy to denounce or ignore national polls in favor of state polls. Lousy handicapping, IMO. If Hillary had plunged in national polling prior to NH, the disbelief would be sensible. Meanwhile, I don't think there has been a single national poll with Hillary trailing Obama, yet pundits apparently were unanimous that Hillary was all but buried on the eve of NH. In my world of sports analysis you'd never get away with that type of bizarro but apparently it passed as legit on the cable programs last week.

BTW, isn't it 95% confidence level? How does that equate to Never-Never Land? Pollsters need to accept the slap and the heat and get on with it. Obviously I could never be a pollster. Hell, in Las Vegas I'm happy to be wrong only 45% of the time.:)

____________________

Kevin :

Congratulations on a great first step. Hopefully this ad hoc committee will advocate for the necessary next step: releasing online raw survey data (or at a minimum, very detailed cross tabs) within a short period of time after the survey results themselves are released (a week? two?). Promulgating such a norm -- and aficter all, promulgating norms for pollsters is what AAPOR does -- will enhance public confidence in the survey research industry and will obviate the need for forming such ad hoc committees in the wake of the next "polling fiasco." Though I suppose given the recommendations of your latest National Journal column this is preaching to the choir.

Also, who is on this ad hoc committee?

____________________

cmmcd:

Mark, this statement appears to rule out any analysis of the exit polls. Can you explain why?

____________________

My theory...

Democratic leaning Independent thought Obama was going to walk away with the win, so they voted for McCain instead.

____________________

Claus Nielsen:

Acoms razor;
either your methodology is flawed,
or the Diebold Voting Machine whit their
"Mickey Mouse" like security, is flawed

Come on; A private company in charge of,
- security certification of their own product

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR