Pollster.com

Articles and Analysis

 

Crowley: Survey Says

Topics: IVR , Michael Crowley , Pollsters

The New Republic's Michael Crowley has produced a nearly 4,000 word must-read on the new and changing world of polls, pollsters and polling websites in the 2008 campaign. For Pollster.com regulars, and anyone intrigued by the business of polling these days, it is truly worth reading in full but here are three paragraphs that capture some of the important conflict in the world of political survey research:

Many pollsters insist that the way to restore public confidence in their profession is to re-impose methodological standards, which treat polling as a social science rather than an amateur's pursuit. But the problem is that, despite the meticulous nature of Gallup and Pew--and the quicker, less orthodox approach of Leve and some other newcomers to the field-the establishment firms don't necessarily have all the answers. Jay Leve['s firm SurveyUSA], as it turns out, actually has a very strong track record. In March, Nate Silver ranked SurveyUSA as the most accurate of 18 major national firms, ahead of more venerable outfits like Gallup and CNN's pollster, Opinion Research.

Leve's winning track record may be the result of the technology he uses; automated polling allows him to tap into a larger pool of voters at a faster rate than live human polling--offering an instant snapshot rather than results blurred by time, as with a slow camera shutter. But Leve's technique is subject to some of the same criticisms as traditional polling. For example, some phone surveys are missing young voters with cell phones but not land lines. Overall, response rates have been declining for years. And, this season, race and gender have added tricky new variables. In short, Leve's success is hard to explain-- affirming a postmodern sense that methodology no longer ensures accuracy more than instinct and dumb luck.

The irony is that this perception, one that critics use to deride polling, is now widespread even among pollsters themselves. Last January, after all the big pollsters failed to predict Clinton's stunning New Hampshire primary victory, aapor asked them to hand over their raw data for evaluation. Leve complied immediately but says others dragged their feet. "That report has never come out," he says with a shake of his head. Mathiowetz, the former aapor president, says the report is coming soon. But, she says, a wider lack of transparency is a sign of changing times. It wasn't just the Chicago Tribune that blew the 1948 Truman-Dewey presidential election call--it was the entire polling establishment. Afterward, a panel of public opinion professionals studied what went wrong. Almost every major polling firm cooperated and submitted data about their research. The thought of this lost era clearly moves Mathiowetz: "What a lovely ..." she says, trailing off. "It just kind of brings tears to my eyes."

Crowley's piece is thorough, smart and covers the most important ground with respect to polls and how they are produced in 2008. I know it's a blogger cliche, but I mean it: go read it all.

 

Comments
Allen:

The article at TNR says "Post Date Wednesday, November 05, 2008". I love reading about the future! :-)

____________________

Gary Kilbride:

Oh come on, you didn't highlight the juicy stuff, things like the laughably defensive assertion that the term robo-calling was like using the N-word, "a street gang with a calculator," angry New York Times op-ed columns, and $1000 accuracy challenges from palm-readers.

Excellent article with plenty of anecdotes, like Alan Abramowitz bugging pollsters by email and CCing the emails all over the planet. That's great stuff, "nerdy ferocity" indeed. We need at least a weekly summary. What good is 4000 words, even if plenty of them are eight or ten syllables?

I have to say I agree with Jay Leve's basic premise. A few days ago I had a problem with cable service, requiring two phone calls hours apart. One customer service operator couldn't have been more thorough and impressive, the other barely able to comprehend my problem. Can you imagine the potential for varying results if that dynamic exists among phone polling? No doubt it does. Exit polling also. The fewer variables the better. I know I'd be more comfortable responding to a simple automated precise recorded message than the few times I've had to wade through live operators, struggling to pronounce names or place proper emphasis on the script in front of them.

Here's the problem with elections, particularly the general election: They haven't been modernized to maximize potential for frenzy and scrutiny and second guessing in regard to polling. The Founding Fathers had no way to anticipate this. Otherwise we'd have staggered elections, like one every day or every few days. Then the pollsters could scramble to poll the next race on the docket and we'd have near-instant ridicule of the most inept pollster and anointment of the king, even if he were a ROBO-CALLER, before moving to the next examination. What a waste. The NFL learned to expand the schedule from 12 to 14 to 16 and soon apparently to 18. Golf tournaments used to end on Saturdays prior to TV saturation. Our government is still stuck in pre-polling mode and it's disgusting.

BTW, I've seen this Crowley guy on TV. He's very sharp and needs to appear more often. But something about him always bugs me. I think it's that he resembles all the clean cut guys who used to be editor of the school papers, when I was writing sports. They all looked like that, with the PGA pro hair. Generally decent guys but determined to kill the occasionally controversial sports headline or story, in a fit of unnecessary precaution, so sometimes I wanted to belt them.

And sorry, but it is robo-calling. That's an easily remembered simple term, as opposed to the forced domestic engineer-type label that Leve tried to substitute. Crowley wimped out by immediately using the force fed term after quoting Leve.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR