Pollster.com

Articles and Analysis

 

A New York Times Op-Ed...and an Eplogue

Topics: 2008 , Disclosure , John Zogby , NY Times , SurveyUSA

I am honored to report that I have op-ed column in today's New York Times on a subject that will be familiar to regular readers: The need for better disclosure of methodological details by pollsters. I hope you will go read it all.

As it happens, John Zogby provides an epilogue on his hugely inaccurate California survey in his post Super Tuesday commentary:

About California: Some of you may have noticed our pre-election polling differed from the actual results. It appears that we underestimated Hispanic turnout and overestimated the importance of younger Hispanic voters. We also overestimated turnout among African-American voters. Those of you who have been following our work know that we have gotten 13 out of 17 races right this year, and so many others over the years. This does happen.

So now he tells us. Although, if you notice, he is still not ready to disclose the racial and ethnic composition of his California survey. By how much, exactly, did they "underestimate" HIspanic turnout and "overestimate" the contribution of younger Hispanics and African Americans? He did not make these details available on his survey release on Tuesday (at least not to non-subscribers), and is apparently not making them available now. SurveyUSA, Field, McClatchy/Mason Dixon, and Suffolk University did report demographic composition details. That ought to tell us something.

Incidentally, it is also worth noting that while the results of the final SurveyUSA poll nailed the final ten-point Clinton margin, and the "sturdy" 13-point Obama lead forecast by Zogby never materialized, in Missouri the roles were reversed. In the final hours, SurveyUSA showed Clinton leading by eleven <s>nine</s> points (54% to 43%), while Zogby gave Obama a slight advantage (45% to 42%). Obama won 49% of the actual vote to 48% for Clinton.

The lesson?: Better disclosure puts us in a better position to understand and interpret the data, but all pollsters are fallible and all polls are subject to error (random and otherwise).

 

Comments
margaret:

Concerning Zogby, I found a page from his website that seems to say he uses '02 directories, only listed telephone nubmers. Am I reading this incorrectly? It says "copyright 2008" at the bottom so it's current. Phone numbers six years old? No unlisted #'s? Please someone else read this and comment.

http://64.233.169.104/search?q=cache:tBsB-qgPOKAJ:www.zogby.com/methodology/index.cfm+zogby+sampling&hl=en&ct=clnk&cd=3&gl=us&ie=UTF-8

____________________

gomathi:

I read your Op-Ed and loved it....here was my letter to the NYT... just in case they don't publish 9i've just had a letter published so it may not make it in).

To the Editor:

I loved Mark Blumenthal's piece on survey methods (Feb 7th). As a relative newcomer to politics but with a background in data analysis, I have been watching the misreading of those polls with incredulity. Clearly the media does not have a strong handle on interpretation of survey results. Mr. Blumenthal's site at pollster.com contained a nice analysis of the surveys -- while everyone was talking about a potential win for Sen. Obama in CA and MA, pollster.com clearly showed that considering the large gap between him and Sen. Clinton for a few weeks, Sen. Obama would come close but not cross over by Feb 5. It also clearly showed the "Kennedy effect". Comments from the media puzzled me -- had they not seen the data? So, unlike the rest of my family and friends I was prepared for Tuesday night � not one of the results surprised me! Thanks Mr. Blumenthal!

____________________

FlyOnTheWall:

Mark:

Thanks, as always, for your continuing crusade for better disclosure. And thanks as well for continuing to wade through the increasingly partisan (and off-topic) comments sections of your posts, and for actually responding to the questions we pose.

I look forward to your post later today on Gallup. I'm continuing to scratch my head over that one. I have trouble seeing how even a mildly-flawed likely voter screen could produce such an off-target result - particularly because it's a sudden and dramatic shift in a very large sample. Any chance you can persuade Gallup to release their demographic model? They're supposed to be the gold standard, after all. I find myself wondering about the ethnic and gender composition of their rolling samples. Does it vary day by day? Is it reflective of those we've seen voting so far this year?

____________________

BDM:

I to am puzzled by Gallup's tracking poll showing Clinton at 52 AND Obama at 39

Today Rasmussen's tracking poll shows a far different result:

Obama 44
Clinton 44

What gives?

____________________

Patrick Gilfillan :

I agree that a definitive statement of methodology should be required. The NYTimes and every major outlet should require one and vet those methods with experts before agreeing to print reference to them as a matter of journalistic ethics. YOu can't just print a number somebody says and call it news, it should be confirmed/vetted as having a valid scientific basis, like any other news story. Maybe they already do that. But even having said that, I was amazed by the results, especially since Survey USA nailed it and Zogby (which has been so generally reliable) didn't. That past reliability of Zogby (as well as Florida 2000) indicate another factor might be involved: irregularities at the voting site/or with ballots. CA had problems with confusion about the ballot that are currently be investigated, with the possibility that crossover voters were not being counted by the computer because they didn't pop a particular bubble on the ballot. This problem would seem to result in a substantial undercounting of votes for Obama. The S of State has agreed to sample one percent of the ballots to determine the extent of the problem, though I don't we will ever hear of this issue again. Guess my point is that sometimes the inaccuracy of a poll that is usually reliable might also indicate a polling problem, like 2000 Fla.

____________________

richard pollara:

The problem is when the news media reports a poll on election day it has the feeling that they are hinting at the outcome of the race. MSNBC used Tom Brokaw, their most credible broadcaster, to announce it on Tuesday afternoon. He said, "If these numbers are correct it is very bad news for Hillary Clinton." My immediate reaction, even though I had seen the SurveyUSA poll, was that the race was over. I think everyone watching TV on election day thinks that the networks have some sort of inside info that they are not disclosing until after the polls close. To use Brokaw to comment on Zogby made me believe their exit polls showed an Obama romp. It is reminiscent of the networks predicting a winner before the polls close. I think both Zogby and MSNBC owe us a greater duty of care when reporting so close to the finish line.

____________________

Tony Rizko:

Rasmussen is a Republican company. look it up

proud member of the RNC

____________________

Matt:

Mark,
I like to remind you that Missouri's election outcome may have been greatly influenced by treacherous road conditions in the midwest due to bad weather - especially in the rural towns where Clinton was polling well. Urban cities where Obama was polling better were not significantly impacted by the weather as much as throughout the rest of the state.

____________________

Rick :

There are a lot of candidates for worst polling of the year and not all of them are commercial pollsters or self styled hacks. how about the University of Minnesota/University of Connecticut being at 33% for Obama and him putting up 67!! They can caveat all they want, but wow. Or UConn reporting 25% undecided in the late January Democratic Primary poll in CT. The vaunted academic pollsters have to come down from the tower and make sure they are doing this right..

____________________

Kudos to Mr. Blumenthal for calling for more transparency in polling methodology. When I was political writer for The (Louisville) Courier-Journal, I had this policy on reporting polls: I had to see the complete survey instrument, up to and including the last question for which a result was provided (this gave me the voter screen; I was willing to keep intervening questions confidential if I thought they had no impact on the results), and the pollster had to certify the results and submit to on-the-record questioning about methodology, including callback routine, location of callers, etc. Reporters shouldn't have to squeeze such information out of pollsters and campaigns; it should be offered in good faith with the voters.

____________________

Daniel T:

Mark:

As you know from my comments, we share a desire for better transparency. I am glad to see the NYT at least gave the issue some coverage and your were a good choice to write that op-ed.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR