Pollster.com

Articles and Analysis

 

Daily Kos: We Were 'Defrauded' by Research 2000

Topics: AAPOR , AAPOR Transparency Initiative , Daily Kos , Disclosure , Markos Moulitsas , Research2000

Daily Kos founder Markos Moulitsas today rocked the polling world by posting an analysis that he says shows "quite convincingly" that the national surveys conducted for his website by Research 2000 since early 2009 were "largely bunk." Just three weeks ago, Moulitsas fired Research 2000 on the basis of low accuracy scores tabulated by Nate Silver. Today, on the basis of the work of "statistics wizards" Mark Grebner, Michael Weissman, and Jonathan Weissman, Moulitsas announced that Daily Kos "will be filing suit" against its former pollster "within the next day or two."

The core of his extraordinary explanation is worth reading in full:

We contracted with Research 2000 to conduct polling and to provide us with the results of their surveys. Based on the report of the statisticians, it's clear that we did not get what we paid for. We were defrauded by Research 2000, and while we don't know if some or all of the data was fabricated or manipulated beyond recognition, we know we can't trust it. Meanwhile, Research 2000 has refused to offer any explanation. Early in this process, I asked for and they offered to provide us with their raw data for independent analysis -- which could potentially exculpate them. That was two weeks ago, and despite repeated promises to provide us that data, Research 2000 ultimately refused to do so. At one point, they claimed they couldn't deliver them because their computers were down and they had to work out of a Kinkos office. Research 2000 was delivered a copy of the report early Monday morning, and though they quickly responded and promised a full response, once again the authors of the report heard nothing more.

While the investigation didn't look at all of Research 2000 polling conducted for us, fact is I no longer have any confidence in any of it, and neither should anyone else. I ask that all poll tracking sites remove any Research 2000 polls commissioned by us from their databases. I hereby renounce any post we've written based exclusively on Research 2000 polling.

Separately, Charles Franklin will soon post some thoughts on the evidence presented by Grebner, Weissman and Weissman, but for the moment let's consider the issues of disclosure raised.

First, there is some further history. Two years ago, when the American Association for Public Opinion Research (AAPOR) launched an investigation into the polling miscues during the 2008 presidential primary elections, it asked 21 firms to provide response rate information. Research 2000 could not calculate specific response rates for the calls they made. The estimates they provided were "about 1 complete of every eight attempts" in New Hampshire and "about" 1 in 9 in Wisconsin. They were also unable to provide "a full set of dispositions" (a tally of how many calls resulted in completed interviews, no answers, refusals to be interviewed, and so on).

Their incomplete answer did not result in a formal censure, since Research 2000 claimed they were sharing whatever information they had. Another half dozen or so pollsters also claimed they failed to keep an accounting necessary to enable such calculations. But whether a failure of disclosure or quality control, the inability to provide such a basic metric speaks to the importance of demanding greater disclosure as a matter of routine. That's why AAPOR's Disclosure Initiative is so important. If you have not yet read my column from last month on this subject yet, I hope you will. It is highly relevant to this story (h/t to Nirml for making the same point on Twitter).

Second, the most damning information for the layperson about Research 2000 provided in today's announcement -- and is certainly most troubling to me -- is their apparent reluctance to share raw data with their own client. AAPOR's Disclosure Initiative will not mandate the release of raw, respondent level data, but it is worth considering that some of the most respected media pollsters already make their raw data available to scholars by routinely depositing it the Roper Center archives. The Pew Research Center makes their raw data available to the general public through its own web site.

As this story broke, NBC's Chuck Todd noted the "stain" that irresponsible pollsters are leaving on the community of credible pollsters. Many are asking what the polling profession can do. Whatever one might conclude about Research 2000, there are two clear answers: Full support for and participation in AAPOR's Transparency Initiative and a greater willingness to deposit raw data to the Roper Archives.

 

Comments
Farleftandproud:

It is bad enough to realize that the scary far right wack jobs are ahead on the polls but to find out that Research 2000 gave us a glimmer of hope that the public's opinion wasn't quite as f'd up as I thought, was troubling. It is not right for any pollster to give people false hope, but I do want Daily Kos to find a new polling web site or create their own.

____________________

Aaron_in_TX:

The unwillingness to show the data to their own client is indeed perplexing. Why would they do that?

____________________

scottkj:

Does it take a rocket scientist to figure out their polling numbers were bunk. I think KOS saw the writing on the wall last week and is saving face.

____________________

Field Marshal:

Hopefully Daily Kos will see the light and hire Rasmussen as their pollster. Man, that would be great to see the silly libies on these boards have a coronary.

____________________

Niikeb:

Honestly it was obvious that R2K was making things up or seriously putting their fingers on the scales. Many people have called it out here that their numbers did not pass sanity check.

R2K really sold liberals a bale of goods, ironically if Kos did not come forward, they could have carried this on all year.

____________________

hoosier_gary:

I never believed those polls and very few people did either. KOS knew that too. They stayed with this pollster because they liked the results that they used to push their political agenda. Now that they have been caught using phony polls, they are screaming that they were defrauded. No one believes that either. KOS got what they wanted and now they want to pretend that they were innocent victims.

____________________

Field Marshal:

Kos and Research 2000 were the one's who did all that polling about what the GOP voters believe- like where Obama was born, whether ACORN stole the election, if Obama should be impeached, if Obama is a socialist, etc. that Daily Kos and the crazy nuts who frequent the site along with much of the mainstream liberal media used to demonize the GOP. Now we see it was all lies. SHOCKING!

____________________

Matt Sheldon:

I think it is obvious why R2K is not sharing the raw data with Markos...

HE IS SUING THEM.

They will have to release it at some point, but the moment the lawyers get involved you are best served to deny the other side any means of advantage.

My guess is that Markos did his part to alienate R2K early on in this process.

The raw data will be turned over in discovery. By them, R2K will have responded to every charge with reams of analysis.

...or they will turn over nothing and plead guilty.

My guess is that R2K has a treasure trove of embarrassing e-mails from Markos pushing for an engineered result or pushing for a favorable methodology.

Markos dropped the polling once R2K could no long prop Obama's approval rating above 50%.

Is it any surprise?

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR