Pollster.com

Articles and Analysis

 

On Removing Research 2000 Polls From Our Charts

Topics: Charts , Daily Kos , Del Ali , Markos Moulitsas , pollster.com , Research2000

When Markos ("Kos") Moulitsas published the analysis last week that convinced him that the polls produced by Research 2000 were "likely bunk" and announced plans to sue his former pollster for fraud, he also made an unusual request:

I ask that all poll tracking sites remove any Research 2000 polls commissioned by us from their databases.

Given the still unexplained patterns in the results uncovered by Grebner, Weissman and Weissman, and the even more troubling response late last week by Research 2000 President Del Ali (discussed here), we have chosen to honor Kos' request, as least as it pertains to the active charts on Pollster.com that we continue to update (such as favorable ratings and vote preference questions for upcoming elections). As of this writing, we have removed the Daily Kos/Research 2000 results from the national Obama favorable rating and national right direction/wrong track charts. The rest should be removed from active charts by close of business today.

We have left in place, at least for now, Research 2000 poll results in active charts sponsored by others organizations, although we will also remove those if they so request. We may also revisit this decision as further developments warrant.

Finally, we will leave in place the results from prior elections as, for better or worse, we consider our final estimates (and the results upon which they were based) to be part of the public record. That said, we will likely follow Brendan Nyhan's lead and add a footnote about the controversy to our charts from 2008 and 2009 that include Daily Kos/Research 2000 data.

 

Comments
Michael Weissman:

Hi Mark- With regard to the old databases, it might be possible to use the male-female even-odd pattern as a screening tool. (Some other time-series tools just aren't usable on singleton polls.) Of course, the peculiar parity-match pattern doesn't prove very much by itself, but in the absence of other quick checks it's a way to screen any polls which have male-female crosstabs for at least several questions.

____________________

Farleftandproud:

Obama has lost white voters, but I don't think they are all ctr/right folks. I can't say I would approve of him on a bad day and times like these in many ways seem worse than when Bush was president. I have lost hope in him, because I thought he was going to bring change to this country, yet it seems like he can't pass any legislation on time and our country's laws are becoming more extreme than they were under Bush. Texas wants to re-criminalize homosexuality, California lowers state worker's wages on state workers instead of raising income tax on the wealthiest 10 percent? Most of all you see Arizona passing an unconstitional law that encourages racial profiling. Did I ever believe that things would actually get worse under Obama? He can be a good president, but he has to have more executive orders on energy reform, and more legislation through reconciliation. He has to show some initiative.

____________________

Mark Grebner:

Michael Weissman's suggestion seems sensible to me. It should be very sensitive to the particular manipulation that was endemic to the data released to Kos; whether R2K engaged in other kinds of misconduct (which would not cause such a pattern) is unknown.

I am too lazy to do the work myself, but I would be very interested if someone searched polling archives to see how widely the gender-parity-matching phenomenon is spread through R2K's previous work. When did it appear? Was it present in a particular type of poll, or for particular clients?

____________________

Mark Blumenthal:

Michael, Mark:

Unless I'm missing something in what you're suggesting, the key limitation is that other sponsoring organizations rarely, if ever, released crosstab data as extensive as what Daily Kos ran routinely. In that sense, Kos was far more transparent.

____________________

Michael Weissman:

Mark- You're right. It's not a general purpose tool. I just raised it for the issue you brought up about what to do with other R2K polls in the database. It could serve as a sort of stopgap proxy indicator.

____________________

tom brady:

Mark,

I understand your reluctance to address the real issue here, but for the sake of your credibility, you really need to do this: at what point, if ever, did you realize that Research2000's polling was consistently an outlier? I understand why Markos ignored the obvious bias in the results, since the bias supported his ideological predispositions. But we expect better from you. Other nonpartisan sites, like Dickinson's Presidential Power blog, (see http://blogs.middlebury.edu/presidentialpower/)picked up on the bias in the Research 2000 polls very early. Anyone with a modicum of knowledge of polling understood that Research 2000 consistently leaned left. At some point Markos should have recognized this, but he ignored the obvious signs. When are you going to address this issue? What, in your view, explains why Research 2000 consistently leaned left in its results? And how much blame should we place on Markos for ignoring the obvious tilt in the results - or do you think the results indicate that Markos had his hand on the scale?

____________________

Mark W:

Hey Tom,
I don't want to speak for Mark B, but I might try to defend him and the rest of Pollster.com a little bit. I see three distinct ideas in play here: incorrect results (a pollster consistently getting outcomes wrong), bad results (those based on fraudulent or sloppy methods), and "biased" results (ie those results that may skew inaccurately in one ideological direction or another). To me, either of the first two are grounds for dumping a pollster but to the best of my knowledge (and correct me if I'm wrong), R2K doesn't have a record that's particularly poor compared to its peers. As for potentially fraudulent data, it took a whistle blower (Markos) and some fairly nuanced mathematical analysis to uncover what now appears to be highly questionable work (at the very least). To me, Pollster.com's job is not to sit around constantly sniffing for phonies.

Finally, what we all call "bias" (the ideological kind) is almost always a mathematical artifact of the pollster's own personal weighting scheme. It's his or her individual (and proprietary) take on what the ideological makeup of the surveyed demographic is. Obviously not everyone has the same view of what those weights should be, and to me Pollster.com has no business defining what an "acceptable" slant should be. Obviously this would be a concern if these pollsters were consistent outliers AND pulling the overall trends in one direction, but I don't think this is backed up by the data, especially considering that many reputable pollsters are left in the trends who are often accused of skewing "right" (eg Rasmussen). Mark and co have addressed all this before (they regularly address the pollster bias topic) and I think its grossly unfair to suggest that they somehow should have seen R2K's demise based solely on having a left slant.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR