Pollster.com

Articles and Analysis

 

A Lesson in Caveat Emptor

Topics: Daily Kos , Frank Newport , Gary Langer , Markos Moulitsas , Research2000

What is the most important lesson to be learned from the emerging Daily Kos - Research 2000 polling scandal? Two prominent pollsters, Gallup Poll Editor-in-Chief Frank Newport and ABC News Polling Director Gary Langer, both chimed in last week with a similar conclusion: More disclosure is good, but poll sponsors need to do a better job checking and verifying what they publish.

First up was Newport, who is also serving this year as president of the American Association for Public Opinion Research:

I would emphasize the ultimate responsibility which rests with the entity commissioning or releasing poll data, just as a newspaper or broadcast outlet has the ultimate responsibility for what it releases or publishes. The current controversy revolves around a client-contractor relationship between Daily Kos and Research 2000. It is unclear what procedures Daily Kos may or may not have used to verify and check the data it received from the survey research firm it employed (Research 2000) before publishing it. (Daily Kos ultimately, it says, fired the research firm). Nevertheless, in general, a news or web outlet has an obligation to check and verify what it puts out. This is often easier said than done, of course. A number of publications have been burned in recent times when outside contractors or freelance writers have not followed standard journalistic procedures.

Langer, as always, was a bit more direct:

Disclosure, then, as necessary as it is, does not in and of itself assure data quality. That takes another step: The need for those who fund and then promote or disseminate these data first to dig deeply into the bona fides of the product.

Indeed to my mind the delivery of methodological details, including original datasets, should be an initial and ongoing requirement of any polling provider, not a demand only when controversy arises. That points to a more basic lesson of this story: the principle of caveat emptor.

Polling is a complex undertaking that can be produced in many ways - some highly valid and reliable, some less so, some not in the least. Anyone buying it needs to take the trouble to ascertain precisely how it's being carried out - in sampling, questionnaire design, respondent selection, interviewing, quality control, weighting and more - and to assess the appropriateness of these methods for the intended use of the research.

Just so you don't miss the point, Langer published his comments under the headline, "Running With Scissors." Snark aside, I can't quibble with the larger argument. The first and most important step in assuring data quality and integrity rests with the organizations that sponsor it. I suspect that Markos Moulitsas agrees, despite Langer's implied argument that only the "adults" of the media world can be trusted to publish surveys.

Markos has taken his share of lumps in this controversy, but we should also give him credit for taking a stand in a way that will ultimately force every ugly detail into the public domain.** Important lessons have and will be learned that have nothing to do with politics and everything to do with how to be better data consumers, whether we are paying to conduct the polls or just reading about them.

I am glad that both Newport and Langer also underscored their longstanding commitments to better disclosure, including AAPOR's emerging Transparency Initiative. Better disclosure is a critical tool for the rest of us who do not fund polling. New media brands are emerging even faster than new polling technologies, and very few of those organizations have in-house experts who can access things like sampling, questionnaire design, respondent selection, and the rest.

Those of us who are part of "The Crowd" can do our part to help others make sense of polling methodology, but only if the designs are transparent and the underlying data available.

**A semi-related post-script: In my post on Saturday I noted the "the apparent lack of a written contract" between Daily Kos and Research 2000, based on the statement in the Daily Kos complaint that they entered into an agreement "reached orally" to conduct national polls in 2009. Markos Moulitsas subsequently emailed to say that while there was no formal "boilerplate" contract, "we hashed out our agreement via email." To be clear, a legally binding contract between two parties does not require a written document.

 

Comments
hoosier_gary:

You're letting Markos off the hook way to easily. As pointed out by Newport, it was up to Markos to verify what he was publishing. The polls may or may not have been garbage. To me the bigger point is that Markos published them anyway.

Who's to say he didn't verify them? Maybe he knew they were bad and just didn't care because they supported the position he wanted to forward.

No reasonable person could look at those polls, compare them to other polls, and not question why they were consistantly off by so much from the average of all the others.

The screams of "victim" by Markos makes me think the man doth protest too much.

____________________

Matt Sheldon:

For all the talk of the transparency initiative, it would not have mattered one single bit in this case...

Which of these did R2K NOT adhere to?

1. Who sponsored the survey, and who conducted it. CHECK

2. The exact wording of questions asked, including the text of any preceding instruction or explanation to the interviewer or respondents that might reasonably be expected to affect the response.
CHECK

3. A definition of the population under study, and a description of the sampling frame used to identify this population.
CHECK

4. A description of the sample design, giving a clear indication of the method by which the respondents were selected by the researcher, or whether the respondents were entirely self-selected.
CHECK (Sort of)

5. Sample sizes and, where appropriate, eligibility criteria, screening procedures, and response rates computed according to AAPOR Standard Definitions. At a minimum, a summary of disposition of sample cases should be provided so that response rates could be computed.
CHECK

6. A discussion of the precision of the findings, including estimates of sampling error, and a description of any weighting or estimating procedures used.
NO EXPLICIT DETAIL, BUT THE STABILITY OF THE DEMOGRAPHICS FROM WEEK TO WEEK CLEARLY POINTED TO THE WEIGHTING PROCEDURES

7. Which results are based on parts of the sample, rather than on the total sample, and the size of such parts.
CHECK

8. Method, location, and dates of data collection.
CHECK

In all likelihood, R2K met all of these criteria, so how is this a solution to this dilemma?

It strikes me that a fraudulent pollster could easily comply with all of these criteria.

...so what does this actually solve?

____________________

Mark Grebner:

@Matt Sheldon: You're exactly right. They need to make explicit one additional standard: the polling firm has to actually perform its duties in accordance with its answers to the questions. Merely answering them - which is all that's literally asked - doesn't quite complete the chain.

____________________

Immanentize:

I disagree with hoosier_gary. When someone contracts for services (Moulitsas' emails are just as righteous as parchment in contract law these days) the purchaser expects the contractor to fulfill their obligations honestly and ethically. Any example fits -- when I get surgery next month, am I really obligated to fully check out my surgeon's techniques and underlying style? Or can I rely on his previous education, reputation and professional credentials? When I had my kitchen redone -- certainly I got refererences from folks who used the contractor, but I did not check out his nail to board length ratios or his framing style choices. In short, we rely on professionals to be professional. When they are not, they stand to be sued (both surgeon and construction contractor fully understand this).

Why then does Langer give a special pass to pollsters who, it seems acording to him, can actually rely on the purchaser of the poll for verification? Poll purchasers are no usually polling experts -- that's why they contract with polling firms! To suggest the onus is on the purchaser turns contract for services on its head perhaps suggests another large problem in the polling industry -- no self-acountability or professional responsibility. After all, if the pollster just makes stuff up, is it really the person who contracted for the poll's fault for not being a polling expert who can catch the mistake?

Nice job if you can get it....

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR