Mark Blumenthal | April 5, 2010
Topics: AAPOR , Disclosure , NCPP
My column today concludes with the argument that news media outlets, including Pollster.com, need to do a better job holding pollsters to the minimal disclosure standards set by organizations like the National Council on Public Polls (NCPP). What follows are some thoughts about how we plan to do better on that score here at Pollster.com.
One challenge we have been confronted with in recent months is what to do about polls released by organizations that are either newly formed or that have not previously released surveys on the campaigns we track. We saw that happen in the special election for U.S. Senate in Massachusetts, and given the emergence of vendors offering to conduct automated surveys for less than a thousand dollars, we will likely see much more over the next six months.
So as a first step, starting today, when we encounter polls from an new organization (or an organization that is new to us), we are going to require that their publicly accessible reports meet all of NCPP's minimal (Level 1) disclosure requirements before including their results in our charts and tables:
Level 1 Disclosure: All reports of survey findings issued for public release by a member organization will include the following information:
- Sponsorship of the survey
- Fieldwork provider (if applicable)
- Dates of interviewing
- Sampling method employed (for example, random-digit dialed telephone sample, list-based telephone sample, area probability sample, probability mail sample, other probability sample, opt-in internet panel, non-probability convenience sample, use of any oversampling)
- Population that was sampled (for example, general population; registered voters; likely voters; or any specific population group defined by gender, race, age, occupation or any other characteristic)
- Size of the sample that serves as the primary basis of the survey report
- Size and description of the subsample, if the survey report relies primarily on less than the total sample
- Margin of sampling error (if a probability sample)
- Survey mode (for example, telephone/interviewer, telephone/automated, mail, internet, fax, e-mail)
- Complete wording and ordering of questions mentioned in or upon which the release is based
- Percentage results of all questions reported
Member organizations reporting results will endeavor to have print and broadcast media include the above items in their news stories.
Note that the last sentence provides something a loophole: Disclosure of the specified information is required in "all reports of survey findings issued for public release," but not necessarily in newspaper and television stories based on those reports. Since virtually every pollster or sponsoring news organization now maintains some sort of web site, we will interpret the rule to mean that while news stories may not disclose all of this detail, the pollster needs to make a more complete report available somewhere on the web.
Discerning readers will immediately see some big shortcomings in this first step. Let's consider the most obvious:
1) It's not fair. Many polls that Pollster.com currently publishes fall short of meeting NCPP's minimal disclosure guidelines.
True. Exhibit A, as reported in today's column, is Insider Advantage, a pollster that almost never discloses their survey mode in their public reports. But we don't have to stop there. Other items on the NCPP list that many pollsters frequently neglect to disclose include the sampling method (or "frame"), the fieldwork provider and -- all too often -- the complete wording and ordering of survey questions.
However, given how little the NCPP code requires, these are shortcomings that pollsters can easily correct, going forward. The sample mode, sample frame and fieldwork provider can be specified in just a sentence or two. And how hard is it to complete text and order of survey questions in the form of a PDF on web site?
To address the inconsistency of applying this rule to some pollsters but not others, I pledge a second step: Over the next month or so, we will examine all of the polls published in Pollster.com charts over the last year to determine more precisely how many pollsters are falling short on the NCPP standards. We will report those findings here and, at that point, consider whether any pollsters merit a "delisting."
2) That's a weak standard. Shouldn't pollsters disclose more about their work?
Absolutely. I am certainly on record asking pollsters to disclose much more, especially with respect to party identification, and the demographics and mechanics of "likely voter" samples. Back in August, I called for a system of scoring the quality of disclosure based on much more than the NCPP Level 1 information.
Also, the American Association for Public Opinion Research (AAPOR), is currently in the process of revising their own disclosure guidelines. Their proposed minimal disclosure standards mandate a few things that NCPP's standards do not, including "a description of the variables used in any weighting or estimating procedures" and the name of the supplier that provided the survey sample.
So I'll pledge two additional steps: First, in examining the polls we have published over the last year, we will also look at whether pollsters are meeting AAPOR's minimal standards and consider whether to require that polls meet both the AAPOR and NCPP minimal standards.
Second, we will gather whatever methodological details pollsters have published, including those listed in NCPP's Level 2 and Level 3 disclosure and the items that AAPOR's proposed code asks pollsters to make available after 30 days.
Again, my ultimate goal is to move toward all of this information to score the quality of disclosure of public polls. The steps described above will move us in that direction.
3) But disclosure isn't quality. A pollster could tell you everything you want to know about a crappy poll, and it would still be a crappy poll.
Unfortunately, that's mostly true. There is probably some correlation between a pollster's ability to answer basic questions about their methodology and the quality of their work. It is hard to have much confidence in a pollster that will not describe their sampling frame or weighting variables or that cannot release a disposition report on the numbers dialed.
But I won't quarrel with the basic point: Disclosure is not quality. The unfortunate problem is that pollsters have a very hard time agreeing among themselves about what defines a quality poll. If we want to make judgments about survey quality, full disclosure is a necessary prerequisite. When a survey's methodology is a mystery, it is much harder to conclude much of anything about its quality.
So we'll start by asking newcomer pollsters to meet the NCPP minimal standards, but that's just a start.