Pollster.com

Articles and Analysis

 

Minimal Disclosure and Pollster.com

Topics: AAPOR , Disclosure , NCPP

My column today concludes with the argument that news media outlets, including Pollster.com, need to do a better job holding pollsters to the minimal disclosure standards set by organizations like the National Council on Public Polls (NCPP). What follows are some thoughts about how we plan to do better on that score here at Pollster.com.

One challenge we have been confronted with in recent months is what to do about polls released by organizations that are either newly formed or that have not previously released surveys on the campaigns we track. We saw that happen in the special election for U.S. Senate in Massachusetts, and given the emergence of vendors offering to conduct automated surveys for less than a thousand dollars, we will likely see much more over the next six months.

So as a first step, starting today, when we encounter polls from an new organization (or an organization that is new to us), we are going to require that their publicly accessible reports meet all of NCPP's minimal (Level 1) disclosure requirements before including their results in our charts and tables:

Level 1 Disclosure: All reports of survey findings issued for public release by a member organization will include the following information:

  • Sponsorship of the survey
  • Fieldwork provider (if applicable)
  • Dates of interviewing
  • Sampling method employed (for example, random-digit dialed telephone sample, list-based telephone sample, area probability sample, probability mail sample, other probability sample, opt-in internet panel, non-probability convenience sample, use of any oversampling)
  • Population that was sampled (for example, general population; registered voters; likely voters; or any specific population group defined by gender, race, age, occupation or any other characteristic)
  • Size of the sample that serves as the primary basis of the survey report
  • Size and description of the subsample, if the survey report relies primarily on less than the total sample
  • Margin of sampling error (if a probability sample)
  • Survey mode (for example, telephone/interviewer, telephone/automated, mail, internet, fax, e-mail)
  • Complete wording and ordering of questions mentioned in or upon which the release is based
  • Percentage results of all questions reported

Member organizations reporting results will endeavor to have print and broadcast media include the above items in their news stories.

Note that the last sentence provides something a loophole: Disclosure of the specified information is required in "all reports of survey findings issued for public release," but not necessarily in newspaper and television stories based on those reports. Since virtually every pollster or sponsoring news organization now maintains some sort of web site, we will interpret the rule to mean that while news stories may not disclose all of this detail, the pollster needs to make a more complete report available somewhere on the web.

Discerning readers will immediately see some big shortcomings in this first step. Let's consider the most obvious:

1) It's not fair. Many polls that Pollster.com currently publishes fall short of meeting NCPP's minimal disclosure guidelines.

True. Exhibit A, as reported in today's column, is Insider Advantage, a pollster that almost never discloses their survey mode in their public reports. But we don't have to stop there. Other items on the NCPP list that many pollsters frequently neglect to disclose include the sampling method (or "frame"), the fieldwork provider and -- all too often -- the complete wording and ordering of survey questions.

However, given how little the NCPP code requires, these are shortcomings that pollsters can easily correct, going forward. The sample mode, sample frame and fieldwork provider can be specified in just a sentence or two. And how hard is it to complete text and order of survey questions in the form of a PDF on web site?

To address the inconsistency of applying this rule to some pollsters but not others, I pledge a second step: Over the next month or so, we will examine all of the polls published in Pollster.com charts over the last year to determine more precisely how many pollsters are falling short on the NCPP standards. We will report those findings here and, at that point, consider whether any pollsters merit a "delisting."

2) That's a weak standard. Shouldn't pollsters disclose more about their work?

Absolutely. I am certainly on record asking pollsters to disclose much more, especially with respect to party identification, and the demographics and mechanics of "likely voter" samples. Back in August, I called for a system of scoring the quality of disclosure based on much more than the NCPP Level 1 information.

Also, the American Association for Public Opinion Research (AAPOR), is currently in the process of revising their own disclosure guidelines. Their proposed minimal disclosure standards mandate a few things that NCPP's standards do not, including "a description of the variables used in any weighting or estimating procedures" and the name of the supplier that provided the survey sample.

So I'll pledge two additional steps: First, in examining the polls we have published over the last year, we will also look at whether pollsters are meeting AAPOR's minimal standards and consider whether to require that polls meet both the AAPOR and NCPP minimal standards.

Second, we will gather whatever methodological details pollsters have published, including those listed in NCPP's Level 2 and Level 3 disclosure and the items that AAPOR's proposed code asks pollsters to make available after 30 days.

Again, my ultimate goal is to move toward all of this information to score the quality of disclosure of public polls. The steps described above will move us in that direction.

3) But disclosure isn't quality. A pollster could tell you everything you want to know about a crappy poll, and it would still be a crappy poll.

Unfortunately, that's mostly true. There is probably some correlation between a pollster's ability to answer basic questions about their methodology and the quality of their work. It is hard to have much confidence in a pollster that will not describe their sampling frame or weighting variables or that cannot release a disposition report on the numbers dialed.

But I won't quarrel with the basic point: Disclosure is not quality. The unfortunate problem is that pollsters have a very hard time agreeing among themselves about what defines a quality poll. If we want to make judgments about survey quality, full disclosure is a necessary prerequisite. When a survey's methodology is a mystery, it is much harder to conclude much of anything about its quality.

So we'll start by asking newcomer pollsters to meet the NCPP minimal standards, but that's just a start.

 

Comments
Evans Witt:

Mark,

Thank you for highlighting the NCPP standards and for using the minimal disclosures list as a starting point for any poll that enters the public debate through pollster.com.

To expand on your third point, a pollster who provides these facts at least provides the basics for judging the value of a poll. Disclosure, indeed, does not guarantee quality, but it does provide the basis for a reasoned judgment of quality. To paraphrase your post, with adequate disclosure, the readers of pollster.com should be able to judge when a poll is a crappy poll.

Evans Witt
President, NCPP

____________________

sfcpoll:

This is a fantastic step for pollster.com - I appreciate the amount of additional work this will take for you and Emily, but it is a tremendous service to readers of your site.
Bravo!

____________________

nimh:

I'm just another reader, so it's not like I have any leverage or anything, but I'd strongly disrecommend leaving some polls out of your tables altogether. I use Pollster.com as a reliable overview of all the known polling conducted - the comprehensiveness is one of its main selling points, where other sites sometimes seem to pick and choose. It's also an easy place to refer people to when making an argument about a polling trend - here you can find all the polls, see for yourself if I am selectively reporting one or the other.

I am, however, enthusiastic about your drive to promote fuller disclosure. As you do have some leverage, being the single most reputable source for polling analysis out there, it's commendable that you are willing to use it for this goal.

But why not simply mark polls in your tables by their standards of disclosure? You could still list every poll out there in your tables, so we can make our own selection and judgement in reporting what's out there; but create two tiers of polls, wherein you clearly mark the second tier polls, conducted by new pollsters that fail basic disclosure criteria. Asterisk them with a footnote explaining what sets them apart.

That's just the tables. It's easy to understand why you would want to filter new, unknown pollsters with dubious disclosure standards from your charts, to avoid possible rotten apples from distorting your trendlines. So why not leave the second tier out of your charts, but include them in the tables, with an asterisk/footnote setting them apart and explaining why they are not calculated into the chart?

____________________

ahb:

I applaud you on this decision. Not only will it require polling organizations to be more transparent, it will also reward those that are and penalize those that are not. This reason alone is why I disagree with the proposal that "nimh" requests.

I remember the post you made awhile back about transparency and one of the comments suggested that such a requirement will give everyone a hammer with which to beat on the more transparent survey organizations. While true, you do earn respect & integrity, for which there is no substitute as a pollster.

____________________

William Ockham:

Why just newcomers? Shouldn't you at least note which big-name pollsters meet these minimal disclosure practices? At the least, you could make this an option in your interactive charts, just the way you do with IVR vs. live polling. Also, the phrase "the pollster.com average" has a certain brand credibility to it. Should you include pollsters who don't measure up in that average?

____________________

Robert Cicippio:

In a world spinning with bluster, bull and babble, I come to pollster.com for solid information. Your effort to block orchestrated attempts at misinformation is the reason I will return to polster.com. Keep up the good work.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR