Pollster.com

Taylor: Do Pollsters Need a Code of Ethics to Prevent "Hired Guns Polls" Designed to Get the Answers the Clients Want?

Guest Pollster | January 28, 2009

Today's guest pollster contribution comes from Humphrey Taylor, who has served as chairman of The Harris Poll, a service of Harris Interactive, since 1994.

In the late 1970s, Louis Harris & Associates (one of the two firms that merged to form Harris Interactive in 1996) was commissioned to conduct a survey of American attitudes to allowing oil companies to drill for oil in wilderness areas that were off-limits. Unfortunately neither Lou Harris nor any of his senior colleagues knew anything about this survey until we read the results in the media -which showed strong public support for opening up wilderness areas for drilling, quite contrary to the findings of other polls by Harris and other firms.

When we looked at the questionnaire it was immediately obvious that this was a particularly egregious example of a "hired gun poll" designed to get the answers the client wanted, a survey designed to mislead rather than to inform policy makers about public opinion.

Lou Harris, to his great credit, publicly disowned the survey and said that the findings did not reflect public opinion. Soon afterwards, he received a phone call from an irate Senator Stevens of Alaska, who was apparently close to our clients, trying unsuccessfully to persuade Lou to back down. And the clients never paid the bill.

Were I a lobbyist with a strong point of view, I (or any competent pollster) could without much difficulty write a series of questions which might be technically acceptable, but which - taken together or separately - would mislead the media and many legislators into believing that public support for my client's position was much stronger than it really is. And here I'm not talking about bad samples or manipulating data to get the answers I want, only about the design of the questionnaire itself.

Over the years I have read many published surveys that fitted the description "hired gun polls." They are often easy for pollsters to spot because the results are surprising and are strikingly different from the results from other polls. Furthermore, they provide powerful arguments that can be used by those who paid for the survey to lobby government, to influence elected officials and to generate favorable publicity. However, it is much harder for most people, including policy makers and influentials, who are not themselves pollsters to recognize, and discount, those "hired gun polls." And this can be made more difficult when the funding source - which may be a company, a trade association, a public relations firm, an NGO or an advocacy group - is hidden behind a supposedly independent third party.

This would not matter if polls had no influence, but I strongly believe that polls sometimes influence the political agenda, policy makers, regulators and the way the media cover issues. At the risk of sounding pompous, I believe all pollsters whose polls are released to the media and the public have a moral obligation that they inform and do not mislead. Unfortunately, hired gun polls are designed to mislead.

In the polling community, some polling firms are seen as particularly bad practitioners of hired gun polling. Some will defend their right to ask whatever questions they and their clients want. Most major polling firms probably try to avoid hired gun work, but in an imperfect world it is not always easy to spot them early enough to prevent them.

Those who are not pollsters may be puzzled by this whole issue. They may believe that the public opinion is what it is and that as long as a representative sample is surveyed the replies will reflect public opinion. Pollsters know better. Different questions on the same issue can produce very different and apparently contradictory results, and questions asked earlier in a survey can have a big impact on questions that are asked later. It is not difficult to write questionnaires that greatly increase the number of people who give a particular response. Fortunately the reputation of our organizations can be hurt if we are seen as to be doing this, and our business may suffer. But, sadly, some firms that have a track record of doing hired gun polls are still in business.

Some pollsters' clients may be puzzled. Once in Latin America, I asked a presidential candidate why he had published polls showing him in the lead, when no other polls did so (he was soundly defeated). His immediate and refreshingly honest reply, "If I pay for a poll, I should be able to get the answers I want and am paying for." He is not unique.

To address this problem Harris Interactive has a set of guidelines and procedures that I summarize below. I would be delighted to hear that other survey firms have similar rules. And I would be thrilled if the leading polling firms could agree on a code of ethics based on similar principles.

We pollsters need to put our house in order so that we do not have to defend polling that is morally and ethically indefensible.

Hired gun polls damage the credibility and reputation of polling and pollsters and, I am glad to say, can - when spotted and criticized - be damaging to the firms that do them and to their clients. A much more serious effect is that they can mislead and misinform policy makers, opinion leaders and the media.

The leading polling organizations have a code but it is a code of disclosure, not a code of ethics. It is necessary but not sufficient. Many years ago they came together to found the National Council on Public Polls (NCPP) and agreed to a code of disclosure that describes what they must publish whenever they release a new poll (they must describe the universe, the methodology, the fieldwork dates and the relevant questions, and who commissioned the research). But complying with the NCPP code of disclosure does not do much to help the readers to recognize a hired gun poll, that is intended to mislead rather than to inform.

Maybe the time has come for pollsters to agree on a code of ethics that would inhibit them from conducting surveys that they know are designed to get the answers the clients want. This is not an argument against conducting and publishing any polls for interest groups and advocacy organizations. They fund many valuable and useful surveys. Rather it is a plea that pollsters do everything they can to ensure that such polls are sufficiently comprehensive and are fair and balanced (although those words seem to have taken on a new meaning) that they do not mislead.

Note: This document provides a summary if the ground rules that bind us and our clients who commission surveys for public release.

By Guest Pollster | January 28, 2009 1:24 PM | Permalink | Comments (2) | TrackBacks (0)