Pollster.com

Articles and Analysis

 

A Red Herring

Topics: Disclosure , Divergent Polls , Measurement

Mark Blumenthal is right to argue in his recent National Journal article ("The Problem With Polling Cap-and-Trade,") that "politicians should proceed with caution when trying to anticipate public opinion on a complex policy issue." In my opinion, however, he goes too far when he implicitly expresses approval (or perhaps tolerance) for such polling, on the grounds that pollsters are just probing latent public opinion.

As Blumenthal notes, CNN and Pew each conducted polls on this issue, with CNN finding 97 percent of the public expressing an opinion, while Pew reported 89 percent with an opinion. Also, CNN found a 23-point margin in favor of cap-and-trade legislation, Pew an 11-point margin. Blumenthal emailed George Bishop, author of  The Illusion of Public Opinion, asking "What should we make of such findings?"

Bishop responded by writing: "Reliable and valid measures of public opinion on such a complex policy issue cannot be so simply simulated by merely telling respondents what it's about and then asking them to react to it on the spot. Down that road lie misleading illusions and the manufacturing of public opinion - a disservice to the Congress, the president and the press that covers them."

Blumenthal takes issue with Bishop's response, arguing that "policy makers have good reason to want to probe the sorts of opinions that the venerable political scientist V.O. Key once termed latent, those likely to be stirred up should the legislation become law or the focus of a future election campaign. Pollsters can attempt to simulate such hypothetical attitudes in a telephone survey, but the results will be very sensitive to the words they use and, more importantly, to the assumptions they make about the competing arguments voters may eventually hear."(italics added)

So far, I agree with everything Blumenthal says. Note especially the words in italics, which emphasize both that pollsters can "attempt" to simulate hypothetical attitudes, and that such results are sensitive to question wording.

But then Blumenthal argues that "It is possible to anticipate how public opinion on a topic like cap-and-trade may evolve, but proceed with caution." (italics added) Unfortunately, the assertion in italics lacks supporting evidence. In fact, the efforts to predict public opinion on complex issues are more likely to generate wildly conflicting polling results, such as polls on support and opposition for a bailout, or the "card check" (EFCA) bill. The reason: As Blumenthal cautions - because polling results in such situations are highly dependent on question wording and context.

More distressing, however, is Blumenthal's advice to "proceed" - even with caution - which implies 1) that pollsters recognize they are measuring "latent" opinion and 2) they are willing to admit as much to the public. But that is not the case.

It's important to remember that "latent" opinion is not "current" opinion. It is hypothetical, speculative, inconclusive - i.e., it is a concept about what opinion might emerge, depending on the varied ways in which the issue is covered in the press. It would be disingenuous for pollsters to (as Blumenthal says above) "attempt to simulate such hypothetical attitudes" using only one method (question wording) and then declare that the results represent current public opinion. Yet, that is exactly what media pollsters do.

Pollsters do not say they are "simulating hypothetical attitudes." Instead, recognizing that many people may not know anything about the issue (which Pew in fact acknowledges), pollsters feed their respondents information, used forced-choice questions to extract an immediate reaction, and then announce the results as though they represent what the public is currently thinking.

CNN announced its hypothetical results by saying - "CNN Poll: 6 in 10 back 'cap and trade'." Pew was no less assertive in presenting its hypothetical results, saying that "the survey finds more support than opposition for a policy to set limits on carbon emissions." Its sub-headline was also quite definite: "Modest Support for 'Cap and Trade' Policy." In no place did either polling organization admit that these results were based on "simulating hypothetical attitudes."

It's important to keep in mind that once respondents are fed information in the context of a survey, they no longer represent the larger population from which they were drawn - because the rest of the public has not been given the exact same information at the exact same time. Objective questions can often be asked of sample respondents without fatally tainting the sample, but giving respondents extra information about an issue cannot avoid contamination. Such a process inevitably means that, at best, pollsters are trying to simulate what public opinion might emerge. But they don't present their results with such a warning label.

So, it seems to be a red herring to justify polls on complex issues, such as 'cap and trade', by suggesting that policy makers may want to probe "latent" public opinion. Yes, perhaps they do. But that's not what the media pollsters admit to doing.

As Blumenthal writes in a previous post, "If subtle changes in wording can produce such different results, then we can assume that many respondents are forming opinions on the spot rather than sharing pre-existing views on the actual legislation. 'Public opinion' in this sense isn't so much 'fluid' (a favorite pollster cliché) as non-existent."

Yes, indeed. Pollsters converting non-existent public opinion into the appearance of current public opinion, exactly the "manufacturing" charge that Bishop makes.

If the media pollsters want to simulate hypothetical public opinion, they should clearly label it as such, instead of presenting their results as reflective of actual (existing) public opinion. Until pollsters are more candid about the nature of their polls, I've got to side with Bishop in characterizing this kind of polling as "manufacturing opinion," which produces "misleading illusions" about the public and is ultimately a "disservice" to the Congress, the president, the press and the people.

 

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR