Mark Blumenthal | January 27, 2009
Topics: Divergent Polls , Economic stimulus , Measurement , Professor M
Last week our colleague David Moore highlighted some contradictory results on the issue of the proposed economic stimulus program to illustrate, as he put it, "how easy it is for pollsters to manipulate public opinion into something different from what it really is." He pointed out that a question asked by Gallup produced just 11% with no opinion, while a similar question posed by the NBC/Wall Street Journal poll produced 30% in that category when it prompted respondents to say when they "do not have an opinion either way."
David is nothing if not provocative, and he comes to his opinions from a long career in the field of survey research. For those reasons we are glad he has become a regular contributor. I also agree with him that pollsters too rarely report probes of overall awareness of specific proposals, or conversely, the reluctance of respondents to offer an opinion when asked questions about complex public policy issues. More often than not, issue questions on media polls measure the way the public reacts to the information presented rather than a pre-existing opinion on the proposal being discussed. That habit, as Moore argues, tends to exaggerate our sense of the public's engagement in matters of public policy. But I part company with David when he implies a nefarious intent, arguing that pollsters "are generally not interested in realistic measures of public opinion," and choose instead to deliberately "create" or "manipulate" public opinion.
By way of explanation, consider all of the questions asked over the last few weeks that attempt to gauge support for the proposed economic stimulus legislation. I have ranked the table below on the percentage expressing support (all are from January, follow the links for details on specific dates, sample sizes and reported margins of error).
|NBC/Wall Street Journal: Do you think that the recently proposed economic stimulus legislation is a good idea or a bad idea? If you do not have an opinion either way, please just say so.||43% good idea||27% bad idea||30% no opinion/ unsure|
|Rasmussen Reports (automated): Do you favor or oppose the economic recovery package proposed by Barack Obama?||45% favor||34% oppose||21% unsure|
|Gallup: Do you favor or oppose Congress passing a new $775-billion dollar economic stimulus program as soon as possible after Barack Obama takes office?||53% favor||36% oppose||11% unsure|
|Hotline/Diageo (asked of half sample): In general, do you support or oppose Congress and the President passing an $825 billion economic stimulus plan to jumpstart the economy, even if it means increasing the federal deficit in order to do so?||54% favor||34% oppose||12% unsure|
|IPSOS/McClatchy: With the current stimulus package being considered, the total cost of the economic stimulus could total one trillion dollars. Is spending this amount of money definitely necessary, probably necessary, probably not necessary or definitely not necessary?||55% necessary||41% necessary||4% unsure|
|NBC/Wall Street Journal (follow-up to NBC question above): When it comes to the economic stimulus plan proposed by the Obama administration, which of these two statements comes closer to your point of view? Statement A: The economic stimulus plan is a good idea because it will help make the recession shorter, get people back to work, and provide money for transportation, education, and Medicaid programs. Statement B: The economic stimulus plan is a bad idea because it will do little to shorten the recession, the jobs are temporary, and it will significantly increase the deficit.||57% good idea||36% bad idea||7% unsure|
|CNN/ORC: Would you favor or oppose a proposal to attempt to stimulate the economy by increasing federal government spending on construction projects and economic assistance to some Americans by about eight hundred billion dollars?||58% favor||40% oppose||1% unsure|
|IPSOS/McClatchy : Do you think that an economic stimulus package is necessary to improve the current state of the economy, or not?||62% necessary||32% not||6% unsure|
|Hotline/Diageo (asked of half sample): As proposed, the $825 billion economic stimulus plan would include $550 billion in new spending for alternative energy technology, roads and bridges, state governments and local school districts, and increasing benefits for the unemployed. The remaining part of the stimulus package contains $275 billion in tax cuts and credits for individuals and for business to help generate more jobs. In general, do you support or oppose Congress and the President passing this $825 billion economic stimulus plan to jumpstart the economy, even if it means increasing the federal deficit in order to do so?||67% favor||27% oppose||6% unsure|
|ABC News/Washington Post - Would you support or oppose new federal spending of about 800 billion dollars on tax cuts, construction projects, energy, education, and health care to try to stimulate the economy?||70% support||27% oppose||3% unsure|
|CNN/ORC (follow-up to CNN question above): And if that economic stimulus plan also included tax cuts for individuals and businesses, would you favor or oppose that proposal?||71% favor||28% oppose||1% unsure|
When you compare the results this way, a few obvious patterns emerge:
1) When asked whether they favor or oppose an "economic stimulus" plan that would cost $800 billion or so (give or take a hundred million), Americans generally express support in the mid-50-percent range.
2) When pollsters also provide an explicit "do not have an opinion" option, as NBC/Wall Street Journal and (presumably) Rasmussen do, support falls to the mid-40 percent range and (on the NBC/WSJ poll at least) opposition also falls proportionately.
3) When the questions provide more information on how the $800 billion (or so) will be spent, usually specifying a combination of tax cuts and transportation, education and energy projects," support grows to mid-60 low-70 percent range.
4) Only one question -- again from NBC/Wall Street Journal -- poses explicit arguments for and against the proposal, and it produces a slightly higher level of support (57%) than the first category of questions that mention only the overall price tag and omits a specific prompt for "no opinion."
5) Every question shows net support for the proposal.
David Moore argues, in effect, that only the second category of question provides a "realistic" measure of public opinion. There we disagree. Yes, it is important to understand that many Americans lack a specific opinion the "economic stimulus" legislation per se, something stressed by too few pollsters. Still, that finding is just one part of "public opinion" on this issue. Reactions to new information are also important, as are the underlying values driving responses to all of the questions reproduced above.
Think about it this way. Suppose you had the chance (and the time and sheer stamina) to sit down and ask a random sample of Americans, one at a time, to share their opinions about what the government in Washington should be doing to "stimulate" the economy. A substantial number, though a minority, would have strong awareness of the details of the plans being debated and would offer highly informed, strongly held opinions. A smaller number, I'm guessing, would offer little more than a blank stare, having no idea what plans are being debated and would express no real opinion about what the government should do even when provided some vague details on the options being considered.
The vast majority, however, would fall somewhere in between. The would offer some opinion about what the government should do, even if they know little about the "economic stimulus" proposal now being debated in Washington. We know (from questions on the recent ABC/Washington Post or the Pew Research Center surveys) that at least three out of four Americans want their government to do something to strengthen the economy. Some are ready to trust (or distrust) whatever the new Obama administration is proposing, on the basis of their trust (or distrust) of our new president alone. Others will have general attitudes about government spending, taxes or specific priorities they want addressed or about other players in the debate.
And the important point is that all of this defines "public opinion" on the issue of economic stimulus. Poll analysts and journalists get in trouble when we try to boil all of public opinion on any legislative proposal down to a a single question, when we try to pick the most "correct" question when their wording differs (see the comments from my old friend Professor M here and here) or when we forget that survey questions sometimes tap both pre-existing opinions and reactions to new information.
PS: To be clear, the polls linked to above asked many questions on the economy and economic stimulus, not just the one or two from each I included above. The Polling Report also has a helpful compilation.