Articles and Analysis


Young and Amic: Polling on fuzzy issues like healthcare reform- You can't measure what doesn't exist

Topics: health care , Health Care Reform , Question wording

Cliff Young is Senior Vice President at Ipsos Public Affairs. Cliff is head of the Public Sector practice and responsible for the Ipsos McClatchy poll. Aaron Amic is Vice President at Ipsos Public Affairs and is responsible for analytics for the Ipsos McClatchy Poll.

When the definitive history of the 2009 healthcare reform debate is written, one footnote will read how varied, even contradictory, the polls had been. We see this now. Indeed, on any given day, different people can cite different polls and come to very different conclusions. "Americans are in favor of healthcare reform-no, wait, they are against it!"

It goes without saying that given this uncertainty, cherry picking of polls has been rife on both the right and the left. Democrats prefer to cite polls on the "public option" which has consistently shown strong majority support. Republicans, on the other hand, point to polls on general support for healthcare reform-most showing only a plurality.

At a methodological level, pollsters have been grappling with this dilemma as well. The original debate centered around the variability of question wording and its effect on levels of support. The overriding question was-what is the ideal healthcare question, if even such a thing exists?

More recently, the debate has shifted to explaining the differences between generic healthcare questions and more specific ones referring to the "public option". The controversy lies in the differential levels of support-generic questions have shown only plurality support, while specific questions referring to the "public option", show majority support. The consensus explanation is that the healthcare debate is quite distant from people's day-to-day lives and so their answers are "uninformed"-in methodological speak, a classic case of "non-attitudes."

Both lines of reasoning have their merit. However, we believe that they miss the mark because they assume that polling on healthcare reform is analogous to polling on presidential elections. In our opinion, it isn't.

Indeed, in presidential elections, our job as pollsters is made easy with ballot questions being basically fixed after the primaries. Simply put, we know which candidates will be running. This, in turn, all but defines our ballot question for us.

In contrast, issues like healthcare reform are quite fuzzy as no bill typically exists at the beginning of the process. This makes the construction of a single question impossible if not simply disingenuous.

Put another way, we have no "true value" to measure against- no concrete bill exists (or at least did not exist until recently). You can't measure what doesn't exist!

The problem is most apparent when looking at generic questions on healthcare[1]. Such questions are broadly worded and lack any concrete anchor. People, consequently, can (and do) read into them what they want, making their meaning variable. To illustrate our point let's look at table 1 below.


The above question shows that only a plurality (34%) of Americans support healthcare reform (or at least the proposals in Congress). Simple Conclusion: Americans do not support healthcare reform.


However, a simple follow up question shows that about a quarter (25%) of those that oppose the reform bills actually think the proposals "do not go far enough" (see table 2 above)! This same 25% actually is much more likely to be Democrat and more likely to support the public option. People, once again, read into the question what they want.

In contrast, questions which refer to "the public option" and other specific policy measures can introduce greater certainty into the ballot question, helping to establish a clear reference point for people (See table 3 below). However, once again, such questions are nothing more than hypothetical as we do not know a priori which items will (and will not) be included.


So what are our takeaways here? What does polling on American healthcare reform teach us about polling on non-electoral policy issues involving the legislative process?

First, polling on healthcare reform is quite different than polling on presidential elections because our "true value" is not fixed. This makes the construction of singlequestions impossible and misleading. Such issues are, well, fuzzy and, therefore, only a multiple indicators approach will tell the entire story-some generic, some specific questions. Here triangulation is key.

Second, genericquestions should be used with caution. At the least, they should include a follow up question in order to determine why people favor or oppose healthcare reform. We only included such a follow up after struggling with interpreting the results.

Are such generic questions valid at all? We think they are but with caveats.

Indeed, before the final bill, such questions seem to be nothing more than a measure of optimism about the reform process, much like "right track, wrong track" questions. Looking forward to a final bill, we do expect that such generic questions will become relevant. Only then will they have a "true value" to be measured against.

Third, questions which reference specifics like the "public option" are hypothetical and have to be understood as such. Indeed, without a final bill, they should be used more for sensitivity analysis than anything predictive-which policy measures garner more support, which ones less so. While such questions say nothing about "general support for healthcare reform," they do help us understand which measures are more (and less) likely to be in the final bill as politicians read polls too.

To this end, we have tracked specific items for most of the healthcare debate. Here we understood that healthcare reform would be fundamentally a debate about the role of government (or lack thereof). All of our items fall along a government intervention continuum. In our experience, polling on "fuzzy" issues places a premium on understanding the underlying value cleavages related to the policy debate at hand. At its essence, healthcare reform is a debate about the proper role of government.

Fourth, from an analytical perspective, the combination of generic and specific (hypothetical) questions makes total sense. They allow us to be both predictive as well as diagnostic with our clients but only make sense when used together.

Fifth, from a media polling perspective, the combination of general and specific ballot questions is much less tidy than a single "up or down" measure and, thus, more complicated to explain. Looking forward to future non-electoral legislative reform debates, we, as an industry, need to do better in explaining these complexities.

[1] Examples of some questions recently fielded:

Ipsos wording: As of right now, do you favor or oppose the healthcare reform proposals presently being discussed?

ABC wording: Overall, given what you know about them, would you say you support or oppose the proposed changes to the health care system being developed by Congress and the Obama administration?

AP-GFK wording: In general, do you support, oppose or neither support nor oppose the health care reform plans being discussed in Congress?

PEW wording: As of right now, do you generally favor or generally oppose the health care proposals being discussed in Congress?

CBS wording: Do you mostly support or mostly oppose the changes to the health care system proposed by Barack Obama, or don't you know enough about them yet to say?