Mark Blumenthal | March 10, 2010
Topics: Health Care Reform , House Effects , Measurement , Rasmussen , YouGov/Polimetrix
Our chart of the favor-or-oppose questions on health care reform has generated a fair amount of discussion this week. Both Chris Bowers and the analysts at Democracy Corps (the Democratic affiliated polling outfit) noticed a slightly tighter margin in recent weeks (support increasing and opposition decreasing), which in turn caught the attention of Jon Chait. Andrew Sullivan leaned heavily on our chart this morning in effort to refute a new Wall Street Journal op-ed by pollsters Scott Rasmussen and Doug Schoen about the "steady" nature of public opinion on health reform, which in turn drew a response from Megan McArdle taking issue with Sullivan's conclusions about which polls are "outliers" on health reform.
All of this commentary gets at two important questions: Is support for health reform growing (and opposition fading)? And do the large "house effects" among pollsters obscure our ability to see trends amidst the noise?
The version reproduced above is a snapshot of our chart as of this writing (click here to see the regularly updated, interactive version). Remember, our chart is something of a mash-up that combines different questions and surveys produced by more than 20 different pollsters. In it, we do something that many pollsters and statisticians advise against, which is to compare apples and oranges in terms of the question text and populations sampled. When we look at horse-race results for election campaigns, most pollsters use very similar questions and ultimately at least try to measure the same population (the likely electorate). In this case, the wording and format of the questions vary widely. Some sample all adults, others sample "likely voters." Look closely at the chart and you will see far more variation in the results than is typical for our horse-race charts -- between 10 and 20 points worth of variation in the favor and oppose percentages at any given time.
That variation also reflects the vague sense that many Americans have of the health care reform legislation now being debated in Congress. When the NBC/Wall Street Journal poll asks about health reform, they prompt respondents to say if they "do not have an opinion either way." As a result, roughly one-in-five adults (23% on their last survey) do not express an opinion. Other pollsters (such as ABC/Washington Post and Rasmussen) report an "unsure" percentage in the low single digits, while another (YouGov/Polimetrix) reports none at all. Thus, the degree to which pollsters push their respondents for an opinion explains some of the "house effect" variation.
All of this makes it prudent to take an apples-to-apples approach in pondering the recent trend. That's what I tried to do in the two charts that follow. I separated the lines the favor and oppose percentages into two charts to make them more legible. I also limited the plotted pollsters to the seven organizations that have updated health reform tracking over the last month. However, I also included the Pollster.com trend line from our interactive chart, which is based on all available polls, not just the seven whose dots are connected below.
1) The trend evident in the grey Pollster.com trend line -- a 4.4 percentage point drop in opposition and a 1.6 percent increase in support -- is more or less consistent with the trends shown by YouGov/Polimetrix and Rasmussen Reports, the two organizations that have polled most often on this topic during 2010. The results from PPP and IPSOS are also consistent with the same trend.
2) The results from Gallup's two polls appear to show a contradictory trend, although we should note that the Gallup changed their question wording slightly between January and March. Their most recent survey asks, "would you advise your representative in Congress to vote for or against a healthcare reform bill similar to the one proposed by President Obama?" In previous surveys, they asked about voting for "a health care bill this year" without reference to the President or either party. Note that Gallup's own analysis does not treat the January and March results as comparable.
3) Rasmussen shows a house effect on the oppose percentage (typically 5-6 point higher than our trend line; early January was an exception), but tends to be in the middle of the pack on the favor percentage. YouGov/Polimetrix shows a similar house effect on the favor percentage (typically 5-6 points higher than average), but not the oppose percentage. Whatever doubts you might have about their methods -- Rasmussen uses automated, recorded voice interviewing and YouGov/Polimetrix conducts online interviews sampled from an opt-in panel -- both are consistent in their respective questions and methods and both shown trends that generally track with those measured by other pollsters.
[Correction: The wording of the question asked on the most recent Economist/YouGov/Polimetrix survey, conducted February 28 to March 2, was slightly different from what they had asked before. Their previous surveys asked, "Overall, given what you know about them, do you support or oppose the proposed changes to the health care system being developed by Congress and the Obama Administration?" On their 2/28-3/2 survey, they dropped the reference to Congress and simply referred to the proposal as "being proposed by the Obama administration." That change could account for the spike in support to 53%].
4) We would still see a closing margin (increased support, falling opposition) if we use our charts filter tool to remove both Rasmussen and Polimetrix (as per the snapshot below). One reason may be the absence in recent weeks of surveys like Quinnipiac and NBC/Wall Street Journal (which typically report lower than average support percentages) and CNN (which typically reports higher than average oppose percentages). Notice how the range of dots is narrower over the last few weeks than in previous months. To be absolutely sure the trend is real, we will need to wait for updates from these organizations.
So yes, there are certainly large "house effects" in the health care favor-or-oppose results, but even though different pollsters gauge different levels of support, most pick up more or less the same trends, especially when they ask exactly the same questions on multiple surveys exactly the same way. Any way you slice it, there does appear to be a real tightening of opinion on health reform although as always, these results are snapshots and subject to change.
Update: Given the correction above regarding the small wording change on the most recent YouGov/Polimetrix survey, I thought it best to create two new charts that connect-the-dots for only those polls with consistent wording. So the two charts that follow drop the two Gallup surveys and the most recent YouGov/Polimetrix survey. The grey Pollster.com trend line, however, is still consistent with the line on our standard chart that is based on all surveys on this questions, regardless of wording. My bottom line remains the same: There does appear to be a small but real tightening of opinion.