Mark Blumenthal | February 15, 2008
Carl Bialik, author of "The Numbers Guy" column for the Wall Street Journal, takes a balanced look today at the pitfalls of something we do here at Pollster, "mashing up surveys from various sources this election year to produce composite numbers meant to smooth out aberrant results." His piece is worth reading in full, as it considers both the benefits and risks of creating composite trends or averages:
Stirring disparate pollsters in one pot has its critics. "That's dangerous." says Michael Traugott, professor at the University of Michigan, and author of a recent guide to election polls. "I don't believe in this technique."
Among the pitfalls: Polls have different sample sizes, yet in the composite, those with more respondents are weighted the same. They are fielded at different times, some before respondents have absorbed the results from other states' primaries. They cover different populations, especially during primaries when turnout is traditionally lower. It's expensive to reach the target number of likely voters, so some pollsters apply looser screens. Also, pollsters apply different weights to adjust for voters they've missed. And wording of questions can differ, which makes it especially tricky to count undecided voters. Even identifying these differences isn't easy, as some of the included polls aren't adequately footnoted.
Bialik quotes both me and Charles Franklin in the column, but here are a few additional thoughts. We do not consider the trend estimates to stand as worthy replacements to the data from individual surveys. The trend lines -- and the estimates derived from their end-points -- are best considered as tools to help make sense of the barrage of often conflicting results from individual surveys. We learned in 2006 that "mashing up" surveys and "smoothing out" the variation between them helps counter the instinct to overreact to variation between individual polls -- some of it clearly aberrant -- that is common in hotly competitive political races. Moreover, while we only plot a few summary measures here such as vote preference and job approval, many of the surveys we report and link to include a wide variety of questions that help illuminate many aspects of public opinion.
Bialik is correct to argue that benefits of averaging lessen when we start to see large and consistent "house effects" separating the results from different pollsters. If a few polls are providing good estimates, while many other polls have misleading results, the mashed up averages may reflect more of the bad than the good. I wrote as much just before the Iowa Caucuses. Bialik correctly notes that the averages were misleading in California, where most polls showed the Clinton-Obama race closer than it turned out to be. His suggestion that we could "bolster" the case for trend estimates or averaging by comparing those numbers "directly against those from individual polling firms in terms of election accuracy" is a good one and something we are working on.
Bialik adds some additional detail in a companion blog item that focuses, among other things, on my calls for greater disclosure of methodological details, which includes a response of sorts from Zogby International:
When I asked Zogby spokesman Fritz Wenzel for further details, such as what those flawed estimates were, and passed along a blog post from Mr. Blumenthal calling for more disclosure from the firm, Mr. Wenzel dismissed sites like Pollster.com as “rivals.” “We are satisfied that we have identified the problem in California,” Mr. Wenzel wrote in an email, “and giving our rivals more ammo in the form of methodological detail, some of which is proprietary, with which to criticize us further doesn’t make the world a better place.”
Bialik is asking his readers comment on the value of composite poll numbers and whether better disclosure would "make the world a better place. Your comments are welcome here or there (or both!).