Mark Blumenthal | April 30, 2009
An update on Monday's post that linked to a ABC News polling director Gary Langer's recent column criticizing polls based on "opt-in" internet panel surveys that report a margin of error. "You need a probability sample to compute sampling error," he wrote. Opt-in panels that claim sampling error are "trying to nose their way" into "the house" of probability sampling and "don't belong there."
Simon Jackman, professor of political science at Stanford University, responded with a blog post that takes a different perspective:
Observation: all survey respondents “opt-in”. Would-be respondents (selected via random sampling or not) decide whether to respond or not, or can’t be reached at all. We then weight the data we get to try to deal with any resulting biases. The resulting standard errors should be computed taking the weighting into account (in almost all media polling I see, they are not, and the standard error is computed a la Stats 101 with the number of completed interviews in the denominator), but in any event, even the correct standard errors are conditional on the way the weights were computed. The Stats 101 “textbook purity” of “simple random sampling” has long been left behind…particularly given some of the horror stories you hear about [random digit dial] RDD [telephone survey] response rates.
So I tend to think the “you can’t trust opt-in Internet polls” line is something of a beat-up. Sure, there is work to be done in understanding the properties of data generated this way, and how to compute a standard error with these data. I don’t see this as an impossible hill to climb. It is critical that this work get done, because if/when we can get comfortable with the bias issues (and we know what the issues are), then I think its game over.
Jackman, we should point out, was a principal investigator for the Cooperative Campaign Analysis Project (CCAP), an academic survey conducted using the opt-in internet panel maintained by YouGov/Polimetrix (the company that also owns Pollster.com). Nevertheless, this conversation is important since, as Jackman points out, "internet polling is not going away" and just about every pollster I know concedes that we need to find a way to "solve the problem" of random sampling over the internet.