January 25, 2007
Sampling Error - What It Isn't
Back to basics for a moment, to a point we cannot make often enough. Yes, polls report a "margin of error," which derives from the random variation that comes from drawing a sample rather than interviewing the full population. But that is all it is. The "margin of error" statistic is not an overall measure of quality and can tell you nothing about the potential for other sources of error, including as a bias from missing certain voters (such as those who lack a landline telephone or who hang up when called) or from asking questions in a way that produces misleading results.
A friend sent this take on sampling error from Good Math, Bad Math, a blog by a computer scientist:
People frequently believe that the margin of error is a measure of the quality of a statistic - that is, that a well-designed poll will have a smaller margin of error than a poorly-designed poll. It doesn't - the MoE only represents sampling errors! A great poll with a sample size of 100 will virtually always have a considerably larger MoE than a terrible poll with a sample size of 1000. If you want to know the quality of a poll, you need to know more information it than just the margin of error; If you want to gauge the relative quality of two different polls, you need to know more than just the margin of error. In either case, you really need to know the sample size, how the sample was collected, and most importantly exactly what they measure.
It's worth a click.
By Mark Blumenthal on January 25, 2007 1:43 PM | Permalink
Comments
The margin of error just gives you a confidence interval which follows from the standard error of measurement. I thought that the standard error was only relevant under the assumption that the sample was taken without bias (IID). Otherwise, the MSE is the VAR + Bias and need not converge to zero as the sample size increases.
On my lecture module for survey sampling within my course page for teaching research methodology, I have a little section on random sampling error (to which the MoE applies) vs. non-random sources of variation between polls.
Near the bottom of the linked page, look for where it says in bold red font, "Why do different polls on the same topic, taken at the same time, not always agree?"
"Margin of Error" speaks of precision, not accuracy.
MoE:"Good poll"::precision:accuracy
Posted on January 25, 2007 3:29 PM