Pollster.com

Articles and Analysis

 

Sampling Error - What It Isn't

Topics: Sampling

Back to basics for a moment, to a point we cannot make often enough. Yes, polls report a "margin of error," which derives from the random variation that comes from drawing a sample rather than interviewing the full population. But that is all it is. The "margin of error" statistic is not an overall measure of quality and can tell you nothing about the potential for other sources of error, including as a bias from missing certain voters (such as those who lack a landline telephone or who hang up when called) or from asking questions in a way that produces misleading results.

A friend sent this take on sampling error from Good Math, Bad Math, a blog by a computer scientist:

People frequently believe that the margin of error is a measure of the quality of a statistic - that is, that a well-designed poll will have a smaller margin of error than a poorly-designed poll. It doesn't - the MoE only represents sampling errors! A great poll with a sample size of 100 will virtually always have a considerably larger MoE than a terrible poll with a sample size of 1000. If you want to know the quality of a poll, you need to know more information it than just the margin of error; If you want to gauge the relative quality of two different polls, you need to know more than just the margin of error. In either case, you really need to know the sample size, how the sample was collected, and most importantly exactly what they measure.

It's worth a click.

 

Comments
Furrier:

"Margin of Error" speaks of precision, not accuracy.

MoE:"Good poll"::precision:accuracy

____________________

The margin of error just gives you a confidence interval which follows from the standard error of measurement. I thought that the standard error was only relevant under the assumption that the sample was taken without bias (IID). Otherwise, the MSE is the VAR + Bias and need not converge to zero as the sample size increases.

____________________

Alan Reifman:

On my lecture module for survey sampling within my course page for teaching research methodology, I have a little section on random sampling error (to which the MoE applies) vs. non-random sources of variation between polls.

Near the bottom of the linked page, look for where it says in bold red font, "Why do different polls on the same topic, taken at the same time, not always agree?"

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR