Mark Blumenthal | January 25, 2007
Back to basics for a moment, to a point we cannot make often enough. Yes, polls report a "margin of error," which derives from the random variation that comes from drawing a sample rather than interviewing the full population. But that is all it is. The "margin of error" statistic is not an overall measure of quality and can tell you nothing about the potential for other sources of error, including as a bias from missing certain voters (such as those who lack a landline telephone or who hang up when called) or from asking questions in a way that produces misleading results.
A friend sent this take on sampling error from Good Math, Bad Math, a blog by a computer scientist:
People frequently believe that the margin of error is a measure of the quality of a statistic - that is, that a well-designed poll will have a smaller margin of error than a poorly-designed poll. It doesn't - the MoE only represents sampling errors! A great poll with a sample size of 100 will virtually always have a considerably larger MoE than a terrible poll with a sample size of 1000. If you want to know the quality of a poll, you need to know more information it than just the margin of error; If you want to gauge the relative quality of two different polls, you need to know more than just the margin of error. In either case, you really need to know the sample size, how the sample was collected, and most importantly exactly what they measure.
It's worth a click.