Pollster.com

Articles and Analysis

 

POQ Free Issue on Non-Response Bias

Topics: Response Rates , Robert Groves

First, let me say a quick but huge thank you to Charles Franklin for his frequent posts over the last two weeks as I took much needed holiday break and attended to two days of AAPOR meetings last week. I'm glad he kept things busy here while I lived the slacker life. I should be back to a more regular schedule this week, with much to catch up on.

While we are on the subject of the American Association for Public Opinion Research, I want to say a quick word about the organization's academic journal, Public Opinion Quarterly, which last week released a special edition on "Non-Response Bias in Household Surveys." For the non-pollsters among you, "non-response bias" is the technical term for the error that can result [especially] when response rates are low and those that respond to a survey differ from those who do not [but it can result even when response rates are high; see the comment from Joel Bloom below]. The POQ special edition includes articles and research from the most respected authorities on this subject, and best of all, the editors have made electronic access to this edition completely free.

One of the ideas that we try to stress here on Pollster is that polls are subject to all sorts of potential error not captured by the so-called "margin of error." The study of non-response may be a bit arcane to ordinary political junkies, but if the POQ Special Edition proves anything, it is that academic survey researchers have been studying it for quite a long time. Consider this summary from the introduction by Eleanor Singer, the editor of the special edition:

Concern about survey nonresponse is of course not new. Smith (2002, pp. 27-28) notes that "early research extends back to the emergence of polling in the 1930s and has been a regular feature in statistical and social science journals since the 1940s. An analysis of JSTOR statistical journals dates the first nonresponse article from 1945 and the Public Opinion Quarterly index's earliest reference is from 1948. The index of Public Opinion Quarterly contains 125 articles on this topic; a full-text search of journals covered in JSTOR finds the following number of articles, by subject area, that included the word 'nonresponse': political science-62, economics-87, sociology-146, and statistics-431

Of course, the complexity of some of the concepts presented make this edition of the journal a tough read for those without a survey background. Equations and Greek letters abound. But for the pollsters in the audience - and I know you're out there - this edition is a must read.

For those thinking about hiring a pollster or survey researcher, I'd also suggest reviewing the article abstracts and skimming enough of the articles to form some pertinent questions to the prospective pollster. If nothing else, if you ask, say, what approaches the pollster takes in "assessing non-response bias" as per Bob Groves' recommendations , and the pollster asks, "Bob who?" then you know you have a problem.

 

Comments
Joel Bloom:

Mark,
Thanks for posting this, but I have to issue one quick but important correction: non-response bias doesn't only occur when response rates are low. Some of the most interesting studies on nonresponse bias show how it can rear its ugly head even with high response rates! Best,
-- Joel

____________________

Ben Ross:

There is a considerable literature about biased sampling among ground water scientists, because sampling is almost never random. When you have to drill a well to get each sample, samples are scarce. Many potential sampling locations are impossible to reach (under buildings, lakes, etc.) and in contamination studies, you want to concentrate your sampling in the area thought to be contaminated.

It would be interesting to see if there's anything useful to be learned by comparing that literature to the polling literature.

____________________

McelveyG:

...well, that 'Public Opinion Quarterly'
summary seems to pooh-pooh the notion that non-response-bias is any significant error source in modern polling -- but no evidence is provided demonstrating that conclusion.

They do state that current response rates by major political polling companies are typically 25%.

That means 75% of their "scientific" random sample is unknown.

Guess it's time to re-write the statistical science books on the definition of a random sample ?

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR