December 15, 2006

Amy Simon: Random Digits or Lists

Today's Guest Pollster's Corner contribution comes from Amy Simon, a partner at Goodwin Simon Victoria Research.

News media and academics hold up Random Digit Dialing (RDD) sampling methodology as the gold standard for survey samples for elections. Meanwhile, many top notch political pollsters have been serving their clients well for years by instead using samples selected from the official list of registered voters (the statewide voter file), often called Registration-Based Sampling (RBS).

RDD samples are created when a computer randomly generates the last four digits of a phone number. The advantage of RDD is that everyone with a working landline phone is included in the sample - it doesn't matter if your phone service was just turned on that morning or if your number is unlisted, since the sample isn't generated from a list of actual phone numbers. An obvious disadvantage is that an RDD sample also includes business numbers, fax numbers, disconnected numbers, and even numbers that have never been connected - so the costs of administering an RDD sample are higher since the built-in inefficiencies bring down your contact rate.

An RBS sample draws a sample from a list of registered voters. The obvious advantage of using voter files for survey samples - one that has been noted for years - is that voter file studies are cheaper to administer than RDD studies. RDD surveys have to churn through not only bad numbers but also have to bear the cost of screening out the large portion of adults who are not registered voters, in order to find their real interview targets: respondents who self report as registered voters and who, after applying their own likely voter models, the pollsters define post-interview as likely voters.

With RBS surveys, when you do reach an actual person on the phone, you already know -- since you ask for them by name - that you have a real live actual registered voter on the line and therefore have a better production rate. (The cost difference between the two methods is even more significant in a primary or other low-turnout election scenario, but the debate about using RDD versus RBS samples in low, medium, and high turnout elections is another topic requiring its own separate discussion.) In states that have high quality voter history showing which registrants have actually voted in different types of elections, pollsters can use a likely voter screen to draw the sample in the first place, further ensuring that they are interviewing people most likely to vote in the kind of election they are attempting to measure.

Yet the news media and academics engaged in polling question whether RBS studies can be as accurate as RDD studies, since no voter registration list is 100% up to date, nor does any voter file include 100% of the phone numbers of voters. In fact, the phone match rate for a voter registration list is not only less than 100% but it can vary significantly across a state based on geography, with suburban areas showing a higher match rate than either urban or rural areas. So drawing an RBS sample requires special expertise in terms of controlling for this and other issues about who is potentially over-or under-represented in your sample. So why is it that so many experienced political pollsters continue to use RBS samples despite these concerns about its accuracy? We do so because we find that in many instances (though certainly not all) it is just as accurate, or even more so, than RDD studies.

In fact, some academics and media outlets have been experimenting with voter file survey samples and have found this to be the case. Several have publicly shared at least some of their findings about the ways in which the results do or do not differ when using RDD versus voter file samples. Several studies worth reviewing are by Mitofsky, Lenski and Bloom, by Gerber and Green in Public Opinion Quarterly and the online archive of Gerber and Green's work maintained by the list vendor Voter Contact Services (VCS). These studies have largely shown that RBS studies can be just as accurate and in some cases, more accurate, than RDD studies. One hypothesis offered is that samples drawn from voter registration lists by definition consist of actual voters, while RDD studies rely entirely on respondents' self-reporting about whether they are in fact registered to vote. Given the larger and larger portion of the adult American population that is not registered to vote, the potential for survey error when relying on self-reported behavior may be introducing larger error than carefully designed RBS studies contain.

In one recent example, we saw virtually no differences between the results of an RDD and an RBS study. We provide here just one example from our own work as the polling firm for Ned Lamont for U.S. Senate in Connecticut. In the course of the general election, at one point in September we simultaneously conducted both an RDD study and an RBS study. The results were dramatically in sync, with a margin of error of +/- 4.0 percent on the n=600 RBS study and a margin of error of +/- 3.5 percent on the n=800 RDD study. Considering the far higher cost of using RDD samples as compared to RBS samples, these results certainly give weight to the common practice among political pollsters of using voter file samples instead of RDD samples in general election campaigns.

-- Guest Pollster

December 15, 2006 in Pollsters, Response Rates, The 2006 Race

TrackBack

TrackBack URL for this entry:
http://www.pollster.com/cgi-bin/mt/mt-tb.fcgi/3222.

Listed below are links to weblogs that reference Amy Simon: Random Digits or Lists:

Comments

Post a comment

URL:

Comments: (you may use HTML tags for style)

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

Please email us to report offensive comments.

See our comment policy here. Note that we require commenters to share their email address via Typekey. We will never share your email address with anyone without your explicit permission.