December 31, 2007
By Mark Blumenthal
Washington Post polling director Jon Cohen reported an easily overlooked but important statistic yesterday, especially to anyone thinking about the reliability of the last round of Iowa polls. Using the Iowa tables here at pollster.com, he determined that public polls in Iowa this year have interviewed nearly 80,000 "likely caucus goer" respondents:
As a ratio of voters polled to expected turnout, this must be something of a record. (In 2004 about 120,000 people participated in the Democratic caucuses, and in 2000 about 90,000 in the GOP contest.)
And it's not just the public pollsters calling. Campaigns have been known to set up a phone bank or two to gauge opinion, solicit support and cajole voters to actually show up and spend hours caucusing in the middle of winter.
A month-and-a-half ago, already deep into the "silly season" but well before the final stretch, eight in 10 likely Democratic caucus goers and nearly six in 10 on the GOP side said they'd been called on the telephone by at least one of the campaigns. And Pew reported the pervasive use of robo-calls (though most Iowans who get such automated calls about the campaign said they usually hang up).
I can add two confirming anecdotes. The first comes from a comment left by "Randy Iowa" here at Pollster just last night:
Is there a Do Not Call list that i can get on? I have received a survey call everyday this week and at least one candidate has called everyday as well.
I emailed Randy, and sure enough, he is an Iowa voter. He says that "80%" of the calls he received were automated. Interestingly, he is also a non-affiliated voter (not registered with a party) registered independent who has never participated in a caucus (though has "voted Republican my entire life"). (By the way, the short answer to Randy is no. Pollsters and political campaigns are exempt from the federal do not call restrictions, though at least one group is trying to change that).
I wonder how many calls those identified as past caucus goers are getting? Here is one possible answer in he form of an email I received about an hour ago from a "help desk" operator at a major residential telephone company. He apparently assumed (mistakenly) that Pollster.com conducts surveys:
Subject: Please stop calling this customer
This customer is getting upwards of 20 calls a day from automated poll services, she lives in Iowa and her phone number is 563-[redacted]. Please stop calling her.
Not surprisingly, the recipient of the calls lives near Davenport Iowa.
Aside from spectacle of the sheer volume of "poll" calls, we might want to think about what all that calling is doing the the response rates the real pollsters are getting. And if pollsters are having a harder time getting voters to respond this week, are those suddenly reluctant voters skewing the results? We may never know, of course, but if nothing else, I would be very nervous were I using an automated (IVR) methodology to collect survey data in Iowa right now. More important: I wonder how many many Iowans have been ignoring their ringing phones altogether the last few days?
By Mark Blumenthal on December 31, 2007 6:03 PM
| Permalink
| Comments (7)
January 8, 2007
By Mark Blumenthal
First, let me say a quick but huge thank you to Charles
Franklin for his frequent posts over the last two weeks as I took much needed
holiday break and attended to two days of AAPOR
meetings last week. I'm glad he kept things
busy here while I lived the slacker life.
I should be back to a more regular schedule this week, with much to
catch up on.
While we are on the subject of the American Association for
Public Opinion Research, I want to say a quick word about the organization's
academic journal, Public Opinion
Quarterly, which last week released a special edition on "Non-Response Bias
in Household Surveys." For the
non-pollsters among you, "non-response bias" is the technical term for the
error that can result [especially] when response rates are low and those that respond to a
survey differ from those who do not [but it can result even when response rates are high; see the comment from Joel Bloom below].
The POQ special edition includes articles and research from the most
respected authorities on this subject, and best of all, the editors have made electronic access to this
edition completely free.
One of the ideas that we try to stress here on Pollster is
that polls are subject to all sorts of potential error not captured by the
so-called "margin of error." The study
of non-response may be a bit arcane to ordinary political junkies, but if the
POQ Special Edition proves anything, it is that academic survey researchers
have been studying it for quite a long time.
Consider this summary from the introduction by Eleanor Singer, the
editor of the special edition:
Concern about survey nonresponse is of course not new. Smith (2002, pp. 27-28) notes that "early research extends back to the emergence of polling in the 1930s and has been a regular feature in statistical and social science journals since the 1940s. An analysis of JSTOR statistical journals dates the first nonresponse article from 1945 and the Public Opinion Quarterly index's earliest reference is from 1948. The index of Public Opinion Quarterly contains 125 articles on this topic; a full-text search of journals covered in JSTOR finds the following number of articles, by subject area, that included the word 'nonresponse': political science-62, economics-87, sociology-146, and statistics-431
Of course, the complexity of some
of the concepts presented make this edition of the journal a tough read for those
without a survey background. Equations and Greek letters abound. But for the pollsters in the audience - and I
know you're out there - this edition is a must read.
For those thinking about hiring a pollster or survey researcher, I'd
also suggest reviewing the article abstracts and skimming enough of the
articles to form some pertinent questions to the prospective pollster. If nothing else, if you ask, say, what approaches the
pollster takes in "assessing non-response bias" as per Bob Groves' recommendations
, and the pollster asks, "Bob who?" then you know you have
a problem.
By Mark Blumenthal on January 8, 2007 11:59 AM
| Permalink
| Comments (3)
| TrackBacks (0)
December 15, 2006
By Guest Pollster
Today's Guest Pollster's Corner contribution comes from Amy Simon, a partner at Goodwin Simon Victoria Research.
News media and academics hold up Random Digit Dialing (RDD) sampling methodology as the gold standard for survey samples for elections. Meanwhile, many top notch political pollsters have been serving their clients well for years by instead using samples selected from the official list of registered voters (the statewide voter file), often called Registration-Based Sampling (RBS).
RDD samples are created when a computer randomly generates the last four digits of a phone number. The advantage of RDD is that everyone with a working landline phone is included in the sample - it doesn't matter if your phone service was just turned on that morning or if your number is unlisted, since the sample isn't generated from a list of actual phone numbers. An obvious disadvantage is that an RDD sample also includes business numbers, fax numbers, disconnected numbers, and even numbers that have never been connected - so the costs of administering an RDD sample are higher since the built-in inefficiencies bring down your contact rate.
An RBS sample draws a sample from a list of registered voters. The obvious advantage of using voter files for survey samples - one that has been noted for years - is that voter file studies are cheaper to administer than RDD studies. RDD surveys have to churn through not only bad numbers but also have to bear the cost of screening out the large portion of adults who are not registered voters, in order to find their real interview targets: respondents who self report as registered voters and who, after applying their own likely voter models, the pollsters define post-interview as likely voters.
With RBS surveys, when you do reach an actual person on the phone, you already know -- since you ask for them by name - that you have a real live actual registered voter on the line and therefore have a better production rate. (The cost difference between the two methods is even more significant in a primary or other low-turnout election scenario, but the debate about using RDD versus RBS samples in low, medium, and high turnout elections is another topic requiring its own separate discussion.) In states that have high quality voter history showing which registrants have actually voted in different types of elections, pollsters can use a likely voter screen to draw the sample in the first place, further ensuring that they are interviewing people most likely to vote in the kind of election they are attempting to measure.
Yet the news media and academics engaged in polling question whether RBS studies can be as accurate as RDD studies, since no voter registration list is 100% up to date, nor does any voter file include 100% of the phone numbers of voters. In fact, the phone match rate for a voter registration list is not only less than 100% but it can vary significantly across a state based on geography, with suburban areas showing a higher match rate than either urban or rural areas. So drawing an RBS sample requires special expertise in terms of controlling for this and other issues about who is potentially over-or under-represented in your sample. So why is it that so many experienced political pollsters continue to use RBS samples despite these concerns about its accuracy? We do so because we find that in many instances (though certainly not all) it is just as accurate, or even more so, than RDD studies.
In fact, some academics and media outlets have been experimenting with voter file survey samples and have found this to be the case. Several have publicly shared at least some of their findings about the ways in which the results do or do not differ when using RDD versus voter file samples. Several studies worth reviewing are by Mitofsky, Lenski and Bloom, by Gerber and Green in Public Opinion Quarterly and the online archive of Gerber and Green's work maintained by the list vendor Voter Contact Services (VCS). These studies have largely shown that RBS studies can be just as accurate and in some cases, more accurate, than RDD studies. One hypothesis offered is that samples drawn from voter registration lists by definition consist of actual voters, while RDD studies rely entirely on respondents' self-reporting about whether they are in fact registered to vote. Given the larger and larger portion of the adult American population that is not registered to vote, the potential for survey error when relying on self-reported behavior may be introducing larger error than carefully designed RBS studies contain.
In one recent example, we saw virtually no differences between the results of an RDD and an RBS study. We provide here just one example from our own work as the polling firm for Ned Lamont for U.S. Senate in Connecticut. In the course of the general election, at one point in September we simultaneously conducted both an RDD study and an RBS study. The results were dramatically in sync, with a margin of error of +/- 4.0 percent on the n=600 RBS study and a margin of error of +/- 3.5 percent on the n=800 RDD study. Considering the far higher cost of using RDD samples as compared to RBS samples, these results certainly give weight to the common practice among political pollsters of using voter file samples instead of RDD samples in general election campaigns.
By Guest Pollster on December 15, 2006 9:12 AM
| Permalink
| Comments (3)
| TrackBacks (0)
