Following-up on my post on Monday, which discussion of Nate Silver's suggestion that pollsters consider weighting on non-traditional variables, the Pew Research Center's Scott Keeter emails to point out they did a post election analysis that suggests strong potential for adding variables like income, marital status and home ownership to reduce the bias assess the effect of adding additional variables to their weighting scheme.
They found that the additional variables reduced the cell-phone-only bias substantially, though as Keeter points out, "this was not a comparison of different weighting schemes, but was instructive of what additional weighting factors might accomplish."
Logistic regression was used to estimate the probability of voting for Obama among landline voters and cell-only voters. As would be expected, the difference is sizeable; the predicted probability of voting for Obama is 16 points higher for cell-only voters than for landline voters. Adding most of the standard demographic variables used in weighting (e.g., age, sex, race, Hispanic [ancestry], education, and region) to the model (labeled the "standard model" in Table 3) reduces this difference to 11 points, a result consistent with the notion that weighting helps reduce but not eliminate the potential for non-coverage bias. Including income, marital status and home ownership in the model reduces the difference even further to 5 points. When these additional demographics are included in the model, being cell phone only is no longer a significant predictor of candidate support, as it was in the first two models.
University of Cincinnati/Institute for Policy Research Ohio Poll
5/11-20/10; 898 adults, 3.3% margin of error
888 likely voters, 3.8% margin of error
Mode: Live telephone interviews
(University of Cincinnati release)
Mason-Dixon for the Las Vegas Review-Journal
5/24-26/10; 625 registered voters, 4.0% margin of error
500 likely Republican primary voters, 4.5% margin of error
Mode: Live telephone interviews
This morning, Nate Silver flagged a pretty glaring difference between two similarly worded and structured questions asked on surveys conducted by Rasmussen Reports and CBS News at roughly the same time. In so doing, he's highlighted a critical question at the heart of an important debate about not just Rasmussen's automated polls, but about all surveys that compromise aspects of their methods: Are the respondents to these surveys skewed to the most attentive, interested Americans? Do Rasmussen's samples skew, to use Nate's phrase, to "political junkies?"
Here is his chart:
CBS News found just 11% of adults who say they are "very closely" following "news about the appointment of U.S. Solicitor General Elena Kagan to the U.S. Supreme Court" in a survey fielded May 20-24. Rasmussen found 37% who say they are "very closely" following "news stories about President Obama's nominee for the Supreme Court" in an automated survey fielded from May 24-25. The answer categories were identical.
In addition to the minor wording differences, the big potential confounding factor is that Rasmussen screened for "likely voters," while CBS interviewed all adults. Nate does some hypothetical extrapolating and speculates that likely voter model alone cannot account for all the difference.
Whether you find that speculation convincing or not, the theory that more politically interested people "self select" into automated surveys is both logical and important. GWU Political Scientist John Sides put it succinctly in a blog post last year about an automated poll by PPP:
A reasonable question, then, is whether this small self-selected sample is -- even with sample weighting -- skewed towards the kind of politically engaged citizens who are more likely to think and act as partisan[s] or ideologues.
It is difficult to answer that question definitively, especially about Rasmussen's surveys, in a way that is based on hard empirical evidence and not just informed speculation. The reason is that the difference in mode (automated or live interviewer) is typically confounded by equally significant differences in question wording (examples here and here) or use of likely voter filtering by Rasmussen but not other polls. The Kagan example is helpful because the question wording is much closer, but the likely voter confound remains.
I have longargued that Rasmussen could help resolve some of this uncertainty by being more transparent about their likely voter samples, which dominate their releases to a far greater degree than almost any other media pollster. What questions do they use to select likely voters? What percentage of adults does Rasmussen's likely voter universe represent? What is the demographic composition of their likely voter sample, by age, gender, race, income? That sort of information is withheld even from Rasmussen's subscribers.
They could also start reporting more results among both likely voters and all adults. They call everyone, and they would incur virtually zero marginal expense in keeping all voters on the phone for a few additional questions.
Back to Silver's post. He includes some extended discussion on some of the differences in methodology that might explain why political junkies would be more prone to self-select to Rasmussen's surveys than those done by CBS. I have to smile a little because I outlined the same issues in a presentation at the Netroots Nation conference last August on a panel that happened to include Silver. I've embedded the video of that presentation below. It's well worth watching if you want more details on the stark methodological differences between Rasmussen and CBS News (my presentation begins at about the 52:00 minute mark; I review much of the same material in the first part of my Can I Trust This Poll series).
Finally, I want to follow-up on two of Nate's comments. Here's the first:
I've never received a call from Rasmussen, but from anecdotal accounts, they do indeed identify themselves as being from Rasmussen and I'm sure that the respondent catches on very quickly to the fact that it's a political poll. I'd assume that someone who is in fact interested in politics are significantly more likely to complete the poll than someone who isn't.
I emailed Rasmussen's communications director and she confirmed that they do indeed identify Rasmussen Reports as the pollster at the beginning of their surveys.
But here's the catch: So does CBS. I also emailed CBS News polling director Sarah Dutton and she confirms that their scripted introduction introduces their surveys as being conducted by CBS News or (on joint projects) by CBS and the New York Times. According to Dutton, they "also tell respondents they can see the poll's results on the CBS Evening News with Katie Couric, or read them in the NY Times."
Second, he argues that CBS will "call throughout the course of the day, not just between 5 and 9, which happens to be the peak time for news programming." I checked with Dutton, and that's technically true: CBS does call throughout the day, up until 10 p.m. in the time zone of the respondent. However, they schedule most of their interviewers to work in the evenings. As such, most of their calling occurs during evening hours because, as Dutton puts it, "that's when people are at home."
More important, CBS makes at least 4 dials to each selected phone number over a period of 3 to 5 days, and they make sure to call back during both evening and daytime hours. The idea is to improve the odds of catching the full random sample at home. That said, if a pollster does not do callbacks -- and Rasmussen does not -- it's probably better to restrict their calling to the early evening because, again, that's when people are at home.
But I don't want to get bogged down in the minutiae. The question of whether automated surveys have a bias toward interested and informed respondents is big and important, especially when we move beyond horse race polling to surveys on more general topics. I'm sure Nate Silver will have more to say on it. So will we.
How would you rate the job each of the following has done in responding to the oil spill, as very good, good, poor or very poor?
BP: 24% Good/Very Good, 73% Poor/Very Poor
Pres. Obama: 43% Good/Very Good, 53% Poor/Very Poor
The federal government: 35% Good/Very Good, 60% Poor/Very Poor
Who do you think should be in charge of efforts to contain the spill and to clean up the oil already spilled -
68% BP, 28% Federal Government
Just your best guess, which comes closest to your view of the impact of this oil spill in the long run -
37% Worst environmental disaster in the U.S. in at least 100 years
35% A disaster but not the worst environmental disaster in the U.S. in the last 100 years
23% A major problem but not a disaster
3% A minor problem
Favorable / Unfavorable
Richard Blumenthal: 61 / 29
Rob Simmons: 36 / 14
Linda McMahon: 32 / 39
Peter Schiff: 15 / 6
How much have you heard or read about the controversy regarding Richard Blumenthal's statements about his military service during the Vietnam war - a lot, some, not much, or nothing at all?
62% A lot, 26% Some, 7% Not much, 4% Nothing
Does this controversy make you think more favorably of Blumenthal, less favorably of Blumenthal, or doesn't it make a difference?
4% More, 43% Less, 51% No difference
Last Thursday the Pew Research Center released an analysis drawing on their extensive ongoing investigation of the impact of the growing cell-phone only population on conventional telephone surveys. It is a must read for anyone in the polling business. You may have also seen commentary on the report on Monday by Nate Silver and Chris Bowers, but I'd like to add a few thoughts.
For those new to it, the crux of the issue is that telephone surveys have traditionally relied on samples of landline telephone numbers. Unfortunately, the explosion of cell phone usage over the last 10 years places a rapidly growing number of "cell phone only" Americans out of reach of those surveys. In pollster lingo, this is a "coverage" problem. A lack of coverage will result in statistical bias if the out-of-reach segment of the population is both large and different from the rest.
What's new in this latest Pew report is the growing evidence of just such bias. Specifically:
The growth in cell-phone only households continues unchecked. The latest estimates from the National Center for Health Statistics shows that 25% of households (and 23% of adults) have cell phone service but no landline (another 2% of households have no telephone service at all). The cell-phone-only population has doubled (from 12% of adults) in just three years.
The side-by-side comparisons by the Pew Research Center, which has interviewed respondents on both landline and cell phones since 2006, now show non-coverage bias "appearing regularly" on landline-only surveys that have been fully weighted to correct for demographic imbalances:
Of 72 questions examined [Since August 2009], 43 show differences of 0, 1 or 2 percentage points between the landline and dual frame weighted samples. Twenty-nine of the differences are 3 percentage points or more, all of which are statistically significant. Only one difference is as large as 7 points, while four others are 5 points and seven are 4 points.
The Pew Research analysts are also confident that "the bias has grown in the last four years." In 2006, they did 46 similar comparisons and found not a single difference that exceeded two percentage points.
Last but not least, the bias appears to extend to one very critical political measure for 2010 (emphasis added):
Weighted estimates from the landline sample tend to slightly underestimate support for Democratic candidates when compared with estimates from dual frame landline and cell samples in polling for the midterm congressional elections this year. The same result was seen in Pew Research Center polls throughout the 2008 presidential election. In the landline sample, Republican candidates have a 47%-to-41% margin over Democratic candidates on the 2010 generic horserace, but in the combined sample voters are evenly divided in their candidate preferences for this November (44% for each party).
Two big cautions about that last bullet. First, the 2010 generic horserace comparison is based on just one survey from March that involved a landline sample of 1,442 registered voters and combined landline-cellphone sample of 2,070 (of whom, just 191 registered voters were cell-phone-only). While it is consistent with similar comparisons in 2008 and 2010 based on far more interviews, is possible that random variation exaggerated the bias in this single measurement.
Second, although the apparent bias is consistent with what the Pew Center found on other measures in 2008, another Pew Research report found no clear evidence the the bias led to greater polling errors in statewide polling in 2008, virtually all of which was conducted over landlines only: "[T]he average candidate error for [237 polls in statewide] races was 1.9 percentage points, about the same as in 2004 (1.7 percentage points)."
But the history of largely unbiased statewide and national polling in 2008 is no guarantee of a repeat performance in 2010, particularly given the rapid increase in the cell-phone only population. While some statewide pollsters -- most notably Quinnipiac University -- are planning to interview using "dual frame" surveys that interview over both landline and cell phones this year, the vast majority of pollsters will not.
Why not? Calling by cell phone adds considerable expense and runs up against a federal law that bars pollsters from dialing a cell phone using any automated means. For live-interviewer polls, that means more time consuming hand dialing of cell phone numbers. For those using an automated method -- like SurveyUSA, Rasmussen and PPP -- the regulation is a total barrier. The automated (IVR) pollsters simply cannot interview respondents by cell phone.
It is also worth reviewing the concluding "discussion" section of the Pew Research report for their review of why "dual frame" surveys of both landline and cell phones provide "no panacea for the coverage problem." The Pew Center estimates that in addition to the 23% of adults that are "cell-phone-only," another 17% are now "cell-phone mostly" -- Americans with both kinds of phone service who say they rely on cell phones for most of their calling. A survey that relies on cell phone sample to reach just the 23% with cell phones only may still under-represent cell-phone mostlys.
Finally, thoughts on some suggestions Nate Silver made on Monday:
Another approach, in the absence of calling cellphones, is to increase the sample size that one uses. Although there's not that much difference between calling an unbiased sample of 500 respondents or 1,000 (the associated margins of error are 4.4 points and 3.1 points, respectively), these differences are magnified if one relies upon upweighting results from smaller subsamples to correct for response bias, such as those for young voters or Hispanics.
Finally, pollsters might want to consider weighting based on "non-traditional" criteria such as urban/rural status, technology usage, or perhaps even media consumption habits.
If you read that quickly, it may have sounded as if simply doing more interviews would eliminate the sort of bias that the Pew Center report describes. I don't think that's the point Nate was trying to make, but just so there's no confusion: If the sample has coverage bias, it doesn't matter whether you do 500 interviews, 1,000 interviews or 100,000. More interviews won't fix the bias.
That said, pollsters are already weighting their samples more severely than ever before to compensate for the purely demographic bias caused by the cell-phone only coverage problem. Bigger and more severe weighting makes for more random error. Pollsters should be increasing their reported "margin of error" to account for the additional "design effect" of all that extra weighting. Unfortunately, few do (but that's another story for another day).
If pollsters can find newer non-traditional weighting schemes that can correct the sort of coverage bias described in the Pew report, those schemes will likely involve weights that are, for some respondents, even bigger and more severe. So Nate is right in recommending larger sample sizes to reduce the effectively larger "margins of error" (that have absolutely nothing to do with coverage bias).
Finally, I ran his suggestion for "non-traditional" weighting by Pew Center's Leah Christian. First, Pew already weights by what is effectively an urban/rural measure: the population density of the respondent's county. Second, they are wary of weighting by technology usage or media consumption:
The main issue with weighting to technology use or media consumption is the lack of reliable national parameters. In theory, someone could produce estimates from another survey that could then be used to weight a landline survey, but the reliability of these would depend a lot on the quality of the sample from the original survey and the stability of the estimates over time. Technology use and media consumption are much more variable over time (unlike many standard demographics), and I would have many of the same concerns as weighting to party identification.
My point here is not to discourage innovation but to offer a reality check. When it comes to coverage bias, there are no easy answers. The very big changes in telephone usage are increasingly challenging our ability to obtain representative samples via telephone.
Update: Pew Research's Scott Keeter emails to point out that their 2008 post-election analysis found that adding variables like marital status, income and home ownership to a regression model reduced cell-phone-only bias in predicting support for Obama. More details here.
What is your preference for the outcome of this year's congressional elections-- a Congress controlled by Republicans or a Congress controlled by Democrats?
44% Republicans, 43% Democrats
As you may know, Barack Obama recently nominated Elena Kagan to serve as a justice on the Supreme Court. Do you strongly support, somewhat support, somewhat oppose, or strongly oppose Elena Kagan's serving as a justice on the Supreme Court, or do you not know enough about her to say?
31% Support, 21% Oppose
Would you say that immigration helps the United States more than it hurts it, OR immigration hurts the United States more than it helps it?
All adults: 43% Helps more, 45% Hurts more
Hispanics: 60% Helps more, 30% Hurts more
Now, as you may know, Arizona recently passed some legislation regarding immigration...The Arizona law makes it a state crime to be in the U.S. illegally. It requires local and state law enforcement officers to question people about their immigration status if they have reason to suspect a person is in the country illegally, making it a crime for them to lack registration documents. Do you support or oppose this law?
All adults: 61% Support, 36% Oppose
Hispanics: 31% Support, 65% Oppose
31% Democrat, 24% Republican, 41% independent (chart)
Do you think the country would be better off if the Republicans controlled Congress, if the Democrats controlled Congress, or would the country be the same regardless of which party controlled Congress?
28% Republicans, 27% Democrats, 44% Same
Would you be more likely to vote for a candidate for Congress who ______ supports, or more likely to vote for a candidate who ______ opposes?
President Obama: 44% Supports, 41% Opposes
Tea Party: 43% Supports, 40% Opposes
Now I'd like you to think about all political offices at the federal, state and local level. Often when you vote for these offices, you must choose between an incumbent who is running for re-election to an office he or she already holds, and a challenger who does not hold that office. In general, are you more likely to vote for the incumbent running for re-election or more likely to vote for the challenger who does not hold that office?
30% Incumbent, 47% Challenger
Job Approval / Disapproval
Dems in Congress: 34 / 57
Reps in Congress: 26 / 62
State of the Country
29% Satisfied, 70% Dissatisfied (chart)
Should Senators support or oppose Elena Kagan's nomination to the Supreme Court based only on whether she is qualified to be a justice, or should they also consider her views on controversial issues like abortion and gay marriage?
43% Qualifications, 43% Consider views
If Senators did not agree with Elena Kagan on controversial issues like abortion and gay marriage, do you think they would be justified, or not justified in using the filibuster to prevent her nomination from coming to a vote?
37% Justified, 50% Not justified
Do you approve or disapprove of the federal health care overhaul?
40% Approve, 51% Disapprove (chart)
As you may know, the state of Arizona recently passed a law that gives the police the power to question someone they have already stopped, detained, or arrested about their legal status in the country. The law requires people to produce documents verifying their status if asked, Do you think this law goes too far in dealing with the issue of ILLEGAL immigration, doesn't go far enough, or is about right?
28% Too far, 17% Not far enough, 52% About right
From what you've heard or read, do you approve or disapprove of the new health care reform bill?
43% Approve, 47% Disapprove (chart)
Would you favor allowing increased drilling for oil and natural gas off the U.S. coast, or do you think the costs and risks are too great?
45% Favor, 46% Costs and risks too great
Do you approve or disapprove of the way ______ is handling the oil spill in the Gulf of Mexico?
Obama Administration: 35% Approve, 45% Disapprove
BP, the company that operated the oil rig: 18% Approve, 70% Disapprove
How would you rate the job ______ has been doing responding to the oil leak in the Gulf of Mexico?
Pres. Obama: 7% Excellent, 24% Good, 31% Only Fair, 26% Poor
B.P.: 3% Excellent, 16% Good, 26% Only Fair, 44% Poor
Please tell me if you think the REPUBLICAN Party or the DEMOCRATIC Party could do a better job in each of the following areas...
Dealing with the economy: 34% Democratic, 33% Republican
Reducing the federal budget deficit: 33% Republican, 30% Democratic
Dealing with the terrorist threat at home: 38% Republican, 27% Democratic
Dealing with the nation's energy problems: 35% Democratic, 28% Republican
Dealing with immigration: 35% Republican, 27% Democratic
From what you know, do you strongly agree, agree, disagree or strongly disagree with the Tea Party movement, or don't you have an opinion either way?
25% Strongly agree / Agree, 28% Strongly Disagree / Disagree
32% Democrat, 23% Republican, 36% independent (chart)
My column for this week looks back at how the polls performed in Pennsylvania's Democratic primary for Senate and one potentially overlooked finding: Several polls showed Joe Sestak doing better as pollsters narrowed their samples to the most likely of voters. Please click through and read it all.
And special thanks for Chris Borick at Muhlenberg University for providing the data featured in the column.
Ipsos Public Affairs for St. Petersburg Times / Miami Herald / Bay News 9 / Central Florida News 13
5/14-18/10; 607 registered voters, 4% margin of error
Mode: Live telephone interviews
(Miami Herald article)