Will this be the year that "cell phone only" voters wreak havoc on the results of pre-election polls? And does the cell phone only problem doom pollsters that depend on automated, recorded voice methodologies? Two new recent polls from SurveyUSA suggest the answers are not as obvious as some may think.
Let's start with the first question. SurveyUSA, a company that has been conducting recorded-voice surveys for local television news stations for nearly twenty years, has recently released two statewide surveys based on dual samples of both landline and mobile phones. In both cases including cell-phone-only voters interviewed over their cell phones did not make much difference in the results. Their recent Washington poll, for example, shows Democratic Senator Patty Murray leading by a not-statistically-significant four-point margin (37% to 33%) over challenger Dino Rossi in a combined sample of landline and mobile phones. Murray's lead would have been a virtually identical five-point margin (39% to 34%) had they interviewed by landline phone only.
Similarly, a North Carolina survey released just yesterday shows Republican Richard Burr leading Democrat Elaine Marshall by ten points (46% to 36%) in the combined sample interviewed over both landline and cell phones. Burr would also have led by a 10-point margin (47% to 37%) had they interviewed all respondents via landline phones only.
These are just two surveys, of course. A more comprehensive assessment of national data gathered by the Pew Research Center earlier this year found that, "weighted estimates" from a large landline sample "tend to slightly underestimate support for Democratic candidates when compared with estimates from dual frame landline and cell samples in polling for the midterm congressional elections this year." But if that slight understatement is real, it may not produce many "significant" differences, either statistically or substantively, in individual statewide surveys.
What is more interesting here, however, is that an automated pollster managed to conduct a "dual frame" survey at all. The underlying story gets us closer to an answer to the issue of the impact of cell phones on automated surveys.
The TCPA forbids calling a cell phone using any automated telephone dialing system (autodialer) without prior express consent. This rule applies to all uses of autodialers and predictive dialers, including survey and opinion research.
Virtually all pollsters use some form of "autodialer" to place calls to landline respondents, so virtually all pollsters are affected by the TCPA's restrictions on calls to cell phones. With the exception of CBS News (the only operation I know of where interviewers still hand dial each number), virtually all pollsters use some form of computerized interviewing system that dials the phone so interviewers don't have to. Some also use "predictive dialers" that place calls and only connect the respondent to an interviewer once a live person answers the phone (a process that creates that annoying pause that anyone who has answered a call from a telemarketer is all too familiar with). Finally, all recorded-voice pollsters use an "automated dialing system" for their complete process, though they could theoretically begin with a live interviewer and then hand off the process, with the respondent's consent, to an automated interview.
So when live-interviewer pollsters want to interview respondents on their cell phone, their interviewers need to place the calls manually. Their process becomes less efficient and more expensive, but they do not face a total barrier.
Pollsters that use a recorded-voice methodology face a much bigger problem. Yet somehow, SurveyUSA managed to interview voters in North Carolina and Washington over their cell phones. How did they do it? They used live interviewers:
Cellphone numbers were dialed one at a time, cellphone respondents were interviewed by call center employees. Landline respondents heard the recorded voice of a SurveyUSA professional announcer.
In North Carolina, SurveyUSA used more expensive live interviewers to conduct 404 out of 1,000 interviews, although only 250 of those were in cell-phone-only households (see their methodology statement for more details).
So while this approach amounts to a technical solution to the challenge of reaching cell-phone-only households, it creates a huge challenge to the underlying business model of automated pollsters like SurveyUSA. Consider the chart below, prepared by SurveyUSA CEO Jay Leve for a presentation last year. It suggests that in this case, their costs were somewhere between triple and quadruple what they would have been had they done all interviews using a recorded voice methodology.
Any other lessons here?
First, this issue provides another demonstration of why all automated surveys are not created equal. In this case, SurveyUSA is actually doing more of a "mixed mode" poll that combines both recorded-voice and live interviewers.
Second, all "dual mode" surveys based on combining landline and cell phone samples are not created equal either. Pollsters have to decide whether to use the cell phone samples to reach just the "cell phone only" households, or whether to also include the "cell phone mostlys" as well. And either way, they need to decide how to weight the combined samples, often without reliable estimates of the percentage of cell-phone-only households at the state level (see the SurveyUSA release and the Pew Research report for more detail).
Third, as is true for many aspects of poll methodology, pollsters could do a better job disclosing the procedures and methods they use to interview Americans over their cell phones and combine those results with interviews conducted via landline phones. CBS News, for example, tells us only that the numbers for their just released survey "were dialed from random digit dial samples of both standard land-line and cell phones." The release for the NBC News/Wall Street Journal poll tells us that their sample of 1,000 adults included "200 reached by cell phone," but nothing more. There are exceptions, of course -- most notably the Pew Research Center -- but they are few and far between.
Following-up on my post on Monday, which discussion of Nate Silver's suggestion that pollsters consider weighting on non-traditional variables, the Pew Research Center's Scott Keeter emails to point out they did a post election analysis that suggests strong potential for adding variables like income, marital status and home ownership to reduce the bias assess the effect of adding additional variables to their weighting scheme.
They found that the additional variables reduced the cell-phone-only bias substantially, though as Keeter points out, "this was not a comparison of different weighting schemes, but was instructive of what additional weighting factors might accomplish."
Logistic regression was used to estimate the probability of voting for Obama among landline voters and cell-only voters. As would be expected, the difference is sizeable; the predicted probability of voting for Obama is 16 points higher for cell-only voters than for landline voters. Adding most of the standard demographic variables used in weighting (e.g., age, sex, race, Hispanic [ancestry], education, and region) to the model (labeled the "standard model" in Table 3) reduces this difference to 11 points, a result consistent with the notion that weighting helps reduce but not eliminate the potential for non-coverage bias. Including income, marital status and home ownership in the model reduces the difference even further to 5 points. When these additional demographics are included in the model, being cell phone only is no longer a significant predictor of candidate support, as it was in the first two models.
Last Thursday the Pew Research Center released an analysis drawing on their extensive ongoing investigation of the impact of the growing cell-phone only population on conventional telephone surveys. It is a must read for anyone in the polling business. You may have also seen commentary on the report on Monday by Nate Silver and Chris Bowers, but I'd like to add a few thoughts.
For those new to it, the crux of the issue is that telephone surveys have traditionally relied on samples of landline telephone numbers. Unfortunately, the explosion of cell phone usage over the last 10 years places a rapidly growing number of "cell phone only" Americans out of reach of those surveys. In pollster lingo, this is a "coverage" problem. A lack of coverage will result in statistical bias if the out-of-reach segment of the population is both large and different from the rest.
What's new in this latest Pew report is the growing evidence of just such bias. Specifically:
The growth in cell-phone only households continues unchecked. The latest estimates from the National Center for Health Statistics shows that 25% of households (and 23% of adults) have cell phone service but no landline (another 2% of households have no telephone service at all). The cell-phone-only population has doubled (from 12% of adults) in just three years.
The side-by-side comparisons by the Pew Research Center, which has interviewed respondents on both landline and cell phones since 2006, now show non-coverage bias "appearing regularly" on landline-only surveys that have been fully weighted to correct for demographic imbalances:
Of 72 questions examined [Since August 2009], 43 show differences of 0, 1 or 2 percentage points between the landline and dual frame weighted samples. Twenty-nine of the differences are 3 percentage points or more, all of which are statistically significant. Only one difference is as large as 7 points, while four others are 5 points and seven are 4 points.
The Pew Research analysts are also confident that "the bias has grown in the last four years." In 2006, they did 46 similar comparisons and found not a single difference that exceeded two percentage points.
Last but not least, the bias appears to extend to one very critical political measure for 2010 (emphasis added):
Weighted estimates from the landline sample tend to slightly underestimate support for Democratic candidates when compared with estimates from dual frame landline and cell samples in polling for the midterm congressional elections this year. The same result was seen in Pew Research Center polls throughout the 2008 presidential election. In the landline sample, Republican candidates have a 47%-to-41% margin over Democratic candidates on the 2010 generic horserace, but in the combined sample voters are evenly divided in their candidate preferences for this November (44% for each party).
Two big cautions about that last bullet. First, the 2010 generic horserace comparison is based on just one survey from March that involved a landline sample of 1,442 registered voters and combined landline-cellphone sample of 2,070 (of whom, just 191 registered voters were cell-phone-only). While it is consistent with similar comparisons in 2008 and 2010 based on far more interviews, is possible that random variation exaggerated the bias in this single measurement.
Second, although the apparent bias is consistent with what the Pew Center found on other measures in 2008, another Pew Research report found no clear evidence the the bias led to greater polling errors in statewide polling in 2008, virtually all of which was conducted over landlines only: "[T]he average candidate error for [237 polls in statewide] races was 1.9 percentage points, about the same as in 2004 (1.7 percentage points)."
But the history of largely unbiased statewide and national polling in 2008 is no guarantee of a repeat performance in 2010, particularly given the rapid increase in the cell-phone only population. While some statewide pollsters -- most notably Quinnipiac University -- are planning to interview using "dual frame" surveys that interview over both landline and cell phones this year, the vast majority of pollsters will not.
Why not? Calling by cell phone adds considerable expense and runs up against a federal law that bars pollsters from dialing a cell phone using any automated means. For live-interviewer polls, that means more time consuming hand dialing of cell phone numbers. For those using an automated method -- like SurveyUSA, Rasmussen and PPP -- the regulation is a total barrier. The automated (IVR) pollsters simply cannot interview respondents by cell phone.
It is also worth reviewing the concluding "discussion" section of the Pew Research report for their review of why "dual frame" surveys of both landline and cell phones provide "no panacea for the coverage problem." The Pew Center estimates that in addition to the 23% of adults that are "cell-phone-only," another 17% are now "cell-phone mostly" -- Americans with both kinds of phone service who say they rely on cell phones for most of their calling. A survey that relies on cell phone sample to reach just the 23% with cell phones only may still under-represent cell-phone mostlys.
Finally, thoughts on some suggestions Nate Silver made on Monday:
Another approach, in the absence of calling cellphones, is to increase the sample size that one uses. Although there's not that much difference between calling an unbiased sample of 500 respondents or 1,000 (the associated margins of error are 4.4 points and 3.1 points, respectively), these differences are magnified if one relies upon upweighting results from smaller subsamples to correct for response bias, such as those for young voters or Hispanics.
Finally, pollsters might want to consider weighting based on "non-traditional" criteria such as urban/rural status, technology usage, or perhaps even media consumption habits.
If you read that quickly, it may have sounded as if simply doing more interviews would eliminate the sort of bias that the Pew Center report describes. I don't think that's the point Nate was trying to make, but just so there's no confusion: If the sample has coverage bias, it doesn't matter whether you do 500 interviews, 1,000 interviews or 100,000. More interviews won't fix the bias.
That said, pollsters are already weighting their samples more severely than ever before to compensate for the purely demographic bias caused by the cell-phone only coverage problem. Bigger and more severe weighting makes for more random error. Pollsters should be increasing their reported "margin of error" to account for the additional "design effect" of all that extra weighting. Unfortunately, few do (but that's another story for another day).
If pollsters can find newer non-traditional weighting schemes that can correct the sort of coverage bias described in the Pew report, those schemes will likely involve weights that are, for some respondents, even bigger and more severe. So Nate is right in recommending larger sample sizes to reduce the effectively larger "margins of error" (that have absolutely nothing to do with coverage bias).
Finally, I ran his suggestion for "non-traditional" weighting by Pew Center's Leah Christian. First, Pew already weights by what is effectively an urban/rural measure: the population density of the respondent's county. Second, they are wary of weighting by technology usage or media consumption:
The main issue with weighting to technology use or media consumption is the lack of reliable national parameters. In theory, someone could produce estimates from another survey that could then be used to weight a landline survey, but the reliability of these would depend a lot on the quality of the sample from the original survey and the stability of the estimates over time. Technology use and media consumption are much more variable over time (unlike many standard demographics), and I would have many of the same concerns as weighting to party identification.
My point here is not to discourage innovation but to offer a reality check. When it comes to coverage bias, there are no easy answers. The very big changes in telephone usage are increasingly challenging our ability to obtain representative samples via telephone.
Update: Pew Research's Scott Keeter emails to point out that their 2008 post-election analysis found that adding variables like marital status, income and home ownership to a regression model reduced cell-phone-only bias in predicting support for Obama. More details here.
One of the things I loved most about the AAPOR conference was the opportunity to learn from pollsters of different disciplines. The lessons one organization learns about how to reach a unique population are often useful to researchers of all varieties. In this case, Pew presented its findings about how best to reach Hispanics in general public opinion surveys. From issues in language and translation to interviewer hand-offs to the prevalence of cell phone use, Pew's findings highlighted the challenges in ensuring Hispanics are properly represented in survey research. For campaign pollsters, particularly those operating in states with a high proportion of Hispanic voters, knowing how to get a representative snapshot is becoming more and more critical to monitoring political attitudes. I had a chance to chat with Jocelyn Kiley about the research and its importance to political polling.
The Quinnipiac University Poll plans to begin calling cell phones with its next Florida poll, later this spring...The Institute for Public Opinion Research [at Florida International University] already does, in almost every poll.
Most of the well-known, national media surveys (including those conducted by the Pew Research Center, Gallup Daily, USA Today/Gallup, ABC/Washington Post, CBS/New York Times, CNN/ORC, NBC/Wall Street Journal, and AP-GFK) now interview respondents on both landline and cell phones, but we have so far seen very little of the same at the state and local level. As such, shifts by pollsters like Quinnipiac that poll at the state level mark a significant milestone.
The cell-phone-only issue presents a special challenge to pollsters that rely on automated, recorded voice methodology (sometimes referred to as interactive voice response or IVR), because the federal Telephone Consumer Protection Act (TCPA) bans any sort of unsolicited call to a cell phone using "automated dialing devices." The ban means that prolific statewide pollsters like Rasmussen Reports, SurveyUSA and Public Policy Polling (PPP) cannot dial voters on their cell phones.
For more detail, see our posts on cell phones and survey and the Pew Research Center report from 2008 on their research on calling via cell phones in 2008.
Update - I emailed Doug Schwartz, the director of the Quinnipiac University Poll and he confirms that they have been or will be adding cell phone samples to their statewide polling in 2010:
We've already started call cell phones in our NYC, NY, NJ, OH, and national polls. And over the next few months we will begin calling cell phones in the rest of our states - PA, CT, and FL.
My column this week looks at the controversy over a series of surveys conducted by SurveyUSA for the liberal web site Firedloglake. Please click through to read the whole thing.
Lost in the attack memos and other questions raised is an important question facing nearly every telephone survey conducted in House, Senate and Gubernatorial races this year: Are we at the point where the majority of true "likely voters" under the age of 35 are out of reach of landline telephone samples? And at what point is simply "weighting up" those younger voters that pollsters can still reach inadequate to solve the problem?
The table below, produced by the Pew Research Center and based on their national surveys, shows that by 2006 their unweighted landline samples were under-representing roughly a third of adults under age 35. And that was as of three years ago, when the percentage of all adults living in landline-only households was estimated at 12%, nine percentage points lower than the most recent estimate:
Now consider the estimated growth in the cell-phone-only population over the last three years. As shown in the chart below (which comes from a report last year by the National Center for Health Statistics of the Centers for Disease Control and Prevention), landline-only samples are most likely to miss voters under age 35.
Now consider this additional statistic reported on Pollster.com by Mike Mokrzycki in December. On the most recent CDC report covering the first half of 2009, nearly two thirds (63.5%) of people age 25-29 live in households with either no landline phone (45.8%) or in "cell-mostly" households (17.7%), those were "all or almost all calls are received on cell phones."
So what should a pollster do if they reach so few 18-to-34-year-old voters that they make up just 1% of the likely voters sample for an election where past turnout suggests that age group should make up roughly 10% of the electorate? If the pollster believes they have under-represented younger voters, can they simply weight to correct the problem? Not if the shortfall is that extreme. In a sample with only 400 or 500 completed interviews, such a weight would multiply 4 or 5 interviews by a factor of 10. As I wrote in the column, you don't need to be a statistician to imagine how those "super respondents" might crate greater error and volatility in the results, especially those produced by cross-tabulations of demographic subgroups.
Let's remember that we are able to pick at SurveyUSA because they were willing to disclose the weighted demographics of their sample and because they opted against any such extreme weighting in this case. So rather than beat up on SurveyUSA, we might do better to ask: How many polls have we seen in recent months that involved a similarly sparse number of younger likely voters and were simply weighted up by factors of 5 or greater to conceal the shortfall? How would we know?
Finally, whatever we want to make of the Firedoglake surveys, it is important to remember that SurveyUSA has maintained an outstanding record of final-poll accuracy, especially in U.S. House elections and in hard-to-model primary elections. For House races, the company's own scorecard -- which I have no reason to doubt -- shows that their average error on the margin in polling 27 House races in 2006 (3.4) was roughly half that of all other pollsters combined (6.3). Their error rate was also significantly lower than the three most prolific public pollsters that year, Research2000 (5.5), Zogby (5.9) and RT Strategies (5.9).
So since we have picked at their work mercilessly, I want to give SurveyUSA's Jay Leve the last word and reproduce the full email he sent me last week in response to my questions about the Firedoglake surveys:
In August 2002, SurveyUSA released a poll showing US Senator Robert Torricelli (D-NJ) trailing. No survey to that point had showed Torricelli trailing. An hour after the poll was released, SurveyUSA's client, CBS-TV in Philadelphia, called SurveyUSA and said, "Put your helmets on. The DSCC is coming after you." And the DSCC did. The DSCC found a journalist willing to write the smack that the DSCC was shoveling, and the message went forth: Nothing wrong with Robert Torricelli, plenty wrong with SurveyUSA.
A few weeks later, Torricelli dropped out of the race. Other polls had the same results as SurveyUSA.
Fast forward to today: In a poll conducted in January 2010, at a time the Democrats were losing the state of Massachusetts, SurveyUSA finds an incumbent Democrat in a tight fight in New York state. The DCCC is unhappy. Partisans start shoveling smack. "Sources" start providing willing journalists with leaked memos. Nothing wrong with Democrat Tim Bishop. Plenty wrong with SurveyUSA.
The highway to high office is littered with the road kill of political operatives who find it easier to campaign against a poll than an opponent.
Lost in the hurly burly is an opportunity for real reflection. To my knowledge, there has never (ever) been a publicly released telephone poll conducted in a U.S. congressional district that included a known subset of interviews with respondents who did not have a home (aka: landline) telephone. An acknowledged limitation of SurveyUSA's work in NY-01, and a known limitation to date of all congressional district polling, is that voters who do not have a home phone are under represented. At a statewide-level (in contrast to the CD level), only one pollster in the 2009 election included a known subset of cellphone-only respondents in its sample (at extraordinary expense, because of the theoretical justification), and that pollster's results were worse than many polling firms who did not include a known subset of cell-phone-only respondents. Whether one anticipates that in 2010 young voters will turn out in record numbers of stay home in record numbers, the problem of how to count those voters is real, and right before us.
One of the major themes in the paper is that understanding the cell-only population is about more than just age. In fact, residential mobility has a strong influence on whether someone has shed their landline. Even after controlling for age and a litany of other demographic variables, we find that respondents who moved within the last year were 24 percentage points more likely to be cell only than those who had lived in the same residence for at least five years. Renters, singles, and those without children were also much more likely to be cell-only.
Our explanation for this pattern:
"There are several reasons that highly mobile Americans may be more likely to go without landlines. First, whenever someone moves from one residence to another, they have an opportunity to reassess their phone needs. Thus, the act of moving provides an opportunity for individuals to shed their landlines. Second, mobile Americans may choose a CPO lifestyle because cell phone numbers tend to be more portable than landlines. When moving from one metropolitan area to another, individuals must change their landline phone number, but do not need to change their cell number. This may provide an incentive for choosing not to maintain a landline in a new residence. Third, those with fewer family and community ties may feel less of a need to have multiple phone lines on which they can be reached by members of their social networks. "
The fact that the cell-only public tends to be more mobile has some important political consequences. Some highlights:
The difference in the percentage of landline and cell-only respondents who reported being registered was fairly small--over 95% in both groups. However, there was a much larger gap in actual registration rates (66.8% versus 53.9%). Since cell-onlys are more likely to have moved recently, they may not have successfully registered to vote at their new addresses despite the fact that they may think they are registered.
Cell-only respondents were significantly more likely to have problems with their registration when attempting to vote. In 2008, over 7% of cell-only respondents indicated that there was a problem with their registration when they attempted to vote, compared to fewer than 4% of respondents with landlines.
Cell-only respondents were more than twice as likely as those with landlines to report that neither campaign contacted them. In short, this group is much less likely to be subjected to mobilization efforts from the campaigns.
Cell-onlys are politically distinct on a variety of measures. However, this distinctiveness is somewhat muted when demographic controls are taken into account. Interestingly, the largest differences between cell-only and landline respondents are not on issues or ideological self-placement, but on reported vote choices.
Ultimately, we argue that weighting for standard demographic measures such as age, education, income, and race may not be sufficient. Pollsters relying on landline samples may want to consider weighting by other factors such as time in residency, renter/home owner, and marital status. But check out the full paper for a more detailed discussion of all of these points.
Today the National Center for Health Statistics (NCHS) of the Centers for Disease Control and Prevention (CDC) released its latest biannual report on the prevalence of households without wireless or standard telephone service. This latest, which covers the latter half of 2008, shows the trend toward "cell phone only" households continuing unabated as hat 18.4% of adults were reachable only by cell phone, while another 1.7% lacked telephone service of any kind.
A refresher for those unfamiliar: CDC monitors the cell-phone-only population because it conducts huge ongoing health "surveillance" surveys via telephone, and as such, asks questions about telephone usage on their ongoing, in-person National Health Interview Survey (NHIS). Traditionally, telephone surveys have relied exclusively on random digit dial (RDD) samples that reach only those with landline phone service. These regular CDC estimates are a big reason why most national media polls are now including samples of cell phone users.
The NCHS estimate of the percentage of American households with only wireless phones increased 2.7 percentage points (from 17.5% to 20.2%), amounting to "the largest 6-month increase observed since NHIS began collecting data on wireless-only households in 2003," according to the report. They also note a big jump in what some call "cell-phone mostlys. "One of every seven American homes (14.5%) received all or almost all calls on wireless telephones, despite having a landline telephone in the home."
This latest report also includes a new feature: A chart with regression lines showing the growth in wireless-only households by age and by year. The chart makes clear that while the wireless-only population remains disproportionately younger, it is also growing rapidly among Americans over 30 years of age as well.
For further reading: More on the latest report from Carl Bialik. And see this link for all of our recent coverage involving cell phones and surveys.
**One reason why it seems like less than six months. In March, NCHS released a supplemental report that provided wireless-only estimates for all 50 states (our summary here).
The survey research community is focusing intently on the challenges posed by the fast-growing share of Americans who are cell-phone-onlys (CPOs). In fact, there are 40 papers being presented on the topic at the AAPOR conference next month. One of the practical issues faced by pollsters is whether the cost of reaching CPOs is worth the payoff. Last week, Scott Keeter, Mike Dimock, and Leah Christian hosted a forum at Pew during which they discussed this tradeoff. But pollsters aren't the only people who have to make cost-benefit decisions when it comes to deciding whether to attempt to contact CPOs. Campaign organizations must make the same calculation.
So how well did the campaigns do at contacting CPOs during the 2008 campaign? The chart below compares the percentage of those with landlines and cell-onlys who reported being contacted by a campaign representative in 2008. The data comes from the National Election Study (NES), which uses residential sampling and face-to-face interviews to interview both landline and CPO respondents. In the chart below, the blue bars show the percentage of each group that reported being contacted while the black lines represent 95% confidence intervals for these percentages.
The chart shows that CPOs were much less likely to be contacted by the campaigns than people with landlines. Over half of landline respondents reported being contacted compared to less than one-in-three CPOs. This sizable difference holds up even when controlling for age, income, education, partisanship, and a variety of other factors.
The next chart (below) indicates that for those CPOs who were contacted, the contact tended to come overwhelmingly from Democrats. Over 80% of CPOs who were reached by the campaigns were contacted by the Democratic side while just a little over one-third were reached by Republicans. Republicans were significantly more competitive with Democrats when it came to contacting those with landlines.
Unfortunately, the NES did not include questions asking respondents how they were contacted by the campaigns. But a subset of respondents to the 2008 Cooperative Congressional Election Study (which I've analyzed in previousposts) were asked these questions. The chart below plots the responses for those who had landlines compared to CPOs.
CPOs who were contacted by one of the campaigns were significantly less likely to have had that contact over the phone compared to those with landlines. Otherwise, there were not major differences between how landline and CPO respondents were contacted. CPOs were somewhat more likely to get an email while those with landlines were a bit more likely to receive snail mail, but neither of these differences are large. The percentage being contacted in-person or by text message were nearly identical for both groups.
Overall, the findings from these surveys suggest that shedding your landline may help you avoid those pesky campaign calls in future election years. While Democrats were a little more successful than Republicans in reaching CPOs, the cell-only crowd was almost as successful avoiding campaign volunteers as they were hiding from pollsters.
I attended a presentation last week at the Pew Research Center (sponsored by the DC AAPOR chapter) on some of the practical issues they have encountered in their innovative work on cell phone polling. I'm still catching up from a few hectic days that have followed but want to pass along a few interesting details they shared.
Most of what was new in the session will be of more interest to pollsters than to political junkies wondering about how pollsters are dealing with the growing number of Americans without landline phone service. Fortunately, for those of you in the latter category, the PRC shared most of their more general data obtained from calling cell phones during the 2008 campaign in a report released this past December (see their summary, full report pdf and our review).
Here are a few highlights that seemed especially noteworthy or new in the presentation by Pew's survey research director Scott Keeter, associate director Michael Dimock and research associate Leah Christian:
Pew has now conducted 18 surveys (14 in 2008 and and 4 in 2009) featured a "dual frame" sample of both landline and mobile phones. In those surveys, they have interviewed approximately 9,400 adults via cell phone. Keeter explained that these dual frame samples are now "standard policy" for Pew's political surveys.
They have found cell phone users just as willing to answer their phones and cooperate as landline users. Their response rate for cell phones (23%) is virtually the same as that for landlines (24%)
Pew's calling center is finding it "cheaper and easier" to interview by cell phone. Just a few months ago, Keeter reported that it cost Pew "two to two and half times" as much for cell phone interviews as landline. Now the cell phone costs are "closer to two times as expensive" as landline interviews. Keeter attributed the improvement to their call centers "getting more familiar with the tricks of doing successful cell phone interviewing." Note that Pew interviews all cell phone users, and weights down respondents less likely to be "cell phone only." If they had screened for just the cell phone only users, the differential would be closer to four times the cost of landline interviews.
Pew continues to offer a $10 per interview incentive to cell phone respondents although, according to Keeter, other pollsters such as Gallup do not pay incentives to the cell phone respondents.
The biggest continuing methodological challenge to pollsters is determining how to combine and weight their landline and cell phone samples, partly because of what some are now calling the "cell phone mostly" problem. This issues involves those who have both mobile and landline phones but make "all" or "almost all" of their calls on the mobile phone (see pp. 6-8 from the December report, and my previous discussion of the issue for more details).
In this presentation, Dimock presented results showing that the cell phone mostly respondents were less likely to say they could be reached "right now" on a landline phone (52%) than those who use their cell phones for only some or few calls (63%). "This is really a spectrum," Dimock explained, without "a clear cut line between the group of people who absolutely can't be reached on a landline and another group who absolutely can't," but rather "a probability across the spectrum of people." As such, Pew's approach is to interview everyone reached via cell phone and weight on the inverse of their probability of selection.
Christian presented data on the problems identifying the actual geography of respondents based on their telephone number. Before the widespread adoption of cell phones, telephone area codes could identify the state and time zone of each number with great accuracy, as landline telephone numbers are closely associated with geography. Cell phone numbers, on the other hand, are assigned based on the geography where you first purchased your cell phone. The relatively new ability to "transport" cell phone numbers from one carrier is creating a growing mismatch between phone numbers and geography.
Pew has been able to confirm and quantify that trend by asking respondents to provide their zip codes. That additional data shows that a 5% mismatch for their cell phone samples on region (presumably census region), a 9% mismatch on state and a 39% mismatch on county. This discrepancy has two practical implications: First, obviously, when interviewing via cell phone, pollsters cannot treat phone numbers as an accurate measure of geography, especially at the county level. Second, they have to be careful about scheduling call time based on area code. A few years ago, pollsters could safely dial west coast area codes until 9:00 p.m. Pacific time. Now it is all to easy to ring someone much later at night, so pollsters have had to modify their procedures.
A hat tip and thanks to Susanna Fox of the Pew Internet Project and our friend Alex Lundry, for their helpful notes posted notes during the session on Twitter
Steve Ansolabehere and I have been working over the past few weeks on a paper we are writing for the AAPOR conference next month. Over the next couple of weeks, I'll share some of our preliminary findings here and I wanted to lead off today by presenting some comparative data we put together from three different surveys with distinct approaches to sampling.
The National Election Study, which has been around since 1948, is a labor- and time-intensive survey that uses a residential sampling approach and conducts face-to-face interviews. The Cooperative Congressional Election Study is an web-based survey conducted by YouGov/Polimetrix. The respondents opt-in to the study and Polimetrix uses a sample matching methodology where they first select a random target sample and then attempt to find a match for each respondent in their pool of opt-in respondents. Finally, Pew's survey work is fairly well known to regular Pollster.com readers. Their efforts to incorporate CPOs into their samples has relied on a dual-frame approach where they randomly select landlines and then cellular lines to build their sample.
So what happens when we compare these surveys, each of which takes a different sampling approach? Each survey I'll present data on here was conducted during the presidential campaign last year. For starters, the chart below compares each survey's estimate of the size of the cell phone only population after sample weights are applied. Note that Pew includes phone status as one of their weighting criteria, but the NES and CCES do not.
CPOs make up 19.7% of the CCES sample after weighting; 17.9% of the Pew sample and 17.3% of the NES sample are CPOs (again, after weights are applied). Thus, the differences are relatively small. Also note that the 95% confidence intervals for each estimate (represented by the darker lines) overlap.
So each of the surveys provides a similar estimate of the size of the CPO population, but what about the composition of that group? The table below compares landline and CPO respondents on a wide array of demographic and socioeconomic factors.
There is a lot of information in the table, but the different surveys are fairly similar across most measures. A few differences do stand out, however. After weighting, the CPO respondents reached by Pew and CCES were less likely to have children, less likely to be married, and less likely to be home-owners than those reached by the NES. Pew's CPOs also tended to have lower incomes and were somewhat more likely to be racial/ethnic minorities. Perhaps NES's face-to-face approach is more likely to pick up CPOs who have "settled down" relative to CCES or Pew.
Speaking of "settling down," one other item deserves attention; that is the information on residential mobility at the bottom of the table. While we weren't able to find this information in any of Pew's surveys conducted in October or November, the data were available for the CCES and NES. Not surprisingly, residential mobility is strongly related to whether one is a CPO or not. Even controlling for age, people who have moved recently are much more likely to have shed their landline and gone with just their cell. In fact, this relationship holds up when you control for all of the other demographic variables in the table. This is something I'll post more about later, but you can probably imagine that there are some significant political consequences arising from the fact that CPOs tend to move around much more frequently.
Oklahoma and DC have the highest percentage of adults living in "cell phone only" households in the nation (25.1% and 25.4% respectively) according to a new report just issued by the Centers for Disease Control (CDC). The report, issued by CDC's National Center for Health Statistics (NCHS) provides estimates of the wireless-only percentage of households and adults for all 50 states and the District of Columbia (which reports a wireless-only percentage even higher than Oklahoma). The states with the lowest number of cell-phone only adults are Delaware (4.0%), Vermont (4.6%) and Connecticut (4.8%).
Five years ago, survey methodologists at CDC started asking questions about telephone usage on the on-going National Health Interview Survey (NHIS). Since NHIS is an in-person survey with a massive sample (interviewing roughly 13,000 Americans every six months), it can provide very precise, national estimates of the cell-phone-only population. The semi-annual estimates of the cell-phone only population from NCHS have become the most anticipated numbers in survey research. Their estimates are critically important to any survey -- including most national media polls last fall -- that interviewed respondents over their cell phones in order to reach the cell-phone only population. Pollsters use the NCHS estimates to determine how to weight their combined landline, cell phone samples.
Until now, however, NCHS has not produced state level estimates of the cell phone only population because "the sample size of NHIS is insufficient for direct reliable annual estimates for most states."
How did they get around that limit? They first used the NHIS data to create statistical models (using logistic regression) that estimate the cell-phone-only percentages using demographic variables (including gender, age, race, education, household size, home ownership, employment and poverty status). They then applied their models to a much larger set of survey data collected as part of the Current Population Study conducted by the U.S. Census Bureau to create "modeled estimates" of the cell phone only population in each state (the first table in their report). The report also includes the "direct state-level estimates" (the results of the NHIS survey interviews in each state).
Like the national NCHS estimates, these new state level statistics will be closely watched -- and used -- by pollsters and survey researchers of all varieties.
Update: Pollster reader Joran sends in two charts he quickly created that plot the state-level NCHS estimates of the wireless only percentages of households and adults, complete with error bars (see the report for details on these not-quite-confidence intervals).
Frequent Pollster.com readers will know that the Cell Phone Only (CPO) population is a subject I blogged onfrequentlyduringthecampaign. One of the reasons for this interest is because there is still a lot we don't know about CPOs at this point (I like a good mystery). In this post, I want to present a little analysis that demonstrates just how much we still have to learn about CPOs.
CPOs wouldn't be nearly as fascinating if their distinctiveness could be easily explained by their age or other basic demographic or political factors. One view is that CPOs are just younger, more urban, and more mobile and that once you account for these factors, CPOs behave pretty much just like their landline counterparts. Last year, however, a Pew report suggested that the differences might not be that simple when they discovered that weighting alone might not be enough to account for CPOs.
To explain why this is the case, I used some of the data Pew relied on to publish that report (in this case, Pew's June 2008 Voter Attitudes Survey). In this survey, Pew interviewed respondents both on landline phones and on cell phones (see more on the methodology here). What I wanted to know was whether CPOs are really that different from landline users once you account for demographic and political factors?
To answer this question, I estimated a multivariate statistical model that would allow me to take into account any demographic or political information that the Pew survey collected that I could imagine would explain the difference between CPOs and the rest of the public. I controlled for age, gender, education, ethnicity, race, income, home ownership, marital status, and whether the respondent lived in an urban area. In addition to these demographic factors, I also accounted for the partisan affiliation of the respondent. If these factors explained the difference between CPOs and landline respondents, then we wouldn't expect to find any measurable difference between these groups once we've controlled for them.
If you enjoy reading output from statistical models, you can get that information View image. However, the key information is presented in the chart below. The chart shows the probability of a landline or CPO respondent registering a vote preference for Obama, McCain, or neither candidate after controlling for the demographic and political factors.
This chart indicates that even after controlling for all of the demographic and political factors listed above, CPOs still had distinctive vote preferences relative to those with landlines in their homes. While landline respondents were 49% likely to prefer Obama in June, CPOs were 65% likely to do so. These differences can't be explained as a result of CPOs being younger, or because they were single, or because they lived in urban areas, or even because they were more likely to be Democrats. Essentially, if you had two people who were the same on all of the factors mentioned above, the one without a landline would still be more likely to support Obama than the one with a landline.
If it isn't age, income, education, or even mobility, then what makes CPOs distinctive from those with landlines? Is it something more inherent about embracing a CPO lifestyle? Perhaps it is an outlook on life that makes CPOs more willing to cast off traditions and venture into something new? Perhaps a higher tolerance for taking risks and embracing change? Or something else entirely? We still don't have a good handle on what makes CPOs so distinctive, which is why this makes for such a great mystery.
(Note: I'm using this Pew data to examine another reason why CPOs may cause pollsters so much trouble--because it may be more difficult to pin down whether they are actually going to vote. I'll present some analysis on that topic in my next post.)
The last two days bring news on the issue of cell phones and their impact on political surveys in the form of new reports from the most respected researchers on the subject.
First, yesterday, the Center for Disease Control and Prevention's National Center for Health Statistics released its latest biannual report on the prevalence of households without wireless or standard telephone service (via Bialik). The CDC monitors the cell-phone-only population because it conducts huge ongoing health "surveillance" surveys via telephone, and as such, ask questions about telephone usage on their ongoing, in-person National Health Interview Survey (NHIS). Their latest report, which covers the first six months of the year, shows that 16.1% of adults were reachable only by cell phone, while another 2.1% lacked telephone service of any kind.
As the chart above shows, the latest survey continues an ongoing, near linear upward growth in the cell-phone-only population. It is worth noting that the hint of a plateau in the trend seen between the second half of 2006 and the first half of 2007 was likely the result of a very slight change in question wording that took effect in 2007. Except for the brief near-pause, the trend has been steadily and consistently upward since 2005.
The second news is the latest report from the Pew Research Center on their efforts to survey voters via cell phone during 2008 (summary, full PDF). It is hard to overstate* the influence of the Pew Center's work on cell phones in the political polling industry. At the PAPOR conference in San Francisco last week, Professor Mike Traugott (a past president of AAPOR and chair of AAPOR's Special Committee on 2008 Primary Polling) noted that earlier this year, "the conventional wisdom in the Spring [among pollsters] was that we didn't have to worry about cell phone only people." At the end of the summer, however, "this conventional wisdom changed drastically" when the Pew Center released its report showing that the omission of cell phone only voters could understate Barack Obama's margin over McCain by two to three percentage points." By the fall, many national media surveys included supplemental cell phone samples in their surveys.
The new Pew report confirms that the patterns they saw, "that estimates based only on landline interviews were likely to have a pro-McCain tilt compared with estimates that included cell phone interviews," persisted through their final survey in late October. It was a difference of a point or two on the margins that "while statistically significant, was small in absolute terms -- smaller than the margin of sampling error in most polls." Of possibly greater importance going forward, they also found similar differences in party identification and self-reported ideology, and bigger differences in (not surprisingly) the use of Internet as a news source and social networking sites like Facebook or MySpace. See the report for the details.
Now that the conventional wisdom on cell-phone only households has shifted among national pollsters, from "it doesn't matter" to "we better account for the them," the Pew Report also points us to the next issue for the c.w. to chew over: Simply interviewing by cell phone (and screening for "cell only" respondents) may not be enough. Pollsters will also need to confront the more difficult issue of how to handle and weight the interviews among what some call "cell phone mostlys:"
Unlike most other polling organizations, Pew's election surveys involved a "full dual frame design," in which people reached by cell phone who also have a landline are interviewed, as well as cell-only respondents. In contrast, most pollsters who included cell phones in their election surveys screened their cell samples for cell-only respondents.
The difference between these two approaches can be seen as a tradeoff in methodological challenges. Supplementing a landline sample with cell-only respondents has the advantage of not "double covering" respondents who have both types of phones. This makes combining the samples more straightforward, but assumes that the landline sample is capable of accurately reaching all adults equally. If some adults have landline phones that they rarely or never answer because they favor their cell phones, they will be underrepresented in these surveys. Pew's approach of interviewing all adults in both the landline and cell phone samples ensures that every adult with a telephone is covered by the survey, but raises challenges in combining the data because some adults had a greater chance to participate if they have more than one telephone. Pew's methodology accounts for this double coverage by weighting respondents with both kinds of phones according to their probability of selection and the regularity with which they use each kind of telephone.
The report goes on to present data showing that the slightly different results produced by the two approaches. Duel users reached by cell phone were more likely to support Obama (53%) than those reached by landline phone (46%). The cell-phone-mostlys reached by cell also identified with the Democrats (54%) more often than those interviewed by landline phone (47%).
The report concludes with a section on the "practical considerations" of interviewing by cell phone. As they have found previously, once they offer a monetary incentive to potential cell-phone respondents ($10), the contact and cooperation rates are comparable to what they get on landline phones. So cell phone interviewing can be done. The downside is that cell-phone interviews cost Pew "nearly two-and-a-half times as much as landline interviews" for reasons they explain in detail.
As always, my summary does little justice to the full report. Go read it all.
Now that most of the national vote has been tabulated, we can get a pretty good sense of which pollsters came closest to pegging the final popular vote. As Mark mentioned in an earlier post, several others have donethisalready, but I thought I'd create these plots for Pollster.com readers.
The final national poll results from individual pollsters are plotted below (these are the last 19 national polls listed on Pollster.com's national trend page). The pollsters represented with red dots are those that included cell phone only (CPO) respondents in their sample. The Obama vote is represented by the y-axis and the McCain vote is the x-axis (UPDATE: I've updated the plots to reflect the updated vote share of 52.5% for Obama and 46.2% for McCain). The horizontal red line is the actual vote that Obama garnered while the vertical line indicates McCain's share of the vote. The closer a poll is located to where the two red lines meet, the more accurate that poll was in predicting the final outcome.
Note that every pre-election poll plotted here underestimated McCain's support. However, the big winners wereRasmussen and Pew, both of whom estimated a 52-46% advantage for Obama. The Pew poll included CPO respondents while the Rasmussen survey did not. CNN and Ipsos/McClatchy also came quite close by estimating a 53-46% advantage for Obama (UPDATE: These polls now come just as close as the Pew/Rasmussen polls). Neither survey reached the CPO population. Indeed, the plot reveals no clear pattern with regard to the CPO issue. Polls including CPO respondents did not appear to be any more accurate than those only reaching landlines.
Before the election, I separated out the Pollster.com national trend into surveys including the CPO population and those who were only calling landlines. The plot below looks at how each of these trends performed.
The trend based on surveys including the CPO population did slightly better at estimating Obama's vote but worse at gauging McCain's support. Overall, the CPO trend was slightly further off the mark than the landline trend.
Of course, there are any number of other factors at play with these different surveys (such as different likely voter screens, weighting, etc), so we can't draw any definitive conclusions from this analysis. But there is no obvious pattern from these initial results that indicate that including CPO respondents helped improve polling accuracy.
Update: Updated to reflect changes in popular vote.
In my column this week, I summarized some recent data provided by both the Pew Research Center and Gallup on the interviews both have been conducting this year via cell phone. Both pollsters have seen a similar pattern. With the interviews among "cell phone only" respondents (those who live in households without landline telephones) included, Obama does a point or two better, McCain does a point or two worse.
One of the more significant findings in this data was from Pew. They found big differences between 18-to-29-year-old voters with landline phone service an those reachable only by cell phone. The cell phone only younger voters favored Obama by a much bigger margin (62% to 27%, n=250) than the landline younger voters (52% to 39%, n=146). While those differences were just large enough to be statistically significant, the sample sizes involved are relatively small, so it is hard to be certain about the magnitude of the difference.
Curious, I asked Gallup's Jeff Jones if they could replicate the Pew table, and he kindly sent along the following data. You do not see differences by party identification. The ballot numbers show the same general pattern, but the differences are not as large as they appear on the Pew survey. Obama, runs better among the Gallup cell-phone only 18-to-29-year-olds (+28) than among those reachable by landline (+20). Those differences still appear to be statistically significant given the larger sample sizes.
The bottom line? Gallup and Pew have produced data supporting the theory of a likely "cell phone effect" in Obama's favor that weighting by age may not eliminate. However, the Gallup data implies a smaller effect than Pew. Of course, without seeing a weighted comparison for all voters by Gallup (something they may not have time to produce until after the election), we won't know for sure.
My NationalJournal.com column for the week looks at what we know about whether the rise in cell-phone only households is causing any skew in polling results. The short answer is that polls that cannot reach cell-phone-only voters may be slightly understating support for Obama and overstating support for McCain, although the difference is small (and likely within the margin of error of any individual poll). As the Pew Research Center's Scott Keeter recent told the Arizona Reporter (in a clip received too late to make the column), "For the first time, we’re actually seeing a difference between cell-only voters and land line voters when you take into account age."
The key takeaway from the recent Pew Report on cell phone only users was not that cell phone only respondents are different, but that even weighting landline only survey data doesn't fully account for excluding cell phone only users. Typically, a survey may be weighted for factors such as age, race, gender, education, and region. This allows pollsters to take a particular sample, and adjust it to look more like what they think the population they are interested in actually looks like. Pew found that even if you weighted a landline sample for all of these factors, that sample still provided results that were 2%-3% less favorable for Obama than one that included cell phone only users.
What exactly makes cell phone only respondents different from those with a landline? If it is simply the case that cell phone only respondents are more likely to be Democrats than those with landlines, then it should be simple enough to correct for not calling cell phones by weighting a sample by party identification. From my reading of the Pew report, they did not examine whether applying a party weight would have accounted for the exclusion of cell phone only respondents. A party weight is something that some pollsters (like Rasmussen) apply, but others do not. However, based on some recent analysis I have conducted using the 2006 Cooperative Congressional Election Study, even weighting by party is not likely to fully account for the differences between cell phone only respondents and those with landlines. There are two reasons for this.
First, it is true that cell phone only respondents are more Democratic than landline respondents. But this relationship is a little more complex than it first seems. In the 2006 survey, cell phone only respondents were just 4% more Democratic than landline respondents and they were 9% less Republican when asked a standard party identification question. However, once you factor in independents who lean towards the Democratic or Republican Party, you find that cell phone only respondents are 10% more Democratic and 12% less Republican. Thus, the party differences are larger when you factor in leaners, a pattern that results because cell phone only respondents are more likely to initially call themselves independents even though they lean Democratic. To fully capture party differences among cell phone only respondents, one would need to factor in leaners.
Second, just looking at party affiliation masks the fact that cell phone only respondents are actually quite a bit more liberal than those with a landline. 35% of cell phone only respondents classified themselves as liberals compared to just 23% of those with a landline. These ideological differences are not completely accounted for by party either. From the table below, you can see that cell phone only Democrats are 10% more liberal than those with landlines. Democratic leaners in the cell phone only sample are 15% more likely to classify themselves as liberal. And even those cell phone only independents who did not express a lean to either party were more likely to be liberal compared to their landline counterparts. Given that cell phone only Democrats and Democratic leaners are more liberal than those with landlines, they should be less likely to defect and vote Republican than landline Democrats.
Thus, this analysis suggests that differences between cell phone only users and those with landlines cannot simply be accounted for by partisanship. In fact, even when I used multivariate models controlling for a wide range of demographic and political factors (party, age, race, gender, income, education, and even religion), cell phone only respondents were still substantially more liberal than those with landlines. Cell phone only respondents are ideologically distinct in ways that cannot be accounted for by party identification or all the other standard demographic factors that pollsters may use to weight samples.
There has been a lot of discussion of the Pew Report released earlier this week that shows that including cell-phone only respondents does appear to make a 2-3% difference in the presidential preference polling (see Mark's post). What's most intriguing to me is how this would play out at the state level. Indeed, it seems very unlikely that every state has the same percentage of cell phone only households. Thus, in states with fewer cell-phone only users, the effect of excluding such respondents may have less of an effect on the poll results. On the other hand, states where there are more cell-phone only households may have polling that is further off the mark.
I was hoping to be able to easily find some survey data with enough respondents to get a sense of the prevalence of cell-phone only households in each state. Unfortunately, the 2007 CDC data that is often cited provides more than enough national interviews to accomplish this task, but the dataset hides the state of the respondent, only allowing users to place respondents in a particular Census Region. Nevertheless, we can learn a little about geographical variance from this data. Specifically, families in the South and Midwest are more likely to have cell phones only compared to states in the West and Northeast. Based on this data, we should expect polling from southern and midwestern states to be more prone to error from the cell-phone only problem than polling in other regions.
Another source of data that may be of some use in answering this question is the 2006 Cooperative Congressional Election Study. (In the interests of disclosure, I should say that I am involved in the 2008 version of this study, though I had no role in the 2006 version.) This was a large (approximately 30,000 respondents) internet survey conducted by YouGov/Polimetrix using a matched random sample design. Because this is an internet survey, it probably isn't as ideal for addressing this question as the CDC survey would be. However, the sample was stratified to assure that there would be a large enough sample from each state and since the state of the respondent is available for this data (and isn't for the CDC data), it is the one decent way I've found of breaking out cell-phone only figures by state.
In this survey, 10.6% of respondents indicated that they only had a cell-phone (this is smaller than the percentage cited in the CDC survey, though the CDC survey was conducted a year later). Most interesting is the variation across states. The map below shows this variance.
Some of the swing states that stand out as having higher than average cell-phone only users are Montana (21%), Oregon (17%), Virginia (15.7%), Wisconsin (15.3%) and Minnesota (15.1%). (Keep in mind that these figures are from two years ago). If this survey is providing reasonably accurate figures on cell-phone only users, then it may be the case that polling of these states would be particularly prone to under-stating support for Obama. It may also explain why the polling in some of these states (for example, Virginia) has been so erratic.
Of course, these data may be problematic and should be taken with some caution. If anyone has ideas about other sources that could be used to compile state-by-state measures of cell-phone only households, please let me know. If excluding cell-phone only respondents does matter, then it would be nice to have a strong sense of where it will matter most.
Few in the field of survey research have examined the problem of cell phones and surveys as closely over the last four years as Scott Keeter and his colleagues at the Pew Research Center. They conducted one of the first large-scale political surveys by cell phone two years ago and have been leaders in testing and developing new methods to conduct political surveys by combining samples of landline telephones and cell phone.
Today they released a new must-read report summarizing findings from "three major election surveys [conducted] with both cell phone and landline samples since the conclusion of the primaries." The verdict? "Pew's surveys this year suggest at least the possibility of a small bias in landline surveys."
The key details (emphasis added):
In each of the surveys, there were only small, and not statistically significant, differences between presidential horserace estimates based on the combined interviews and estimates based on the landline surveys only. Yet a virtually identical pattern is seen across all three surveys: In each case, including cell phone interviews resulted in slightly more support for Obama and slightly less for McCain, a consistent difference of two-to-three points in the margin.
Pollsters have long understood that the cell phone only population -- those who have cell phone but no landline telephone service -- tend to be younger, and that the growth of that population has made it more difficult to reach 18-29-year olds. However, the conventional wisdom among pollsters has held that weighting by age could mostly alleviate any potential bias, as they did they did in 2004.
The new Pew report shows why weighting by age may not have the same effect now:
Traditional landline surveys are typically weighted to compensate for age and other demographic differences, but the process depends on the assumption that the people reached over landlines are similar politically to their cell-only counterparts. These surveys suggest that this assumption is increasingly questionable, particularly among younger people. [...]
In the pooled [August-September] data, cell-only young people are considerably less likely than young people reached by landline to identify with or lean to the Republican Party, and even less likely to say they support John McCain. Among landline respondents under age 30, there is an 18-point gap in party identification - 54% identify or lean Democratic while 36% are Republican. Among the cell-only respondents under age 30, there is a 34-point gap - 62% are Democrats, 28% Republican. The difference among registered voters on the horserace is similar: 39% of registered voters under 30 reached by landline favor McCain, compared with just 27% of cell-only respondents. Obama is backed by 52% of landline respondents under 30, compared with 62% of the cell-only.
The roughly two-to-three point difference in the margin favoring Obama is, as it happens, very close to the effect Nate Silver obtained over the weekend by comparing results from pollsters that have been interviewing by cell phone (including Pew) with a control group that has not.
Finally, the Pew Report has a bit of a bonus with implications for the ongoing debate over how to model "likely voters" this year:
While 18-29-year-olds reached by cell phone tend to have less experience voting than their landline counterparts, they are just as interested in the 2008 campaign, and express just as much intention to vote this year.
We have followed the challenge posed to survey for the last four years, both here at Pollster and at its forerunner, my old blog MysteryPollster. Over the last four years, survey researchers have been developing techniques for interviewing respondents on their cell phone, and over the last few months, many of the well known national media surveys have been including samples of voters contacted on their cell phones in their national samples or conducting side-by-side tests. These include the Pew Research Center, Gallup (both the Gallup Daily and USA Today/Gallup surveys), CBS/New York Times, Time/SRBI and most recently NBC/Wall Street Journal, ABC/Washington Post and the new AP/GfK poll.
Over the last two weeks, some of these pollsters have provided updates on the impact of their cell phone samples (or lack thereof):
ABC News polling director Gary Langer today describes their cell phone interviewing test in a new blog post today and describes the impact on the overall results as "negligible....The precise changes were 0 for Obama and -1 for McCain among registered voters, +0.7 for Obama and -0.8 for McCain among likely voters." These differences fell well within the survey's margin of error.
NBC's First Read included this line in their recap of the latest NBC/WSJ poll: "[T]he poll included some cellphone surveys (we found no significant difference in cell phone respondents as we have from landline respondents." More details on the cell phone sample at the end of the filled-in questionnaire provided by the Wall Street Journal.
Keep in mind that these are relatively small scale tests, in which the margins of error for both the base land-line sample and the supplemental cell-phone test samples are probably larger than any likely effect. Gallup and the Pew Research Center have released similar tests based on larger samples that suggest a small benefit (perhaps 2 to 3 points on the margin) benefiting Barack Obama from the inclusion of cell phone only interviewing.
We will definitely have more to say on this subject in the weeks ahead. Those looking for all the gory details, on this subject may want to start with my series from last year on cell phones and political surveys (Part I and Part II).
The Pew Research Center, which has been at the forefront of efforts to measure the impact of "cell phone only" households on political surveys, has a new report out on subject today. Like Gallup they have found evidence that including interviews of cell-only Americans as a "modest affect" on results in the presidential race:
Pollsters are continuing to monitor changes in telephone use by the U.S. public, since most surveys are still conducted using only landline telephones. Growing numbers of Americans are reachable only by cell phone, and an even larger number who have both a landline and a cell phone may be "functionally cell-only" because of their phone use habits. The latest Pew Research Center national survey, conducted June 18-29 with a sample of 2,004 adults including 503 on a cell phone, finds that the overall estimate of voter presidential preference is modestly affected by whether or not the cell phone respondents are included. Obama holds a 48% to 40% lead in the sample that includes cell phones, and a 46% to 41% advantage in the landline sample. Estimates of congressional vote are the same in the landline and combined samples. [Emphasis added].
The numbers noted above are based on interviews with registered voters. When they narrow the universe to more likely voters, however, the difference [mostly] disappears:
Narrowing the analysis to voters who are certain about their vote choice, there is almost no difference between the landline and combined samples: Obama has a 38%-28% advantage in the combined sample, while the margin is 38%-30% in the landline sample.
For more detail on the challenge of cell-phone only households to political polling, see my two-part series last year, as well as any of our more recent posts on the subject.
[I added the word "mostly" to my second paragraph based on comments below. The report does not speak to the statistical significance of either set of numbers].
"Numbers Guy" Carl Bialik has a nice round-up of the latest developments in the "steady displacement of landline usage by cellphones" that is pushing some pollsters to "try to reach Americans on their cell phones."
Carl noticed something I had overlooked, namely that the most recent CBS/New York Times poll included a supplemental sample of cell phone numbers. Unlike Gallup, they do not yet see an impact on the results from the greater sample coverage:
Kathy Frankovic, director of surveys for CBS News, told me that “I haven’t seen any great difference in results, but it is still early.”
Frankovic confirms that all future national CBS polls will include a supplemental cell phone sample. In an email, she notes that they "have been working on incorporating a cell phone sample into our polls since late last year" and reminds us they incorporated a cell phone sample into their Iowa poll last year.
For those keeping track, that means that three national survey organizations -- Gallup, The Pew Research Center and CBS/New York Times -- now routinely include supplemental samples of cell phone numbers as part of their national political surveys. More will presumably follow.
The trend toward cell phone sampling raises a special challenge for the pollsters using the automated Interactive Voice Response (IVR) method in statewide surveys, because of the one regulatory barrier that affects cell phone interviewing. The federal Telephone Consumer Protection Act (TCPA) bans any sort of unsolicited call to a cell phone using "automated dialing devices." As Bialik points out, some pollsters are pushing for a change in that regulation:
The Council for Marketing and Opinion Research, a lobbying group for the survey industry based in Washington, D.C., is pushing Congress to exempt pollsters from the auto-dialer ban. LaToya Lang, the state legislative director for the group, calls this issue a “high priority.”
Bialik also highlights another bit of cell phone survey news I neglected to pass along from the AAPOR conference:
Recognizing that cellphone surveying is on the rise, last month the American Association for Public Opinion Research, a professional group, released a report offering guidelines for the practice. The report doesn’t call these “standards,” because more research is needed to determine how to conduct surveys via cellphone and how to blend the results with landline interviews.
After last year's AAPOR conference, I wrote a two-part series summarizing both the challenge to political surveys from the growth of cell-phone-only households and the experimental approaches pollsters are using to conduct interviews on cell phones. My interviews at this year's conference include a update from Steven Blumberg on the most recent data from the National Center for Health Statistics (NCHS) and a report from Gallup's Jeff Jones comparing results on the presidential race with and without the supplemental cell phone interviews.
I'm blogging from an airport with just enough time to like and block quote two helpful explanations of what the missing "cell phone only" households mean to the vast majority of surveys that do not attempt to interview Americans on their cell phones.
First, Wall Street Journal "Numbers Guy" Carl Bialik has a very clear, concise review of the central issues. He draws on conversations with both Jeff Jones of Gallup and Scott Keeter of the Pew Research Center, the two organizations that have done the most significant political surveys of Americans over their cell phones. It is well worth reading in full if you are new to this issue. Bialik stresses that the evidence so far shows that the missing "cell onlys" have little impact on political survey results:
The impact of losing cellphone-only respondents, however, may be exaggerated. Their numbers aren't big enough to budge most poll results by more than a point or two, Gallup has found.
People who use only cellphones, on average, are younger, more likely to rent their homes and have lower incomes than their tethered-telephone peers. But once you adjust for age, cellphone-only users have similar political viewpoints. Although he thinks cellphones should be included, Jeffrey M. Jones, managing editor of the Gallup Poll, asks, "It's still a lot of cost and effort, and what's the payoff?"
Meanwhile the Pew Research Center yesterday released another mega-study of the cell-phone-only issue. It's a must read for those who want all the nitty-gritty details. Here's the bottom line:
On key political measures such as presidential approval, Iraq policy, presidential primary voter preference, and party affiliation, respondents reached on cell phones hold attitudes that are very similar to those reached on landline telephones. Analysis of two separate nationwide studies shows that including interviews conducted by cell phone does not substantially change any key survey findings.
Update: Carl Bialik has much more on his blog, and also reminds me to link to the recent Public Opinion Quarterly special issue on cell phone surveys and my own series from last summer on the cell phones and political surveys (Part I and Part II).
One interesting new wrinkle on the latest USA Today/Gallup survey. Editor in chief Frank Newport tells us on his Gallup Guru blog that, starting with this survey, Gallup will regularly sample cell phones:
[A]s of Jan. 1, 2008, Gallup has made the decision to include cell phone interviewing as part of the sample used for its general population studies.
This is a complex and costly modification in methodology. Our statisticians and methodologists have spent a great deal of time reviewing the procedures and implications of the change. Essentially, in addition to sampling from the traditional database of all landline telephone exchanges, Gallup now also adds in sampling from a new database of all cell phone telephone exchanges in the country. We screen for those individuals using cell phones who report not having a landline, and then interview a random sample thereof. We then weigh into the sample a proportionate percentage of these interviews conducted via cell phone.
For now, at least, this change is not likely to produce dramatic differences in the results. The ongoing cell phone surveys conducted by the Pew Research Center have shown that the missing cell-phone-only population rarely makes a difference of more than a point or two. But that point or two may sometimes make a difference, especially in a close race. Consider last week's Gallup poll in New Hampshire. USA Today polling editor Jim Norman let us know, via email, that they included a cell-phone sample on that survey:
[I]t added a point to Obama's total and took one away from Clinton. In other words, without the cell-phone-only respondents, Obama's lead among likely voters was 11, not 13.
The bigger significance in this change is symbolic. Gallup is the granddaddy of all polling firms. Their polling "time series" goes back to the 1930s. As such, they are typically the most cautious about changes in methodology, so their move to regular cell-phone sampling is likely to have a big ripple effect on the polling industry. At very least, this most closely watched poll will provide a regular source of data on the potential impact of the cell-phone-only households that will be missing from other surveys.
Norman also sent a long the actual text of questions used to identify and screen for cell-phone-only households. I have posted it after the jump.
Yesterday, the National Center for Health Statistics (NCHS) today released its biannual update on the still growing number of Americans living in households without a landline telephone:
Preliminary results from the January–June 2007 National Health Interview Survey (NHIS) indicate that more than one out of every eight American homes (13.6%) had only wireless telephones during the first half of 2007.
The NCHS is an invaluable source of data on the growing mobile-phone only population because it conducts more than 13,000 in-person interviews every six months that reach all Americans, regardless of their telephone service (or the lack thereof). The Centers for Disease Control (of which NCHS is a part) is concerned about the growth of "cell phone only" households because of the massive ongoing health "surveillance" studies it conducts via telephone.
When we last checked in with the NCHS data, the rate of growth appeared to be increasing, as evident in the upturn in the trend in the chart above from July 2005 through December 2006. That rate of growth appears to have slowed on the most recent release, but the NCHS report explains that the change may be due to a changes in the NCHS questionnaire:
The observed increase in the percentage of adults living in wireless-only households from the last 6 months of 2006 to the first 6 months of 2007 was not statistically significant. Other observed increases over time in the percentage of adults living in wireless-only households were statistically significant. These results suggest a possible recent decline in the rate of increase. However, questionnaire changes in 2007 could have contributed to the observed decline. Therefore, conclusions about trends cannot be made until data from the last 6 months of 2007 are released in May 2008.
The full NCHS report has details on those questionnaire changes.
AP also published concise summary yesterday of the findings from NCHS Report. For more information on the impact of the growing "cell phone only" population on political surveys, see my two-part series earlier this year.
ponders the subgroup shifts in the Democratic presidential race found in the
latest Pew Research Center
Bialik questions (appropriately) another Zogby online survey about another
College senior Mara
Gordon considers the implications of cell-phone only users excluded from
And speaking of cell-phone-only
households: The recent Pew Research Center survey used a combined sample of respondents interviewed by landline phone and
cell-phone. They promise "a
detailed analysis of the landline and cell phone samples in the coming weeks."
The biannual health study, which
provides data on more than 42,000 of the state's roughly 12 million households,
is used to "drive decision-making and to drive the recommendations we make
to the Legislature or to the governor," said Sandra Shewry, director of
the California Department of Health Care Services.
If data are inaccurate, a survey paints a nonrepresentative picture of
populations, which can in turn be used as the basis for an improper allocation
of funds or just bad decisions, she said.
"The implications are that if we do not include people with cellphones
only, then we are likely to be underreporting un-insurance rates, and we're
likely to underreport smoking prevalence rates in the state," survey
director David Grant said. The numbers won't be dramatic, he said, "but it
will introduce some level of bias in our estimates.
Two weeks ago, I took a long look at the cell-phone
only problem and whether the absence of those without landline phones is
affecting survey results. Today, I want to conclude with a look at how
pollsters conduct surveys via cell phone. Like Part I, this article is long,
even by Pollster.com standards. So it continues after the jump.
An intriguing footnote to the first
part of my post on the cell-phone-only problem (alas, a shortened yet
crowded week has pushed Part II until next week). The bottom line is that even
at 12% of adults, the cell-phone-only population appears neither large nor distinctive
enough to throw off most political survey results by more than a point or two. And
while that conclusion may not change drastically should the cell-phone only
population double over the next year or two, all bets if "cell phone only" comes
to describe the majority of U.S.
households (a point reader Chris G made in the comments).
Could that happen? An article
yesterday by The New York Times' technology
writer, David Pogue, suggests a potential pathway. Last week, the cell-phone
carrier T-Mobile announced a new service called T-Mobile HotSpot @Home,
something Pogue described as an "absolutely ingenious" and as potentially "game
changing" to the technology world as Apple's iPhone. "It could save you
hundreds or thousands of dollars a year," he wrote, "and yet enrich T-Mobile at
the same time." How?
Here's the basic idea. If you're willing to pay $10
a month on top of a regular T-Mobile voice plan, you get a special cellphone.
When you're out and about, it works like any other phone; calls eat up your
monthly minutes as usual.
But when it's in a Wi-Fi wireless Internet hot spot, this phone offers a huge
bargain: all your calls are free. You use it and dial it the same as always -
you still get call hold, caller ID, three-way calling and all the other
features - but now your voice is carried by the Internet rather than the
These phones hand off your calls from Wi-Fi network
to cell network seamlessly and automatically, without a single crackle or pop
to punctuate the switch.
And what does this have to do with the cell-phone only
problem? Read on...
O.K., but how often are you in a Wi-Fi hot spot?
With this plan, about 14 hours a day. T-Mobile gives you a wireless router
(transmitter) for your house - also free, after a $50 rebate. Connect it to
your high-speed Internet modem, and in about a minute, you've got a wireless
home network. Your computer can use it to surf the Web wirelessly - and now all
of your home phone calls are free.
You know how people never seem to have good phone
reception in their homes? How they have to huddle next to a window to make
calls? That's all over now. The free router is like a little T-Mobile cell
tower right in your house.
Pogue goes on to explain that HotSpot @Home will work with essentially any
existing Wi-Fi (802.11b/g) router. What could this mean for the cell-phone-only
problem? In outlining four ways this product can save consumers money, Pogue
does everything but connect the dots:
SAVING NO. 4 T-Mobile's
hope is that you'll cancel your home phone line altogether. You'll be all cellphone,
all the time. And why not, since you'll now get great cell reception at home
and have only one phone number and voicemail? Ka-ching: there's an additional
$500 a year saved.
While the new Apple iPhone, which went on sale last week, does not aim
to replace home phone service, it does provide a very similar hand-off from the
AT&T wireless network to home or office Wi-Fi hotspots for its built in
Internet connection. If these features prove popular with consumers, if HotSpot@Home
"enriches" T-Mobile as Pogue speculates, then other cell-phone carriers (with
the possible exception of Verizon) are sure to offer similar services.
And if that happens, pollsters may look back with
great nostalgia on the days when the cell-phone-only population was just 12%.
UPDATE: A very alert survey researcher emails and notes that the Pogue column is "the #1 most e-mailed story on the New York Times web site today!"
Since the AAPOR conference back in May, I have promised severaltimes
to take a closer look at the challenge that the growth of "cell phone only"
households are presenting to political polling. Today, finally, I am posting
the first of a two-part review of some of the latest research.
As this item is a bit long even by Pollster.com standards,
it continues in full after the jump.
During last month's conference of the American Association
for Public Opinion Research (AAPOR), I mentioned
the many papers being presented on the growth of cell-phone-only households and
promised to report back. I have been digging through the many papers ever
since, and have been working on some analysis on the subject which, in the
crush of new charts and other activity around here, I have yet to post.
Fortunately, Scott Keeter, the director of survey research
at the Pew Research Center,
has posted a concise
and accessible review of the cell phone challenge to surveys. The Pew Center
has been at the forefront of research and development on this subject,
conducting four pilot studies over the last two years that interviewed people
with cell phones over their cell phones.
The summary is well worth reading in full, but for those in
a rush, here is Keeter's view of where things are heading:
Pollsters recognize that some type
of accommodation for the cell-only population will have to be made eventually,
as was clear from the large amount of research on the topic presented at the
AAPOR conference last month. In addition to the use of so-called "dual
frame samples" such as those described above (calling both a cell phone
sample and a landline sample), practitioners are discussing other alternatives,
including the establishment of panels of cell-only respondents that can be
surveyed periodically to track their opinions, and employing mail or internet
surveys to reach the cell-only population.
For those who want more detail, I can also highly recommend
the longer paper he
presented at the AAPOR conference (co-authored with Courtney Kennedy and April
Clark of the Pew Center and Trevor Thompson and Mike
Mokrzycki of the Associated Press) which Keeter has now posted online. Of all
the papers I have reviewed, it was easily the best review of the issues most
relevant to the political surveys we all obsess over.
On Monday, I linked
to a new report
from the National
Center for Health Statistics
(NCHS) showing that 12.8% of American households had only wireless phones (and
no land line phones) during the second half of 2006. "I don't know how this
impacts traditional polling techniques," MyDD's Matt Stoller noted on Tuesday, "but
I am curious."
Well, funny he should ask. The released of the NCHS report
was timed, in part, to coincide with the annual conference of
the American Association for Public Opinion Research (AAPOR) which I am attending in Anaheim, California.
One thing I can report is that the community of survey researchers continues to
take the trend toward wireless only households very seriously (as they have at
AAPOR conferences for the last four or five years).
Today, Stephen Steven Blumberg of NCHS presented his findings in
greater detail to a standing room audience of academic and professional survey
researchers at the conference. Their report is one of twenty-two research
papers, plus a panel discussion and a mini-course being presented here just on
the subject of mobile phones and their impact on polling. Many of these papers
describe pilot studies involving interviews conducted with respondents on their cell phones.
We often say that political polling is a mix of both science
and art. One of the things I appreciate most about these AAPOR meetings is the exposure
I get to the true science of this profession, which as of late has focused on
the issue of how to conduct surveys on cell phones. One thing that many here I
spoke with seem to agree on is that by 2008, many of the national news media pollsters
are considering adding some "cell phone component" to their surveys.
I am certain I will have more to report in the weeks
following the conference, although as a result of the conference, my blogging
has been light this week. For better or worse, I wear many hats at AAPOR: I
serve on the organization's executive council, I will be presenting a paper
with Charles Franklin tomorrow (which we hope to roll out on the blog next
week) and I try to attend as many sessions as possible and absorb all that the
conference has to offer.
Center for Health
Statistics (NCHS) today released another of its regular updates on the ever
growing number of Americans living in households without
landline telephones (hat tip to alert reader BS):
During the last 6 months of 2006, more than 3 out of every
20 American homes (15.8%) did not have a landline telephone. Of those homes
without a landline telephone, most had at least one working wireless telephone.
Preliminary results from NHIS suggest that more than one out of every eight
American homes (at least 12.8%) had only wireless telephones during the second
half of 2006. These are the most up-to-date estimates available from the
federal government concerning the size of this population.
The ongoing NCHS survey, which involves roughly 13,000 interviews
every six months, conducts its interviews in person. Thus, it potentially
reaches all Americans, regardless of phone service. As the following chart
shows, the upward trend in wireless only households shows no signs of slowing:
For thoughts on what this trend may mean for the telephone
surveys we all obsess over, see my previous commentary here and here.
Yesterday, the National
Center for Health
Statistics (NCHS) released its latest estimates of the number of Americans
living in households without landline telephones, as well as a statistic
closely watched by pollsters: During the first six months of 2006, "approximately
10.5 percent of households do not have a traditional landline telephone, but do
have at least one wireless telephone."
Pollsters have been watching the growth in "cell phone only" households
because cell phones are largely out of reach of the traditional random digit
dial sampling methods used in most conventional telephone surveys. As such, the continuing upward tend in such
households illustrated by the NCHS surveys (which involve massive monthly in-person
samples of Americans), should be of great interest to anyone who follows public
Although I have written about these issues previously (here,
the best analysis of how this trend has affected the accuracy of public polling
has been done by the Pew Research Center.
Last year, in partnership with the
Associated Press and America Online, they conducted parallel surveys: One
using conventional telephone sampling and another that interviewed a random
sample of 750 mobile phone users over their mobile phones. The study produced a report by the Pew Research
Center (available in
or PDF format)
reached the following conclusions:
[Cell only Americans] are younger, less affluent,
less likely to be married or to own their home, and more liberal on many
Yet despite these differences, the
absence of this group from traditional telephone surveys has only a minimal
impact on the results. Specifically, the study shows that including cell-only
respondents with those interviewed from a standard landline sample, and weighting
the resulting combined sample to the full U.S. public demographically, changes
the overall results of the poll by no more than one percentage point on any of
nine key political questions included in the study.
Of course, given the trend reported by NCHS, the cell-phone only adult population appears to be a moving target. It has more than doubled in the last two
years, and the trend shows no signs of slowing.
How big a problem will cell-phone only households be in 2008? Will their absence from traditional phone surveys begin to impact results? Will pollsters begin to routinely incorporate more expensive cell phone samples into their surveys? Time will tell.