May 12, 2009
By Brian Schaffner
A few weeks ago, I highlighted some preliminary findings from a paper written by myself and Stephen Ansolabehere for this week's AAPOR conference. The paper is now finished and you can check out a copy here. The data we use for the paper is the 2006 and 2008 Cooperative Congressional Election Study.
One of the major themes in the paper is that understanding the cell-only population is about more than just age. In fact, residential mobility has a strong influence on whether someone has shed their landline. Even after controlling for age and a litany of other demographic variables, we find that respondents who moved within the last year were 24 percentage points more likely to be cell only than those who had lived in the same residence for at least five years. Renters, singles, and those without children were also much more likely to be cell-only.
Our explanation for this pattern:
"There are several reasons that highly mobile Americans may be more likely to go without landlines. First, whenever someone moves from one residence to another, they have an opportunity to reassess their phone needs. Thus, the act of moving provides an opportunity for individuals to shed their landlines. Second, mobile Americans may choose a CPO lifestyle because cell phone numbers tend to be more portable than landlines. When moving from one metropolitan area to another, individuals must change their landline phone number, but do not need to change their cell number. This may provide an incentive for choosing not to maintain a landline in a new residence. Third, those with fewer family and community ties may feel less of a need to have multiple phone lines on which they can be reached by members of their social networks. "
The fact that the cell-only public tends to be more mobile has some important political consequences. Some highlights:
The difference in the percentage of landline and cell-only respondents who reported being registered was fairly small--over 95% in both groups. However, there was a much larger gap in actual registration rates (66.8% versus 53.9%). Since cell-onlys are more likely to have moved recently, they may not have successfully registered to vote at their new addresses despite the fact that they may think they are registered.
Cell-only respondents were significantly more likely to have problems with their registration when attempting to vote. In 2008, over 7% of cell-only respondents indicated that there was a problem with their registration when they attempted to vote, compared to fewer than 4% of respondents with landlines.
Cell-only respondents were more than twice as likely as those with landlines to report that neither campaign contacted them. In short, this group is much less likely to be subjected to mobilization efforts from the campaigns.
Cell-onlys are politically distinct on a variety of measures. However, this distinctiveness is somewhat muted when demographic controls are taken into account. Interestingly, the largest differences between cell-only and landline respondents are not on issues or ideological self-placement, but on reported vote choices.
Ultimately, we argue that weighting for standard demographic measures such as age, education, income, and race may not be sufficient. Pollsters relying on landline samples may want to consider weighting by other factors such as time in residency, renter/home owner, and marital status. But check out the full paper for a more detailed discussion of all of these points.
By Brian Schaffner on May 12, 2009 10:14 PM
| Permalink
| Comments (0)
| TrackBacks (0)
May 6, 2009
By Mark Blumenthal
Has it been another six months already?**
Today the National Center for Health Statistics (NCHS) of the Centers for Disease Control and Prevention (CDC) released its latest biannual report on the prevalence of households without wireless or standard telephone service. This latest, which covers the latter half of 2008, shows the trend toward "cell phone only" households continuing unabated as hat 18.4% of adults were reachable only by cell phone, while another 1.7% lacked telephone service of any kind.
A refresher for those unfamiliar: CDC monitors the cell-phone-only population because it conducts huge ongoing health "surveillance" surveys via telephone, and as such, asks questions about telephone usage on their ongoing, in-person National Health Interview Survey (NHIS). Traditionally, telephone surveys have relied exclusively on random digit dial (RDD) samples that reach only those with landline phone service. These regular CDC estimates are a big reason why most national media polls are now including samples of cell phone users.
The NCHS estimate of the percentage of American households with only wireless phones increased 2.7 percentage points (from 17.5% to 20.2%), amounting to "the largest 6-month increase observed since NHIS began collecting data on wireless-only households in 2003," according to the report. They also note a big jump in what some call "cell-phone mostlys. "One of every seven American homes (14.5%) received all or almost all calls on wireless telephones, despite having a landline telephone in the home."
This latest report also includes a new feature: A chart with regression lines showing the growth in wireless-only households by age and by year. The chart makes clear that while the wireless-only population remains disproportionately younger, it is also growing rapidly among Americans over 30 years of age as well.
For further reading: More on the latest report from Carl Bialik. And see this link for all of our recent coverage involving cell phones and surveys.
**One reason why it seems like less than six months. In March, NCHS released a supplemental report that provided wireless-only estimates for all 50 states (our summary here).
By Mark Blumenthal on May 6, 2009 2:33 PM
| Permalink
| Comments (0)
| TrackBacks (0)
April 27, 2009
By Brian Schaffner
The survey research community is focusing intently on the challenges posed by the fast-growing share of Americans who are cell-phone-onlys (CPOs). In fact, there are 40 papers being presented on the topic at the AAPOR conference next month. One of the practical issues faced by pollsters is whether the cost of reaching CPOs is worth the payoff. Last week, Scott Keeter, Mike Dimock, and Leah Christian hosted a forum at Pew during which they discussed this tradeoff. But pollsters aren't the only people who have to make cost-benefit decisions when it comes to deciding whether to attempt to contact CPOs. Campaign organizations must make the same calculation.
So how well did the campaigns do at contacting CPOs during the 2008 campaign? The chart below compares the percentage of those with landlines and cell-onlys who reported being contacted by a campaign representative in 2008. The data comes from the National Election Study (NES), which uses residential sampling and face-to-face interviews to interview both landline and CPO respondents. In the chart below, the blue bars show the percentage of each group that reported being contacted while the black lines represent 95% confidence intervals for these percentages.
The chart shows that CPOs were much less likely to be contacted by the campaigns than people with landlines. Over half of landline respondents reported being contacted compared to less than one-in-three CPOs. This sizable difference holds up even when controlling for age, income, education, partisanship, and a variety of other factors.
The next chart (below) indicates that for those CPOs who were contacted, the contact tended to come overwhelmingly from Democrats. Over 80% of CPOs who were reached by the campaigns were contacted by the Democratic side while just a little over one-third were reached by Republicans. Republicans were significantly more competitive with Democrats when it came to contacting those with landlines.
Unfortunately, the NES did not include questions asking respondents how they were contacted by the campaigns. But a subset of respondents to the 2008 Cooperative Congressional Election Study (which I've analyzed in previous posts) were asked these questions. The chart below plots the responses for those who had landlines compared to CPOs.
CPOs who were contacted by one of the campaigns were significantly less likely to have had that contact over the phone compared to those with landlines. Otherwise, there were not major differences between how landline and CPO respondents were contacted. CPOs were somewhat more likely to get an email while those with landlines were a bit more likely to receive snail mail, but neither of these differences are large. The percentage being contacted in-person or by text message were nearly identical for both groups.
Overall, the findings from these surveys suggest that shedding your landline may help you avoid those pesky campaign calls in future election years. While Democrats were a little more successful than Republicans in reaching CPOs, the cell-only crowd was almost as successful avoiding campaign volunteers as they were hiding from pollsters.
By Brian Schaffner on April 27, 2009 10:08 AM
| Permalink
| Comments (0)
| TrackBacks (0)
April 21, 2009
By Mark Blumenthal
I attended a presentation last week at the Pew Research Center (sponsored by the DC AAPOR chapter) on some of the practical issues they have encountered in their innovative work on cell phone polling. I'm still catching up from a few hectic days that have followed but want to pass along a few interesting details they shared.
Most of what was new in the session will be of more interest to pollsters than to political junkies wondering about how pollsters are dealing with the growing number of Americans without landline phone service. Fortunately, for those of you in the latter category, the PRC shared most of their more general data obtained from calling cell phones during the 2008 campaign in a report released this past December (see their summary, full report pdf and our review).
Here are a few highlights that seemed especially noteworthy or new in the presentation by Pew's survey research director Scott Keeter, associate director Michael Dimock and research associate Leah Christian:
- Pew has now conducted 18 surveys (14 in 2008 and and 4 in 2009) featured a "dual frame" sample of both landline and mobile phones. In those surveys, they have interviewed approximately 9,400 adults via cell phone. Keeter explained that these dual frame samples are now "standard policy" for Pew's political surveys.
- They have found cell phone users just as willing to answer their phones and cooperate as landline users. Their response rate for cell phones (23%) is virtually the same as that for landlines (24%)
- Pew's calling center is finding it "cheaper and easier" to interview by cell phone. Just a few months ago, Keeter reported that it cost Pew "two to two and half times" as much for cell phone interviews as landline. Now the cell phone costs are "closer to two times as expensive" as landline interviews. Keeter attributed the improvement to their call centers "getting more familiar with the tricks of doing successful cell phone interviewing." Note that Pew interviews all cell phone users, and weights down respondents less likely to be "cell phone only." If they had screened for just the cell phone only users, the differential would be closer to four times the cost of landline interviews.
- Pew continues to offer a $10 per interview incentive to cell phone respondents although, according to Keeter, other pollsters such as Gallup do not pay incentives to the cell phone respondents.
- The biggest continuing methodological challenge to pollsters is determining how to combine and weight their landline and cell phone samples, partly because of what some are now calling the "cell phone mostly" problem. This issues involves those who have both mobile and landline phones but make "all" or "almost all" of their calls on the mobile phone (see pp. 6-8 from the December report, and my previous discussion of the issue for more details).
In this presentation, Dimock presented results showing that the cell phone mostly respondents were less likely to say they could be reached "right now" on a landline phone (52%) than those who use their cell phones for only some or few calls (63%). "This is really a spectrum," Dimock explained, without "a clear cut line between the group of people who absolutely can't be reached on a landline and another group who absolutely can't," but rather "a probability across the spectrum of people." As such, Pew's approach is to interview everyone reached via cell phone and weight on the inverse of their probability of selection.
- Christian presented data on the problems identifying the actual geography of respondents based on their telephone number. Before the widespread adoption of cell phones, telephone area codes could identify the state and time zone of each number with great accuracy, as landline telephone numbers are closely associated with geography. Cell phone numbers, on the other hand, are assigned based on the geography where you first purchased your cell phone. The relatively new ability to "transport" cell phone numbers from one carrier is creating a growing mismatch between phone numbers and geography.
Pew has been able to confirm and quantify that trend by asking respondents to provide their zip codes. That additional data shows that a 5% mismatch for their cell phone samples on region (presumably census region), a 9% mismatch on state and a 39% mismatch on county. This discrepancy has two practical implications: First, obviously, when interviewing via cell phone, pollsters cannot treat phone numbers as an accurate measure of geography, especially at the county level. Second, they have to be careful about scheduling call time based on area code. A few years ago, pollsters could safely dial west coast area codes until 9:00 p.m. Pacific time. Now it is all to easy to ring someone much later at night, so pollsters have had to modify their procedures.
A hat tip and thanks to Susanna Fox of the Pew Internet Project and our friend Alex Lundry, for their helpful notes posted notes during the session on Twitter
By Mark Blumenthal on April 21, 2009 3:35 PM
| Permalink
| Comments (0)
| TrackBacks (0)
April 16, 2009
By Brian Schaffner
Steve Ansolabehere and I have been working over the past few weeks on a paper we are writing for the AAPOR conference next month. Over the next couple of weeks, I'll share some of our preliminary findings here and I wanted to lead off today by presenting some comparative data we put together from three different surveys with distinct approaches to sampling.
The National Election Study, which has been around since 1948, is a labor- and time-intensive survey that uses a residential sampling approach and conducts face-to-face interviews. The Cooperative Congressional Election Study is an web-based survey conducted by YouGov/Polimetrix. The respondents opt-in to the study and Polimetrix uses a sample matching methodology where they first select a random target sample and then attempt to find a match for each respondent in their pool of opt-in respondents. Finally, Pew's survey work is fairly well known to regular Pollster.com readers. Their efforts to incorporate CPOs into their samples has relied on a dual-frame approach where they randomly select landlines and then cellular lines to build their sample.
So what happens when we compare these surveys, each of which takes a different sampling approach? Each survey I'll present data on here was conducted during the presidential campaign last year. For starters, the chart below compares each survey's estimate of the size of the cell phone only population after sample weights are applied. Note that Pew includes phone status as one of their weighting criteria, but the NES and CCES do not.
CPOs make up 19.7% of the CCES sample after weighting; 17.9% of the Pew sample and 17.3% of the NES sample are CPOs (again, after weights are applied). Thus, the differences are relatively small. Also note that the 95% confidence intervals for each estimate (represented by the darker lines) overlap.
So each of the surveys provides a similar estimate of the size of the CPO population, but what about the composition of that group? The table below compares landline and CPO respondents on a wide array of demographic and socioeconomic factors.
There is a lot of information in the table, but the different surveys are fairly similar across most measures. A few differences do stand out, however. After weighting, the CPO respondents reached by Pew and CCES were less likely to have children, less likely to be married, and less likely to be home-owners than those reached by the NES. Pew's CPOs also tended to have lower incomes and were somewhat more likely to be racial/ethnic minorities. Perhaps NES's face-to-face approach is more likely to pick up CPOs who have "settled down" relative to CCES or Pew.
Speaking of "settling down," one other item deserves attention; that is the information on residential mobility at the bottom of the table. While we weren't able to find this information in any of Pew's surveys conducted in October or November, the data were available for the CCES and NES. Not surprisingly, residential mobility is strongly related to whether one is a CPO or not. Even controlling for age, people who have moved recently are much more likely to have shed their landline and gone with just their cell. In fact, this relationship holds up when you control for all of the other demographic variables in the table. This is something I'll post more about later, but you can probably imagine that there are some significant political consequences arising from the fact that CPOs tend to move around much more frequently.
By Brian Schaffner on April 16, 2009 9:42 AM
| Permalink
| Comments (0)
| TrackBacks (0)
March 11, 2009
By Mark Blumenthal
Oklahoma and DC have the highest percentage of adults living in "cell phone only" households in the nation (25.1% and 25.4% respectively) according to a new report just issued by the Centers for Disease Control (CDC). The report, issued by CDC's National Center for Health Statistics (NCHS) provides estimates of the wireless-only percentage of households and adults for all 50 states and the District of Columbia (which reports a wireless-only percentage even higher than Oklahoma). The states with the lowest number of cell-phone only adults are Delaware (4.0%), Vermont (4.6%) and Connecticut (4.8%).
Why does the CDC care about cell phone usage? As regular readers know (and as the report explains), CDC conducts a variety of critical health surveillance surveys via telephone, including the including the Behavioral Risk Factor Surveillance System, the National Immunization Survey, and the State and Local Area Integrated Telephone Survey. They need to monitor the growth of the cell-phone-only population in order to check for potential bias in their telephone surveys and guide any transition to cell phone interviewing.
Five years ago, survey methodologists at CDC started asking questions about telephone usage on the on-going National Health Interview Survey (NHIS). Since NHIS is an in-person survey with a massive sample (interviewing roughly 13,000 Americans every six months), it can provide very precise, national estimates of the cell-phone-only population. The semi-annual estimates of the cell-phone only population from NCHS have become the most anticipated numbers in survey research. Their estimates are critically important to any survey -- including most national media polls last fall -- that interviewed respondents over their cell phones in order to reach the cell-phone only population. Pollsters use the NCHS estimates to determine how to weight their combined landline, cell phone samples.
Until now, however, NCHS has not produced state level estimates of the cell phone only population because "the sample size of NHIS is insufficient for direct reliable annual estimates for most states."
How did they get around that limit? They first used the NHIS data to create statistical models (using logistic regression) that estimate the cell-phone-only percentages using demographic variables (including gender, age, race, education, household size, home ownership, employment and poverty status). They then applied their models to a much larger set of survey data collected as part of the Current Population Study conducted by the U.S. Census Bureau to create "modeled estimates" of the cell phone only population in each state (the first table in their report). The report also includes the "direct state-level estimates" (the results of the NHIS survey interviews in each state).
Like the national NCHS estimates, these new state level statistics will be closely watched -- and used -- by pollsters and survey researchers of all varieties.
PS: See also the reports today by AP's Mike Mokrzycki, The Washington Post's Jon Cohen Jennifer Agiesta and the Wall Street Journal's Carl Bialik. Our colleague Brian Schaffner pondered the issue of cell phone only households by state last September. We have written a lot on the subject of cell-phone-only households and surveys, including a NationalJournal.com column in October.
Update: Pollster reader Joran sends in two charts he quickly created that plot the state-level NCHS estimates of the wireless only percentages of households and adults, complete with error bars (see the report for details on these not-quite-confidence intervals).
By Mark Blumenthal on March 11, 2009 3:05 PM
| Permalink
| Comments (0)
| TrackBacks (0)
February 3, 2009
By Brian Schaffner
Frequent Pollster.com readers will know that the Cell Phone Only (CPO) population is a subject I blogged on frequently during the campaign. One of the reasons for this interest is because there is still a lot we don't know about CPOs at this point (I like a good mystery). In this post, I want to present a little analysis that demonstrates just how much we still have to learn about CPOs.
CPOs wouldn't be nearly as fascinating if their distinctiveness could be easily explained by their age or other basic demographic or political factors. One view is that CPOs are just younger, more urban, and more mobile and that once you account for these factors, CPOs behave pretty much just like their landline counterparts. Last year, however, a Pew report suggested that the differences might not be that simple when they discovered that weighting alone might not be enough to account for CPOs.
To explain why this is the case, I used some of the data Pew relied on to publish that report (in this case, Pew's June 2008 Voter Attitudes Survey). In this survey, Pew interviewed respondents both on landline phones and on cell phones (see more on the methodology here). What I wanted to know was whether CPOs are really that different from landline users once you account for demographic and political factors?
To answer this question, I estimated a multivariate statistical model that would allow me to take into account any demographic or political information that the Pew survey collected that I could imagine would explain the difference between CPOs and the rest of the public. I controlled for age, gender, education, ethnicity, race, income, home ownership, marital status, and whether the respondent lived in an urban area. In addition to these demographic factors, I also accounted for the partisan affiliation of the respondent. If these factors explained the difference between CPOs and landline respondents, then we wouldn't expect to find any measurable difference between these groups once we've controlled for them.
If you enjoy reading output from statistical models, you can get that information View image. However, the key information is presented in the chart below. The chart shows the probability of a landline or CPO respondent registering a vote preference for Obama, McCain, or neither candidate after controlling for the demographic and political factors.
This chart indicates that even after controlling for all of the demographic and political factors listed above, CPOs still had distinctive vote preferences relative to those with landlines in their homes. While landline respondents were 49% likely to prefer Obama in June, CPOs were 65% likely to do so. These differences can't be explained as a result of CPOs being younger, or because they were single, or because they lived in urban areas, or even because they were more likely to be Democrats. Essentially, if you had two people who were the same on all of the factors mentioned above, the one without a landline would still be more likely to support Obama than the one with a landline.
If it isn't age, income, education, or even mobility, then what makes CPOs distinctive from those with landlines? Is it something more inherent about embracing a CPO lifestyle? Perhaps it is an outlook on life that makes CPOs more willing to cast off traditions and venture into something new? Perhaps a higher tolerance for taking risks and embracing change? Or something else entirely? We still don't have a good handle on what makes CPOs so distinctive, which is why this makes for such a great mystery.
(Note: I'm using this Pew data to examine another reason why CPOs may cause pollsters so much trouble--because it may be more difficult to pin down whether they are actually going to vote. I'll present some analysis on that topic in my next post.)
By Brian Schaffner on February 3, 2009 9:44 AM
| Permalink
| Comments (3)
| TrackBacks (0)
December 18, 2008
By Mark Blumenthal
The last two days bring news on the issue of cell phones and their impact on political surveys in the form of new reports from the most respected researchers on the subject.
First, yesterday, the Center for Disease Control and Prevention's National Center for Health Statistics released its latest biannual report on the prevalence of households without wireless or standard telephone service (via Bialik). The CDC monitors the cell-phone-only population because it conducts huge ongoing health "surveillance" surveys via telephone, and as such, ask questions about telephone usage on their ongoing, in-person National Health Interview Survey (NHIS). Their latest report, which covers the first six months of the year, shows that 16.1% of adults were reachable only by cell phone, while another 2.1% lacked telephone service of any kind.
As the chart above shows, the latest survey continues an ongoing, near linear upward growth in the cell-phone-only population. It is worth noting that the hint of a plateau in the trend seen between the second half of 2006 and the first half of 2007 was likely the result of a very slight change in question wording that took effect in 2007. Except for the brief near-pause, the trend has been steadily and consistently upward since 2005.
The second news is the latest report from the Pew Research Center on their efforts to survey voters via cell phone during 2008 (summary, full PDF). It is hard to overstate* the influence of the Pew Center's work on cell phones in the political polling industry. At the PAPOR conference in San Francisco last week, Professor Mike Traugott (a past president of AAPOR and chair of AAPOR's Special Committee on 2008 Primary Polling) noted that earlier this year, "the conventional wisdom in the Spring [among pollsters] was that we didn't have to worry about cell phone only people." At the end of the summer, however, "this conventional wisdom changed drastically" when the Pew Center released its report showing that the omission of cell phone only voters could understate Barack Obama's margin over McCain by two to three percentage points." By the fall, many national media surveys included supplemental cell phone samples in their surveys.
The new Pew report confirms that the patterns they saw, "that estimates based only on landline interviews were likely to have a pro-McCain tilt compared with estimates that included cell phone interviews," persisted through their final survey in late October. It was a difference of a point or two on the margins that "while statistically significant, was small in absolute terms -- smaller than the margin of sampling error in most polls." Of possibly greater importance going forward, they also found similar differences in party identification and self-reported ideology, and bigger differences in (not surprisingly) the use of Internet as a news source and social networking sites like Facebook or MySpace. See the report for the details.
Now that the conventional wisdom on cell-phone only households has shifted among national pollsters, from "it doesn't matter" to "we better account for the them," the Pew Report also points us to the next issue for the c.w. to chew over: Simply interviewing by cell phone (and screening for "cell only" respondents) may not be enough. Pollsters will also need to confront the more difficult issue of how to handle and weight the interviews among what some call "cell phone mostlys:"
Unlike most other polling organizations, Pew's election surveys involved a "full dual frame design," in which people reached by cell phone who also have a landline are interviewed, as well as cell-only respondents. In contrast, most pollsters who included cell phones in their election surveys screened their cell samples for cell-only respondents.
The difference between these two approaches can be seen as a tradeoff in methodological challenges. Supplementing a landline sample with cell-only respondents has the advantage of not "double covering" respondents who have both types of phones. This makes combining the samples more straightforward, but assumes that the landline sample is capable of accurately reaching all adults equally. If some adults have landline phones that they rarely or never answer because they favor their cell phones, they will be underrepresented in these surveys. Pew's approach of interviewing all adults in both the landline and cell phone samples ensures that every adult with a telephone is covered by the survey, but raises challenges in combining the data because some adults had a greater chance to participate if they have more than one telephone. Pew's methodology accounts for this double coverage by weighting respondents with both kinds of phones according to their probability of selection and the regularity with which they use each kind of telephone.
The report goes on to present data showing that the slightly different results produced by the two approaches. Duel users reached by cell phone were more likely to support Obama (53%) than those reached by landline phone (46%). The cell-phone-mostlys reached by cell also identified with the Democrats (54%) more often than those interviewed by landline phone (47%).
The report concludes with a section on the "practical considerations" of interviewing by cell phone. As they have found previously, once they offer a monetary incentive to potential cell-phone respondents ($10), the contact and cooperation rates are comparable to what they get on landline phones. So cell phone interviewing can be done. The downside is that cell-phone interviews cost Pew "nearly two-and-a-half times as much as landline interviews" for reasons they explain in detail.
As always, my summary does little justice to the full report. Go read it all.
*[Mangled use of "understate" corrected]
By Mark Blumenthal on December 18, 2008 3:07 PM
| Permalink
| Comments (1)
| TrackBacks (0)
November 6, 2008
By Brian Schaffner
Now that most of the national vote has been tabulated, we can get a pretty good sense of which pollsters came closest to pegging the final popular vote. As Mark mentioned in an earlier post, several others have done this already, but I thought I'd create these plots for Pollster.com readers.
The final national poll results from individual pollsters are plotted below (these are the last 19 national polls listed on Pollster.com's national trend page). The pollsters represented with red dots are those that included cell phone only (CPO) respondents in their sample. The Obama vote is represented by the y-axis and the McCain vote is the x-axis (UPDATE: I've updated the plots to reflect the updated vote share of 52.5% for Obama and 46.2% for McCain). The horizontal red line is the actual vote that Obama garnered while the vertical line indicates McCain's share of the vote. The closer a poll is located to where the two red lines meet, the more accurate that poll was in predicting the final outcome.
Note that every pre-election poll plotted here underestimated McCain's support. However, the big winners were Rasmussen and Pew, both of whom estimated a 52-46% advantage for Obama. The Pew poll included CPO respondents while the Rasmussen survey did not. CNN and Ipsos/McClatchy also came quite close by estimating a 53-46% advantage for Obama (UPDATE: These polls now come just as close as the Pew/Rasmussen polls). Neither survey reached the CPO population. Indeed, the plot reveals no clear pattern with regard to the CPO issue. Polls including CPO respondents did not appear to be any more accurate than those only reaching landlines.
Before the election, I separated out the Pollster.com national trend into surveys including the CPO population and those who were only calling landlines. The plot below looks at how each of these trends performed.
The trend based on surveys including the CPO population did slightly better at estimating Obama's vote but worse at gauging McCain's support. Overall, the CPO trend was slightly further off the mark than the landline trend.
Of course, there are any number of other factors at play with these different surveys (such as different likely voter screens, weighting, etc), so we can't draw any definitive conclusions from this analysis. But there is no obvious pattern from these initial results that indicate that including CPO respondents helped improve polling accuracy.
Update: Updated to reflect changes in popular vote.
By Brian Schaffner on November 6, 2008 11:46 PM
| Permalink
| Comments (9)
| TrackBacks (0)
October 17, 2008
By Mark Blumenthal
In my column this week, I summarized some recent data provided by both the Pew Research Center and Gallup on the interviews both have been conducting this year via cell phone. Both pollsters have seen a similar pattern. With the interviews among "cell phone only" respondents (those who live in households without landline telephones) included, Obama does a point or two better, McCain does a point or two worse.
One of the more significant findings in this data was from Pew. They found big differences between 18-to-29-year-old voters with landline phone service an those reachable only by cell phone. The cell phone only younger voters favored Obama by a much bigger margin (62% to 27%, n=250) than the landline younger voters (52% to 39%, n=146). While those differences were just large enough to be statistically significant, the sample sizes involved are relatively small, so it is hard to be certain about the magnitude of the difference.
Curious, I asked Gallup's Jeff Jones if they could replicate the Pew table, and he kindly sent along the following data. You do not see differences by party identification. The ballot numbers show the same general pattern, but the differences are not as large as they appear on the Pew survey. Obama, runs better among the Gallup cell-phone only 18-to-29-year-olds (+28) than among those reachable by landline (+20). Those differences still appear to be statistically significant given the larger sample sizes.
The bottom line? Gallup and Pew have produced data supporting the theory of a likely "cell phone effect" in Obama's favor that weighting by age may not eliminate. However, the Gallup data implies a smaller effect than Pew. Of course, without seeing a weighted comparison for all voters by Gallup (something they may not have time to produce until after the election), we won't know for sure.
By Mark Blumenthal on October 17, 2008 3:19 PM
| Permalink
| Comments (39)
| TrackBacks (0)
October 15, 2008
By Mark Blumenthal
My NationalJournal.com column for the week looks at what we know about whether the rise in cell-phone only households is causing any skew in polling results. The short answer is that polls that cannot reach cell-phone-only voters may be slightly understating support for Obama and overstating support for McCain, although the difference is small (and likely within the margin of error of any individual poll). As the Pew Research Center's Scott Keeter recent told the Arizona Reporter (in a clip received too late to make the column), "For the first time, we’re actually seeing a difference between cell-only voters and land line voters when you take into account age."
I hope you read it all.
By Mark Blumenthal on October 15, 2008 11:29 AM
| Permalink
| Comments (13)
| TrackBacks (0)
October 10, 2008
By Brian Schaffner
The key takeaway from the recent Pew Report on cell phone only users was not that cell phone only respondents are different, but that even weighting landline only survey data doesn't fully account for excluding cell phone only users. Typically, a survey may be weighted for factors such as age, race, gender, education, and region. This allows pollsters to take a particular sample, and adjust it to look more like what they think the population they are interested in actually looks like. Pew found that even if you weighted a landline sample for all of these factors, that sample still provided results that were 2%-3% less favorable for Obama than one that included cell phone only users.
What exactly makes cell phone only respondents different from those with a landline? If it is simply the case that cell phone only respondents are more likely to be Democrats than those with landlines, then it should be simple enough to correct for not calling cell phones by weighting a sample by party identification. From my reading of the Pew report, they did not examine whether applying a party weight would have accounted for the exclusion of cell phone only respondents. A party weight is something that some pollsters (like Rasmussen) apply, but others do not. However, based on some recent analysis I have conducted using the 2006 Cooperative Congressional Election Study, even weighting by party is not likely to fully account for the differences between cell phone only respondents and those with landlines. There are two reasons for this.
First, it is true that cell phone only respondents are more Democratic than landline respondents. But this relationship is a little more complex than it first seems. In the 2006 survey, cell phone only respondents were just 4% more Democratic than landline respondents and they were 9% less Republican when asked a standard party identification question. However, once you factor in independents who lean towards the Democratic or Republican Party, you find that cell phone only respondents are 10% more Democratic and 12% less Republican. Thus, the party differences are larger when you factor in leaners, a pattern that results because cell phone only respondents are more likely to initially call themselves independents even though they lean Democratic. To fully capture party differences among cell phone only respondents, one would need to factor in leaners.
Second, just looking at party affiliation masks the fact that cell phone only respondents are actually quite a bit more liberal than those with a landline. 35% of cell phone only respondents classified themselves as liberals compared to just 23% of those with a landline. These ideological differences are not completely accounted for by party either. From the table below, you can see that cell phone only Democrats are 10% more liberal than those with landlines. Democratic leaners in the cell phone only sample are 15% more likely to classify themselves as liberal. And even those cell phone only independents who did not express a lean to either party were more likely to be liberal compared to their landline counterparts. Given that cell phone only Democrats and Democratic leaners are more liberal than those with landlines, they should be less likely to defect and vote Republican than landline Democrats.
Thus, this analysis suggests that differences between cell phone only users and those with landlines cannot simply be accounted for by partisanship. In fact, even when I used multivariate models controlling for a wide range of demographic and political factors (party, age, race, gender, income, education, and even religion), cell phone only respondents were still substantially more liberal than those with landlines. Cell phone only respondents are ideologically distinct in ways that cannot be accounted for by party identification or all the other standard demographic factors that pollsters may use to weight samples.
By Brian Schaffner on October 10, 2008 12:48 PM
| Permalink
| Comments (24)
| TrackBacks (0)
September 26, 2008
By Brian Schaffner
There has been a lot of discussion of the Pew Report released earlier this week that shows that including cell-phone only respondents does appear to make a 2-3% difference in the presidential preference polling (see Mark's post). What's most intriguing to me is how this would play out at the state level. Indeed, it seems very unlikely that every state has the same percentage of cell phone only households. Thus, in states with fewer cell-phone only users, the effect of excluding such respondents may have less of an effect on the poll results. On the other hand, states where there are more cell-phone only households may have polling that is further off the mark.
I was hoping to be able to easily find some survey data with enough respondents to get a sense of the prevalence of cell-phone only households in each state. Unfortunately, the 2007 CDC data that is often cited provides more than enough national interviews to accomplish this task, but the dataset hides the state of the respondent, only allowing users to place respondents in a particular Census Region. Nevertheless, we can learn a little about geographical variance from this data. Specifically, families in the South and Midwest are more likely to have cell phones only compared to states in the West and Northeast. Based on this data, we should expect polling from southern and midwestern states to be more prone to error from the cell-phone only problem than polling in other regions.
Another source of data that may be of some use in answering this question is the 2006 Cooperative Congressional Election Study. (In the interests of disclosure, I should say that I am involved in the 2008 version of this study, though I had no role in the 2006 version.) This was a large (approximately 30,000 respondents) internet survey conducted by YouGov/Polimetrix using a matched random sample design. Because this is an internet survey, it probably isn't as ideal for addressing this question as the CDC survey would be. However, the sample was stratified to assure that there would be a large enough sample from each state and since the state of the respondent is available for this data (and isn't for the CDC data), it is the one decent way I've found of breaking out cell-phone only figures by state.
In this survey, 10.6% of respondents indicated that they only had a cell-phone (this is smaller than the percentage cited in the CDC survey, though the CDC survey was conducted a year later). Most interesting is the variation across states. The map below shows this variance.
Some of the swing states that stand out as having higher than average cell-phone only users are Montana (21%), Oregon (17%), Virginia (15.7%), Wisconsin (15.3%) and Minnesota (15.1%). (Keep in mind that these figures are from two years ago). If this survey is providing reasonably accurate figures on cell-phone only users, then it may be the case that polling of these states would be particularly prone to under-stating support for Obama. It may also explain why the polling in some of these states (for example, Virginia) has been so erratic.
Of course, these data may be problematic and should be taken with some caution. If anyone has ideas about other sources that could be used to compile state-by-state measures of cell-phone only households, please let me know. If excluding cell-phone only respondents does matter, then it would be nice to have a strong sense of where it will matter most.
By Brian Schaffner on September 26, 2008 1:11 PM
| Permalink
| Comments (7)
| TrackBacks (0)
September 23, 2008
By Mark Blumenthal
Few in the field of survey research have examined the problem of cell phones and surveys as closely over the last four years as Scott Keeter and his colleagues at the Pew Research Center. They conducted one of the first large-scale political surveys by cell phone two years ago and have been leaders in testing and developing new methods to conduct political surveys by combining samples of landline telephones and cell phone.
Today they released a new must-read report summarizing findings from "three major election surveys [conducted] with both cell phone and landline samples since the conclusion of the primaries." The verdict? "Pew's surveys this year suggest at least the possibility of a small bias in landline surveys."
The key details (emphasis added):
In each of the surveys, there were only small, and not statistically significant, differences between presidential horserace estimates based on the combined interviews and estimates based on the landline surveys only. Yet a virtually identical pattern is seen across all three surveys: In each case, including cell phone interviews resulted in slightly more support for Obama and slightly less for McCain, a consistent difference of two-to-three points in the margin.
Pollsters have long understood that the cell phone only population -- those who have cell phone but no landline telephone service -- tend to be younger, and that the growth of that population has made it more difficult to reach 18-29-year olds. However, the conventional wisdom among pollsters has held that weighting by age could mostly alleviate any potential bias, as they did they did in 2004.
The new Pew report shows why weighting by age may not have the same effect now:
Traditional landline surveys are typically weighted to compensate for age and other demographic differences, but the process depends on the assumption that the people reached over landlines are similar politically to their cell-only counterparts. These surveys suggest that this assumption is increasingly questionable, particularly among younger people. [...]
In the pooled [August-September] data, cell-only young people are considerably less likely than young people reached by landline to identify with or lean to the Republican Party, and even less likely to say they support John McCain. Among landline respondents under age 30, there is an 18-point gap in party identification - 54% identify or lean Democratic while 36% are Republican. Among the cell-only respondents under age 30, there is a 34-point gap - 62% are Democrats, 28% Republican. The difference among registered voters on the horserace is similar: 39% of registered voters under 30 reached by landline favor McCain, compared with just 27% of cell-only respondents. Obama is backed by 52% of landline respondents under 30, compared with 62% of the cell-only.
The roughly two-to-three point difference in the margin favoring Obama is, as it happens, very close to the effect Nate Silver obtained over the weekend by comparing results from pollsters that have been interviewing by cell phone (including Pew) with a control group that has not.
Finally, the Pew Report has a bit of a bonus with implications for the ongoing debate over how to model "likely voters" this year:
While 18-29-year-olds reached by cell phone tend to have less experience voting than their landline counterparts, they are just as interested in the 2008 campaign, and express just as much intention to vote this year.
For all the details, go read the whole thing.
By Mark Blumenthal on September 23, 2008 10:37 PM
| Permalink
| Comments (20)
September 19, 2008
By Mark Blumenthal
We have followed the challenge posed to survey for the last four years, both here at Pollster and at its forerunner, my old blog MysteryPollster. Over the last four years, survey researchers have been developing techniques for interviewing respondents on their cell phone, and over the last few months, many of the well known national media surveys have been including samples of voters contacted on their cell phones in their national samples or conducting side-by-side tests. These include the Pew Research Center, Gallup (both the Gallup Daily and USA Today/Gallup surveys), CBS/New York Times, Time/SRBI and most recently NBC/Wall Street Journal, ABC/Washington Post and the new AP/GfK poll.
Over the last two weeks, some of these pollsters have provided updates on the impact of their cell phone samples (or lack thereof):
- ABC News polling director Gary Langer today describes their cell phone interviewing test in a new blog post today and describes the impact on the overall results as "negligible....The precise changes were 0 for Obama and -1 for McCain among registered voters, +0.7 for Obama and -0.8 for McCain among likely voters." These differences fell well within the survey's margin of error.
- NBC's First Read included this line in their recap of the latest NBC/WSJ poll: "[T]he poll included some cellphone surveys (we found no significant difference in cell phone respondents as we have from landline respondents." More details on the cell phone sample at the end of the filled-in questionnaire provided by the Wall Street Journal.
Keep in mind that these are relatively small scale tests, in which the margins of error for both the base land-line sample and the supplemental cell-phone test samples are probably larger than any likely effect. Gallup and the Pew Research Center have released similar tests based on larger samples that suggest a small benefit (perhaps 2 to 3 points on the margin) benefiting Barack Obama from the inclusion of cell phone only interviewing.
We will definitely have more to say on this subject in the weeks ahead. Those looking for all the gory details, on this subject may want to start with my series from last year on cell phones and political surveys (Part I and Part II).
By Mark Blumenthal on September 19, 2008 5:33 PM
| Permalink
| Comments (24)
| TrackBacks (0)
July 17, 2008
By Mark Blumenthal
The Pew Research Center, which has been at the forefront of efforts to measure the impact of "cell phone only" households on political surveys, has a new report out on subject today. Like Gallup they have found evidence that including interviews of cell-only Americans as a "modest affect" on results in the presidential race:
Pollsters are continuing to monitor changes in telephone use by the U.S. public, since most surveys are still conducted using only landline telephones. Growing numbers of Americans are reachable only by cell phone, and an even larger number who have both a landline and a cell phone may be "functionally cell-only" because of their phone use habits. The latest Pew Research Center national survey, conducted June 18-29 with a sample of 2,004 adults including 503 on a cell phone, finds that the overall estimate of voter presidential preference is modestly affected by whether or not the cell phone respondents are included. Obama holds a 48% to 40% lead in the sample that includes cell phones, and a 46% to 41% advantage in the landline sample. Estimates of congressional vote are the same in the landline and combined samples. [Emphasis added].
The numbers noted above are based on interviews with registered voters. When they narrow the universe to more likely voters, however, the difference [mostly] disappears:
Narrowing the analysis to voters who are certain about their vote choice, there is almost no difference between the landline and combined samples: Obama has a 38%-28% advantage in the combined sample, while the margin is 38%-30% in the landline sample.
For more detail on the challenge of cell-phone only households to political polling, see my two-part series last year, as well as any of our more recent posts on the subject.
[I added the word "mostly" to my second paragraph based on comments below. The report does not speak to the statistical significance of either set of numbers].
By Mark Blumenthal on July 17, 2008 5:33 PM
| Permalink
| Comments (13)
May 20, 2008
By Mark Blumenthal
"Numbers Guy" Carl Bialik has a nice round-up of the latest developments in the "steady displacement of landline usage by cellphones" that is pushing some pollsters to "try to reach Americans on their cell phones."
Carl noticed something I had overlooked, namely that the most recent CBS/New York Times poll included a supplemental sample of cell phone numbers. Unlike Gallup, they do not yet see an impact on the results from the greater sample coverage:
Kathy Frankovic, director of surveys for CBS News, told me that “I haven’t seen any great difference in results, but it is still early.”
Frankovic confirms that all future national CBS polls will include a supplemental cell phone sample. In an email, she notes that they "have been working on incorporating a cell phone sample into our polls since late last year" and reminds us they incorporated a cell phone sample into their Iowa poll last year.
For those keeping track, that means that three national survey organizations -- Gallup, The Pew Research Center and CBS/New York Times -- now routinely include supplemental samples of cell phone numbers as part of their national political surveys. More will presumably follow.
The trend toward cell phone sampling raises a special challenge for the pollsters using the automated Interactive Voice Response (IVR) method in statewide surveys, because of the one regulatory barrier that affects cell phone interviewing. The federal Telephone Consumer Protection Act (TCPA) bans any sort of unsolicited call to a cell phone using "automated dialing devices." As Bialik points out, some pollsters are pushing for a change in that regulation:
The Council for Marketing and Opinion Research, a lobbying group for the survey industry based in Washington, D.C., is pushing Congress to exempt pollsters from the auto-dialer ban. LaToya Lang, the state legislative director for the group, calls this issue a “high priority.”
Bialik also highlights another bit of cell phone survey news I neglected to pass along from the AAPOR conference:
Recognizing that cellphone surveying is on the rise, last month the American Association for Public Opinion Research, a professional group, released a report offering guidelines for the practice. The report doesn’t call these “standards,” because more research is needed to determine how to conduct surveys via cellphone and how to blend the results with landline interviews.
After last year's AAPOR conference, I wrote a two-part series summarizing both the challenge to political surveys from the growth of cell-phone-only households and the experimental approaches pollsters are using to conduct interviews on cell phones. My interviews at this year's conference include a update from Steven Blumberg on the most recent data from the National Center for Health Statistics (NCHS) and a report from Gallup's Jeff Jones comparing results on the presidential race with and without the supplemental cell phone interviews.
By Mark Blumenthal on May 20, 2008 11:51 AM
| Permalink
| Comments (1)
February 1, 2008
By Mark Blumenthal
I'm blogging from an airport with just enough time to like and block quote two helpful explanations of what the missing "cell phone only" households mean to the vast majority of surveys that do not attempt to interview Americans on their cell phones.
First, Wall Street Journal "Numbers Guy" Carl Bialik has a very clear, concise review of the central issues. He draws on conversations with both Jeff Jones of Gallup and Scott Keeter of the Pew Research Center, the two organizations that have done the most significant political surveys of Americans over their cell phones. It is well worth reading in full if you are new to this issue. Bialik stresses that the evidence so far shows that the missing "cell onlys" have little impact on political survey results:
The impact of losing cellphone-only respondents, however, may be exaggerated. Their numbers aren't big enough to budge most poll results by more than a point or two, Gallup has found.
People who use only cellphones, on average, are younger, more likely to rent their homes and have lower incomes than their tethered-telephone peers. But once you adjust for age, cellphone-only users have similar political viewpoints. Although he thinks cellphones should be included, Jeffrey M. Jones, managing editor of the Gallup Poll, asks, "It's still a lot of cost and effort, and what's the payoff?"
Meanwhile the Pew Research Center yesterday released another mega-study of the cell-phone-only issue. It's a must read for those who want all the nitty-gritty details. Here's the bottom line:
On key political measures such as presidential approval, Iraq policy, presidential primary voter preference, and party affiliation, respondents reached on cell phones hold attitudes that are very similar to those reached on landline telephones. Analysis of two separate nationwide studies shows that including interviews conducted by cell phone does not substantially change any key survey findings.
Update: Carl Bialik has much more on his blog, and also reminds me to link to the recent Public Opinion Quarterly special issue on cell phone surveys and my own series from last summer on the cell phones and political surveys (Part I and Part II).
By Mark Blumenthal on February 1, 2008 3:00 PM
| Permalink
| Comments (3)
January 14, 2008
By Mark Blumenthal
One interesting new wrinkle on the latest USA Today/Gallup survey. Editor in chief Frank Newport tells us on his Gallup Guru blog that, starting with this survey, Gallup will regularly sample cell phones:
[A]s of Jan. 1, 2008, Gallup has made the decision to include cell phone interviewing as part of the sample used for its general population studies.
This is a complex and costly modification in methodology. Our statisticians and methodologists have spent a great deal of time reviewing the procedures and implications of the change. Essentially, in addition to sampling from the traditional database of all landline telephone exchanges, Gallup now also adds in sampling from a new database of all cell phone telephone exchanges in the country. We screen for those individuals using cell phones who report not having a landline, and then interview a random sample thereof. We then weigh into the sample a proportionate percentage of these interviews conducted via cell phone.
For now, at least, this change is not likely to produce dramatic differences in the results. The ongoing cell phone surveys conducted by the Pew Research Center have shown that the missing cell-phone-only population rarely makes a difference of more than a point or two. But that point or two may sometimes make a difference, especially in a close race. Consider last week's Gallup poll in New Hampshire. USA Today polling editor Jim Norman let us know, via email, that they included a cell-phone sample on that survey:
[I]t added a point to Obama's total and took one away from Clinton. In other words, without the cell-phone-only respondents, Obama's lead among likely voters was 11, not 13.
The bigger significance in this change is symbolic. Gallup is the granddaddy of all polling firms. Their polling "time series" goes back to the 1930s. As such, they are typically the most cautious about changes in methodology, so their move to regular cell-phone sampling is likely to have a big ripple effect on the polling industry. At very least, this most closely watched poll will provide a regular source of data on the potential impact of the cell-phone-only households that will be missing from other surveys.
Norman also sent a long the actual text of questions used to identify and screen for cell-phone-only households. I have posted it after the jump.
Continue reading "Gallup Adds Cell Phone Interviewing"
By Mark Blumenthal on January 14, 2008 4:34 PM
| Permalink
| Comments (4)
December 13, 2007
By Mark Blumenthal
Yesterday, the National Center for Health Statistics (NCHS) today released its biannual update on the still growing number of Americans living in households without a landline telephone:
Preliminary results from the January–June 2007 National Health Interview Survey (NHIS) indicate that more than one out of every eight American homes (13.6%) had only wireless telephones during the first half of 2007.
The NCHS is an invaluable source of data on the growing mobile-phone only population because it conducts more than 13,000 in-person interviews every six months that reach all Americans, regardless of their telephone service (or the lack thereof). The Centers for Disease Control (of which NCHS is a part) is concerned about the growth of "cell phone only" households because of the massive ongoing health "surveillance" studies it conducts via telephone.
When we last checked in with the NCHS data, the rate of growth appeared to be increasing, as evident in the upturn in the trend in the chart above from July 2005 through December 2006. That rate of growth appears to have slowed on the most recent release, but the NCHS report explains that the change may be due to a changes in the NCHS questionnaire:
The observed increase in the percentage of adults living in wireless-only households from the last 6 months of 2006 to the first 6 months of 2007 was not statistically significant. Other observed increases over time in the percentage of adults living in wireless-only households were statistically significant. These results suggest a possible recent decline in the rate of increase. However, questionnaire changes in 2007 could have contributed to the observed decline. Therefore, conclusions about trends cannot be made until data from the last 6 months of 2007 are released in May 2008.
The full NCHS report has details on those questionnaire changes.
AP also published concise summary yesterday of the findings from NCHS Report. For more information on the impact of the growing "cell phone only" population on political surveys, see my two-part series earlier this year.
By Mark Blumenthal on December 13, 2007 4:36 AM
| Permalink
| Comments (1)
December 8, 2007
By Mark Blumenthal
The New York Times' Megan Thee looks at the challenge to telephone poll sampling posed by cell phones.
My neighbor Joel Achenbach blogs on political polls as only Joel Achenbach can. Don't miss his discussion of "squishiness bias."
Carl Bialik takes a more serious look at the term "statistical tie" and finds that some pollsters (including yours truly) prefer to avoid it.
Frank Newport posts "ten things Americans think about Mormons and a Mormon president."
The Pew Research Center finds that 4 out of 5 likely caucus goers in Iowa say they have receiving recorded phone calls from the campaigns.
First Read's Aswini Anburajan goes to Nielsen "to get ratings and demographic information on just who watches Oprah Winfrey" in Iowa, South Carolina and the nation.
Dante Scala has evidence of a coming Huckabee media buy in New Hampshire.
Compete (a web analytics company) has posted an Election 2008 Data Hub that allows comparison of candidates by their web traffic (via Swampland).
John Distaso and Tom Fahey have the latest on the criminal investigation being conducted in New Hampshire into survey calls that allegedly conveyed negative information about Mitt Romney.
By Mark Blumenthal on December 8, 2007 3:29 AM
| Permalink
| Comments (0)
November 2, 2007
By Mark Blumenthal
Frank Newport blogs
on the importance of immigration to Democrats and promises
a new Gallup
poll on the presidential race early next week.
Jonathan
Martin blogs comments from Winthrop
University professor Scott Huffman on
the methodology of his recent poll
of South Carolina.
Doyle
McManus warns of the shaky predictive value of polls "purporting to say
who's winning the race."
Kathy
Frankovic considers how religious faith will affect this year's
presidential campaign.
Chris
Cillizza parses some recent CNN poll tabulations of perceptions of Hillary
Clinton by gender.
Margie
Connolly examines the independents in a recent CBS News poll that expect to
"weigh in on the selection of presidential nominees next year."
David
Hill considers the political viability of the No Child Left Behind Act.
Mark
Mellman reports results of two surveys on "looming crisis of long term care"
(surveys were sponsored by Americans
for Secure Retirement and Genworth
Financial).
Chris Bowers
ponders the subgroup shifts in the Democratic presidential race found in the
latest Pew Research Center
survey.
Carl
Bialik questions (appropriately) another Zogby online survey about another
online activity.
College senior Mara
Gordon considers the implications of cell-phone only users excluded from
national surveys.
And speaking of cell-phone-only
households: The recent Pew Research Center survey used a combined sample of respondents interviewed by landline phone and
cell-phone. They promise "a
detailed analysis of the landline and cell phone samples in the coming weeks."
By Mark Blumenthal on November 2, 2007 8:19 PM
| Permalink
| Comments (0)
| TrackBacks (0)
July 23, 2007
By Mark Blumenthal
An update from the Los
Angeles Times on plans by the California Health Interview Survey (CHIS) to reach cell-phone only households
with interviews
conducted over cell phones:
The biannual health study, which
provides data on more than 42,000 of the state's roughly 12 million households,
is used to "drive decision-making and to drive the recommendations we make
to the Legislature or to the governor," said Sandra Shewry, director of
the California Department of Health Care Services.
If data are inaccurate, a survey paints a nonrepresentative picture of
populations, which can in turn be used as the basis for an improper allocation
of funds or just bad decisions, she said.
"The implications are that if we do not include people with cellphones
only, then we are likely to be underreporting un-insurance rates, and we're
likely to underreport smoking prevalence rates in the state," survey
director David Grant said. The numbers won't be dramatic, he said, "but it
will introduce some level of bias in our estimates.
For more information, see my recent posts on the challenges
of the missing cell
phone only households and lessons
learned from pilot studies conducted via cell phone.
By Mark Blumenthal on July 23, 2007 1:57 PM
| Permalink
| Comments (0)
| TrackBacks (0)
July 13, 2007
By Mark Blumenthal
Two weeks ago, I took a long look at the cell-phone
only problem and whether the absence of those without landline phones is
affecting survey results. Today, I want to conclude with a look at how
pollsters conduct surveys via cell phone. Like Part I, this article is long,
even by Pollster.com standards. So it continues after the jump.
Continue reading "Cell Phones and Political Surveys: Part II"
By Mark Blumenthal on July 13, 2007 8:14 PM
| Permalink
| Comments (4)
| TrackBacks (0)
July 6, 2007
By Mark Blumenthal
An intriguing footnote to the first
part of my post on the cell-phone-only problem (alas, a shortened yet
crowded week has pushed Part II until next week). The bottom line is that even
at 12% of adults, the cell-phone-only population appears neither large nor distinctive
enough to throw off most political survey results by more than a point or two. And
while that conclusion may not change drastically should the cell-phone only
population double over the next year or two, all bets if "cell phone only" comes
to describe the majority of U.S.
households (a point reader Chris G made in the comments).
Could that happen? An article
yesterday by The New York Times' technology
writer, David Pogue, suggests a potential pathway. Last week, the cell-phone
carrier T-Mobile announced a new service called T-Mobile HotSpot @Home,
something Pogue described as an "absolutely ingenious" and as potentially "game
changing" to the technology world as Apple's iPhone. "It could save you
hundreds or thousands of dollars a year," he wrote, "and yet enrich T-Mobile at
the same time." How?
Here's the basic idea. If you're willing to pay $10
a month on top of a regular T-Mobile voice plan, you get a special cellphone.
When you're out and about, it works like any other phone; calls eat up your
monthly minutes as usual.
But when it's in a Wi-Fi wireless Internet hot spot, this phone offers a huge
bargain: all your calls are free. You use it and dial it the same as always -
you still get call hold, caller ID, three-way calling and all the other
features - but now your voice is carried by the Internet rather than the
cellular airwaves.
These phones hand off your calls from Wi-Fi network
to cell network seamlessly and automatically, without a single crackle or pop
to punctuate the switch.
And what does this have to do with the cell-phone only
problem? Read on...
O.K., but how often are you in a Wi-Fi hot spot?
With this plan, about 14 hours a day. T-Mobile gives you a wireless router
(transmitter) for your house - also free, after a $50 rebate. Connect it to
your high-speed Internet modem, and in about a minute, you've got a wireless
home network. Your computer can use it to surf the Web wirelessly - and now all
of your home phone calls are free.
You know how people never seem to have good phone
reception in their homes? How they have to huddle next to a window to make
calls? That's all over now. The free router is like a little T-Mobile cell
tower right in your house.
Pogue goes on to explain that HotSpot @Home will work with essentially any
existing Wi-Fi (802.11b/g) router. What could this mean for the cell-phone-only
problem? In outlining four ways this product can save consumers money, Pogue
does everything but connect the dots:
SAVING NO. 4 T-Mobile's
hope is that you'll cancel your home phone line altogether. You'll be all cellphone,
all the time. And why not, since you'll now get great cell reception at home
and have only one phone number and voicemail? Ka-ching: there's an additional
$500 a year saved.
While the new Apple iPhone, which went on sale last week, does not aim
to replace home phone service, it does provide a very similar hand-off from the
AT&T wireless network to home or office Wi-Fi hotspots for its built in
Internet connection. If these features prove popular with consumers, if HotSpot@Home
"enriches" T-Mobile as Pogue speculates, then other cell-phone carriers (with
the possible exception of Verizon) are sure to offer similar services.
And if that happens, pollsters may look back with
great nostalgia on the days when the cell-phone-only population was just 12%.
UPDATE: A very alert survey researcher emails and notes that the Pogue column is "the #1 most e-mailed story on the New York Times web site today!"
By Mark Blumenthal on July 6, 2007 8:30 PM
| Permalink
| Comments (0)
| TrackBacks (0)
July 3, 2007
By Mark Blumenthal
Since the AAPOR conference back in May, I have promised several
times
to take a closer look at the challenge that the growth of "cell phone only"
households are presenting to political polling. Today, finally, I am posting
the first of a two-part review of some of the latest research.
As this item is a bit long even by Pollster.com standards,
it continues in full after the jump.
Continue reading "Cell Phones and Political Surveys: Part I"
By Mark Blumenthal on July 3, 2007 7:49 PM
| Permalink
| Comments (12)
| TrackBacks (0)
June 22, 2007
By Mark Blumenthal
During last month's conference of the American Association
for Public Opinion Research (AAPOR), I mentioned
the many papers being presented on the growth of cell-phone-only households and
promised to report back. I have been digging through the many papers ever
since, and have been working on some analysis on the subject which, in the
crush of new charts and other activity around here, I have yet to post.
Fortunately, Scott Keeter, the director of survey research
at the Pew Research Center,
has posted a concise
and accessible review of the cell phone challenge to surveys. The Pew Center
has been at the forefront of research and development on this subject,
conducting four pilot studies over the last two years that interviewed people
with cell phones over their cell phones.
The summary is well worth reading in full, but for those in
a rush, here is Keeter's view of where things are heading:
Pollsters recognize that some type
of accommodation for the cell-only population will have to be made eventually,
as was clear from the large amount of research on the topic presented at the
AAPOR conference last month. In addition to the use of so-called "dual
frame samples" such as those described above (calling both a cell phone
sample and a landline sample), practitioners are discussing other alternatives,
including the establishment of panels of cell-only respondents that can be
surveyed periodically to track their opinions, and employing mail or internet
surveys to reach the cell-only population.
For those who want more detail, I can also highly recommend
the longer paper he
presented at the AAPOR conference (co-authored with Courtney Kennedy and April
Clark of the Pew Center and Trevor Thompson and Mike
Mokrzycki of the Associated Press) which Keeter has now posted online. Of all
the papers I have reviewed, it was easily the best review of the issues most
relevant to the political surveys we all obsess over.
PS: When I return to the cell phone issue next week, I also
want to say more about a question starting to bounce around the Internet (via Sullivan): "Are Cell Phones Killing
Ron Paul's campaign."
By Mark Blumenthal on June 22, 2007 5:50 PM
| Permalink
| Comments (7)
| TrackBacks (0)
May 18, 2007
By Mark Blumenthal
On Monday, I linked
to a new report
from the National
Center for Health Statistics
(NCHS) showing that 12.8% of American households had only wireless phones (and
no land line phones) during the second half of 2006. "I don't know how this
impacts traditional polling techniques," MyDD's Matt Stoller noted on Tuesday, "but
I am curious."
Well, funny he should ask. The released of the NCHS report
was timed, in part, to coincide with the annual conference of
the American Association for Public Opinion Research (AAPOR) which I am attending in Anaheim, California.
One thing I can report is that the community of survey researchers continues to
take the trend toward wireless only households very seriously (as they have at
AAPOR conferences for the last four or five years).
Today, Stephen Steven Blumberg of NCHS presented his findings in
greater detail to a standing room audience of academic and professional survey
researchers at the conference. Their report is one of twenty-two research
papers, plus a panel discussion and a mini-course being presented here just on
the subject of mobile phones and their impact on polling. Many of these papers
describe pilot studies involving interviews conducted with respondents on their cell phones.
We often say that political polling is a mix of both science
and art. One of the things I appreciate most about these AAPOR meetings is the exposure
I get to the true science of this profession, which as of late has focused on
the issue of how to conduct surveys on cell phones. One thing that many here I
spoke with seem to agree on is that by 2008, many of the national news media pollsters
are considering adding some "cell phone component" to their surveys.
I am certain I will have more to report in the weeks
following the conference, although as a result of the conference, my blogging
has been light this week. For better or worse, I wear many hats at AAPOR: I
serve on the organization's executive council, I will be presenting a paper
with Charles Franklin tomorrow (which we hope to roll out on the blog next
week) and I try to attend as many sessions as possible and absorb all that the
conference has to offer.
By Mark Blumenthal on May 18, 2007 6:57 AM
| Permalink
| Comments (2)
| TrackBacks (0)
May 14, 2007
By Mark Blumenthal
The National
Center for Health
Statistics (NCHS) today released another of its regular updates on the ever
growing number of Americans living in households without
landline telephones (hat tip to alert reader BS):
During the last 6 months of 2006, more than 3 out of every
20 American homes (15.8%) did not have a landline telephone. Of those homes
without a landline telephone, most had at least one working wireless telephone.
Preliminary results from NHIS suggest that more than one out of every eight
American homes (at least 12.8%) had only wireless telephones during the second
half of 2006. These are the most up-to-date estimates available from the
federal government concerning the size of this population.
The ongoing NCHS survey, which involves roughly 13,000 interviews
every six months, conducts its interviews in person. Thus, it potentially
reaches all Americans, regardless of phone service. As the following chart
shows, the upward trend in wireless only households shows no signs of slowing:
For thoughts on what this trend may mean for the telephone
surveys we all obsess over, see my previous commentary here and here.
By Mark Blumenthal on May 14, 2007 10:00 PM
| Permalink
| Comments (0)
| TrackBacks (0)
January 19, 2007
By Mark Blumenthal
Yesterday, the National
Center for Health
Statistics (NCHS) released its latest estimates of the number of Americans
living in households without landline telephones, as well as a statistic
closely watched by pollsters: During the first six months of 2006, "approximately
10.5 percent of households do not have a traditional landline telephone, but do
have at least one wireless telephone."
Pollsters have been watching the growth in "cell phone only" households
because cell phones are largely out of reach of the traditional random digit
dial sampling methods used in most conventional telephone surveys. As such, the continuing upward tend in such
households illustrated by the NCHS surveys (which involve massive monthly in-person
samples of Americans), should be of great interest to anyone who follows public
opinion polling.
Although I have written about these issues previously (here,
here,
here
and here),
the best analysis of how this trend has affected the accuracy of public polling
has been done by the Pew Research Center.
Last year, in partnership with the
Associated Press and America Online, they conducted parallel surveys: One
using conventional telephone sampling and another that interviewed a random
sample of 750 mobile phone users over their mobile phones. The study produced a report by the Pew Research
Center (available in
either HTML
or PDF format)
reached the following conclusions:
[Cell only Americans] are younger, less affluent,
less likely to be married or to own their home, and more liberal on many
political questions.
Yet despite these differences, the
absence of this group from traditional telephone surveys has only a minimal
impact on the results. Specifically, the study shows that including cell-only
respondents with those interviewed from a standard landline sample, and weighting
the resulting combined sample to the full U.S. public demographically, changes
the overall results of the poll by no more than one percentage point on any of
nine key political questions included in the study.
Of course, given the trend reported by NCHS, the cell-phone only adult population appears to be a moving target. It has more than doubled in the last two
years, and the trend shows no signs of slowing.
How big a problem will cell-phone only households be in 2008? Will their absence from traditional phone surveys begin to impact results? Will pollsters begin to routinely incorporate more expensive cell phone samples into their surveys? Time will tell.
By Mark Blumenthal on January 19, 2007 5:02 PM
| Permalink
| Comments (4)
| TrackBacks (0)
