Articles and Analysis


Pew Research: Missing Cell-Phone Onlys Matter

Topics: 2008 , Cell Phones , Pew Research Center , Scott Keeter

Few in the field of survey research have examined the problem of cell phones and surveys as closely over the last four years as Scott Keeter and his colleagues at the Pew Research Center. They conducted one of the first large-scale political surveys by cell phone two years ago and have been leaders in testing and developing new methods to conduct political surveys by combining samples of landline telephones and cell phone.

Today they released a new must-read report summarizing findings from "three major election surveys [conducted] with both cell phone and landline samples since the conclusion of the primaries." The verdict? "Pew's surveys this year suggest at least the possibility of a small bias in landline surveys."

The key details (emphasis added):

In each of the surveys, there were only small, and not statistically significant, differences between presidential horserace estimates based on the combined interviews and estimates based on the landline surveys only. Yet a virtually identical pattern is seen across all three surveys: In each case, including cell phone interviews resulted in slightly more support for Obama and slightly less for McCain, a consistent difference of two-to-three points in the margin.


Pollsters have long understood that the cell phone only population -- those who have cell phone but no landline telephone service -- tend to be younger, and that the growth of that population has made it more difficult to reach 18-29-year olds. However, the conventional wisdom among pollsters has held that weighting by age could mostly alleviate any potential bias, as they did they did in 2004.

The new Pew report shows why weighting by age may not have the same effect now:

Traditional landline surveys are typically weighted to compensate for age and other demographic differences, but the process depends on the assumption that the people reached over landlines are similar politically to their cell-only counterparts. These surveys suggest that this assumption is increasingly questionable, particularly among younger people. [...]

In the pooled [August-September] data, cell-only young people are considerably less likely than young people reached by landline to identify with or lean to the Republican Party, and even less likely to say they support John McCain. Among landline respondents under age 30, there is an 18-point gap in party identification - 54% identify or lean Democratic while 36% are Republican. Among the cell-only respondents under age 30, there is a 34-point gap - 62% are Democrats, 28% Republican. The difference among registered voters on the horserace is similar: 39% of registered voters under 30 reached by landline favor McCain, compared with just 27% of cell-only respondents. Obama is backed by 52% of landline respondents under 30, compared with 62% of the cell-only.

The roughly two-to-three point difference in the margin favoring Obama is, as it happens, very close to the effect Nate Silver obtained over the weekend by comparing results from pollsters that have been interviewing by cell phone (including Pew) with a control group that has not.

Finally, the Pew Report has a bit of a bonus with implications for the ongoing debate over how to model "likely voters" this year:

While 18-29-year-olds reached by cell phone tend to have less experience voting than their landline counterparts, they are just as interested in the 2008 campaign, and express just as much intention to vote this year.

For all the details, go read the whole thing.


richard pollara:

Can the Pew findings be retroactively tested against this years primary results?



But what about Caller ID? It has been mentioned here before, but not much discussion on its impact on both landlines and cell phones.

If certain demographics are more likely to have CID (and answer an unknown number), wouldn't that skew results?

Anyone know what CID displays when a polling company calls?



@ GradStudent

What you are saying about CID would only affect the sample if there was some data that suggests that one group of voter are more likely to ignore unknown calls than another. Otherwise whether or not there is CID would not change the outcome of the poll. Even before this article I was incline to believe that there are more cell phone only users in the 18-30 age group than other age groups mostly because the younger group tend to be more accepting of new technology. (The number of times I have to fix minor problems on my parent's computer gave me this hint, and I'm on my way out of the 18-30 age group!).

On the other hand I don't think there is a specific age group that are more likely to answer an unknown call than another. I couldn't think of any reason why one age group would be more likely to answer an unknown call than another.



Let's take this with a grain of salt and just mentally give Obama an edge in states that are way too close to call. In VA - where it is way too close to make any predictions just predict Obama but don't be too surprised if you're wrong.

Think of cell-phone only people you know. What demographic? They seem like Obama people but then again not always. So just error on that side when its tight.



@ bmcavan
i disagree that we should mentally give obama the edge for close polls because of the bradley effect - it is likely that obama will suffer a 2-3 pt "silent vote for mccain" disadvantage come election day because of his race
so what this means is that the race effect and the cell phone effect will probably cross each other out
lets take the polls as they are :)



This is an interesting article, but unfortunately rather lacking in detail.

My first concern is in trying to draw much of a conclusion from such a small sample of cell phone only users (94, 163, and 176).

My second concern again is the small number of cell phone only users who were polled for this research. In the Sept 9-14 poll, which had the highest percentage of cell phone only participants, only 3.8% of people polled were cell phone only. Yet the article notes that government surveys estimated about 15% of the population was cell phone only, rising by 2% each year (so as high as 17% this year). The discrepancy between 3.8% and 17% is pretty staggering. If the number is in fact anywhere near 17%, then the difference between cell phone and land line poll numbers could be a good bit larger than what is shown.

This low number of cell phone only results still raises some major questions, such as just how are the researchers acquiring the cell phone numbers? Are they gathered from voter registration records? If so, wouldn't this bias the conclusion that "they are just as interested in the 2008 campaign, and express just as much intention to vote this year")? Since they tend to be younger and have had less experience voting, if they registered, it means that they likely were a self selected group of interested people.

A final major issue is how geography affects the results of cell phone only individuals. It may not be unreasonable to imagine that a higher proportion of cell phone only individuals are in more urban/densely populated areas where cell phone service is more reliable, and these larger urban areas (think LA, Chicago, NYC) are in states that will go democrat anyway, hence the effect of cell phone only people may be less important.

The point is, while this report is interesting, it probably just raises more questions than answers about how cell phone only individuals might affect the polls and vote totals.



Polls serve three purposes in my experience.

1 ) Outcome prediction
2 ) Trend following
3 ) Comedic value

The vast majority of political polls fall in to category 2 & 3.

Only the final poll before an actual vote can legitimately be called predictive. Even in those cases I'm being generous as they should be correctly identified as attempts at prediction.

If you are a serious pollster interested in the science of polling these are your touchstones. The validation that comes from comparing the actual vote result is invaluable. It is here, after each primary and election, that a pollster gets to hone his art.

The three polls chosen by pew for this analysis of cell phone effects are at best reasonable Trend polls. The lack of post poll validation impairs any conclusion you might make from this data.

Also troubling is their statement on one hand that ...

"In each of the surveys, there were only small, and not statistically significant, differences between presidential horserace estimates based on the combined interviews and estimates based on the landline surveys only."

and then trying to impute a significance by saying ...

"Yet a virtually identical pattern is seen across all three surveys: In each case, including cell phone interviews resulted in slightly more support for Obama and slightly less for McCain, a consistent difference of two-to-three points in the margin.".

Had these been three pre primary polls where the actual vote could be used to validate the result we might have something worth thinking about. As it stands we can only say that including cell phones produces different results. As to which method has a higher correlation with election results is still anybodies guess. I see no advancement in the science of polling here.

This isn't a partisan conclusion. Obama may indeed under-poll because of cell phone demographics but you can't use Pew's data to make that case.

One thing is certain. This election cycle is confounding all Pollsters. Not just the Cellular issue but they are all trying to gage whether race and social expectations are causing Obama to over poll. All in all it makes for an interesting year. Enjoy.

About my self. I cut my polling teeth under the wing of Dr. Richard Wirthlin at D.M.I. when Dr. Wirthlin was Ronald Reagan's pollster and one of his most trusted advisor's. I am also the founding member of the Quarter Poll Club. Membership is free and anyone can join. You need only honor the following oath.

"I do swear to answer any poll according to the result of a coin toss or other answer randomizing method"

PS I love your web site...



Cell-phone only voters may become a major problem in the future for pollsters, but I don't think that fact skewes the results significantly today. The possible Bradley effect will be at least as strong a bias in these polls as cell-phone only people.



But will they vote? My guess based on this article that they haven't voted as much means that they are much less likely to vote, as has always been the case with Democratic youth. The best way to judge someone's future behaviour is by their past behaviour.

We will see.




18-29 year old cell phone only users are not younger than 18-29 year old land line or land line and cell phone users.

This study is also ignoring the causal vs correlative effect.

There may be other socio-economic facotrs that relate to the cell phone only users that make them more pro-obama that could be easily factored into other polls. Are they more likely black, lower income, inner city, urban, more or less educated on average. Perhaps they are more likely to be urban blacks at university away from home.

My guess is that the underlying issue is not which phone they use but other factors that are already taken account of in other polls and counting these facors twice introduces another error into the data.




I'm sorry, we are living in 21st century.
There is no bradly effect. you can check in wikipedia.....

By the way, I can give you the real example here.
In Primary, Obama was outperforming in the states like VA, NC, IN, OR where there are big urban areas or colleges or AA population.
I think that is due to cell phone users are missed in most state polls.

So the result were far better than the polls.
You can check all the polls and compare with the results.



s.b., you do make a valid point about "will they show up?" I think your argument may have had more credence in 2004, when the youth vote, energized as it "seemed", just didn't show in November. That said, the "energized youth vote" meme of 2004 was more CW than actual evidence. Also, a report I read recently suggested that only ~7% of the population was cell-phone only in 2004.

Now, contrast with 2008. The "energized youth vote" meme is less of a pipe-dream, and more of a reality. Youth vote in primaries this year was through the roof (as were a lot of other demos, mind you), and, anecdotally, people are way more "jazzed" about Sen. Obama on college campuses (one of which I am on). The youth vote may not disproportionally overpower every other demographic, but may be just enough to give Obama the 1-2% boost (or 2-3% as this article suggests) in those all important battleground states. Also, analysists estimate the percentage of cell-phone only users has doubled since 2004, to about 15% in 2008.

I agree that we need to be cautious. But not because predictions of youth turnout are pessimistic, but because this is uncharted territory. When Gallup first pioneered the whole "polling method" in the 30s, it was a new and daring concept, and people didn't believe it at first. It had many flaws. Now that we are entering into unchartered territory will cell users, we need to exhibit the same level of cautious optimism, in my humble opinion....



If getting the young out to vote was nothing more than wishful thinking, Hillary Clinton would be the Democratic Nominee.

Obama is the Democratic nominee in no small part because he emerged on the scene as a legitimate candidate after he won the Iowa Caucus. He won the Iowa Caucus because he got more than 2X the under 25 vote to get out and vote than 2004. 60% of them voted for Obama.




In 2004 the "youth vote" did in fact show up, if you are talking about 18-34 year olds. The issue was that although they broke a multi-year downward trend, they still weren't polling at levels seen in the past.

I think part of the issue is tracking against the number of registered voters, and motor voter bills have increased the registered youth, but in turn decreased their so-called turnout.

I believe this age group was 18% of the electorate in 2004, up from 16% in 2000, but don't quote me exactly on that. In 1984 and earlier, it was regularly above 20%, but this might have been a reflection of a younger population.

I believe it may be possible that we go even higher this year, maybe back up to 20% of the electorate.



I think the other major question is not just how the cell-only demographic will vote, but does this have other effects on turnout?

If someone is cell-only, are they likely to use text messages or use their phones to watch video? If so, are thy receiving text messages from a campaign? (I certainly am) And does that help reinforce turnout, as they can receive a personalized text with directions to their polling place?




You're right, my phrasing was bad. The youth vote DID come out in 2004, unfortunately it was somewhat mitigated by massive turnout among evangelicals, for example. Also, the low turnout from the 80s through 04 is probably due to a very apathetic, and small (magnitude-wise) youth contingency, i.e. Generation X.

A little history on American generations. The baby boomers, which were once the largest generation in the US, were born mostly in the 50s and early 60s. Their generation was followed by a "baby lull" through the 60s and 70s, Gen X. Consequently, most of those Gen Xers came of voting age in the 80s and 90s, and thus the youth vote was "depressed". Gen Xers are notoriously politically apathetic, and Republican-leaning.

The Millennium Generation, now the largest generation in the US by far, which begins around 1982 and runs through 2002 or so, is now coming online, so to speak. In 2004, only people born before 1986 could vote, meaning only a small portion of millennials were voting (but nonetheless, the sheer magnitude of millenials allowed for a slight uptick in "youth vote"). In 2008, we add another 4 years of millenials to the rolls, hence the upward trend.

I think it's highly possible we get to 20% or higher.


KS Rose:

Interesting. I wonder what effect there might be for folks that have only internet phones. It is my guess that they are not included in polls since I never get any annoying sales calls. In fact, no calls from anyone unless I've given them my number. I'm sure there are fewer folks that have internet phones than have cell phones but I don't have any idea of the numbers.

I'm not convinced of the Bradley effect. They can point to 3-4 races where they suspect it occurred. I'm sure we can point to several dozen where the vote outcome between two white guys was a surprise. That's why they let people vote not just take a poll and declare someone winner.



@ zen:
racism is more subtle today - its called "modern racism"
type "bradley effect" in wikipedia and you will find it
ps: im a hardcore Obama fan ;)



Mark, does random digit dialing mitigate this effect in some way? Or do polling organizations not include cell phones in the polling universe? This seems to be the crux of the issue.



May I ask any experts out there a couple of questions:

1. I read somewhere that polling organisations under-represent first time voters in their polls. True? Does this mean they under-represent Obama's current support?

2. Is there any evidence that younger potential voters (especially cell only people) are less likely to respond to pollsters (even when they do attempt to poll cell-only people) because they are more socially active than older voters and those who are at home to receive and respond to pollster's calls(out having fun in the evening)? Further, are less socially active young adults more likely to be Republicans?


Post a comment

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.