Pollster.com

Articles and Analysis

 

More Pollsters Interviewing By Cell Phone

Topics: Cell Phones

We have followed the challenge posed to survey for the last four years, both here at Pollster and at its forerunner, my old blog MysteryPollster. Over the last four years, survey researchers have been developing techniques for interviewing respondents on their cell phone, and over the last few months, many of the well known national media surveys have been including samples of voters contacted on their cell phones in their national samples or conducting side-by-side tests. These include the Pew Research Center, Gallup (both the Gallup Daily and USA Today/Gallup surveys), CBS/New York Times, Time/SRBI and most recently NBC/Wall Street Journal, ABC/Washington Post and the new AP/GfK poll.

Over the last two weeks, some of these pollsters have provided updates on the impact of their cell phone samples (or lack thereof):

  • ABC News polling director Gary Langer today describes their cell phone interviewing test in a new blog post today and describes the impact on the overall results as "negligible....The precise changes were 0 for Obama and -1 for McCain among registered voters, +0.7 for Obama and -0.8 for McCain among likely voters." These differences fell well within the survey's margin of error.
  • NBC's First Read included this line in their recap of the latest NBC/WSJ poll: "[T]he poll included some cellphone surveys (we found no significant difference in cell phone respondents as we have from landline respondents." More details on the cell phone sample at the end of the filled-in questionnaire provided by the Wall Street Journal.

Keep in mind that these are relatively small scale tests, in which the margins of error for both the base land-line sample and the supplemental cell-phone test samples are probably larger than any likely effect. Gallup and the Pew Research Center have released similar tests based on larger samples that suggest a small benefit (perhaps 2 to 3 points on the margin) benefiting Barack Obama from the inclusion of cell phone only interviewing.

We will definitely have more to say on this subject in the weeks ahead. Those looking for all the gory details, on this subject may want to start with my series from last year on cell phones and political surveys (Part I and Part II).

 

Comments
JoeJustice:

I still don't know a single person who has ever been polled. I'm 29. I've never been polled in my entire life. None of my friends or associates have ever been polled. I have not had a land line for 8 years.

Look, there's just no way in heck the cell phone results would be negligible.

____________________

Tyler:

Your ancedotal evidence is not convincing. Here's the math:

There are about 300,000,000 people in the US.
About 75% or 225,000,000 are adults (18+)

In total, the four national tracking polls used by this site interview about 4000 every 3 days, or about 1333 people per day.

Let's be generous and say that each of these tracking polls did interviews from January 1 to election day every presidential election year. That's 309 days.

Therefore, every election year 411,897 people get interviewed by a tracking pollster. But that's only 0.18% of the adult population of the US. Therefore every adult has a probability of about 0.0018 being polled each presidential election.

So your chance of being called by a tracking pollster in the roughly 3 presidential elections that have occurred since you turned 18 is about 0.005 (i.e. a half a percent chance). Even if you include say, twenty friends your age, the probability that any of you were ever polled is still only about 0.1 (10% chance).

Therefore it's not to surprising that neither you nor your friends have ever been polled, cell phones or not.

____________________

axt113:

I wouldn't call a 1% swing negligible in an election, a 1% swing can mean the difference between victory and defeat in a close election, and since McCain suffers a negative, even a small one from cellphones and Obama either stays the same or gets only a tiny boost that may bode poorly for McCain in the final result

____________________

Urquhart:

How do they get the numbers? Not as if there's a directory. Or is there? Is the pool of available cell numbers therefore self-selecting, like the Rasmussen interactive polls?

____________________

Urquhart:

Sorry, Zogby interactive.

____________________

Mark Blumenthal:

@Urquhart:

The pollsters listed above get their sampled cell phone numbers essentially the same way they do for land lines: They start with telephone exchanges (the first three numbers after the area code) used for cell phones and randomly generate the last four digits.

That way every working number has a chance of being selected.

Yes, phone number portability will make this selection process less precise in the future, but if you're calling all numbers (wired and wireless) all phones get in the mix.

____________________

carl29:

"The precise changes were 0 for Obama and -1 for McCain among registered voters, +0.7 for Obama and -0.8 for McCain among likely voters."

As some people here had suggested, it seems that Obama benefits when cell phones are included in the mix.

____________________

John:

I realise that the sample size is very small, but unless my maths is failing me again, in the ABC poll, McCain's support among cellphone only users is 6.67% below that of those with landlines (RVs) and the margin among LVs who are cellphone only users is 10 points higher to Obama than those with landlines.
Of course, the margin of error with small samples will be high, but surely if there really are those difference between cellphone only and landline users then this indicates that polls should try to contact cellphone users. Especially as the number of cellphone users is likely to increase in the future.

____________________

Lappel1:

A pair of issues are being ignored here: 1) distribution of cellphone use; 2) the difference between those who own both cellphones and landlines and those who own only cellphones.

____________________

Laurette:

I've seen a lot about the effect of cell phones on polls, but what about caller ID? Personally, if it even smells like a telemarketer, I don't answer the phone. It seems that caller ID is just as much a generational tool as cell phones. My parents don't use it (and even if they did, they still have a Pavlovian response to answering the phone), but younger people just expect to be able to screen their calls.

____________________

arctango:

Of course, the people who have both get oversampled x 2

____________________

Matt P:

The small sample sizes individual pollsters are using for these tests are likely to render small, but potentially important differences statistically insignificant. But what about the broader combined experience? Nate Silver has attempted to analyze this over at 538, and concludes that the bias introduced by not sampling cell phone users is about 2.2% against Obama, consistent with the Gallup and Pew numbers. What's your take on his approach, Mark?

____________________

LesAnne:

Additionally, it is illegal for anyone to solicit cell phone users using automatic dialing systems, and it's hard for me to believe that the larger polls will put the resources into individuals dialing numbers by hand. And what about the national Do Not Call List that has millions of cell phone users blocking their numbers? I just think this can't be accurate sampling of those who only have cell phones.

____________________

GradStudent:

@ Laurette:
_what about caller ID?_

Exactly! Has this been discussed? If so, how are polls taking this into account?

Caller ID should have a significant influence on the population that responds.

Thanks to random number generating calling banks, unlisted cell phones should not have as large of an impact as those selecting their communication through Caller ID. Not sure if it skews towards either party/candidate, but it seems like a significant factor that is not being discussed.

____________________

Cugel:

Since Obama is leading heavily in under the 30 age demographic and since those voters are mostly "cellphone only" the pollsters are full of crap when they say their polling shows "no effect."

If that's what their polling is showing, then it's flat wrong.

We would expect that a failure to poll under 30's at the same rate as older voters with landlines would inevitably lead to improper sampling of that age demographic. To compensate for the millions of voters they can't poll, they have to extrapolate from those they can reach -- those with landlines.

Perhaps they are saying that there's no difference between under 30's with landlines and those without. That might be.

But, it's NOT statistically valid to rely on a smaller sampling of available voters because you can reach them, and then say that the larger sample of unreachable voters is "the same" so it doesn't matter.

Its bad enough when voters don't answer the phone to pollsters. Part of the "convention bounce" is bound to be suddenly energized supporters of one party or another being more willing to sit through a poll.

We are in a constant war between consumers and companies targeting them for a wide variety of things. Polling competes with spam, advertising for various things, and other junk mail that people don't want.

But, the failure to be able to reach many "cellphone only" voters who are on DO NOT CALL lists has an effect and to deny that is statistically invalid.

It might not affect the election if those with landlines vote the same way in the same percentages as those who are "cellphone only" but that would be LUCK, not science.

____________________

mohillDem:

Cugel is exactly right.

I think the point regarding Obama's, so far, unmeasured appeal among young voters is that these people tend to have ONLY a cell phone and NO land line.

While polls of cell phone users might partially tap into this group, they are still include a certain percentage of people who have BOTH a cell phone AND a land line and so their poll is "tainted" with those who are included in 'traditional' land line telephone polls.

Until the pollsters can figure out how to distinguish between these groups, they will never have a true picture of the support among these young voters.

____________________

Matt Herlihy:

I also agree with everything Cugel has said.

Clearly, there are strong demographic trends among which voters elect to only use a cell phone, and those people have a markedly different preference than those of the general landline populace.

In addition to essentially excluding most people under the age of 30, not sampling cell-phone only voters is also excluding many lower income voters who cannot afford two separate phone services for people to contact them on, or those who elect to have a month-to-month service.

____________________

hobetoo:

Tyler,

Your math is reasonable except for one major omission. You've calculated the ratio of completed interviews per capita but not the ratio of the cumulative total sample per capita.

Depending on whether individual numbers are called more than once in a given survey, if we assume roughly a 33% response rate then you should triple your ratio. And you should further multiply the probability of being called to take into consideration the numerous other polls that occur -- not just "the four tracking polls."

I've only had a cellphone about 5 years, and I've never been called for an interview on my cellphone. I don't block such calls. But during that same time I've been called for several surveys on my land line. Perhaps it's because I live in a "battleground state," but I expect to get called for one or another survey a few times per year.

I'm also in the Zogby Interactive sample, and get surveys avery few weeks from them online.

But again, never on my cell phone in the last 5 years. Yes, the odds are long, but they're not as infinitesimal as you suggest.

____________________

damitajo1:

To JoeJustice - There are 300 million people in the US. Odds are - you probably won't know some who has been polled (or maybe they don't think it's big enough of a deal to talk about).

PS: I am 41, and I have never been polled either!

____________________

damitajo1:

Some of you seem to think that cell phone usage only began recently. Pollsters have demonstrated remarkable accuracy during previous national elections, congressional races, and during the recent primaries -- despite or in spite of the usage of cell phones or even "cell only" young people. If you're still in college - take a course in statistics and learn about sampling techniques. A good random sampling will produce better results than "my friends are all voting for Obama, so these polls are wrong" anecdotal evidence.

____________________

falcon79:

I agree with damita - a good random sample will provide a valid representation of the population at hand.
however, the emphasis is on the word "good".
by underpolling or not polling younger voters with landlines, an inevitable bias is introduced into the poll.
pollsters may say there is no "statistically significant" difference, but statistical significance is a moot point, for statistical significance is solely a function of 3 factors:
the magnitude of the difference between groups
the magnitude of sampling error
the level of significance at which the results are evaluated (typically, 5%)

in short, the issue is not whether differences are 'significant", but rather the size of the difference
it should be apparent from the analyses presented by the author that there is a difference, albeit a very small one - perhaps somewhere between 1-3 points, in obama's favor.

so the real question is; is a 1-3 point difference a big deal?
my answer is "YES"!
assuming the votes are essentially tied (a yellow "toss-up" state on the map). the 1-3 point difference will tilt the results blue if taken into account!

bottom line - if you see a state as a toss up with mccain only leading at the most by 2 points, then it is likely that obama will carry that state during the general election

of course, a caveat must be introduced into this analysis: the bradley effect.
if the number of undecideds is high (about 10%) expect the result to favor mccain on election day, regardless of sampling biases by pollsters

____________________

damitajo1:

falcon - good try, but you conclude on your own that the sample is not good. why not? because you believe they are undercounting cell-only people. But the pollsters report otherwise. NBC said there was no signficant difference at all. And I will reiterate a point I made earlier. Some pollsters predicted the '06 midterm and recent primary elections with great accuracy. SUSA and Ras did great jobs. The same people who will have cellphones in November had them during the primaries. If the cellphone issue is as dramatic as you guys claim it is, why then were pollsters able to predict the primary races -- some of them were spot on.

____________________

falcon79:

damita:
NBC said there was no "significant difference" which means there was no statistically significant difference
that is misleading, because given a small sample size and large error terms, it is very likely that NBC made a Type II error - concluding that there was no difference when in fact there is a difference
that is why i said the real issue is not whether or not there is a difference but rather how big of a difference is there?
there probably is a difference, albeit a small one... my point is that even a small difference can be meaningful if the race is close

if the pollsters were able to predict the 06 races and primary elections with accuracy, it could be that the races they predicted with accuracy were not so close as to allow for the small effect of cell phone usage becoming an issue that could change the results

so the issue at hand is not whether or not cell phone usage makes a difference - it is how much of a difference does it make, and is that difference large enough to swing the results one way or the other? the answer to the latter question in "yes" if the races and up being extremely close

____________________

VonnegutIce9:

Pardon my ignorance, but doesn't this new "inclusion" of cell phones contradict their methodology? I was under the impression that they called numbers in a random dialer ???-???-????...so, unless cell phone numbers have an extra digit, and they don't...why WOULDN'T they have been calling cell phones in the first place?

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR