Pollster.com

Articles and Analysis

 

Did Including Cell Phone Only Respondents Increase Polling Accuracy?


Now that most of the national vote has been tabulated, we can get a pretty good sense of which pollsters came closest to pegging the final popular vote. As Mark mentioned in an earlier post, several others have done this already, but I thought I'd create these plots for Pollster.com readers.

The final national poll results from individual pollsters are plotted below (these are the last 19 national polls listed on Pollster.com's national trend page). The pollsters represented with red dots are those that included cell phone only (CPO) respondents in their sample. The Obama vote is represented by the y-axis and the McCain vote is the x-axis (UPDATE: I've updated the plots to reflect the updated vote share of 52.5% for Obama and 46.2% for McCain). The horizontal red line is the actual vote that Obama garnered while the vertical line indicates McCain's share of the vote. The closer a poll is located to where the two red lines meet, the more accurate that poll was in predicting the final outcome.

accuracy1b.PNG

Note that every pre-election poll plotted here underestimated McCain's support. However, the big winners were Rasmussen and Pew, both of whom estimated a 52-46% advantage for Obama. The Pew poll included CPO respondents while the Rasmussen survey did not. CNN and Ipsos/McClatchy also came quite close by estimating a 53-46% advantage for Obama (UPDATE: These polls now come just as close as the Pew/Rasmussen polls). Neither survey reached the CPO population. Indeed, the plot reveals no clear pattern with regard to the CPO issue. Polls including CPO respondents did not appear to be any more accurate than those only reaching landlines.

Before the election, I separated out the Pollster.com national trend into surveys including the CPO population and those who were only calling landlines. The plot below looks at how each of these trends performed.

accuracy3b.PNG

The trend based on surveys including the CPO population did slightly better at estimating Obama's vote but worse at gauging McCain's support. Overall, the CPO trend was slightly further off the mark than the landline trend.

Of course, there are any number of other factors at play with these different surveys (such as different likely voter screens, weighting, etc), so we can't draw any definitive conclusions from this analysis. But there is no obvious pattern from these initial results that indicate that including CPO respondents helped improve polling accuracy.

Update: Updated to reflect changes in popular vote.

 

Comments
Brackdurf:

Sorry if I missed this elsewhere -- so why did everyone so badly undershoot for McCain?

____________________

Gary Kilbride:

To quote from the Monty Python firing squad skit, "Next time, definitely."

____________________

DTM:

Are we sure the count is almost final? I thought there were still millions of ballots to count, and some reason to believe they may shift the margin a bit (e.g., a lot of the uncounted ballots are in the West Coast states, plus scattered provisional votes).

____________________

thoughtful:

I'm with DTM and look forward to the absolutely final count.

The graphic does tell a story though!

____________________

drdr:

Interesting how Pew had Obama +13 and shifted to +6 for their final survey. They get credit for having the final number right, but I doubt there was such a large shift in public opinion in the last weekend. I think they were seeing a difference in enthusiasm between Obama and McCain supporters, with the weak McCain supporters polling as undecided until the very end.

Also Rasmussen ended up pretty accurate in the national number, but wasn't his PA poll way off? His Obama +6 didn't make any sense given that PA was likely to be more Democratic than the national average.

Just looking at the final national number doesn't show how accurate they were overall in the last 2 months.

____________________

Centrist_Dem:

@DTM

Obama's margin has been creeping up as more votes are reported. His plurality is up to nearly 8 mil., and his % share has moved from about 52.3 to about 52.6, though it could of course still move back again.

____________________

RussTC3:

I agree. We need to wait for the final votes. After all, Obama's lead has already increased to +7 without decimals (53/46). The actual tally right now is +6.4% (52.5/46.1).

____________________

Basil:

I can think of three reasons McCain may have outperformed the polls.

One is that there really were some voters who just didn't want to say they supported McCain, because of Obama's popularity and maybe his race as well. It's not exactly a Bradley effect, but sort of.

The second involves third party candidates like Barr and Ron Paul, whom some voters couldn't bring themselves to vote for when it came down to it. Did they underperform in the election?

The third has to do with unlikely-to-vote R's seeing the early D turnout and coming out at the last minute in a vain attempt to make up for it. My 87 year old mother in Atlanta was targeted by GOP GOTV activists (of some sort), something I haven't heard before. Maybe they believed the desperate McCain line about how close the national polls and PA were.

On the other hand, it seemed that western states R's saw the east coast handwriting on the wall before the polls closed (something I hate, as an Oregonian) and stayed away, knowing it was lost.

Any evidence of any of this?

____________________

Some polls hit McCain's 46 (like our R2K poll, Rasmussen and Ipsos.)

I like the idea that not using cell phones cost an Obama point or two. But Gary Langer (ABC) is right that all the polls got it close and that the horse race was not the only thing the polls told us.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR