Articles and Analysis


Mokrzycki: Are MA Senate Polls Prone to Non-response Bias?

Topics: 2010 , Likely Voters , Martha Coakley , Massachusetts , non-response bias , representativeness , Scott Brown

Mike Mokrzycki is an independent consultant who was the founding director of the Associated Press polling unit. He may be reached at mike@mikemokr.com. His guest contribution is cross-posted from his blog, MJM Survey Musings.

One thing is certain about the polling in the last days before Tuesday's special election in Massachusetts to fill the late Ted Kennedy's U.S. Senate seat: Someone's going to end up being very, very wrong.

Polls completed in the past week and recorded at Pollster.com range from a 14-point lead for Democrat Martha Coakley - just weeks ago considered a shoo-in in heavily Democratic Massachusetts - to a 15-point advantage for Republican Scott Brown, who has become a darling and major fund-raising beneficiary of conservatives nationwide.

I'm not going to do a deep methodological dive into all these polls to try to explain the differences. Pollster.com and Fivethirtyeight.com have done their usual stellar job with that already, including analyzing the extraordinary uncertainty inherent in trying to determine who really will vote in this mid-January special election.

I will try to provide a little perspective as someone on the ground in Massachusetts who also knows a thing or two about polls.

My hypothesis: While Brown supporters clearly are more enthusiastic than Coakley backers, that may serve him relatively better in the pre-election telephone polls than it will Tuesday.

I've lived in Massachusetts on and off since 1980 and I can't ever recall Republicans here as energized as they are now. Sure, they had a 16-year run in the governor's office despite the state's overall leftward tilt. But Bill Weld, elected in 1990, was fairly unusual - socially liberal enough that "Weld Republican" became its own label. Paul Cellucci sure didn't inspire a lot of passion and I can't say Mitt Romney did either, with his eye on the White House all along. Brown seems an agile campaigner but I don't think his personal charisma is what's charging up Republicans here and elsewhere; rather, it's the once almost-unthinkable notion that any Republican might actually win the seat Ted Kennedy held for nearly half a century, especially with such extremely high stakes for policy and politics nationally.

This enthusiasm is abundantly evident in internal data from numerous polls. I'd add a couple anecdotes:  I don't put stock in lawn signs but when you see a voter (like someone on the main street in my town) posting a handmade Scott Brown placard, or an ice cream stand using its roadside sign to advertise "VOTE FOR SCOTT BROWN," it may be an indication something beyond rote partisanship is at work.

This race has been Coakley's to lose, and she's seemingly been doing her best to do that. The most recent example was in a radio interview the other day when she called Curt Schilling - famous for helping pitch the Boston Red Sox to a long-awaited World Series championship in 2004 on an ankle stitched together and visibly bleeding through his sock - a New York Yankees fan, of all things. In little more than the time it used to take a Schilling fastball to reach the plate, his recorded voice was on my phone telling me this faux pas was proof Coakley was out of touch with Massachusetts voters. Silly, weighed against the import of issues such as health care reform? Perhaps. But - last baseball metaphor, I promise - Coakley served up a big fat meatball and I sure don't blame the Brown campaign for hitting it out of the park.

Schilling's was one of countless phone calls we've gotten on this race since before the primaries last month. Many have been "robo-calls" like his (as I write this paragraph I just got one from Brown's daughter), though plenty feature live human beings (like someone from Coakley's phone bank who called as I started writing this post).

At this point it's hard to blame people in Massachusetts for screening incoming calls even more so than usual. For years there's been plenty of screening, part of the reason why response rates for all kinds of telephone polls have declined dramatically. (An article in the Winter 2009 issue of the journal Public Opinion Quarterly (subcription required) gives response rates for numerous respected telephone polls it cites and many of them barely crack 10 percent. A response rate greater than 20 percent now is extraordinarily good.) Response rates are even lower for automated polls, which use a recorded voice for interviews and require respondents to punch in answers on the touchtone keypad.

But - and this is an important "but" - a growing body of research indicates decreasing response rates have not hurt the accuracy of survey estimates.  That happens when there's no systematic difference between those who cooperate and take the survey and those who decline.

I'm thinking the Massachusetts Senate race may be a case where we do see non-response bias in surveys. It comes down to relative enthusiasm for the candidates. It's tough to prove, but I'd venture a guess the dynamic works like this:

  • Republicans are excited Brown might win and thus more likely to answer their phone and listen to political messages -possibly be invited to take a survey - when the phone is practically ringing off the hook with such calls. I suspect they'd be particularly enthused to participate in a poll and tell the world they're voting for Brown, to help build the sense he has unstoppable momentum. These folks certainly will vote but there's no upside in Brown's election-day numbers compared to the pre-election poll estimates.

  • Democrats may be demoralized and scared after several weeks of Coakley campaign missteps and bad headlines. They may not be all that eager to pick up the phone for political calls. They also might be more skeptical of or angry about polls since they've been such downers for Coakley and President Obama lately, and thus, I would speculate, more likely to take a pass if invited to participate in one. None of that means these folks are less likely to vote, though - by now any sentient Democratic-leaning voter will know Coakley needs all their votes, and what's at stake. They might not be happy about how Coakley has run her campaign but they'll still be motivated to vote by a desire to deny a Republican the chance to do serious harm to Obama's agenda from Ted Kennedy's old seat. Obama is in Massachusetts today to remind them of exactly that (not that they're necessarily all that enthused about him at this point, either).

Of course, truly independent or "swing" voters are another vital factor, and if Brown wins enough of them he could overcome the inherent Democratic advantage in Massachusetts.  But I'd think enthusiasm, or lack of it, would be more of an issue among stronger partisans.

In pollster speak, what this boils down to is "differential non-response," where one candidate's supporters are more likely than the other's to take a survey.  It's suspected to be a big reason why exit polls in recent years have tended to overstate support for Democratic candidates.  In the Massachusetts special Senate election I suspect it's inflating the Republican's poll numbers. Coakley has room to outperform the polls Tuesday even if her natural base is motivated by nothing more than fear of what would happen if her opponent pulls off an historic upset.



One issue that intrigues about the telephone polls, but live and automated. And that's the cell phone issue. Especially in a place like Mass., with a gazillion colleges and very high numbers of high-tech workers, wouldn't the percentage of cell phone-only people be much higher than the rest of the country?

And are either or both of the automated or live polling groups taking that situation into account? I have seen nothing in any of the poll stories that talks about the cell phone only issue. I had thought that live interviews have a possibility of using cell numbers, but automated polls cannot. Is that correct?

Also, where there are surveys of cell only people vs. landline people, haven't the cell only group been more liberal/Democratic? Or is the data unclear on that issue?



OK, so Coakley's Dem turnout might be more complete, at least on the liberal side of that party's continuum. That analysis breaks somewhat on the moderate side of the Dems, though. I don't see any reason to assume that all Mass Dems are on board with the recent Dem agenda.

Presumably Brown gets most of the Repubs. There might be some moderates who buy into the Dem argument that he's too conservative, but it doesn't seem to me that his record or campaign particularly support that.

Where do the independents and unregistereds go? Some data seems to indicate that they're heavily going for Brown. Is there a reason why that data would be skewed by mood, such that the polls would overstate the leaning toward Brown and/or overstate the likely turnout?


Mike Mokrzycki:

@nelcon1551: Always wise these days to consider potential bias from excluding cell phones but Massachusetts came in quite low in the one available state-by-state estimate of wireless-only incidence. According to modeled estimates from the National Health Interview Survey (which benchmarks national estimates of landline abandonment) and the Current Population Survey, as of 2007 fewer than 10 percent of households and adults in Massachusetts were cell-only. That's only about half the incidence of cell-only nationally at the time.

Yes there are a lot of college students and tech workers here but MA is a big state with a lot more besides.

Cell-only voters do lean a bit more Dem in some recent data, but I'd be careful about making blanket statements. In the 2008 national election-day exit poll the difference (greater likelihood of cell-only voters than those with landlines to back Obama) actually was greater in the 30-39 age group than among under-30 voters. It was more of a mixed bag on some other variables related to partisanship and ideology. (Two co-authors and I have written about this extensively in an article just published in Public Opinion Quarterly.)

Also keep in mind that the usual demographic weighting done on poll results can eliminate at least some of whatever bias may exist from excluding cell phones. Bottom line: I doubt non-coverage of cell-only voters makes much difference in the MA Sen polls, but you never know.

You are correct that all kinds of automated dialers - for "robo-calls" and also used by some live-interviewing firms - are prohibited by federal law from calling cell phones.


tom brady:


There's another response bias that I think you are missing: it appears that Brown does better in automated polls, suggesting people may be reluctant to admit to a live interviewer that they are contemplating supporting a Republican for Kennedy's seat.

Note also that the PPP results are in and it appears that Brown may have slightly widened his lead.


Mike Mokrzycki:

@JWilly48519: I can only speculate - to know for sure we'd need a survey of people who decline to take surveys ... (As an aside, I used to scoff at that notion until I saw exactly such a survey presented at a conference last year! The design was ingenious, and the cost was in the millions. I'll leave it at that.)

Perhaps it would clarify if instead of "Republicans" and "Democrats" I refer respectively to Brown's and Coakley's core supporters - those who under pretty much no circumstances would vote for the other candidate (or for Joseph L. Kennedy). In this election as it has unfolded, this population likely includes some independents/moderates - maybe going more for Brown, at least somewhat offsetting and perhaps entirely overcoming his disadvantage among partisans in this blue state. In any event, however these two groups are comprised, the enthusiasm gap between them is clear. That to me is a recipe for differential non-response in the polls.

As for voters who might still be undecided (whether they self-identify as independent/moderate, are registered as unenrolled, or are nominally partisans), by definition their mood about the candidates is less intense. How these "persuadable" voters will break is of course a very important question in the election but I'd surmise they're less likely to contribute to differential non-response in the pre-election polls.

Unfortunately, as Mark Blumenthal tweeted the other day, it appears we won't have exit polls to help us sort out what happens by ideology, partisanship and other factors. This race heated up only after it was too late to set up the exit polls.


Mike Mokrzycki:

@tom brady: Your social desirability hypothesis is appealing intuitively but I don't think there are enough recent live-interviewer polls for a reliable comparison to IVR. And even a couple of the live-interviewer polls in the last week had Brown up, numerically if not to a statistically significant degree.

I'd also note a couple of the most recent IVR polls are by apparent newcomers to the survey scene and were conducted in one night, while PPP's was two-night but the Saturday and Sunday of what for some is a holiday weekend.



Not being on the ground in Massachusetts, do you still think that Coakley can still win this election and do you believe that the Democrats are finally getting motivated? I also have two more questions. First, special elections are only voted on the day of the election, right? And second, I have done phone banking for Coakley from Tennessee and I have been getting a lot of responses on Saturday and Sunday. Is that a good sign? Thanks.


Mike Mokrzycki:

@Tallman Boyd, ever since this became a race I've been thinking that even if they're not terribly enthusiastic about their candidate, Dems would be motivated to vote by abject fear of losing the seat (and health care reform, etc) to the GOP. Again, I can only speculate but while Coakley has run a lackluster campaign, I don't think she's done anything to piss off Dem regulars into staying home tomorrow. Whether she persuades enough persuadables is a separate, important, question.

There was absentee voting but because of the MLK holiday, the deadline was 5 p.m. Friday.

Can't really comment re your phone banking. It's all relative ...


Post a comment

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.