What Could Go Wrong?
Mark Blumenthal | November 3, 2008
Topics: Bradley/Wilder , Cell Phones , Likely Voters , Response Rates
Every morning for the last month, I have posted updates looking at poll trends. When poll methodology is standardized, polls should be good at helping discern trends, whether we are making "apples-to-apples" across among polls by a single pollster or using the regression trend lines we plot on charts. But today, more than any day of the last two years, we are focused intently on the level of support that each poll estimates for each candidate.
With that focus in mind, I have been using my NaitonalJournal.com column this month to focus on those things that might throw off polls as a "point estimate" of the outcome of the election. I have been asking, in effect, what could go wrong. This week's column attempts to wrap it all up, and consider more potential issue -- the nightmare scenario for pollsters -- that non-response bias might throw off the estimates.
Given the intense interest, I want to blog my own first few paragraphs:
• Will the growing number of voters reachable only by cell phone make polls less accurate? If they do, it will be in Obama's favor in polls that are not interviewing by cell phone. National pollsters, such as the Pew Research Center and Gallup, have reported a slight 1-2 point increase in Obama's margin when they include interviews conducted by cell phone.
The inclusion of cell phone interviews may not be the only explanation, but our trend estimates on Pollster.com show Obama leading by a wider margin on national surveys that interview "cell phone only" voters via cell phone (+9.2 as of this writing) than those that do not (+5.4).
• Are likely voter models missing a surge of new voters? Probably not, since most surveys use screens or models that will capture new registrants or newly energized voters if they express strong interest and enthusiasm in voting. Even then, Obama is leading by wide margins even on the more traditionally restrictive likely voter models (such as those used by Gallup, Newsweek and the Pew Research Center) that include measures of past voting in their models.
• What about the so-called "Bradley Effect?" Will the undecided vote break decisively to McCain, as it did for many white Republicans running against African-American Democrats in the 1980s and early 1990s? Charles Franklin and I looked hard for current evidence of either a race-of-interviewer effect (which was present in some of those races 20 years ago) or a hidden McCain vote among currently undecided voters and found none. The final survey by the Pew Research Center did a similar sort of analysis and found a slight "break" of the remaining undecided vote to McCain, but not enough to make much dent in Obama's lead: It allocated 4 of the 7 undecided percentage points to McCain, 3 to Obama.
We do see some hints of a possible break of undecided voters to McCain in a few battlegrounds. In Pennsylvania, most of the recent movement has represented a shift from undecided to McCain. In Ohio, we notice consistently closer margins on automated telephone surveys than on surveys conducted with a live interviewer, and again the difference looks like a shift to McCain from undecided. Still, Obama is at or above 50 percent in both states and could still carry both even with a decisive "break" of undecideds to McCain
The column goes on to consider the theoretical possibility of non-response bias that would occur if those that pollsters cannot interview -- because they hang up or are unavailable when called -- have different political views than those who are interviewed. The Pew Researcher's Andrew Kohut has often speculated that non-response bias may have been at least partly responsible for some of the Bradley Effect seen in the late 1980s and early 1990s. He shares his thoughts about the potential for such an effect this week. I hope you'll read it all.
Bottom line: We may see small effects from any of these, but so far at least the potential problems are likely to be offsetting or to increase Obama's lead, not reduce it.
One more thing: In the column, I say that the most rigorous national surveys struggle to achieve response rates over 30 percent. We know this mostly from a now five year old study [PDF] by three academic survey methodologists, Jon Krosnick, Allyson Holbrook and Alison Pfent. In a paper presented at the 2003 AAPOR Conference, they analyzed response rates from 20 national surveys contributed by major news media pollsters. They found response rates (based using AAPOR's Response Rate 3 formula) that ranged from 5% to 39% with an average of 22% (see slides 8-9).
What do response rates look like for the surveys we are looking at now? Good luck finding an answer. The only pollster I have seen include an AAPOR response rate is the Boston Globe/University of New Hampshire Survey Center partnership. They reported an AAPOR #4 response rate of 32% on a recent New Hampshire polls. Would it be so hard for other pollsters to do the same?
By Mark Blumenthal | November 3, 2008 2:31 PM | Permalink | Comments (24) | TrackBacks (0)
Comments
How hard would it be? I agree with DTM, but it's ironic that pollsters whose nominal raison d'etre is to provide accurate data refuse to reveal their response rates.
Will the undecided vote break decisively to McCain, as it did for many white Republicans running against African-American Democrats in the 1980s and early 1990s?
I am not a professional pollster, but I was under the impression that the Bradley Effect refers to a phenomenon where voters will falsely tell pollsters they are planning to vote for a black candidate when they have no such intention, and instead vote for the white candidate. That is not the same thing as undecideds breaking decisively for McCain.
So which is it?
Couldn't another "thing" be the overestimation of election day turnout for one candidate's strong base of support? For example, it seems that many Democrats - especially blacks - have voted early (in record numbers). Does this mean that Democrats will dominate the polls on election day or that they just voted early due to enthusiasm and expected long lines, etc? Depending on how a pollster resolves this question, the results of election day voters could tilt strongly towards McCain or Obama.
With all the publicity about it's different now will this affect the nature of the non-responses?
Take a 'normal' election. There may be overall a neutral list of reasons for non-response much of which can be compensated for by demographic weighting. So for example if older people are more likely to refuse to talk to a machine their numbers can be boosted according to their expected level of contribution to the voting population.
But I am sure that a lot of calls are refused by people who are not alone and who do not want to air their political views in front of somebody who will disagree with them.
So suppose you are AA but don't intend voting for Obama but you don't want to argue with your spouse or your friends about it. You will simply not take the survey.
Suppose you are in a liberal environment but want to support McCain or Palin then you might refuse the call. There will be some situations the other way of course where somebody doesn't want to advertise his support for Obama.
Now this might all cancel out but of course it might not. And it might well be exaggerated by the use of cellphones. A cellphone call does not necessarily go to somebody's home. It can go to somebody at work, out for the evening or visiting friend's homes. In short it is more likely to occur in a public environment than a land line call.
The fact is we are on new ground here and no pollster really can know all the ups and downs of this. They all have made assumptions but the particular circumstances of this election might throw things off.
How about likely voter models assuming increased youth and black turnout that isn't going to happen as a percentage of the total vote.
We already know the youth vote isn't happening. And the assumption that black vote will increase as a percentage of the whole is just bad math. If black turnout increases 20% over 2004, guess how much of an increase in white voter turnout negates it? 2.6% increase. Blcaks voted for Kerry 88%. So even if blacks vote for Obama 98% and increase turnout by 20%, a 3% increase in white voter turnout negates it.
How about Republian vs Democrat turnout. Some polls have a 15% spread, won't happen, ever. Last time it was 37%-37%. A 5% split is wishful thinking. It might be 4%, but no where near what some of these polls are predicting.
Also some polls still have over 9% undecideds. If you don't reassign them and you don't push them, almost 10% of people are undecided or not talking to pollsters. How about this, Fox news found in it's latest poll that McCain supporters were 10% less likely to answer polls.
I think a lot of the assumptions pollsters have made are just plain wrong. There won't be increased youth turnout, or black turnout as a percentage of the whole, and there won't be the dem-rep split many are calling.
You failed to list will young voters actually vote in proportion to their "enthusiasm" for the FIRST time in history? You failed to list will newly registered voters actually show up to vote and will their votes actually be honestly counted? Leaving these off your list is leaving the elephant in the living room out of the conversation. Only a fool would do that.
Hello everyone...Can someone explain how the GOP's internal polling can differ so much from any other reputable pollsters? Is it just "hype" from them to make it at least appear McCain is still in the game, or do the numbers have some bearing? Thank you!
btw a 3% "Bradley effect" with no polling margin of error, or methodological errors puts McCain over the top in enough states to surpass 270 EV's.
These are the reasons I've been trolling for months to your true believers that McCain will win. What part of "White House" don't you get? I've never believed that enough of my fellow whites would vote for the first time in their life for a black man, the FIRST TIME, at the level of President when they have never done it for dog catcher, Mayor, councilmen etc.
Another thing I have not seen any comment about: As I recall, in 2004 there was a lot of feeling against Bush because of the Iraq War and because of what happened in 2000.
A lot of people vote not for somebody but against somebody else. Obama supporters are fired up to support him but it was ever thus. It is understandable that AA voters are more fired up than ever but as s.b. says they already vote pretty solidly Dem anyway. There is no feeling against McCain, no wish to keep him out. Sure millions would prefer Obama but I don't think most Obama voters think of McCain in the same way as they thought of Bush in 2004.
On the other hand Obama has been campaigning for so long against Hillary Clinton and then McCain that there are PUMAs and others that won't accept him at any price. This is partly an outcome of being better known, partly because of his stance on issues such as abortion and partly his list of sleazy friends. There seems to be a definite anti-Obama section of the electorate which I would say is much stronger than any sort of anti-Kerry feeling in 2004.
I have seen comments by Dems hoping that the Rep GOTV will be less effective for McCain than it was for Bush. I don't see why it would be less effective. Quite the contrary.
Mark raises an excellent point about response rates. We have been carefully watching all of the data from the various polls and examining the impact of various voter turnout models, but if all of our analysis is based on only 10% to 20% of those originally sampled (that is a response rate of 10% to 20%) AND those who do participate are different from those who do not (the dreaded case of differential nonresponse or nonresponse bias), then we could (and I emphasize COULD) be surprised by the results of the actual vote. Many academic articles have failed to find substantive differences between those who do and those who do not participate. Lets hope Election Day continues this same trend....(that is, NO nonresponse bias!)
Observer, good point! I have maintained for quite some time that unfortunately a negative votye against something or someone is much more reliable and motivating than a positive vote for someone. That's why the opposition never wins an election, the ruling party loses it. There is no incumbent this time, and the Dems control the house and the senate. This election has been about Obama, undeniably. Will voter turnout for Obama surpass the voter turnout as a percentage that was against Bush and the Iraq war? I maintain that it will not. In fact voter turnout against Obama's presumed Presidency, may be a far greater motivating factor than the turnout for him. We shall see.
I do not see the increase in youth turnout and black turnout, or even Democratic turnout as a percentage of the whole that most of these polls assume.
@LiberalTiger
I am not saying that it is but but in theory it can be more accurate. I remember a McCain spokesman saying they poll everyday. They might have far larger samples than most or all of the state polls that have been done and thus be able to break down the race in particular cities, towns and disticts.
They know where they need to do really well and where they need to minimise their losses. They can also compare to how Bush did in 2000 and 2004 and see if McCain is doing better. Smaller sample polls cannot do this. Smaller sample polls might not poll each area of the state in the same depth.
But don't forget the 'they would say that wouldn't they' factor.
That assumption is outside the fact that those very people may also be the ones most likely to answer pollsters questions.
@s.b.:
You're correct that a small percentage increase in the white vote would "negate" a large percentage increase in the AA vote. But the use of the word "negate" assumes that the increased white turnout would be voting entirely for McCain, while the increased AA turnout votes entirely for Obama. The latter assumption is likely true, but the former is not.
@s.b.:
It is true that the improvement in the situation in Iraq has diminished the anti-Iraq sentiment which fueled Democratic gains in the 2006 mid-term elections. You forgot to mention that the Economy has taken its place in voter concerns, and that Democrats are seen as having the edge over Republicans on economic questions.
(Socialism and Joe the Plumber aside.)
From the blog The Patriot Room. This is basically what could go wrong.
The author is Ebonya Washington, Assistant Professor of Economics and Political Science (Yale). She received her PhD in economics at MIT in 2003. She specializes in public finance and political economy with research interests in the interplay of race, gender and political representation.
The article is, How Black Candidates Affect Voter Turnout And Candidate Positioning.
"I find that each black candidate significantly increases black voter turnout by 2.5 percentage points. The white turnout response, 2.1 percentage points, is nearly as high and statistically indistinguishable from the black result.
Given that blacks and whites comprise 10 and 83 percent, respectively, of the population in the average district, the numerical increase in white voters is seven times that of black voters."
polls_apart
No that is not what I assume at all. I assume a split vote among whites, similar to 2004.
@s.b.:
You said that a 5% split (Democrat over Republican) was wishful thinking. While I agree that a 15% split is ridiculous, even Rasmussen indicates a 6%-8% split for Democrats in their polling.
@s.b.: "btw a 3% "Bradley effect" with no polling margin of error, or methodological errors puts McCain over the top in enough states to surpass 270 EV's."
According to this site's morning update for today it would take a 6.9% "Bradley Effect" to give McCain 270. This is his margin behind in his weakest supporting state to give 270 (Colorado)
Polling might be correct - but it won't matter if people can't vote due to the GOP's massive voter-disenfranchisement efforts:
Mark, the one thing that I haven't heard anyone talking about recently is the enthusiasm gap, and how much this may favor Obama over what the polls show.
Take for instance North Carolina where 73.1% of the 2004 total has already participated in early voting, more than double the early voting total of 2004. The stats show clearly that Democrats and especially AA's are way outperforming their demographics, and there is even a 14 point female to mail advantage (though that was the same as 2004). A conservative estimate is that Obama is ahead by 10 points among those that have already voted, or quite possibly over 15 points.
So we know that North Carolina will have a huge surge in turnout and likely come in at 30% to 50% over the 2004 total vote when all is said and done. McCain needs to win the remainder by more than 10 points in order to make up that deficit, though all the recent polls and your trend suggest the race is tied.
The problem in North Carolina for McCain is that this is the enthusiasm gap at work. Those that haven't felt the urge to vote early like over 2.6 million others have, should be overwhelmingly less enthusiastic, and it follows that if these are disproportionally Republicans, that Republicans are going to have a hard time turning out their votes.
I think this likely means that Obama may outperform the polls by as many as 5 points in North Carolina, and to a lesser extent in some other new battleground states.
Colorado, Nevada, Florida, Georgia and New Mexico have all shown uncommon rates of early voting turnout with increases of 50% to 200% over 2004 early voting totals. Even in Ohio which has been pressed hard for two cycles in a row, we are seeing 200% increases in heavily democratic counties like Cuyahoga, and even 300% more in Franklin, the vast majority of which is in absentee votes.
Obama is a positive and inspiring candidate. This makes voters enthusiastic about supporting him, and therefore turns out voters. Obama's ground game and GOTV efforts appear to be by far the best that ever existed. McCain's campaign has been focused on robo-calling attacks on Obama, while Obama has hundreds of thousands of volunteers using their own phones to make calls to either encourage participation or GOTV wherever appropriate. This has clearly paid dividends in early voting for Obama, and there is no reason to believe that his efforts will be below average on election day.
So as far as "what could go wrong", here's my list:
1) Huge lines, lack of ballots and legal battles about keeping polls open later, especially in states like Virginia and Pennsylvania where there was no early voting. Virginia will probably have absolutely huge problems in urban areas.
2) The possibility that polls may have missed the enthusiasm gap and therefore underpolled Obama. The polls may well have to rethink likely voter models to focus on where the enthusiasm lies instead of who voted before or more regularly. Some assume that RV samples equalize for a Democrat favoring year, but why can't LV models give a stronger advantage to the Democrat when that Democrat registers record numbers of voters, has record turnout in the primaries, and has a big advantage in enthusiasm? I believe that traditional models will fall on their face in the new battleground states (though not in less competitive states).
3) The massive increase in early voting in certain states may very well throw off the exit polls by even larger margins than in recent elections. Without a lot of careful study of early voting turnout, especially by party breakdown, race and sex, and the sheer numbers which vary wildly state by state, they will have a very hard time rectifying exit polls with the final poll results. I am guessing that the list of states that I gave with huge increases in early voting are going to have big discrepancies in the exit polls that lean towards McCain.
In short, the focus of the polling and commentary communities seems to have been focused almost entirely on discrediting Obama's strengths, but in reality, they may very well be underestimating them.
Here's a quick link to a nice page that collects early voting data with links back to the appropriate state agencies:
Spencer, you're correct. The Bradley effect refers to voters who tell pollsters that they will vote for the black candidate, and then vote for the white candidate. Thus it refers not to how undecided voters split, but to a potential shift in support for one candidate to the other.
Thanks! Sounds like a great review--I look forward to reading the whole thing.
But first, in answer to your concluding rhetorical question: Hard? No. Contrary to their marketing strategy? Possibly yes.
Posted on November 3, 2008 2:46 PM