May 06, 2008

What's Up With The National Polls?

Last week, Democratic pollster Mark Mellman used his column in The Hill to argue that one could use "same-poll to same-poll comparisons" to argue that the controversy over Barack Obama's "bitter" comments had either no impact or a great deal of impact on the Pennsylvania primary. "In addition to providing jobs for pollsters," he wrote, "proliferating polls now give everyone the evidence to prove their favorite theory." He drew specific examples from the lesser known pollsters active in statewide contests that are typically derided by the polling establishment and the mainstream media.

Over the last 48 hours, however, we have seen Mellman's point proved by the country's most respected and well established polls and media organizations. As Gallup editor in chief Frank Newport noted in his Gallup Guru blog:

Two different headlines today in the New York Times and USA Today about the impact of the Jeremiah Wright controversy came to different conclusions about Jeremiah Wright and Barack Obama. The Times’ headline: “ In Poll, Obama Survives Furor; but Fall Is the Test”, while USA today headlines: “Flap over pastor pulls Obama down, poll finds”.

And for those who missed it, ABC's Gary Langer briefly summarized the conflicting numbers:

Briefly: Times/CBS has Barack Obama +12 vs. Hillary Clinton, with a headline saying Obama “survives furor” over the Rev. Jeremiah Wright. USAT/Gallup has Clinton +7, saying the flap over Wright “pulls Obama down.” Adding to the mix is Gallup’s daily poll, which has Obama +4.

These polls also differ in their general election match-ups: Times/CBS has Obama +11 and Clinton +12 vs. John McCain, while USAT/Gallup has them basically tied. Gallup daily has Clinton-McCain tied, McCain +5 vs. Obama.

And just in overnight: AP/IPSOS weighs in with a national poll showing Clinton +7 over Obama, with Clinton +5 and Obama +4 over McCain, spawning a thousand different headlines, no doubt (as AP stories always do).

Why all the confusion? Langer has a nice summary of the various methodological quirks that may or may not explain the differences between the Times/CBS and Gallup polls, though his bottom line is the most important take-away point. This episode, he writes,

[Is] a reminder that all polls – even good-quality ones – are done differently, and don’t always get the same results or engender the same analysis. And that horse-race results, in the midst of a close and unsettled campaign, may be particularly vulnerable to these kinds of influences.

Another way to make the same point. Look at our chart for the national Democratic primary polls as captured this morning It plots points for every survey released since January 2007 (captured this morning).

Let your eyes focus, for a moment, not on the the lines but on the cloud of dots surrounding each line. Each dot represents an individual poll. The dots are a bit more dense lately, a pattern explained mostly by our inclusion of daily tracking polls by Gallup and Rasmussen Reports since January 2008 (note: we plot only ever third or fourth day of each survey so we their rolling average samples do not overlap). However, the pattern of variation -- the spread of points around each line -- is considerable but not any wider now than at any other point over the last 17 months.

What is different right now is that the gap between Obama and Clinton is very close. So some polls show Clinton ahead, some show Obama ahead and given the sate of the race and we are paying much more attention to the small differences in individual polls than we usually do. Random error may explain some of the variation, small differences in methodology (question wording, order, the particular sub-population that answered he question) explains the rest. Either way, the cloud of variation -- bringing with it sometimes odd and conflicting "same-poll to same-poll comparisons" -- of "is an inherent part of political polling.

The variation is also the reason why we favor looking at all the polls in this "mashed up" graphical form rather than debating endlessly over which individual poll is closest to "right" at any moment.

-- Mark Blumenthal

May 06, 2008 in Divergent Polls, The 2008 Race

Comments

Post a comment

URL:

Comments: (you may use HTML tags for style)

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button.

Please email us to report offensive comments.

See our comment policy here. Note that we require commenters to share their email address via Typekey. We will never share your email address with anyone without your explicit permission.