Pollster.com

Articles and Analysis

 

ABC/Post Poll: Surge or Outlier?

Topics: 2008 , Iowa , The 2008 Race

About that new ABC News/Washington Post poll released yesterday, the one showing Hillary Clinton winning the support of 53% of "leaned Democrats" nationally to 20% for Barack Obama, 13% for John Edwards and single digits for the other candidates. Since this survey was the first to show a majority supporting Clinton, a lot of bloggers let loose with the adjectives, writing that she "pulls away" with a lead that is "crushing" or "explodes," telling us that we have reached "knockout punch time," that "she's killin' it," or declaring simply, "oh baby, when she moves, she moves." My friend Chris Cillizza parsed Clinton's "surge" in his Washingtonpost.com blog and concluded:

It doesn't take a polling expert to understand that Sen. Hillary Rodham Clinton's showing in the latest Washington Post/ABC News poll is a major moment in the campaign.

There is obviously no denying that the poll has created the perception of a major change in the race, but has a Clinton really experienced a "surge" of support over the last month, as indicated by the 12-point increase (from 41% to 53%) in her support on the Post/ABC poll? I'm not so sure.

I checked our national chart for the Democrats late yesterday and saw that this new ABC/Post poll appears to be very different from the others conducted over the last month. I quickly realized, however that this particular comparison highlights a frustrating limitation of our chart, which is that about half of the national polls plotted include Al Gore as an option and about half do not. Since Clinton typically picks up roughly five points on polls - including the new ABC/Post survey - that do not offer Gore as a choice, our standard chart may make the ABC/Post survey look like more of an outlier than it is.

So we decided to devote some time today to producing a chart that displays only the results for polls without Gore (using recalculated votes based on second choice when available). The result below plots 66 national polls released so far this year (about a third fewer than our standard version - as not all provide a "vote without Gore"):

AUS2TopzDems400.png

The chart makes a few things clear. First, the 53% result for new Post/ABC poll is more of an "outlier" from the regression trend line than any poll conducted this year (it's the purple dot at the extreme top right of plot area). At 53%, the polls estimate of Clinton's support falls a full ten percentage points higher than our current estimate of the trend (42.5%) even without Gore in the race.

Second, the addition of the new poll has not budged our trend line. That is mostly by design. Professor Franklin set the sensitivity of the regression model to prevent the inevitable outlier from causing wild gyrations in the trend line. That is one of the reasons we prefer to use regression estimates over simple rolling averages.

One of the reasons we put so much effort into these plots is that they paint a picture of the random variation that is inherent in polling. You can actually see the random variation. While it typically appears as a random and predictable "cloud" of points, bigger outliers still occur from time to time. The more experience you have looking at poll numbers, the more you learn to understand that even with well designed polls, outliers happen. Our advice to poll consumers - as reflected in the design of the charts -- is to try to avoid over-reacting to any one poll.

Of course, we do not yet know whether this poll is really a statistical outlier. Other polls have obviously been showing a more gradual increase in her support recently, and it is still theoretically possible that Clinton's support suddenly lurched up ten points last week. Senator Clinton had appeared on all of the Sunday morning talk shows on September 23, just four days before this poll went into the field. Earlier that week sho also rolled out her plan to reform health care, an issue that ABC/Post and other polls confirm as one her great perceived strengths. But a sudden upward surge of this magnitude does not seem very likely, if only because no other news event so far this year has caused anywhere near as much change in the Democratic race.

We should know soon enough. Unfortunately, the AP-IPSOS survey released yesterday does not help resolve this issue, as Clinton's Sunday talk show tour de force occurred in the middle of their field period and (I'm told) after they had completed most of the interviews on the main sample.

So we will wait and see. But I'll wager that a month from now the real trend will not look nearly as dramatic as the one suggested by yesterday's news.

PS: It will probably take a few days, but we will have a full, regularly updating version of the national "vote without Gore" up soon.

UPDATE (10-16): More recent polls do move the Clinton trend line up and make the Post/ABC poll look much less like an outlier. More here.

 

Comments
Andrew:

Saying that the poll results "do not seem very likely" is vague, since it can be interpreted as being likely, though not very much so.

I assume you meant this outcome was not likely.

____________________

Andrew:

Great analysis though.

____________________

Gary Kilbride:

He wrote an upward surge does not seem likely. It was more of an analysis of the status of the race, than the poll itself. At this point last year we were at the height of the Mark Foley scandal, which understandably caused massive shifts in polling. There's been nothing similar this time, for Hillary's benefit or the detriment of her pursuers. I think that was the point.

____________________

Danielle Clarke:

How Polls Use "Don't Know" Answers

For those wondering why Obama's poll numbers seem surprisingly low,
check the raw results as opposed to the press releases. If the raw results
aren't posted on the polling company's site, there's probably a reason
for that. ;-)

Awhile back I learned it's considered acceptable to drop, re-allocate
or report "Don't Know" answers - it all depends on the polling company's
choice, and they don't tend to disclose which they did. This can cause
misleading results, especially at this point when there are still a
lot of people who don't know enough about Obama (and other candidates) to
want to commit to one.

For example, a poll may find 5% support Obama, 10% Clinton, 10% others,
and 75% don't know. It's considered acceptable to report this as 20%
support Obama, 40% Clinton, and 40% others - which implies everyone
asked had an opinion of some sort.

Another reason to insist on seeing the raw data is to know what
questions were asked before the question one's interested in - those can
clearly shape the answer.

Here's an article I found that discusses these sorts of issues,
relative to the Australian election:
http://www.aph.gov.au/library/pubs/rn/2003-04/04rn52.pdf
Interpreting opinion polls: some essential details

Four pages long, but important stuff to know - especially if you're
replying to some post claiming Clinton has it all sewn up (or whatever),
and especially if there's no mention of undecideds.

____________________

Mark Blumenthal:

Danielle:

You are absolutely right to suggest that we all look carefully at the verbatim text of survey questions and the way pollsters calculate their results, but there is a difference between a table or reference in a story that omits the undecided category and those that reallocate undecideds as you describe above.

I can't speak for common practice in Australia, but in the US, virtually all polls now report their trial heat question results based on calculations that include the undecided or "don't know" category. Some will put out projections based on their final poll before the election that reallocate undecideds as you describe, but none that I know of are doing so now.

One way to check, if the undecided category is omitted, is to add up the results to each candidate. If they total 100%, then the results have been recalculated.

More to the point, all of the results posted and plotted here on Pollster.com at any time -- including the numbers cited above -- are based on calculations that include the undecided category.

Pollsters sometimes include additional questions to probe how certain voters are about their choice. I have written about that here and here, but that's a different issue.

Where available, we include links to the most complete report of results in the tables that appear below the charts in our Polls section. Follow those links to see the full results and verbatim text.

____________________

Nick P:


I have a problem with polls that are including Al Gore who was Bill Clinton's VP in the initial heat questions. ABC and others don't do this while AP and others do.

Using recalculated votes based on second choice "when available" to calculate the race without Gore means some survey respondents didn't have a second choice. It means that respondents without a second choice are excluded from the heat question about candidates who are, in fact, in the race.

In September, polls not including Gore in the mix show Hillary leading by +17 to +33 points, +17 to +21 excluding ABC. Polls that included Gore and then recalculate based on second choice "when available", show Hillary leading by +13 to +23 points. The second choice method produces similar spreads only if ABC is an outlier and it's way to soon to make that judgment.

My preference would be to first ask about known candidates in the race followed by second heat question with the preface "Suppose Al Gore becomes a candidate...". Then the result does not depend on how many respondents have a second choice.



____________________

Nick P:

Want to add to my post. I did a quick search of second choice questions in my Illinois primary polls and found that no second choice runs as high as 20%. This question was asked for reasons other than recalculation.

These polls did include both governor and president and the ballots were considerably shorter than 8-9 candidates as in the current primaries; i.e., fewer candidates means fewer possible second choices.

Mark. I think, in the interest of full disclosure, you should find out how many respondents don't have a second choice in these national polls. That is, how many respondents are excluded from the heat question covering candidates who are actually in the race.


____________________

Mark Blumenthal:

Just a note that I promoted Danielle's comment and added more thoughts here.

____________________

Mark Blumenthal:

Regarding Nick's question:

I'm not exactly sure what information, "in the interests of full disclosure," he wants me to report. To be clear, here is the way the Post/ABC poll asked about second choice on their July poll, a procedure followed essentially the same way by other pollsters that probe for second choices:

27. (ASKED OF LEANED DEMOCRATS) If the 2008 Democratic presidential primary or caucus in your state were being held today, and the candidates were: (Hillary Clinton, John Edwards, Barack Obama, Al Gore, Bill Richardson, Joe Biden, Chris Dodd, Dennis Kucinich, or Mike Gravel), for whom would you vote?

28. (ASKED OF GORE SUPPORTERS) If Gore does not run, for whom would you vote?

Every Gore supporter had a chance to express a second choice. Those with no second choice would be classified as "undecided" in the recalculated vote.

I cannot vouch for the procedure used by the ABC/Post poll, but interviewers are usually instructed to repeat this list of candidates (without Gore in this instance) if the respondent requests it.

I suppose it is theoretically possible that some of the 2% that were undecided on the first question might offer preference if Gore were not included, but I doubt the number would round up to a single percentage point.

I emailed Nick and asked him to clarify.

____________________

Nick P:


Here is the problem.

This description "using recalculated votes based on second choice when available" may not be entirely descriptive.

So-called "second choice" questions, as evident in my second post above, read: "Which candidate is your second choice for...?" after the initial heat question. My experience is that many respondents don't have a second choice.

In your post above, question 28 asks "If Gore does not run, for whom would you vote?" The condition of Gore not running was in fact given to respondents. It's explicit and similar to what I suggested.

All is well.

____________________

My comment on a "knockout punch" was more in reference to Sen. Clinton being in a position to deliver one, but ALSO to Obama/Edwards need to administer one to her before it's too late.

____________________

jose:

if you take ALL the polls, this isnt much of an outlier if you include the numbers of the polls with gore included that ask "What if Al Gore isnt in the race" which usually follows the questions that include al gore

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR