Pollster.com

Articles and Analysis

 

Why So Much Variation? (Redux)

Topics: Divergent Polls , Gallup , LA Times , Measurement , Newsweek , Party Weighing , Pew Research Center , Pollster.com , Rasmussen

There has been quite a bit of speculation this week over that perennial we enjoy here at Pollster.com: Why are the polls so divergent? In the last week or so, we have had national polls showing everything from a dead even race (Gallup Daily) to a 15 point Obama lead (Newsweek) and a half-dozen or so results in between.

Here are three more sets of reactions and analyses from pollsters themselves:

  • Los Angeles Times pollster Susan Pinkus, as quoted in a blog item from the Times' Don Frederick
  • Richard Morin, now of the Pew Research Center, formerly polling director for The Washington Post, as interviewed by CQ Politics.

Cumulatively, this discussion covers the usual suspects: differences in party identification as measured by the surveys, as well differences in question wording and question order. I wrote about the party ID debate in my column last week, something that Pinkus, Morin and Rasmussen address at the links above.

The Gallup editors raise the issue of question wording and order. Frank Newport points out the Gallup Daily begins with the vote question, while the L.A. Times poll started with the standard question about the direction of the country. He also points out that the L.A. Times vote question includes the phrase, "or would you vote for a candidate from some other party," an option not offered explicitly by the Gallup question.

The decision of the L.A. Times pollsters to ask the vote question second, after the "right-direction, wrong track" question, struck Nate Silver as a "big no-no." It is true that most of the media pollsters have now shifted to asking the presidential vote question at the very beginning of the survey, a practice they generally follow in surveys in the last few months before the election.

But not all do. Most of the pollsters doing internal "benchmark" surveys for campaigns take a different approach that usually involves asking questions about the direction of the country and favorable ratings of the candidates before asking about vote preference. You can see examples of this philosophy in the survey out today from Democracy Corps poll conducted by Democratic firm Greenberg-Quinlan-Rosner, or the most recent NBC/Wall Street Journal poll.

The rationale for that question order is that voters do not make their choices on a whim. They cast their ballots after some reflection at the end of the campaign, not by being ambushed and asked to make an immediate decision in the middle of the summer. If neutral questions that get respondents thinking about the candidates and the direction of the country for a minute or two produce a result different than asking immediately for vote preference, then the effect gets us closer to the way they will approach the real campaign. As Silver put it in a second thought, posted after some reflection:

If the mere suggestion that the country might be on the wrong track is enough to send scores of independents into the Obama column, imagine what a concerted effort to frame the discourse that way might do.

It is always important to consider question wording and question order, although I'm dubious that the placement of the "right direction wrong track" question alone explains why the L.A. Times had a better result for Obama than most other recent polls. After all, the Newsweek poll started with the vote questions, and they showed an even bigger (15 point ) margin for Obama.

 

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR