Pollster.com

Articles and Analysis

 

National GOP Contest: Why are ABC/Post & Rasmussen So Different?

Topics: 2008 , IVR , IVR Polls , Likely Voters , The 2008 Race

A suggestion from alert reader and frequent commenter Andrew:

I write to suggest that you analyze the huge discrepancy between the latest Rasmussen and Washington Post/ABC polls. I'm talking about the Republican nomination. Rasmussen says Thompson is up by 4 over RG, while WP/ABC says Rudy is up by 20 pts over FT, who isn't even in second place here (36 RG to 14 FT). One of these pollsters is obviously very wrong. Two polls cannot both be accurate, if their margin of victory do not approximate each other. This is a humongous 24 point discrepancy.

Here, with a little assist from Professor Franklin, is a chart showing the discrepancy that Andrew noticed. The two surveys do seem to show a consistent difference that is clearly about more than random sampling error. The ABC News/Washington Post survey shows Giuliani doing consistently better, and Thompson doing consistently worse, than the automated surveys conducted by Rasmussen Reports, although the discrepancy has been largest in terms of how the most recent ABC/Post poll compares to Rasmussen surveys conducted over the last month or so.

ARasVsWaPo400.png

To try to answer Andrew's question, it makes sense to take two issues separately. First, why are the surveys producing different results for the Republican primary?

At the most basic level, these surveys seem to be measuring the same thing: Where does the Republican nomination contest stand nationally? And both surveys begin with a national sample of working telephone numbers drawn using a random digit dial (RDD) methodology. Take a closer look, however, and you will see some pretty significant difference in methodology:

  • The ABC/Post survey uses live interviewers. Rasmussen uses an automated recorded voice that asks respondents to enter their answers by pushing buttons on a touch tone keypad. This method is known as Interactive Voice Response (IVR). The response rates -- and more importantly, the kinds of people that respond -- are likely different, although neither pollster has released specific response rates for any of the results plotted above.
  • The ABC/Post survey attempts to select a random member of each household to be interviewed by asking "to speak to the household member age 18 or over at home who's had the last birthday" (more details here). Rasmussen interviews whatever adult member of the household answers the telephone. Both organizations weight the final data to reflect the demographics of the population.
  • Rasmussen Reports weights each survey by party identification, using a rolling average of recent survey results as a target (although their party weighting should have little effect on a sub-group of Republican primary voters). The ABC/Post survey does not weight national surveys at this stage in the campaign by party ID.
  • [Update -- one I overlooked: The ABC/Post survey includes Newt Gingrich on their list of choices. Gingrich receives 7% on their most recent survey. If the Rasmussen survey prompts Gingrich as a choice, they do not report it. It is also possible that Rasmussen omits other candidates as well, as t Their report provides results for just Giuliani, Thompson, Romney and McCain. Update II -- Scott Rasmussen informs via email: "We include all announced candidates plus Fred Thompson"].
  • And perhaps most important for Andrew's question: The ABC/Post survey asks the presidential primary question of all adults that identify with or "lean" to the Republicans. The Rasmussen survey screens to a narrower slice of the population: Those they select as "likely Republican primary voters."

Unfortunately, neither pollster tells us the percentage of adults that answered their Republican primary question, but we can take a reasonably educated guess: "Leaned Republicans" have been somewhere between 35% and 42% of the adult population on surveys conducted in recent months by Gallup and the Pew Research Center. If Rasmussen's likely voter selection model for Republican is analogous to their model for Democrats, their "likely Republican primary" subgroup probably represents 20% to 25% of all adults.

Consider also that, even before screening for "likely voters" and regardless of the response rate, those willing to complete an IVR study may well represent a population that is better informed or more politically interested than those who complete a survey with an interviewer.

Put this all together, and it is clear that the Rasmussen survey is reaching a very different population, something I would wager explains much of the difference in the results charted above.

Now, the second question, which result is more "accurate?" It is tempting to say that this question is impossible to answer, since we will never have a national primary election to check it against. But a better answer may be that "accuracy" in this case depends on what we want to use the data for.

If we were trying to predict the outcome of a national primary, and if all other aspects of methodology were equal (which they're not), I would want to look at the narrower slice of "likely voters" rather than all adult "leaned Republicans." Since the nomination process involves series of primaries and caucuses starting with Iowa and New Hampshire, and since the results from those early contests typically influence preferences in the states that vote later, we really need to focus on early states for a more "accurate" assessment of where things stand now. While interesting and fun to follow, these national measurements provide only indirect indicators of the current status of the race for the White House.

Why would the ABC/Post survey want to look at all Republicans, rather than likely voters? Here is the way ABC polling director Gary Langer explained it in his online column this week:

I like to think there are two things we cover in an election campaign. One is the election; the other is the campaign.

The campaign is about who wins. It's about tactics and strategy, fundraising and ad buys, endorsements and get-out-the-vote drives. It's about the score of the game - the horse race, contest-by-contest, and nothing else. We cover it, as we should.

The election is the bigger picture: It's about Americans coming together in their quadrennial exercise of democracy - sizing up where we're at as a country, where we want to be and what kind of person we'd like to lead us there. It's a different story than the horse race, with more texture to it, and plenty of meaning. We cover it, too.

We ask the horse race question in our national polls for context - not to predict the winner of a made-up national primary, but to see how views on issues, candidate attributes and the public's personal characteristics inform their preferences.

Questions like Andrew's are more consequential in the statewide surveys we are tracking here at Pollster.com, and those surveys have been producing some discrepancies even bigger than the one charted above. We will all be in a better to make sense of those differences if we know more about the methodologies pollsters use. I'll be turning to that issue in far more detail next week.

 

Comments
partha:

Clearly, a responsible course of action would be for pollsters to estimate and disclose honestly what the uncertainties in their methodologies are; and refrain from publishing meaningless numbers when the uncertainties approach the numbers themselves.

____________________

Anonymous:

How does the Rasmussen poll, being fully automated, know that the person who is pushing the buttons on the phone is in fact the person they intended to poll? I would think the margin of error has to be much higher in an automated poll. Also in such a poll, if people are in a hurry to get off the phone, they might be likely to push the first name they hear of and hang up.

____________________

Andrew :

Thanks for the methodology breakdown between these two pollsters.

On second thought, I agree with the opinion that these discrepancies are more relevant in statewide contests. Sweeping predicitons are impossible at this time, since one state primary exerts an influence on the next state primary that cannot be measured yet.

Also important is the Gingrich factor.

____________________

Eric:

darn. And i was expecting some good bomb throwing to erupt when i read the headline.

I know rasmussen has been very accurate in the past, but to me it's clear they've become an op-ed piece in 2007.

Case in point:
http://www.rasmussenreports.com/public_content/politics/immigration_bill_failure_proves_rasmussen_s_first_law_of_politics

I just don't think this article quite explains what's going on over at rasmussen. and it's not just the GOP. they had obama over hillary back in May. if the ones responding to rasmussen's poll are in fact the most engaged, why are they by far the most volatile pollers?

____________________

Alex:

Eric: completely agree.

____________________

Nick Panagakis:


All good comments.

Re: "The Rasmussen survey screens to a narrower slice of the population: Those they select as "likely Republican primary voters."

But what about respondents in states without primaries that vote in caucuses instead?

Nick Panagakis

____________________

I think that there has always been something fishy going on related to ABC, nevertheless, I still do not find it out

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR