Pollster.com

Articles and Analysis

 

Dueling Gallups, Part III

Topics: 2008 , ABC/Washington Post , Divergent Polls , Gallup , IVR , National Journal , Pollster , PPP , Rasmussen , SurveyUSA , USAToday Gallup

I am crashing on a deadline for my National Journal column, but want to pass along the analysis just posted by Gallup about the differences in their recent polls on the Democratic nomination. Here is the gist:

Gallup probably does more national polling than any other firm does, so this is certainly not the first time and won't be the last time that one of our polls produces results inconsistent with other polls. When this does happen, though, we do our best to analyze why those differences might have occurred. Unfortunately, these investigations rarely yield a proverbial smoking gun, and, beyond normal sampling error, it is not clear why one poll might differ from other polls -- even in cases when the differences between polls appear to be beyond the margin of error. This appears to be the case with Gallup Poll Daily tracking and USA Today/Gallup poll as well.

Having read my two posts on this subject earlier this week, a pollster friend of mine emailed to ask, "in spirit of Occam’s Razor, am I correct in simply saying 'they don’t have a clue why there is a difference?'"

That's about right.

To their credit, Gallup's analysis thoroughly reviews the subtle methodological differences between the two surveys, and includes a spreadsheet comparing the demographic composition of the two samples. That is more than we can say for some other pollsters. Still, their bottom line is uncertainty:

To reiterate, none of the known differences between Gallup Poll Daily tracking and the USA Today/Gallup poll, based on our analysis, are obvious causes for the disparity in the Democratic ballot estimate between the two samples. In fact, the estimates of the Republican horse race between the two samples were almost identical.

I will let our always throughout readers chew this new analysis over while I go back to drafting my column.

 

Comments
Berkeleyguy:

Whatever their explanation, unless they say one of them is wrong for some particular reasons and retract it, I just don't see how this will help their clients, the public, and the reputation of this industry. If such minor difference in administration of the polls can result in such huge differences in their implications, how can we know what they are telling us? According to Gallup, what is the state of the race? Is the race tied, as the daily tracking suggests, or has Obama pulled up with a huge lead? How can they justify the MOEs? Why should I, as a consumer, pay any attention to their numbers? Can they say, with any level of certainty, that the numbers that they produce have anything to do with the facts on the ground?

____________________

Berkeleyguy:

Whatever their explanation, unless they say one of them is wrong for some particular reasons and retract it, I just don't see how this will help their clients, the public, and the reputation of this industry. If such minor difference in administration of the polls can result in such huge differences in their implications, how can we know what they are telling us? According to Gallup, what is the state of the race? Is the race tied, as the daily tracking suggests, or has Obama pulled up with a huge lead? How can they justify the MOEs? Why should I, as a consumer, pay any attention to their numbers? Can they say, with any level of certainty, that the numbers that they produce have anything to do with the facts on the ground?

____________________

Erik:

Reading this post and the gallup post, I personally think the strong Obama showing was from two factors

1. Different samples- the daily tracking poll samples more highly motivated, high information voters who are generally more polarized between the two candidated
2. Question order...they asked the candidate preferences LAST in the USA Today poll after favorables!..they totally primed respondents select Obama as he has higher favorables and in the same survey 3/4 thought he was more electable than hillary

____________________

InfoGiant:

I tried to explain the difference here, and really don't think it's surprising that polling two groups of people on the same day yield different results when you have one candidate falling and one rocketing upwards. Did they sample in the same county? Had Barack stumped in the cities where they had called? Getting different results is hardly a surprise.

http://infogiant.wordpress.com/2008/02/26/different-polls-different-answers-the-elevator-theory/

http://infogiant.wordpress.com/2008/02/25/would-the-real-hillary-please-stand-up/

____________________

BitJam:

ISTM the simplest explanation is that the discrepancy was due to statistical variations that lay outside the margin of error. Although rare, such events are bound to occur. It would be very suspicious if they never occurred.

Bell curves do have tails.

____________________

VAR:

Is it not correct that in a typical poll, the margin of error typically represents the 95% confidence interval margin for the estimates provided, no? So, given a survey organization taking 20 polls on the same subject on the same day, 1 of the 20 will produce results outside the confidence interval for the other 19.

Is that not correct? Or is the margin of error represented at some other level (99% confidence, 90% confidence)?

____________________

Erik (the other Erik):

You are right, that's the only big difference. Are we looking at a Polling 101 kind of problem?

The only way this could be significant is if people had a tendency to state their preference on the ballot question based on some kind of loyalties or identities OTHER than their personal preference. I don't know that this is what's going on here, but there is a possibility that people want to vote for Clinton out of loyalty to the family or because they feel they would regret not voting for a woman the first time (which came up in my family!).

Bottom line: It's the only big difference, and while we can't definitely explain why it would be important we can at least imagine a possibility.

(thankfully, this thread hasn't been taken over by people who think other people might care about their own personal political choices, so we can talk about polling!)

____________________

Berkeleyguy:

Whatever their explanation, unless they say one of them is wrong for some particular reasons and retract it, I just don't see how this will help their clients, the public, and the reputation of this industry. If such minor difference in administration of the polls can result in such huge differences in their implications, how can we know what they are telling us? According to Gallup, what is the state of the race? Is the race tied, as the daily tracking suggests, or has Obama pulled up with a huge lead? How can they justify the MOEs? Why should I, as a consumer, pay any attention to their numbers? Can they say, with any level of certainty, that the numbers that they produce have anything to do with the facts on the ground?

____________________

mary:

approximately 60% of the us population do not vote due to a total disillusion with the system
all Obama needs is 10% of them and the White House is his.

____________________

mary:

approximately 60% of the us population do not vote due to a total disillusion with the system
all Obama needs is 10% of them and the White House is his.

____________________

Daniel T:

Thanks Gallup. This is the point I made in response to the first article Mark wrote on the topic. Not only don't they have a clue, IT'S NOT A BIG DEAL. What part of "random variation" don't people understand? Anybody who thinks there is a problem here doesn't understand polling. If this had happened multiple times, that would be a problem. But a one-off is just that, a one-off; forget it until you see a pattern.

____________________

Jeff L:

What strikes me as I look at the spreadsheet is the difference in the 2 polls for "Independents." The daily tracking poll shows Obama leading 51-39 (12 point spread) with a 24% weight and the USA Today poll shows Obama leading 62-27 (35 point spread) with a 31% weight. It is also striking that there is almost no difference between the polls for Democrats - Clinton +2 in the daily tracker and Obama +2 in the USA Today poll.

To quote from their analysis: "The USA Today/Gallup is reported on the basis of all Democrats and Democratic-leaning independents. The Gallup Poll Daily tracking is reported on the basis of Democrats and Democratic-leaning voters -- those who say they are "extremely likely," "very likely," or "somewhat likely" to vote in their state's primary or caucus or report that they have already voted."

I'd like to understand more about the screening for "Democratic-leaning independents" vs. "Democratic-leaning voters" in the 2 polls. It would seem that this difference could explain at least some, if not most, of the disparity.

____________________

You guys have to remember that most of the population only understands who is ahead in the state polls and could care less about how it was polled. They trust that you are not attempting to sway their vote by reporting that Obama (or Hillary) is way ahead. The naive want to go with the Winner! CNN has clearly demostrated anti-Hillary tactics during this entire campaign and polls can be just as misleading. The so-called 'educated' group,who were in their puberty years during Clinton's balanced budget years, are gullible and naive enough to swallow their toxic pill. I am conscienable, thank God, and I am weighing the truth. Maybe we can repeat California, inspite of the polls having given Obama the edge.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR