Pollster.com

Articles and Analysis

 

Deuling Gallups

Topics: 2008 , Barack Obama , CNN , Divergent Polls , Frank Newport , Hillary Clinton , USAToday Gallup

In the midst of a very busy day, the polling mystery de jour is certainly the dueling national Democratic primary results from Gallup. The new USA Today/Gallup poll out this afternoon (and conducted Thursday to Sunday, 2/21-24) shows Barack Obama leading Hillary Clinton nationally by 12 points (51% to 39%), while the latest Gallup "Daily" tracking release (conducted Friday to Sunday, 2/22-24) shows Obama edging Clinton by just two points (47% to 45%).

Our comments section and my email are overflowing with questions about why that may be. The answer is not obvious. Gallup editor-in-chief Frank Newport devotes an "editor's blog" item to this question (hat tip to Brian Schaffner who posted a link in the comments). Newport seems to be stumped:

There are some differences in the methodology between these two Gallup polls, including different days of interviewing and some differences in question order. But generally speaking, when Gallup conducts separate polls measuring the same variables at roughly the same times, the estimates are usually and predictably quite similar.

[...]

Sampling differences and the impact of random factors inherent in the survey process can sometimes explain why two polls are different. Given the similarities in the two polls' Republican estimates, however, I think the Democratic differences may well be another indicator of the conflicted nature of the Democratic race this year. It's not an easy choice, and as pre-election polls have shown in reference to the actual vote in some of the primary states, there is a lot of volatility out there among Democrats.

I have written quite a bit about the "hard choice" that Democrats face, and you can see continuing evidence of it in today's new Texas poll from CNN/ORC. Among likely Democratic primary voters in Texas, both Barack Obama and Hillary Clinton enjoy 80% favorable ratings. However, the mere potential for "volatility" does not explain as big a difference as is evident on the two Gallup results. It might take very little to shift vote preference but, in this case, what factors triggered the difference?

As such, I am curious about the "differences in question order" that Newport alludes to. It would be helpful to know what questions came just before the primary vote question on both surveys. One clue may be that the new USA Today/Gallup poll applies their standard "likely voter" model to the general election results. Gallup's model involves eight questions about past voting, vote intent, political interest and knowledge of voting procedures. Did any of those precede the primary vote preference question?

UPDATE - Gallup's Frank Newport clarifies: The Gallup Daily screen is a bit different from the procedure used for the USA Today/Gallup poll. Details -- and more information on the questions that came first on the USAT/Gallup poll -- now posted here.

 

Comments
erik:

I think this passage from the gallup poll blog may be telling:

"At the same time, I think a telling question comes from the USA Today/Gallup poll. It asks Americans who they "think will win the Democratic nomination for president this year." The key point: a large majority of Republicans/Republican leaners (82%) and Democrats/Democratic leaners (70%) assume Obama will win the nomination."

If Gallup asked this question BEFORE they asked their nomination preferences, this could have primed people to state a preference for Obama when this consideration was made salient.

It would be good to know at what point this question was asked in relation to the nomination preference question.

____________________

I'd have thought that if there was a significant difference in question order, that would have jumped out right away as the answer.

I wonder what the internal politics are with regard to Newport saying which poll is "right" and which is "wrong."

____________________

Alcibiades:

Given most states and voters have already voted, asking "Who do you think will win the nomination" first may actually yield a more accurate result. I'm puzzled as to why Gallup, Rasmussen, and presumably other pollsters are still polling voters in Iowa, California, Illinois, etc when we have already have actual vote in from those states, which are obviously the best tool in figuring out how people did vote in a primary. Right now, it seems that a lot of Democrats are voting for Obama because they see him as the presumptive nominee whether or not they say they prefer Clinton in polling.

____________________

mike in ohio:

I think Alcib is on to something, and maybe we can get some numbers later on this. There IS a difference between who a voter might prefer, who the voter thinks will prevail, and who the voter is going to vote for, especially at this point in the cycle.

Maybe that can explain the discrepancy. I'd love to see more work on that theory. THANKS ALCIB.

____________________

Uri:

the sense in counting states that have already voted is to see how people feel now. It's possible that former Clinton voters have switched sides.

Either way, all the polls show Hillary losing Texas, so it's all over anyway in two weeks. Waste of money to go on doing those national polls anyway.

____________________

Ike:

Mark,
I'm getting confuseder. Aalthough I haven't looked back to your earlier post on the Gallup daily tracker, I seemed to recall that the daily sample is "national adults" (as today's Gallup post also says) --but the Pollster tables with your national trends lists Gallup daily's as "LV". The little blurb from Susan Page now on the USAT webpage doesn't explicitly say that the Gallup/USAT survey is "LV", although your post above seems to suggest that it is. I guess I should get un-lazy and go look at the USAT pdf when it comes available. [But of course: if different samples --adults versus LV-- were the reason, Newport would have mentioned it in the post you link above]

____________________

Jesse:

Ok, here's my numskull answer:
Each poll is off, a bit, but that's why they have MOEs. In fact, the MOEs of each Obama and Clinton's numbers in these polls do overlap, Obama's substantially.

So while the 47-45 result with a +/- of 3 negates the possibility of a 51-39 result and vice versa, the MOEs do indicate possible results that aren't negated by either poll: Obama at 48, 49 or 50%, and Clinton at (exactly) 42%.

Now, I'm sure that this is not what any pollster hopes to achieve when they conduct a poll, it does seem to salvage some validity to the system, as well as to the practice of averaging polls together.

____________________

Daniel T:

Jesse. I'm with you. I don't understand the need to make a big deal out of this yet. If we had a consistent pattern where the daily tracking was off from the USAT poll then it would be a problem. While the difference is fairly large, once you take into account MOE it shrinks down. There is such a thing as random variation. So this much time wasted on what is so far a one-off is not useful.

____________________

distantobserver:

Ike, both polls are described as DL by Gallup ('Based on national Democratic and Democratic-leaning voters' for the daily, 'Based on 1,009 Democrats and Democratic leaners' for the USA TODAY/Gallup). So you've got a point there, when you crit the LV label.

However, the USAT poll seems to consistently lean towards Obama. Two weeks ago they had him +3 while the 'daily' still had Clinton between +5 and +2.

The last USAT before Super Tuesday had Clinton +1 - not so bad when I look at the results.

A possible explanation could be in this passage from the USAT methodology statement:

'Interviews are conducted with respondents on landline telephones (for respondents with a landline telephone) and cellular phones (for respondents who are cell phone only).'

I'm not sure whether the daily tracker includes cell phone only DLs. At least they don't explicitely mention them. But I think that the percentage of cell phone only people (cpo) could be actually higher in the Obama leaning slice of the electorate.

PPO (http://people-press.org/reports/pdf/391.pdf) had these numbers:

61% of the cpo were male
80% of the cpo were younger than 50, 85 below 65
19% were 'black', 13% "hispanic'

____________________

InfoGiant:

I think one of the main answer is that you have to elevators passing eachother in Hillary and Barack. If you just were measuring two elevators at standstill, then it would likely be easy to measure what floor they are currently on. When one is going up, and one is going down (swinging, by the way), then there's no way to get the same answer with everyone as they are both rising and falling at different speeds. Full theory:

http://infogiant.wordpress.com/

____________________

Ike:

We probably need another label other than "LV" for Gallup, because when Gallup uses the term "likely voters" it means something specific: The famous likely voter "cut off" model based on an eight question scale. As far as I know, Gallup does not use the vaunted "likely voter model" for the primary vote questions on either the Gallup Daily or USA Today Gallup poll.

The Gallup Daily tracking, as per our discussion here a few weeks ago, uses party identification combined with a simple screen question to select a subgroup that they identify as primary "voters." Quoting from Jeff Jones in that post:

Republicans or Republican-leaning independents who say they are extremely, very or somewhat likely to vote in their state�s primary or caucus when it is held.

Democrats or Democratic-leaning independents who say they are extremely, very or somewhat likely to vote in their state�s primary or caucus when it is held.

We [also] make provisions for those residing in states that have already held their primary caucus � those who indicate they have already voted are considered extremely likely to vote, and those who did not vote in their state�s primary or caucus would be excluded from the base.

Based on Newport's comments, I assume that Gallup used the same primary voter procedure for the USA Today Gallup poll question -- but to be precise about it -- we do not know for certain. Let me see if I can get them to confirm.


____________________

Kabindra:

This underscores how unreliable polls are this season. Close Obama win predictions have become large Obama wins in actuality, large Obama win predictions have translated into large Clinton victories.

Gallup should really be worried about its methodology.

____________________

Ike:

Everybody have their calculators handy? Good. Here's the fine print on yesterday's USAT/Gallup:

http://www.usatoday.com/news/polls/tables/live/2008-02-24-poll.htm

____________________

Mark Blumenthal:

I assumed wrong. The Gallup Daily screen is a bit different from the procedure used for the USA Today/Gallup poll. Details -- and more information on the questions that came first on the USAT/Gallup poll -- now posted here.

____________________

Caroline:

March 07, 2008
POLL: Gallup Daily Tracking

Gallup Poll

National 3/4 - 3/6

Clinton 48, Obama 44

Hillary is gaining.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR