Articles and Analysis


Re: Gallup Daily Vs. Super Tuesday

Topics: 2008 , Likely Voters

While I was finishing my National Journal column late yesterday afternoon, Gallup posted a longer than usual Gallup Daily update that answers most of the questions we asked here yesterday. It is a must read for those closely following the Gallup Daily numbers and other national surveys. Our readers blogged the key passages in the comments last night, but for those who missed it here are the key paragraphs:

The vote opinions of those in Gallup Daily tracking will not, of course, represent the actual vote in various states or in particular combinations of states on Election Day. One reason is that the tracking represents a broad sample of all respondents who say they are at least somewhat likely to vote, removing a small percentage who are unable to vote or not engaged in the campaign to any degree. The "not likely to vote" group is less than 20% in general (among both Republicans and Democrats), meaning that over 80% of American adults are included in the voter figures Gallup reports, making it similar to a typical "registered voter" figure.

Those who track voter turnout in various states that voted on Super Tuesday estimate that actual turnout was around 30%, and varied considerably among states. Thus, a broad sample of over 80% of American adults would not be expected to match the actual voting patterns of the much smaller group that turn out to vote in either party's primary.

There is, in fact, strong evidence in the tracking data from the days prior to Super Tuesday that Obama did significantly better when those who reported the highest likelihood of voting are isolated in the sample. Retrospectively, Gallup analysis can isolate just voters who say they are extremely likely to vote -- about 50% of the sample (this still overestimates actual turnout). The vote preferences of Democrats within that smaller slice for the five days prior to Super Tuesday (and after John Edwards left the race) show that Clinton (45%) and Obama (48%) were basically tied [emphasis added].

This finding is significant since it says something important, not just about the Gallup Daily tracking but about most of the other national surveys that ask about the Democratic primary vote preference among similarly broad samples (that overrepresent primary turnout). Back in April of last year, Open Left blogger Chris Bowers (then with MyDD) wondered whether these overly broad samples in national polls might be inflating Hillary Clinton's advantage. At the time narrower slices of national surveys -- like the one that Gallup did above -- did not support the theory. However, this new evidence, coupled with Obama's consistently better performance in lower-turnout caucuses on Tuesday, suggests that other national surveys may be overstating Clinton's advantage.

Two weeks ago I wondered again if the national screens are "tight" enough. This new evidence from Gallup suggests that if we are interested in the preferences and opinion of Democratic primary voters nationwide, they are not tight enough.

Reacting to these new findings, FlyOnTheWall, the Pollster reader whose question started this discussion, asked:

If Gallup is saying that the sample which includes 80% was wildly off the mark as a predictor of actual voting, but that the sample which included just the 50% of highly likely voters came darn close to predicting how actual voters actually vote - then why the heck don't they use the tighter screen all the time?

If they're trying to find out how all Americans feel, they shouldn't use any screen. But if they're tracking voter sentiment, then they should be screening for voters. And since a loose screen produces results that aren't predictive, and a tight screen produces those that are, I really wish they'd just use the tight screen going forward.

To report daily results based on a rolling average of "extremely likely" to vote respondents, Gallup would either need to call twice as many Americans every night or report a rolling six-day average in order to keep the sample size the same. Read my National Journal column later today (I will add a link when it's up) to get a sense for why it would be a bad idea to do daily tracking based on a smaller sample.

However, Fly makes a very good point. It would certainly be helpful if Gallup could report a weekly average based on just the "extremely likely" to vote respondents. Since they interview 7,000 adults a week, they are uniquely positioned to regularly compare high turnout Democrats to all the rest.



I understand that Gallup could not reliably use its current methodology with a tighter screen.

But I am not sure that really addresses FlyOnTheWall's issue. To slightly reframe the question, it appears that Gallup may have put together a methodology that supports a somewhat odd poll, one that is sampling neither all adults, nor likely primary voters, but rather some group in between (and there is a lot of room in between).

Now I also understand that for well-informed consumers of polls, this may not be a problem. Data is data, and such consumers will undoubtedly be happy to get Gallup's data even if it does not directly address any of the more intuitive questions we might be interested in asking.

But I think it is also worth being concerned about what less well-informed consumers of polls might do with these numbers. And unfortunately in my view that includes most pundits and members of the press.



As we goforward less and less states use caucuses, infact very few left do and more and more are closed to only democrats.

The numbers have been of in states where republicans and independents can show up to caucuses of very few people and skew the results. This may not in fact reflect the actual preferrence of the voters in that state.

In fact many of these Republicans that show up to vote for Obama in Iowa or Idaho, will probabl yvote Republican in a general election and in fact have come out to vote against Clinton.

As I have said this will less of an issue in primaries and certainly closed primaries, where you have to register a month a head of time. I believe, for example that Obama 's lead will be much less in Louisiana a closed pre-registration primary than one would expect from just the similar demographics to SC.




Thanks for putting that more eloquently than I was able.

Yes, that's about the size of it. I'd just add that Rasmussen uses a four-day rolling average, presumably for precisely this reason. It's something Gallup might consider. (I know, heresy. Gallup taking polling pointers from Rasmussen.)

Personally, I like a poll that produces horse-race numbers, and wouldn't mind the lowered accuracy of a smaller sample in exchange for more predictive results. (After all, the larger sample they're using is non-predictive, so there's not much advantage to its greater accuracy). Or, Gallup could decide to get out of the horse-race business, and mine its sample for truly-representative public opinion on a whole range of issues. But I do think that it's irresponsible to go on releasing the loose-screen numbers, given that they're going to be siezed upon by the media, which lacks sophistication, as evidence of things they (evidently) don't actually show. That would still hold true even if they released both loose- and tight-screen numbers; I'm sure only the more dramatic spread would be reported, whichever screen had produced it. That's just the nature of the news cycle. If we can't persaude reporters to cite margins of error or sample size, I don't think we're going to be able to educate them about likely-voter screens.

For the curious, I've been blogging about this (and some unrelated issues) over here:


Mark Blumenthal:

Re - Rasmussen vs. Gallup:

Keep in mind that Rasmussen's calls are automated, so the marginal costs of increasing their sample size are relatively trivial. Gallup pays live human beings to conduct their interviews, so doubling the sample size would be very expensive. As it is, their commitment to interviewing 1,000 adults a night is considerable. As I wrote, among those doing traditional live-interviewer surveys, they are now uniquely positioned to regularly release results based on a relatively tight national screen.




But I think that just returns us to the central problem: what Gallup is doing is undoubtedly uniquely valuable to informed consumers of polls like yourself, so I can understand the reluctance to call on them to abandon these efforts in the absence of a practical alternative that would provide the same value. And actually, I wouldn't want them to abandon these efforts either, because I think they are going to build up a very useful data set for those seeking to improve their understanding of the American electorate.

At the same time, I really think that these numbers may end up misleading a lot of people who are currently engaged in the nomination process. And that strikes me as a problem.

Maybe then the best we can do is just try to educate people about this issue, which of course is consistent with the general idea behind pollster.com. But I might suggest considering being relatively aggressive (and perhaps even monotonous) in pushing the appropriate caveats in this particular case. Among other reasons, I would make that suggestion because Gallup is a well-known and respected name among the general public, and because these numbers are going to be coming out daily for the indefinite future and therefore are likely to have an ongoing impact.



When in doubt, Gallup should release both. It wouldn't be too difficult to use a rolling six-day sample of extremely likely voters to give everyone a more predictive estimate of voter sentiment, even if a six-day period likely introduces substantial error based on the time lag. However, if the public could compare both numbers simultaneously it would raise awareness of the typical three-day poll's failings.

My more general comment is that the sudden trend toward REALLY close national elections suggests that Gallup and other national pollsters need to make their sample selection more rigorous. The divergences in polls throughout the country post facto revealed them to be fairly useless indicators, even though they were being closely parsed by everyone in the days leading up to the election.

I was especially astonished to see a national polling agency release state-by-state results from a 1,500 person survey. There is simply no way those results could pass muster as a legitimate sample for each state. I think it was a bit unethical to release those poll results when they were likely to be statistically inconclusive.


Post a comment

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.