Pollster.com

In response to the dialogue we've been having about the Gallup Daily tracking survey (here and here), Gallup's editor-in-chief Frank Newport sent the following response. Say what you will about Gallup, they are consistently among the most transparent and responsive of the public pollsters.

We are always glad to discuss and analyze Gallup poll data. We generally learn from the insights, comments and questions of others.

The particular reader to whom Mark spends time responding was focusing on the fact that Gallup's daily election tracking was not in exact sync with the vote totals across the 22 Super Tuesday states.

We never reported the Daily Tracking results as projective of what would happen on Super Tuesday. Had that been our intention, we would have used a strict likely voter screen. We would have made specific assumptions about what turnout would be in each state and adjusted each state accordingly. This is what we normally do when trying to predict the actual vote in a state or national election. We did not design the tracking survey methods for that purpose. The general patterns of trends among the broad sample of voters we look are extremely important. But the exact numbers are not projections of the vote in any state or combination of states.

As we reported, candidate support levels in the Super Tuesday states were not dramatically different from the national support levels. This suggests that the momentum and trends observed nationally could be hypothesized to be reflected in the Super Tuesday states.

But for a reader to take that as a prediction by Gallup about the precise vote outcome in all Super Tuesday states (or certainly any individual state) is incorrect.

Our data suggested that among all voters across the country and in Super Tuesday states prior to Feb. 5th, Hillary Clinton had a lead over Barack Obama. Of course not all voters went to the polls -- they never do. Initial estimate are that there was only an average 30% turnout - and a turnout which varied widely across states.

The Gallup Daily election tracking uses a mild screen that filters out just those respondents who say they are not likely to vote in response to a four part question. For Republican voters in February so far that has been 16.9%. For Democratic voters it has been 13.7%. In other words, the screen leaves in more than 80% of national adults, making it functionally similar to the typical registered voter screen.

It certainly wouldn't be expected that a large sample of 80% + of all adults would mirror the actual vote total in a widely disparate group of states with on average just about 30% turnout - and with different turnout within each state. By way of example, when we retrospectively go back and look at the sample of voters from Super Tuesday States from the last five days before Super Tuesday -- screened only among those who are extremely likely to vote -- we find that the vote totals are near a tie, with Obama at 48% and Clinton at 45%.

But we didn't get into that before Super Tuesday because that was not our purpose. The purpose of the national tracking is to monitor the mood of all Democratic and all Republican voters across the country as this primary season progresses. After Jan 3rd, of course, some of these people had already voted, and that proportion continues to go up.

One of the great values of Gallup's tracking is the ability to monitor on a daily basis the changing dynamics of the campaign and to see where the momentum is. (The second value is to be able to aggregate data and look at detailed subgroup analysis). Obama had been gaining in the week or two prior to Super Tuesday to the point where he was essentially tied with Clinton among the broad sample of all voters. But then Clinton retook the momentum. Thus, we hypothesize that had the election been held on Saturday, for example, it looks like Obama would have done better than he eventually ended up doing. But we were not attempting to say what the exact vote totals would be.

[UPDATE (2/10)]: The comments left for this this entry are unusually well-expressed and definitely worth a read. They have inspired a few additional thoughts of my own (delayed, admittedly, by a much needed 36 hour break):

First, we ought not pick just on Gallup. Gallup's broad approach to selecting the "voters" that get asked presidential primary questions is more or less what the other national polls do. I first wrote about this issue almost a year ago and warned about it just last week, on the eve of Super Tuesday when headlines told us of a "dramatic shift" toward Obama.

Second, I am certainly sympathetic to the nearly insurmountable challenges that would be involved in creating a combination actual (past) voter/"likely voter"/"likely caucus goer" model that would apply at the national level and somehow take into account the myriad of different rules for participation and historically varying turnout rates. It would not be at all easy.

Also, be careful what you wish for: Those who remember Gallup's daily during the 2000 election will recall that they applied their "likely voter model" to data as early as Labor Day. Critics made a strong case that while the model works well a week before the election it introduces a lot of variation in the kinds of voters selected as "likely," much of it questionable.

Third, I agree with Mark Lindeman that there is value to Gallup's approach. "it's very interesting," he wrote, "to know what Democrats and Republicans (including leaners) around the country are thinking of "their" candidates, whether their states have already voted or not." However, I tend to agree even more with reader DTM's reaction:

[Quoting Newport] "One of the great values of Gallup's tracking is the ability to monitor on a daily basis the changing dynamics of the campaign and to see where the momentum is."

I think it is fair to say the campaigns are directed at eventually getting actual votes in caucuses and primaries, and the kind of momentum the campaigns care about is the kind of momentum that would further such an end. But given the way in which Gallup is defining "voters", the relationship between what is going on in their tracking polls and what the campaigns are actually trying to accomplish is less than clear.

And this is precisely the sort of confusion which worries me. Indeed, they seem to be more or less encouraging people to use these tracking polls for "horse race" coverage, while at the same time admitting they are not really even trying to screen for actual voters in the upcoming contests, which is what the "race" is all about.

Most people who follow the national poll numbers -- including journalists and political professionals -- treat them as if they measure the views of actual voters in party primaries or caucuses. Pollsters could do a much better job making it clear that they also include far more "leaned partisans" than are likely to actually participate in the party primaries and caucuses (regardless of what respondents claim on vote likelihood questions).

By Mark Blumenthal on February 8, 2008 3:20 PM | Permalink | Comments (12) | TrackBacks (0)

POLL: Gallup Daily Tracking

By Eric Dienstfrey

Gallup Poll

National
Clinton 49, Obama 42

Consumer Confidence
Economy getting worse 77, getting better 14

By Eric Dienstfrey on February 8, 2008 2:50 PM | Permalink | Comments (2) | TrackBacks (0)

My National Journal column, which discusses the Gallup Daily national survey, is now online.

By Mark Blumenthal on February 8, 2008 12:45 PM | Permalink | Comments (3) | TrackBacks (0)

POLL: TIME/SRBI National Survey

By Eric Dienstfrey

TIME/SRBI (analysis, results)

DEMS: Clinton 48, Obama 42
GE: Obama 48, McCain 41... Clinton 46, McCain 46

By Eric Dienstfrey on February 8, 2008 10:26 AM | Permalink | Comments (16) | TrackBacks (0)

Re: Gallup Daily Vs. Super Tuesday

By Mark Blumenthal

While I was finishing my National Journal column late yesterday afternoon, Gallup posted a longer than usual Gallup Daily update that answers most of the questions we asked here yesterday. It is a must read for those closely following the Gallup Daily numbers and other national surveys. Our readers blogged the key passages in the comments last night, but for those who missed it here are the key paragraphs:

The vote opinions of those in Gallup Daily tracking will not, of course, represent the actual vote in various states or in particular combinations of states on Election Day. One reason is that the tracking represents a broad sample of all respondents who say they are at least somewhat likely to vote, removing a small percentage who are unable to vote or not engaged in the campaign to any degree. The "not likely to vote" group is less than 20% in general (among both Republicans and Democrats), meaning that over 80% of American adults are included in the voter figures Gallup reports, making it similar to a typical "registered voter" figure.

Those who track voter turnout in various states that voted on Super Tuesday estimate that actual turnout was around 30%, and varied considerably among states. Thus, a broad sample of over 80% of American adults would not be expected to match the actual voting patterns of the much smaller group that turn out to vote in either party's primary.

There is, in fact, strong evidence in the tracking data from the days prior to Super Tuesday that Obama did significantly better when those who reported the highest likelihood of voting are isolated in the sample. Retrospectively, Gallup analysis can isolate just voters who say they are extremely likely to vote -- about 50% of the sample (this still overestimates actual turnout). The vote preferences of Democrats within that smaller slice for the five days prior to Super Tuesday (and after John Edwards left the race) show that Clinton (45%) and Obama (48%) were basically tied [emphasis added].

This finding is significant since it says something important, not just about the Gallup Daily tracking but about most of the other national surveys that ask about the Democratic primary vote preference among similarly broad samples (that overrepresent primary turnout). Back in April of last year, Open Left blogger Chris Bowers (then with MyDD) wondered whether these overly broad samples in national polls might be inflating Hillary Clinton's advantage. At the time narrower slices of national surveys -- like the one that Gallup did above -- did not support the theory. However, this new evidence, coupled with Obama's consistently better performance in lower-turnout caucuses on Tuesday, suggests that other national surveys may be overstating Clinton's advantage.

Two weeks ago I wondered again if the national screens are "tight" enough. This new evidence from Gallup suggests that if we are interested in the preferences and opinion of Democratic primary voters nationwide, they are not tight enough.

Reacting to these new findings, FlyOnTheWall, the Pollster reader whose question started this discussion, asked:

If Gallup is saying that the sample which includes 80% was wildly off the mark as a predictor of actual voting, but that the sample which included just the 50% of highly likely voters came darn close to predicting how actual voters actually vote - then why the heck don't they use the tighter screen all the time?

If they're trying to find out how all Americans feel, they shouldn't use any screen. But if they're tracking voter sentiment, then they should be screening for voters. And since a loose screen produces results that aren't predictive, and a tight screen produces those that are, I really wish they'd just use the tight screen going forward.

To report daily results based on a rolling average of "extremely likely" to vote respondents, Gallup would either need to call twice as many Americans every night or report a rolling six-day average in order to keep the sample size the same. Read my National Journal column later today (I will add a link when it's up) to get a sense for why it would be a bad idea to do daily tracking based on a smaller sample.

However, Fly makes a very good point. It would certainly be helpful if Gallup could report a weekly average based on just the "extremely likely" to vote respondents. Since they interview 7,000 adults a week, they are uniquely positioned to regularly compare high turnout Democrats to all the rest.