Pollster.com

Articles and Analysis

 

A Response from Gallup's Frank Newport

Topics: 2008 , Barack Obama , Frank Newport , Gallup , Hillary Clinton , Likely Voters , Mark Lindeman , Pollsters

In response to the dialogue we've been having about the Gallup Daily tracking survey (here and here), Gallup's editor-in-chief Frank Newport sent the following response. Say what you will about Gallup, they are consistently among the most transparent and responsive of the public pollsters.

We are always glad to discuss and analyze Gallup poll data. We generally learn from the insights, comments and questions of others.

The particular reader to whom Mark spends time responding was focusing on the fact that Gallup's daily election tracking was not in exact sync with the vote totals across the 22 Super Tuesday states.

We never reported the Daily Tracking results as projective of what would happen on Super Tuesday. Had that been our intention, we would have used a strict likely voter screen. We would have made specific assumptions about what turnout would be in each state and adjusted each state accordingly. This is what we normally do when trying to predict the actual vote in a state or national election. We did not design the tracking survey methods for that purpose. The general patterns of trends among the broad sample of voters we look are extremely important. But the exact numbers are not projections of the vote in any state or combination of states.

As we reported, candidate support levels in the Super Tuesday states were not dramatically different from the national support levels. This suggests that the momentum and trends observed nationally could be hypothesized to be reflected in the Super Tuesday states.

But for a reader to take that as a prediction by Gallup about the precise vote outcome in all Super Tuesday states (or certainly any individual state) is incorrect.

Our data suggested that among all voters across the country and in Super Tuesday states prior to Feb. 5th, Hillary Clinton had a lead over Barack Obama. Of course not all voters went to the polls -- they never do. Initial estimate are that there was only an average 30% turnout - and a turnout which varied widely across states.

The Gallup Daily election tracking uses a mild screen that filters out just those respondents who say they are not likely to vote in response to a four part question. For Republican voters in February so far that has been 16.9%. For Democratic voters it has been 13.7%. In other words, the screen leaves in more than 80% of national adults, making it functionally similar to the typical registered voter screen.

It certainly wouldn't be expected that a large sample of 80% + of all adults would mirror the actual vote total in a widely disparate group of states with on average just about 30% turnout - and with different turnout within each state. By way of example, when we retrospectively go back and look at the sample of voters from Super Tuesday States from the last five days before Super Tuesday -- screened only among those who are extremely likely to vote -- we find that the vote totals are near a tie, with Obama at 48% and Clinton at 45%.

But we didn't get into that before Super Tuesday because that was not our purpose. The purpose of the national tracking is to monitor the mood of all Democratic and all Republican voters across the country as this primary season progresses. After Jan 3rd, of course, some of these people had already voted, and that proportion continues to go up.

One of the great values of Gallup's tracking is the ability to monitor on a daily basis the changing dynamics of the campaign and to see where the momentum is. (The second value is to be able to aggregate data and look at detailed subgroup analysis). Obama had been gaining in the week or two prior to Super Tuesday to the point where he was essentially tied with Clinton among the broad sample of all voters. But then Clinton retook the momentum. Thus, we hypothesize that had the election been held on Saturday, for example, it looks like Obama would have done better than he eventually ended up doing. But we were not attempting to say what the exact vote totals would be.

[UPDATE (2/10)]: The comments left for this this entry are unusually well-expressed and definitely worth a read. They have inspired a few additional thoughts of my own (delayed, admittedly, by a much needed 36 hour break):

First, we ought not pick just on Gallup. Gallup's broad approach to selecting the "voters" that get asked presidential primary questions is more or less what the other national polls do. I first wrote about this issue almost a year ago and warned about it just last week, on the eve of Super Tuesday when headlines told us of a "dramatic shift" toward Obama.

Second, I am certainly sympathetic to the nearly insurmountable challenges that would be involved in creating a combination actual (past) voter/"likely voter"/"likely caucus goer" model that would apply at the national level and somehow take into account the myriad of different rules for participation and historically varying turnout rates. It would not be at all easy.

Also, be careful what you wish for: Those who remember Gallup's daily during the 2000 election will recall that they applied their "likely voter model" to data as early as Labor Day. Critics made a strong case that while the model works well a week before the election it introduces a lot of variation in the kinds of voters selected as "likely," much of it questionable.

Third, I agree with Mark Lindeman that there is value to Gallup's approach. "it's very interesting," he wrote, "to know what Democrats and Republicans (including leaners) around the country are thinking of "their" candidates, whether their states have already voted or not." However, I tend to agree even more with reader DTM's reaction:

[Quoting Newport] "One of the great values of Gallup's tracking is the ability to monitor on a daily basis the changing dynamics of the campaign and to see where the momentum is."

I think it is fair to say the campaigns are directed at eventually getting actual votes in caucuses and primaries, and the kind of momentum the campaigns care about is the kind of momentum that would further such an end. But given the way in which Gallup is defining "voters", the relationship between what is going on in their tracking polls and what the campaigns are actually trying to accomplish is less than clear.

And this is precisely the sort of confusion which worries me. Indeed, they seem to be more or less encouraging people to use these tracking polls for "horse race" coverage, while at the same time admitting they are not really even trying to screen for actual voters in the upcoming contests, which is what the "race" is all about.

Most people who follow the national poll numbers -- including journalists and political professionals -- treat them as if they measure the views of actual voters in party primaries or caucuses. Pollsters could do a much better job making it clear that they also include far more "leaned partisans" than are likely to actually participate in the party primaries and caucuses (regardless of what respondents claim on vote likelihood questions).

 

Comments
DTM:

I do appreciate the responsiveness, but with all due respect to Gallup, this really doesn't clarify or address the issue. The bottomline is that they seem to be defining groups they are calling "Democratic voters" and "Republican voters". They have explained their screening methodology and some of the implications, but what exactly would be the intuitive definition of these groups of "voters"? Again, it is not all adults, but also not really an attempt to get down to likely voters. They analogize it to registered voters, but of course that isn't accurate (and couldn't be, since many primaries are open).

So, we are left with Gallup defining groups of "voters" that do not appear to map onto any intuitive real world groups, and by Gallup's admission are not really intended to map onto such groups. Which for the reasons we have discussed, strikes me as a problem.

____________________

Raj:

Mark,

Gallup's response should be appreciated (as should pollster.com's work on this point), but two things strike me as problems about this reply. First, their use of the term "voters" is simply bizarre, for example, in the sentence "Of course not all voters went to the polls -- they never do." Given this claim, is a "voter" simply someone who says they have voted at some point in time, or that they might vote someday, or that they will not swear that they won't vote?

Second, Gallup is ducking the question and pretending they aren't ducking. We all know that polls can't be strictly "predictive," but the whole point of polling Dems before an election is to get a sense of how that election might go. To flesh out this example, the reason that Gallup polls Americans, and not people in Tanzania, about the upcoming election is that the opinions of Tanzanians are generally irrelevant to the outcome. Similarly, who on earth cares what "Democratic-identifiers" think if more than half of them can't even be bothered to vote? They might as well poll Republican-identifiers on Democratic candidates! I realize the cost/sample size issue, but Gallup should just be up front about this, and should release weekly or "every-several- days-ly" compilations of "extremely likely" voters only.

A final point - it seems more worthwhile for Gallup to stop doing fully-national polls and to poll only in places that haven't voted yet - so do "national" polling but exclude states that have already voted. These are the places that matter and where results aren't set in stone. Even if 100% of Iowans shifted to supporting Candidate X tomorrow, that would not have any effect on remaining elections. I realize it may be too late this cycle for Gallup to shift how they set up the polling, but this way makes a lot more sense and is what they should do next time, IMHO.

____________________

Chris:

"the whole point of polling Dems before an election is to get a sense of how that election might go."

According to Gallup, that is not the purpose for their polling. Their actions cannot make sense as long as you assume that they are doing something that they are not.

It is up there in black and white: "The purpose of the national tracking is to monitor the mood of all Democratic and all Republican voters across the country as this primary season progresses." You can argue that there is no good reason to do this, but you have to analyze their methods in light of their clearly stated purpose.

You cannot monitor the collective mood of the electorate (assuming such a thing exists) if you are polling different segments of the population each week. You cannot accurately follow small changes in the trends if your methodology is too sensitive to your assumptions (e.g. tightening up the likely voter model may give you a better predictor in absolute terms, but a noisier predictor which obscures small changes in the trend).

If you are not interested in what Gallup is trying to measure, ignore their polls. But don't complain about their methods because they aren't measuring the thing that you are interested in.

____________________

AF:

The point that Raj made about the term "voter" seems very appropriate. Gallup polls "all Democratic and all Republican voters." A voter is someone who votes. If you don't vote, you're not a voter. Gallup can say they're not trying to predict primary outcomes, but their questions ask people whom they would vote for in the primary. Then Gallup calls the people they have polled "voters." But for every person that Gallup polls who does not end up voting, in some ways, they have erroneously labeled the person a "voter." At least within the context of the primary, they are not a "voter."

____________________

DTM:

Chris,

I think what many people are pointing out is that Gallup is being less than clear about what in fact they are trying to do.

Take for instance this comment:

"One of the great values of Gallup's tracking is the ability to monitor on a daily basis the changing dynamics of the campaign and to see where the momentum is."

I think it is fair to say the campaigns are directed at eventually getting actual votes in caucuses and primaries, and the kind of momentum the campaigns care about is the kind of momentum that would further such an end. But given the way in which Gallup is defining "voters", the relationship between what is going on in their tracking polls and what the campaigns are actually trying to accomplish is less than clear.

And this is precisely the sort of confusion which worries me. Indeed, they seem to be more or less encouraging people to use these tracking polls for "horse race" coverage, while at the same time admitting they are not really even trying to screen for actual voters in the upcoming contests, which is what the "race" is all about.

____________________

Raj:

Chris,

You seem to be missing a few key points. Perhaps I needed to be clearer.

First, in the sentence "The purpose of the national tracking is to monitor the mood of all Democratic and all Republican voters across the country as this primary season progresses," what is a voter, if not someone who votes? Sure, Gallup can *define* a voter as whatever they want, but a definition of "voter" that yields 80% of people being defined as voters is flat-out a bad definition. Period.

Second, I simply don't see the point of tracking the "national mood" of Democratic leaners, when a huge chunk of those leaners won't vote in the upcoming primaries and another huge chunk already voted. See the Tanzania example above, or the example of polling Republicans about what Dem candidate they like. Yes, those surveys give SOME information, but is is far less useful than the information they could get if they altered how they did the survey (which, of course, would mean departing from the purpose they state in their message, but so what?). Gallup has good resources and my push is for them to use those resources to get more- rather than less-useful information.

Third, your claim that I should "ignore their polls" if I think they have set-up flaws is silly. I think the way they currently have their methods and purpose set up could be improved, and so I will "complain" (or make suggestions for improvement) if I want to. If you don't like my comments, and if my criticism bothers you so much, feel free to ignore it or go read another website instead of this one.

Finally, their polls will be used by many in the media, etc., to prognosticate, and people will consume them relatively uncritically. Their disclaimer about the purpose of their poll will get lost and people will in general have far less important info than they would if Gallup designed their survey better.

The bottom line is that Gallup's poll would be more useful if it geared itself towards measuring something more meaningful that "the national mood of people who might someday vote or maybe have already voted." This site has been a place where discussions about survey design have been gaining prominence, and my suggestion for Gallup to change what they are doing (or how they are doing it) is surely appropriate here.

-Raj

____________________

Daniel T:

I see both sides of this issue. I think that Raj is right and that regardless of disclaimers people will misuse Gallups daily tracking data. On the other hand, Gallup is correct in that such an issue really isn't their problem. The suvery is doing what it is designed to do and if people misue or misunderstand it how are they supposed to stop that. Raj and I might find what Gallup is doing less than helpful, and confusing in light of other Gallup polls, but Gallup has a right to do whatever polls it wishes.

I don't see the daily tracking polls as academically meaningful but rather a nice feature to talk about along with my horoscope. But then this type of edutainment is what many people are after, and it makes good business sense for Gallup to provide it.

____________________

JimS:

The Gallup Daily only became an 'issue' when the trend reversed away from Obama to Clinton. Before that you seemed happy and pleased to report the numbers. Maybe it was a coincidence but it doesn't look like it.

____________________

Joseph E:

The issue, JimS, is not that the poll changed in favor of Clinton, but that the results of the Feb 5th elections did not correlate with the poll. As others have described previously, a tighter screen of "very likely" voters would have more closely predicted the results: an approximate 50/49 split of the overall vote (Clinton/Obama). The failure of the poll to properly forecast voter intentions on the very day of the election suggests that the trends shown may not be valid, either.

____________________

Mark Lindeman:

@ Chris, Raj, et al.: Right now I pretty much agree with Chris. I don't think what Gallup is doing is without interest. I think it's very interesting to know what Democrats and Republicans (including leaners) around the country are thinking of "their" candidates, whether their states have already voted or not. Gallup can't very well use a general election likely voter screen in February; as Newport says, this screen is functionally similar to a registered-voter screen. Fair enough.

@ Jim S: Basically, we launched into a discussion of why the Gallup results didn't seem to jibe with the Super Tuesday results (as per Joseph E), and realized that we didn't know something basic about Gallup's methodology -- so Mark found out. That sort of thing is what this site is for.

@ Joseph E: That's a succinct problem definition, except that since the survey isn't designed to "forecast voter intentions," we can't lean on that very hard in deciding whether the trends are "valid." One does wonder what they mean. They may hint that because Clinton is so much better known than Obama, she tends to run ahead of him (among Democrats) in states where he isn't actively campaigning.

____________________

JimS:

Joseph E's logic is faulty. The Super Tuesday elections were not a national forum but only 22 states whereas the Gallup poll was national. Second the break in the Gallup poll correctly showed that late breaking deciders were going to Clinton. I saw commentators on ABC who refused to believe this because of their pre-conceived ideas of Obama's momentum in spite of what the exit polls showed. Third the polls that were really wrong, the Zogby polls, clearly biased towards Obama, recieved no scrutiny after the vote except a link to a lame disclaimer by Zogby from Huffington Post that was self serving, dishonest and which Pollster accepted at face value. I'm glad you at least admit that only polls that are favorable to Clinton are likely to be examined.

____________________

DTM:

First, I want to take a moment to once again thank Mark and everyone else at pollster.com for being so responsive to those of us post comments here. In general, this is far and away the best place for an informed discussion about polling I have ever found, and that is a credit to the people behind pollster.

Second, as a final wrap-up thought, I believe that polling organizations like Gallup provide an extremely useful service, but they are also subject to the same market forces as any other businesses. So, we will only get the maximum benefit from them if we are informed (and sometimes critical) consumers. That said, I may be overly optimistic, but I have this feeling that people are paying more attention to this extended primary contest than perhaps any other time in my lifetime, and along the way I think that we are collectively learning a lot more about polls and their limitations than ever in the past (again in no small part thanks to the efforts of people like Mark). Accordingly, I am reasonably hopeful that people are indeed becoming better consumers of polls, and in turn I would expect that to have productive effects on Gallup and their fellow polling organizations.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR