DTM:

I understand that Gallup could not reliably use its current methodology with a tighter screen.

But I am not sure that really addresses FlyOnTheWall's issue. To slightly reframe the question, it appears that Gallup may have put together a methodology that supports a somewhat odd poll, one that is sampling neither all adults, nor likely primary voters, but rather some group in between (and there is a lot of room in between).

Now I also understand that for well-informed consumers of polls, this may not be a problem. Data is data, and such consumers will undoubtedly be happy to get Gallup's data even if it does not directly address any of the more intuitive questions we might be interested in asking.

But I think it is also worth being concerned about what less well-informed consumers of polls might do with these numbers. And unfortunately in my view that includes most pundits and members of the press.

Posted on February 8, 2008 7:52 AM

 s.b.:

As we goforward less and less states use caucuses, infact very few left do and more and more are closed to only democrats.

The numbers have been of in states where republicans and independents can show up to caucuses of very few people and skew the results. This may not in fact reflect the actual preferrence of the voters in that state.

In fact many of these Republicans that show up to vote for Obama in Iowa or Idaho, will probabl yvote Republican in a general election and in fact have come out to vote against Clinton.

As I have said this will less of an issue in primaries and certainly closed primaries, where you have to register a month a head of time. I believe, for example that Obama 's lead will be much less in Louisiana a closed pre-registration primary than one would expect from just the similar demographics to SC.

Posted on February 8, 2008 8:55 AM

 FlyOnTheWall:

DTM:

Thanks for putting that more eloquently than I was able.

Yes, that's about the size of it. I'd just add that Rasmussen uses a four-day rolling average, presumably for precisely this reason. It's something Gallup might consider. (I know, heresy. Gallup taking polling pointers from Rasmussen.)

Personally, I like a poll that produces horse-race numbers, and wouldn't mind the lowered accuracy of a smaller sample in exchange for more predictive results. (After all, the larger sample they're using is non-predictive, so there's not much advantage to its greater accuracy). Or, Gallup could decide to get out of the horse-race business, and mine its sample for truly-representative public opinion on a whole range of issues. But I do think that it's irresponsible to go on releasing the loose-screen numbers, given that they're going to be siezed upon by the media, which lacks sophistication, as evidence of things they (evidently) don't actually show. That would still hold true even if they released both loose- and tight-screen numbers; I'm sure only the more dramatic spread would be reported, whichever screen had produced it. That's just the nature of the news cycle. If we can't persaude reporters to cite margins of error or sample size, I don't think we're going to be able to educate them about likely-voter screens.

For the curious, I've been blogging about this (and some unrelated issues) over here:
http://tpmcafe.talkingpointsmemo.com/talk/blogs/flyontnewall/

Posted on February 8, 2008 9:23 AM

 Mark Blumenthal:

Re - Rasmussen vs. Gallup:

Keep in mind that Rasmussen's calls are automated, so the marginal costs of increasing their sample size are relatively trivial. Gallup pays live human beings to conduct their interviews, so doubling the sample size would be very expensive. As it is, their commitment to interviewing 1,000 adults a night is considerable. As I wrote, among those doing traditional live-interviewer surveys, they are now uniquely positioned to regularly release results based on a relatively tight national screen.

Posted on February 8, 2008 9:40 AM

 DTM:

Mark,

But I think that just returns us to the central problem: what Gallup is doing is undoubtedly uniquely valuable to informed consumers of polls like yourself, so I can understand the reluctance to call on them to abandon these efforts in the absence of a practical alternative that would provide the same value. And actually, I wouldn't want them to abandon these efforts either, because I think they are going to build up a very useful data set for those seeking to improve their understanding of the American electorate.

At the same time, I really think that these numbers may end up misleading a lot of people who are currently engaged in the nomination process. And that strikes me as a problem.

Maybe then the best we can do is just try to educate people about this issue, which of course is consistent with the general idea behind pollster.com. But I might suggest considering being relatively aggressive (and perhaps even monotonous) in pushing the appropriate caveats in this particular case. Among other reasons, I would make that suggestion because Gallup is a well-known and respected name among the general public, and because these numbers are going to be coming out daily for the indefinite future and therefore are likely to have an ongoing impact.

Posted on February 8, 2008 10:01 AM

 Braden:

When in doubt, Gallup should release both. It wouldn't be too difficult to use a rolling six-day sample of extremely likely voters to give everyone a more predictive estimate of voter sentiment, even if a six-day period likely introduces substantial error based on the time lag. However, if the public could compare both numbers simultaneously it would raise awareness of the typical three-day poll's failings.

My more general comment is that the sudden trend toward REALLY close national elections suggests that Gallup and other national pollsters need to make their sample selection more rigorous. The divergences in polls throughout the country post facto revealed them to be fairly useless indicators, even though they were being closely parsed by everyone in the days leading up to the election.

I was especially astonished to see a national polling agency release state-by-state results from a 1,500 person survey. There is simply no way those results could pass muster as a legitimate sample for each state. I think it was a bit unethical to release those poll results when they were likely to be statistically inconclusive.

Posted on February 8, 2008 10:31 AM