Pollster.com

Articles and Analysis

 

The "Loopy" Zogby Polls


All pollsters, it seems, eventually find themselves with what Andy Kohut once referred to as "loopy" results. His comment was about the Gallup polls in the 2000 election, though in September 2004, Pew experienced such results itself, and of course several polls this campaign season have produced inexplicable or "wrong" numbers, as indicated by the subsequent primary election vote counts.

 

This time, it's Zogby's turn to confuse the masses. His latest Reuters/Zogby poll, based on a sample of 1,089 "likely voters" drawn from listed telephone numbers, conducted Aug. 14-16, 2008, shows McCain over Obama by 46% to 41%.

 

Two days earlier, Zogby reported substantially different results. His online poll (of self-selected people who want to be part of his Internet polling sample) of 3,339 "likely voters," conducted Aug. 12-14, showed Obama with a three-point lead, 43% to 40%.

 

By Zogby's own calculation of the margins of error of each poll, the difference between the two polls in McCain's support (46% in the later telephone poll vs. 40% in the earlier online poll) is statistically significant. The difference in Obama's support (41% vs. 43% respectively) would not be statistically significant. Still, the 8-point difference in the margin of McCain's lead would be significant - a McCain 5-point lead vs. an Obama 3-point lead in the earlier poll.

 

If we believe both polls, the period of Aug. 13-14 must have been a real bummer for Obama and an electoral high for McCain. Whatever it was that caused millions of voters to "change" their minds and gravitate toward the Republican candidate in the two-day period, however, escaped my notice. Perhaps others have been more observant.

 

Of course, there are reasons to discount both polls. Zogby has long been known for refusing to use sound methods in designing his samples. The use of only listed telephone numbers, and the self-selected samples of voters in his online surveys, are the two most salient problems. Still, his last pre-election polls often come close to the actual election results, and many news media outlets regularly publish his results.

 

Regardless of how loopy are Zogby's results, or his sampling methods, his polls contribute to what Kathy Frankovic, in her AAPOR presidential address in 1993,[i] referred to as the "noise and clamor" of the polls. Thus, they're worth noting, if only in disbelief.



[i] Kathleen A. Frankovic, Presidential Address "Noise and Clamor: The Unintended Consequences of Success," Public Opinion Quarterly, Vol. 57, No. 3 (Autumn, 1993), pp. 441-447.

 

Comments
David:

If "his last pre-election polls often come close to the actual election results", why are his results "loopy"?

____________________

PatrickM:

What David neglected to point out is that Zogby's polls just prior to his final poll can be wildly different, but his final polls "somehow" seem to snap into matching the average of the other polls in that race (I think this year's New Hampshire primary is an example).

A poll he did for an advocacy group in New Jersey was released yesterday, and shows he's as adept at biased question wording as he is at sample manipulation. A full-service snake oil salesman.

____________________

David:

no, more just a comment on how the last w paragraphs are written - I understand how his polls in the middle of the election are loopy, it just seemed to read odd the way the article ended.

____________________

ScheduleC:

A couple of days before the 2004 election, Zogby went on The Daily Show and confidently predicted John Kerry's victory. Who wastes their money paying for his opinion?

____________________

jsh1120:

I'm in the enviable position of never having believed Zogby when his results supported my preferences so I have no problem dismissing them when they run counter to my views.

His "online" poll is a joke; a marketing vehicle and nothing more. His phone polls veer wildly from one to the next, enabling him to claim huge shifts in opinion that no one else finds.

____________________

jsh1120:

P.S. "Listed telephone numbers?" What is this? The rebirth of the Literary Digest? That's literally ludicrous.

____________________

richard pollara:

The "loopiness" characterization avoids the real issue which is that the Obama campaign is in serious trouble. Two points: One Obama's problems illustrates once again how the Democratic Primary process can nominate the least electable candidate. Second, that constructing an intelligent primary campaign is no guarantee that a sound general election strategy is forthcoming. The warning signs were there as soom as the advertising expenditures were released. Obama has had a terrible 6 weeks. He can turn it around with an exciting Vice Presidential pick. Unfortuantely for him, the only VP candidate that will really help him is Mrs. Clinton. I wonder if he has the courage to make such a pick...

____________________

PatrickM:

@Richard:
No it doesn't avoid the "real" issue. It's an entirely different issue. This website is an invaluable resource on the art and craft of polling. Yes, Obama appears to be on the decline. But we know that from the reliable national polls. Zogby adds nothing to that, which is the "real issue" as far as this website is concerned.

____________________

player:

Is there any such thing as a valid poll. I mean to do an accurate poll, the pollster would have to randomly call blocks of phone numbers in a given locale. As far as nationwide goes, it would almost seem impossible or cost a lot of money. The piece hints that Zogby fudges his results by using listed telephone numbers. Therefore he isn't using the random block calling method, so he shouldn't be very accurate with his results.

____________________

richard pollara:

PatrickM:

There is a tendency to characterize as "loopy" or flawed polls or projections that most posters don't agree with or buck the conventional wisdom. Why, for example has no one commented upon Election Projection, which gives McCain an less than a 20% chance of winning (up from 9% last week) or In-trade where you can still get great odds on John McCain or Pollsters electoral map which a week ago showed a triple digit electoral lead for Obama? The one thing that I have seen over the last 8 months is that there is no art and craft of polling. Polls do fairly well when 90% of the electorate is already determined by party ID but fail miserably when they have to quantify likely voters, undecideds and turnout.

That would be fine if polls were only an academic exercise but they are not. They influence policy, the media and elections themselves. Yet the caveats are not there. The plus or minus 3.5% is nonsense. It implies that polling is such a science that you can pin-point the possible error. Does anyone mention that almost none of the polls operate within the margin of error?

Patrick in your post you talk about "reliable national polls". In my opinon that is an oxymoron.

____________________

player:

@richard pollara:
Basically what you are saying is that public consumption polls create a self fulfilling prophecy for those who order the polls? This could be the reasoning that a lot of them come up with the same numbers even though some use loaded questions like the recent NBC/WSJ poll or use shoddy polling methods like Zogby does by using registered phone numbers only.

____________________

eugene:

its the demographics people,mccain cant win by 5% unless he gets 63% of the white vote,the media is playing this down so is all these polls,the gop is in trouble because the minority vote is rising bigtime,its going to be 27% this election theres no way mccain can over come that,bush had to beat kerry with the white vote by 18% to squeak by him,if bush would have beaten kerry that much in 1980 he would have won over 425 electoral votes instead of 286,the minority vote is taken over people,the latinos will be 35% of the usa population by 2040,its all over for the gop in they know it

____________________

saywhat90:


every electoral map has a different result. some maps have mccain gaining,some have not changed and others have given obama the advantage.the bottomline is this.zogby is the only poll showing a mccain lead. yes obama's lead has lessened in most polls but that doesnt mean the election now over and its all going to mccain. just means he had a bad month and a half. i dont know who is going to win because the simmpl fact is that there is no reall way of knowing untul november 4. just like in football even when one team is better then the other. you still have to play the game to know the reuslt.

____________________

Thaddeus Windt:

David Moore's article is quite correct. Sampling frames are crucial to getting an accrurate read on the electorate.

____________________

I'm anything but a poll expert, but right now it looks like Zogby is on a NASCAR and WalMart trip to the Outer Limits. The survey I just took started out with normal questions, then quickly veered into my stock car racing and shopping habits, and hypothetical "Mr. Smith and Mr. Jones" points of view about natural law vs. government control (in essence - with 2-3 sentence questions and double-negatives, who can tell?).

I had to say "Yes, I'm a NASCAR fan," but sadly, no - I don't shave my back.

http://asterling.typepad.com/incipit_vita_nova/2009/07/are-u-a-nascar-fan.html

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR