Pollster.com

Articles and Analysis

 

Zogby's Misleading Knowledge Test


Nate Silver has taken Zogby International to task for a telephone survey of 512 Obama voters that claims to "gauge their knowledge of statements and scandals associated with the presidential tickets during the campaign." Silver is right....but not right...

The summary and statement posted on the Zogby web site claims that "only 54% of Obama voters were able to answer at least half or more of the questions correctly," and more specifically that "statements linked to Republican presidential candidate John McCain and his vice-presidential running-mate Sarah Palin were far more likely to be answered correctly by Obama voters than questions about statements associated with Obama and Vice-President-Elect Joe Biden."

The survey was paid for by John Ziegler, a former talk radio host and publisher of a conservative web site. The Zogby summary quotes Ziegler claiming that "the poll really proves beyond any doubt the stunning level of malpractice on the part of the media in not educating the Obama portion of the voting populace."

The problem, as Silver points out, is that the survey does no such thing. It proves only that Obama voters surveyed were less likely to attribute to Obama or Biden a half dozen statements that were "at best debatable, yet apparently represented as factual to the respondent," such as the following:

"Which of the four [candidates] said his policies would likely bankrupt the coal industry and make energy rates skyrocket?"

"Which of the four [candidates] started his political career at the home of two former members of the Weather Underground?"

"Which of the four [candidates] quit a previous campaign because of plagiarism?"

"Which of the four [candidates] won his first election by getting opponents kicked off the ballot?"

Silver concludes -- appropriately -- that Zogby's survey appears to be less an unbiased measurement than part of "a viral marketing effort to discredit the intelligence of Obama supporters."

Zogby's defense is to deny that he conducted a "push poll" (more on that below), claiming instead that his survey represents "a legitimate effort to test the knowledge of voters who cast ballots for Barack Obama" He claims that "respondents were given a full range of responses and were not pressured or influenced to respond in one way or another." That's a little like describing the question, "when did you stop beating you wife," as fair (and as a fair test of "knowledge") by saying the husband has an opportunity to offer any date on the calendar as a response.

Zogby also claims to be a passive agent that just conducted research on behalf of a client. "The client," Zogby writes,"is free to draw his own conclusions about the research, as are bloggers and other members of society." Really? Then why does the analysis posted on Zogby's website repeatedly support "the client's" conclusions?

Unfortunately, Silver's case would have been stronger had he not reached reflexively, as so many do, for the "push poll" label to describe the Zogby poll. That's a bit like confusing assault with murder. A push poll isn't a survey at all, but negative telemarketing calls made under the guise of the survey. And the Random House Dictionary definition that Silver linked to is at odds with those of the American Association for Public Opinion Research, the Council for Marketing and Opinion Research, and the American Association of Political Consultants, (see also the work of Stu Rothenberg of Roll Call, Kathy Frankovic of CBS News and yours truly).

Zogby's survey does not amount to a "push poll" in that sense, but using the term allows him to respond -- predictably -- with a denial that "this was not a push poll." It wasn't, but that's beside the point. Describing his biased, leading questions as a legitimate test of knowledge is hugely misleading, at best.

 

Comments
GaMeS:

Thank you for making this point so well. I said something very similar this morning on Silver's post:


"Let's not get too caught up in the terminology here: It doesn't matter if this is technically "push-polling," "message testing," or what have you.

The bottom line is that it is grossly unethical."
[LINK]


But, as you point out, there is one critical problem caused by calling it the wrong thing: It gives Zogby an easy out and a way to change the subject instead of addressing his increasingly questionable business practices.

So, again, thank you for speaking so well on this topic; I think Silver was making a very good point, and putting the wrong label on it unfortunately undermined its validity.

____________________

Gary Kilbride:

Zogby seems to be a convenient source for this type of polling, frustrated parties coming out of an election cycle believing their pet theme hasn't been given sufficient weight in the media. I remember similar post 2004, or perhaps post 2006, when a prominent member of the election fraud crew paid Zogby to poll voters whether they have confidence in the electoral system, that their vote will be accurately counted. It had some similarities to this one, contrived wording in the questions and bloated post-survey conclusions from the group who commissioned it.

Anyway, after reading the subsequent thread on 538 I can't decide who came across worse, Nate Silver for weak questioning and a ridiculous gotcha attempt, or John Ziegler for imploding with insults and profanity. Actually it was clear cut at the end. Nate remained calm and Ziegler forfeited early advantage by turning into a mean spirited jackass.

But if Nate is going to branch out like this he needs better instincts and professionalism. You don't ask local yokel-type questions like who are the senators of specific states, and then reply with a juvenile and condescending, "Very good," when the subject gets it right.

____________________

thoughtful:

Mark

Actually the interview Nate Silver did with Ziegler is the most entertaining of this sad episode.Although maybe what Zogby would have to say would be more pertaining to your comment and probably even funnier. If this was not so sad and pathetic.

Two questions for me:

Is John Zogby out of his mind allowing his organization to be used in this way? First time I have ever known a credible pollster to allow a customer total freedom to phrase and ask any question virtually unedited as Ziegler has stated to 517 respondents in a survey all exclusively Obama/Biden voters.

But the bottom line question is this: you can prove anything you want if you commission a survey to prove it?

third question Please ask Zogby?

____________________

DTM:

It might be helpful if we had an alternative phrase for polls like this, ones designed not to gather information but rather to conjure up support for a preconceived claim. I think people gravitate to "push poll" because it sounds descriptively accurate (i.e., by design the poll has been "pushed" toward a particular result). So, introducing a similarly descriptive but distinct term might be a good idea ("agenda poll"? "propaganda poll"? something like that).

____________________

Centrist_Dem:

This "poll" is so laughable, it is hardly worthy of comment. One commenter on Nate Silver's site, discussing the "poll's" ludicrous distortion of Obama's remarks on coal-burning emissions, put it particularly well: "The interviewees were not asked about Obama's energy policy. They were asked to identify him with a right-wing caricature of his energy policy, abusing a quote taken out of context -- and then labeled as "uninformed" when they rightly refused or failed to do so."

That Zogby would conduct this poll at all should be a shattering blow to his credibility. That he would self-righteously defend it as a valid measure of opinion shows his total disregard for anything but the commission fee. The character of the commissioner is shown rather clearly by his interview with Nate Silver, in which Ziegler quickly gave vent to his obscenity-laced paranoia and hate rants.

It should be remembered that Zogby has a history of similar conduct. After the Terri Schiavo mess, nearly all national polls showed strong public opposition to Republican attempts to intervene. Zogby, however, suddenly produced a poll commissioned by a right-wing Christian group, finding that an overwhelming majority DID support the Republican position. This was instantly pounced on by the far-right as "proof" that every other poll was fixed by the "liberal media," and that only Zogby's poll reflected true public opinion.

The trouble was, the poll Zogby conducted had questions along the lines of: "Do you think a husband should be allowed to murder his disabled wife?" Yet, then as now, Zogby piously defending the poll as a fair and balanced survey.

Whether or not this should be called a "push poll," it is a blatant abuse of ethics and standards.

____________________

swinehartj:

There is a world of evidence, from dozens of studies on a variety of topics, that people tend to seek information and sources that confirm their views, and avoid or selectively recall those that support alternative views. Thus even if the Zogby questions for Obama supporters had been phrased without bias, they would probably have produced a similar result. The same questions asked of McCain/Palin supporters would most likely have shown the same selectivity, but in the other direction. Of course, that would have undercut Ziegler's claim of media bias, which might explain why only Obama supporters were questioned.

____________________

edsbowlingshoe:

Ziegler is a joke. Are you kidding me? How can a guy this totally off his rocker be taken seriously? Answer - he can't. And the Right digs its hole ever deeper...

____________________

Centrist_Dem:

I agree Ziegler is a joke, but he really isn't the point: the point is Zogby, a supposedly professional pollster, legitimizing Ziegler's distortions.

____________________

jamzo:

the knowledge test knowledge of mccain campaign negative statements about obama

"Which of the four [candidates] said his policies would likely bankrupt the coal industry and make energy rates skyrocket?"

"Which of the four [candidates] started his political career at the home of two former members of the Weather Underground?"

"Which of the four [candidates] quit a previous campaign because of plagiarism?"

"Which of the four [candidates] won his first election by getting opponents kicked off the ballot?"

surprise!

republican respondents had better knowledge of mccain campaign negative statements about obama

do all pollsters make their money doing surveys like this for talk radio

____________________

Another Mike:

Shame of Zogby for associating his name with such a disreputable person as Ziegler. Zogby's own statement said "We don't have to agree or disagree with the questions, we simply ask them." Seldom does a pollster admit that he's simply a pollster whore who's happy to do whatever the client asks for the right money.

____________________

Another Mike:

I like the suggestion to come up with a label for this type of poll. "Agenda poll" seems most descriptively accurate since its purpose is to push a particular agenda.

____________________

hobetoo:

There is a name for this kind of poll: it's a political information (knowledge) survey. Although certainly there are problems with several of the questions (e.g., were they truly testing relevant information about candidate positions?), the most fundamental problem is that choosing only Obama supporters meant that there was no comparative context for assessing how much they knew.

A test of political information that seeks to dhow that one political group is more ignorant than another needs to ask a common or comparable set of questions of respondents across the political (partisan or ideological) spectrum. The sample design was thus fundamentally flawed, and the client (I hesitate to dignify Ziegler by the term "investigator") should have been told by the survey organization that he can't prove what he seeks to prove by only asking questions to respondents who are on one side of the political spectrum.

Further, the survey organization owed it to the client to mention that surveys of political knowledge that are very specific (e.g., were he to ask the respondents how many justices are on the Supreme Court, or the name of the Chief justice, or the name of the Congressman from their own district) typically find a very low number of "knowledgeable" respondents.

It turned out, of course, that even the self-described highly informed Mr. Ziegler didn't know the names of both Senators from South Carolina. So he came off as a filthy mouthed ignoramus. Nate was only giving Ziegler a sample of what he had given to Zogby's respondents. And it was very revealing.

As for Zogby, he is entitled to field almost any kind of poll he chooses to sell. But he doesn't demand any respect by the survey research community for providing this kind of client service and then hyping the results as if they were meaningful.

Three cheers to Nate for taking both Zogby and Ziegler on.

____________________

Fioreb:

the sample of the 4 questions certainly discredits any effective 'knowledge' seeking. I take #4 refers to the Illinois Senate race were the republican candidate quit the race after his ex wife brought up quite a bit of issues with his moral fiber and character. The question however spins the event as if the democrat (Sen Obama) had a causative or triggering role in the event. A big heap of rubbish that is what most likely Zogby's poll was/is.

____________________

Loyal:

The true problem is the pollster's characterization of this as a knowledge test.

A classic epistemological definition of knowledge is a true verifiable belief. These statements reflect attitudes that if not shared might cause intelligent and informed respondents to not believe them as formulated.

These are as much attitudinal assessments as tests of knowledge.

That's why this is so nonsensical -- a self fulfilling survey.

Loyal

____________________

hobetoo:

The poll was designed for a single purpose: propaganda. It was designed to make one group look dumb by asking some tricky questions. It was not designed to learn how informed people were, or to compare one group with another group.

Zogby failed by being complicit in it.

____________________

jsh1120:

This is just one more example of Zogby's unprofessional conduct. As others noted, it has been going on for years. Any group/individual with the money to commission a poll with Zogby can get the results they pay for.

Unfortunately, despite a long history of such conduct, his firm manages to retain the patina of a professional polling organization.

I recognize pollster.com's willingness to post his results during the recent election campaign stemmed from a desire not to become involved in a slippery slope of determining which pollsters' methodology passed muster, but I have to say that after years and years of this sort of ridiculous behavior Zogby deserves to be written off as a professional polling organization.

____________________

Another Mike:

Fioreb, that question was not about Obama's US Senate race. It was about Obama's first political race for Illinois state senator. I'm not certain of all the details, but basically he challenged his Democratic primary opponents for failing to submit the required number of valid signatures to get on the ballot. They didn't and were kept off the ballot. I've never understood what was so nefarious about this or why it would be newsworthy in the presidential campaign. It was one of the questions I don't think contained a false or misleading premise.

____________________

pbcrunch:

I actually think this question was the most misleading and nefarious:

"Which of the four [candidates] started his political career at the home of two former members of the Weather Underground?"

It required a very astute observer of politics and this campaign to know enough information to answer this one "correctly":

1. That Ayers was a former member of the Weather Underground. It's possible many just knew him to be accused of being a "former terrorist," just plain "terrorist," or a "Weatherman."

2. That Ayers' wife was also a member of the Weather Underground, with the same qualifiers as #1.

3. That people believed the McCain/right wing version of those events. What if you didn't believe Ayers was ever a terrorist for his acts? (Possible as he was never convicted of being one.)

4. That people believed that Obama "started his political career" in such a setting. What if you heard that Obama called upon Ayers the same day he called upon many members of his neighborhood (a version that's closer to the truth)?

Regardless, Zogby should be ashamed of running this poll.

____________________

Another Mike:

The Weather Underground question is poor because whether Obama "started his politcal career" at that particular meeting is at best questionable. My understanding is that there were lots of meetings at lots of different places. At best, it's Republican spin.

I actually thought the coal question was the worst. It asks about what Obama supposedly "said." In fact, Obama did not say what the question assumes he said. It's just factually wrong, as much as the Palin question about seeing Russia from her house is wrong.

____________________

todji:

Perhaps we should pay Zogby to run a poll asking one Question:

How many people think that Zogby has lost all credibility?

____________________

Vicente Duque:

Mr Mark Blumenthal :

Thank you very much for this article that I find extremely interesting.

First, Polls have to be reliable and there has to be a system of ethics for Pollsters. If polls are going to be credible.

Second, the Psychology of all this confrontation is very interesting. Nate Silver is very Rational and Cerebral.

Nate Silver has published another article about a confrontational guy John Ziegler, he is a former "Radio Talk Host" turned pollster or strategist. :

Did Talk Radio Kill Conservatism?
http://www.fivethirtyeight.com/2008/11/did-talk-radio-kill-conservatism.html

It is extremely interesting for someone like me with a website RACIALITY.COM to study these super right wing guys, including :

Bill O'Reilly, Sean Hannity, Rush Limbaugh,

That are mentioned by Nate Silver in his second article and using his very prudent ways, Nate gives clues and hints about the operation of these fast media "conservatives", and their psychology.

Raciality.com

Vicente Duque

____________________

GlassHalfFull:

Zogby already had a crappy reputation, but until now it was just that his methodology was poor. Now that he has sold his meager services to the highest bidder, there is absolutely no reason to pay attention to him anymore.

____________________

Basil:

I'd call it a Prank Poll.

____________________

Basil:

Or maybe a Punk Poll...

____________________

Basil:

Faux Poll?

Mot-Faux Poll?

____________________

archaeoman:

It seems like the Republicans are intent on forgetting rule number 1 of democracy: it is never the voters' fault if you lose an election.

____________________

thoughtful:

Self Serving Survey

____________________

Centrist_Dem:

Zogby has begun backing off, trying to distance himself from this mess. He also refused Ziegler's attempt to commission a second "poll" to "prove" how WELL-informed McCain voters are, due to their reliance on the superior Fox News. Ziegler immediately decreed his poll perfect, was shocked that Zogby insisted that any new poll be co-sponsored by a liberal group, and declared that Zogby had caved in to liberal pressure.

Check out Nate Silver's site for links to the latest.

____________________

terje:

Using the Zogby/Zeigler laughable questions, I guess I'd be considered one of those "ignorant" Obama voters because I would answer many of the questions "wrong" by Zogby/Zeigler's political interpretation of the the answers.


#203 - "Which candidate had to quit a previous political campaign because they were found to have plagiarized a speech?"

The technically correct answer here would be "none" -- Biden actually was accused of (not "found") plagiarizing the Neil Kinnock speech, even though he had often used identical lines in many campaign speeches before while fully crediting Kinnock -- but twice was found to have omitted that credit from his stump speech -- something the Dukakis campaign attempted to turn into a plagiarism meme. But the real reason Biden withdrew from the race was when it was found that he has significantly exaggerated his academic history -- following the Kinnock incident, it raised questions of credibility that led to his withdraw. So if I answered accurately based on that detailed knowledge, I'd be considered wrong by Zogby/Zeigler because I'd didn't accept their simplistic account of a complex incident.

#208 - "Which candidate claimed to have campaigned in 57 states?"

Zogby would consider me uninformed since I would honestly answer none of them, because the reference was to Obama talking about the number of delegations being sent to the Democratic national convention - including in all 50 states, DC, Puerto Rico, US Virgin Islands, Guam, Democrats Abroad, American Samoa, and the party "unassigned" category. Anyone who listened to his statement in context would know that Obama wasn't claiming there are 57 states.

#209 - "Which candidate said their policies would likely bankrupt the coal industry and make energy rates skyrocket?"

Zogby would consider me a wrong answer because I would accurately note that none of the candidates said this - contrary to Republican talking points, I know that the comment was in reference to the need for clean coal technologies to be developed and what would happen if energy companies decided they wanted to build new dirty coal-fired plants. Once more, because I actually read the original SF Chronicle ed board interview, I would be considered stupid by Zogby/Zeigler because I actually bothered to know the details of what was said, instead of the Fox News version.

#210 - "Which candidate said that the government should redistribute the wealth?"

Once more, I'd answer none, but that correct answer would be considered wrong under the loaded Zogby/Zeigler scale. Obama actually talked about "spreading the wealth around" -- it was Republicans who used the word "redistribute." Spreading the wealth around is the concept of fueling economic growth that benefits many people instead of the few, including using progressive taxation as part of that tool. Using the same standard, it would have been just as accurate to claim that McCain/Palin wanted to "redistribute" by giving huge tax cuts to the wealthy.

#211 - "Which candidate started their political career at the home of two former members of the Weather Underground?"

Here the correct answer in the reality-based world would be "none", but Zogby/Zeigler seem to prefer attacks over facts. Ayers/Dohrn hosted a coffee during Obama first State Senate campaign -- but that was during the campaign, not the campaign kick-off. The campaign wasn't "started" there by any definition.

I'd flunk the Zogby/Zeigler test, getting these 5 questions "wrong" and the remaining 6 correct.

So how fair is it to grade a test when the teacher is using inaccurate (and biased) answers to determine what is right and wrong?

This may not have been a "push poll" -- but it is a far greater insight into the lack of knowledge of Zogby and Zeigler than into what Obama voters know.

____________________

DTM:

Thanks to everyone who replied to my naming challenge, and of course to Mark Blumenthal, who took notice and incorporated our suggestions in a subsequent post.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR