Push "Polls": November 16, 2008 - November 22, 2008


Zogby "Knowledge Test" Update

The story of the nefarious Zogby/John Ziegler "knowledge test" poll has generated some useful follow-up:

On Tuesday, Nate Silver posted the verbatim text of a contentious, occasionally profane interview with Ziegler, the sponsor of the survey.

Yesterday, Politico's Mike Allen reported that Zogby had "rejected [Ziegler's] offer to sponsor a poll to test the knowledge of people who voted for John McCain," and included an unhappy reaction from Ziegler:

"I am happy to do a poll of both Obama voters and McCain voters, with questions that I formulated and sponsored either by an objective third party or by someone on the left, in tandem with a John Ziegler on the right -- but poll questions that have my signature," Zogby said.

"I believe there was value in the poll we did," Zogby added. "I also believe it was not our finest hour. This slipped through the cracks. It came out critical only of Obama voters."

Ziegler responded: "I am shocked by John's statement that he would do another poll but not an exact duplication. What is the point of that? Not their finest hour? This a was great poll. This didn't fall through any 'cracks,' they just got scared. ... The point of the poll was for my documentary on the media's impact on voter knowledge."

This morning, The Wall Street Journal's Carl Bialik blogged a thorough review of the whole story that, as Silver points out, is "the fairest and most comprehensive summary of the issue to date" and is worth reading in full.

Over the course of the week, our own commenters took up the challenge from reader DTM, who argued that "it might be helpful if we had an alternative phrase for polls like this, ones designed not to gather information but rather to conjure up support for a preconceived claim...('agenda poll'? 'propaganda poll'? something like that)." These included "agenda poll," "prank poll," "punk poll," "faux poll," "self serving survey" and "advocacy poll" (via email). Put me down as favoring "propaganda poll."

Finally, on the issue of testing "knowledge," ABC's Gary Langer critiques the annual survey on "American Civic Literacy" from the Intercollegiate Studies Institute that claims "the majority of Americans - including elected officials - failed a test of basic knowledge about American history and economics." See Langer's post for the specifics, but he makes a point that could also apply to the Zogby/Ziegler controversy:

The reality is that the ISI itself has failed a test of basic knowledge about the definition and measurement of just what knowledge is. I've blogged about this before - see it here - but the key point is that these folks are confusing knowledge (the ability to draw on information to make considered judgments) with recall (the ability to recite disassociated facts); and then doubling down by using an inappropriate method of measurement.

Zogby's Misleading Knowledge Test

Nate Silver has taken Zogby International to task for a telephone survey of 512 Obama voters that claims to "gauge their knowledge of statements and scandals associated with the presidential tickets during the campaign." Silver is right....but not right...

The summary and statement posted on the Zogby web site claims that "only 54% of Obama voters were able to answer at least half or more of the questions correctly," and more specifically that "statements linked to Republican presidential candidate John McCain and his vice-presidential running-mate Sarah Palin were far more likely to be answered correctly by Obama voters than questions about statements associated with Obama and Vice-President-Elect Joe Biden."

The survey was paid for by John Ziegler, a former talk radio host and publisher of a conservative web site. The Zogby summary quotes Ziegler claiming that "the poll really proves beyond any doubt the stunning level of malpractice on the part of the media in not educating the Obama portion of the voting populace."

The problem, as Silver points out, is that the survey does no such thing. It proves only that Obama voters surveyed were less likely to attribute to Obama or Biden a half dozen statements that were "at best debatable, yet apparently represented as factual to the respondent," such as the following:

"Which of the four [candidates] said his policies would likely bankrupt the coal industry and make energy rates skyrocket?"

"Which of the four [candidates] started his political career at the home of two former members of the Weather Underground?"

"Which of the four [candidates] quit a previous campaign because of plagiarism?"

"Which of the four [candidates] won his first election by getting opponents kicked off the ballot?"

Silver concludes -- appropriately -- that Zogby's survey appears to be less an unbiased measurement than part of "a viral marketing effort to discredit the intelligence of Obama supporters."

Zogby's defense is to deny that he conducted a "push poll" (more on that below), claiming instead that his survey represents "a legitimate effort to test the knowledge of voters who cast ballots for Barack Obama" He claims that "respondents were given a full range of responses and were not pressured or influenced to respond in one way or another." That's a little like describing the question, "when did you stop beating you wife," as fair (and as a fair test of "knowledge") by saying the husband has an opportunity to offer any date on the calendar as a response.

Zogby also claims to be a passive agent that just conducted research on behalf of a client. "The client," Zogby writes,"is free to draw his own conclusions about the research, as are bloggers and other members of society." Really? Then why does the analysis posted on Zogby's website repeatedly support "the client's" conclusions?

Unfortunately, Silver's case would have been stronger had he not reached reflexively, as so many do, for the "push poll" label to describe the Zogby poll. That's a bit like confusing assault with murder. A push poll isn't a survey at all, but negative telemarketing calls made under the guise of the survey. And the Random House Dictionary definition that Silver linked to is at odds with those of the American Association for Public Opinion Research, the Council for Marketing and Opinion Research, and the American Association of Political Consultants, (see also the work of Stu Rothenberg of Roll Call, Kathy Frankovic of CBS News and yours truly).

Zogby's survey does not amount to a "push poll" in that sense, but using the term allows him to respond -- predictably -- with a denial that "this was not a push poll." It wasn't, but that's beside the point. Describing his biased, leading questions as a legitimate test of knowledge is hugely misleading, at best.