November 16, 2008 - November 22, 2008
The story of the nefarious Zogby/John Ziegler "knowledge test" poll has generated some useful follow-up:
On Tuesday, Nate Silver posted the verbatim text of a contentious, occasionally profane interview with Ziegler, the sponsor of the survey.
Yesterday, Politico's Mike Allen reported that Zogby had "rejected [Ziegler's] offer to sponsor a poll to test the knowledge of people who voted for John McCain," and included an unhappy reaction from Ziegler:
"I am happy to do a poll of both Obama voters and McCain voters, with questions that I formulated and sponsored either by an objective third party or by someone on the left, in tandem with a John Ziegler on the right -- but poll questions that have my signature," Zogby said.
"I believe there was value in the poll we did," Zogby added. "I also believe it was not our finest hour. This slipped through the cracks. It came out critical only of Obama voters."
Ziegler responded: "I am shocked by John's statement that he would do another poll but not an exact duplication. What is the point of that? Not their finest hour? This a was great poll. This didn't fall through any 'cracks,' they just got scared. ... The point of the poll was for my documentary on the media's impact on voter knowledge."
This morning, The Wall Street Journal's Carl Bialik blogged a thorough review of the whole story that, as Silver points out, is "the fairest and most comprehensive summary of the issue to date" and is worth reading in full.
Over the course of the week, our own commenters took up the challenge from reader DTM, who argued that "it might be helpful if we had an alternative phrase for polls like this, ones designed not to gather information but rather to conjure up support for a preconceived claim...('agenda poll'? 'propaganda poll'? something like that)." These included "agenda poll," "prank poll," "punk poll," "faux poll," "self serving survey" and "advocacy poll" (via email). Put me down as favoring "propaganda poll."
Finally, on the issue of testing "knowledge," ABC's Gary Langer critiques the annual survey on "American Civic Literacy" from the Intercollegiate Studies Institute that claims "the majority of Americans - including elected officials - failed a test of basic knowledge about American history and economics." See Langer's post for the specifics, but he makes a point that could also apply to the Zogby/Ziegler controversy:
The reality is that the ISI itself has failed a test of basic knowledge about the definition and measurement of just what knowledge is. I've blogged about this before - see it here - but the key point is that these folks are confusing knowledge (the ability to draw on information to make considered judgments) with recall (the ability to recite disassociated facts); and then doubling down by using an inappropriate method of measurement.
The chart for the Georgia Senate runoff election between Saxby Chambliss (R-i) and Jim Martin (D) is now up and running on our site.
The runoff election is scheduled for December 2nd.
We have just added a new chart for Louisiana's 4th Congressional District, which will be holding an election on December 6th to fill the seat of retiring Congressman Jim McCrery (R).
For those who are curious, we will also be posting a chart for the Georgia Senate run-off soon.
11/18/08; 1,013 Adults, 3%
Mode: Live Telephone Interviews
Do you favor or oppose President-elect Barack Obama appointing Senator Hillary Clinton as his secretary of state?
11/18/08; 700 LV, 4%
Sen: Chambliss (R-i) 50, Martin (D) 46
Public Policy Polling (D)
11/15/08; 600 LV, 4%
Generally speaking, do you have a positive of negative impression of the United Auto Workers?
If you had to pick one group most responsible for the problems the auto companies are having, would you blame it on the management of the companies, the United Auto Workers, or the government?
If a candidate for office was endorsed by the UAW, would that make you more or less likely to vote for them?
18% More likely
39% less likely
43% No difference
Nate Silver has taken Zogby International to task for a telephone survey of 512 Obama voters that claims to "gauge their knowledge of statements and scandals associated with the presidential tickets during the campaign." Silver is right....but not right...
The summary and statement posted on the Zogby web site claims that "only 54% of Obama voters were able to answer at least half or more of the questions correctly," and more specifically that "statements linked to Republican presidential candidate John McCain and his vice-presidential running-mate Sarah Palin were far more likely to be answered correctly by Obama voters than questions about statements associated with Obama and Vice-President-Elect Joe Biden."
The survey was paid for by John Ziegler, a former talk radio host and publisher of a conservative web site. The Zogby summary quotes Ziegler claiming that "the poll really proves beyond any doubt the stunning level of malpractice on the part of the media in not educating the Obama portion of the voting populace."
The problem, as Silver points out, is that the survey does no such thing. It proves only that Obama voters surveyed were less likely to attribute to Obama or Biden a half dozen statements that were "at best debatable, yet apparently represented as factual to the respondent," such as the following:
"Which of the four [candidates] said his policies would likely bankrupt the coal industry and make energy rates skyrocket?"
"Which of the four [candidates] started his political career at the home of two former members of the Weather Underground?"
"Which of the four [candidates] quit a previous campaign because of plagiarism?"
"Which of the four [candidates] won his first election by getting opponents kicked off the ballot?"
Silver concludes -- appropriately -- that Zogby's survey appears to be less an unbiased measurement than part of "a viral marketing effort to discredit the intelligence of Obama supporters."
Zogby's defense is to deny that he conducted a "push poll" (more on that below), claiming instead that his survey represents "a legitimate effort to test the knowledge of voters who cast ballots for Barack Obama" He claims that "respondents were given a full range of responses and were not pressured or influenced to respond in one way or another." That's a little like describing the question, "when did you stop beating you wife," as fair (and as a fair test of "knowledge") by saying the husband has an opportunity to offer any date on the calendar as a response.
Zogby also claims to be a passive agent that just conducted research on behalf of a client. "The client," Zogby writes,"is free to draw his own conclusions about the research, as are bloggers and other members of society." Really? Then why does the analysis posted on Zogby's website repeatedly support "the client's" conclusions?
Unfortunately, Silver's case would have been stronger had he not reached reflexively, as so many do, for the "push poll" label to describe the Zogby poll. That's a bit like confusing assault with murder. A push poll isn't a survey at all, but negative telemarketing calls made under the guise of the survey. And the Random House Dictionary definition that Silver linked to is at odds with those of the American Association for Public Opinion Research, the Council for Marketing and Opinion Research, and the American Association of Political Consultants, (see also the work of Stu Rothenberg of Roll Call, Kathy Frankovic of CBS News and yours truly).
Zogby's survey does not amount to a "push poll" in that sense, but using the term allows him to respond -- predictably -- with a denial that "this was not a push poll." It wasn't, but that's beside the point. Describing his biased, leading questions as a legitimate test of knowledge is hugely misleading, at best.
Here is another way to consider the structure and change in group support for candidates in 2008. The horizontal axis shows the group's vote margin in 2004 (Kerry minus Bush). The vertical axis shows how much the margin CHANGED between 2004 and 2008. The general upward movement of almost all points shows how widespread Democratic gains were, with only three groups moving down (below zero on the vertical axis.)
The colors show 2008 winners. Groups to the left of the vertical reference line at zero that are blue switched winner from 2004 to 2008. All red groups also voted Republican in 2004, but most reduced the Republican margin.
Biggest Dem gains: Income over $200k, first time voters, Latinos, high school education only and age 18-29. The only three decreasing Dem groups were small towns and late deciders (though the late deciders still gave a majority to Obama.) And gays dropped their Dem support the most, by over 10 points though still favored Obama by a substantial margin. Fair warning that with only 4% of the national exit poll identifying themselves as gay/lesbian/bisexual/transgendered the sampling error here is pretty large.
While the labels are hard to read, a careful look replays some study. Click on the graph for a large size version.
11/13-16/08; 1,009 Adults, 3%
Mode: Live Telephone Interviews
Congressional Job Approval: 19% Approve, 74% Disapprove
Republicans are more likely to disapprove of the job Congress is doing than are Democrats (79% to 70%).
Compared to Gallup Poll'a national survey released just after the 2006 midterm elections (conducted 11/9-12/06), Congressional job approval has dropped from 26% to 19% while disapproval has risen from 67% to 74%.