April 6, 2007
Zogby, Hillary and the Judicial Watch Poll
Today brings another controversy involving pollster John Zogby with two potential lessons, first, about a set of transparently biased and leading questions and, second, on the limits of such efforts to manipulate opinions.
This morning, the Washington Post's Dana Milbank tells the story of a new poll conducted by Zogby Interactive and sponsored by Judicial Watch, a group that "back in the day filed drawers full of lawsuits alleging Clinton corruption." Milbank describes the poll as "rather loaded in its language:"
"Some people believe that the Bill Clinton administration was corrupt," one question begins. In another question about Hillary Clinton, every answer included the word "corrupt," and the question was not asked about other candidates so that a comparison could be made.
The pollster, John Zogby, defended the questions as "balanced" -- a label Fitton [president of Judicial Watch] made no attempt to earn. As he presented the results yesterday, he announced that Bill Clinton's financial conflicts of interest "make the issues of Halliburton and Dick Cheney . . . pale in comparison."
Let's take a look at the first two questions:
304. Some people believe that the Bill Clinton administration was corrupt. Whether or not you believe the Clinton administration was corrupt, how concerned are you that there will be high levels of corruption in the White House if Hillary Clinton is elected President in 2008?
26% Very concerned
19% Somewhat concerned
20% Not very concerned
33% Not at all concerned
1% Not sure305. When thinking about Hillary Clinton as a politician, which of the following best describes her?
17% Very corrupt
25% Somewhat corrupt
21% Not very corrupt
30%51%Not at all corrupt
7% Not sure
You can pretty much stop after the first sentence. The suggestion that "some believe the Clinton administration was corrupt" is an obvious effort to lead the respondents to the desired answer. The drumbeat of "corrupt" and "corruption" that follows - implying that the issue is not whether Clinton is corrupt but how much - makes the bias almost comic. MyDD's Jonathan Singer has it exactly right:
[T]he apparently unbalanced wording of the polling conducted by Zogby International belies the notion that the organization is serious about coming up with results that actually reflect the views of the American public rather than just the views of those who paid for its services. To harp on one example, beginning a question on the scruples of a politician by saying that some people believe his or her spouse was corrupt inserts such a bias to void the results of the question -- and perhaps even the questions that follow. Simply put, the questions in the poll were not, as Zogby insists, "balanced."
But this episode also raises a second issue. How effective were these leading questions in producing the desired response? Put another way, did Judicial Watch get their money's worth?
Putting aside the obvious - that a 53% majority is not concerned about corruption in a Hillary Clinton White House - consider how the Zogby results compare to a set of balanced (though somewhat dated) questions about honesty and trust (via Polling Report):
ABC News/Washington Post (May 11-15, 2006. n=1,103 adults) - Please tell me if the following statements apply to Hillary Clinton or not... She is honest and trustworthy
52% applies
42% does not apply
6% unsureCNN/USA Today/Gallup (Aug. 5-7, 2005. n=1,004 adults) Thinking about the following characteristics and qualities, please say whether you think each applies or doesn't apply to Hillary Clinton. How about...Is honest and trustworthy?
53% applies
43% does not apply
4% unsure
So a year (or more) ago, roughly the same percentage of Americans considered Hillary Clinton "honest and trustworthy" as expressed little or no concern about Clinton corruption in the Zogby/Judicial Watch survey. While the comparison is obviously imperfect, the lesson here may be that well developed opinions tend to be more resistant to manipulation by leading questions. If you are convinced that Hillary Clinton is honest (or dishonest), the leading language is unlikely to alter your answer either way.
Don't get me wrong. I am not defending the Zogby questions, which are obviously and comically biased. However, the similarity in results when compared to fairly worded questions about honesty and trust suggest that opinions toward Hillary Clinton are well developed and resist manipulation. Voters have a pretty clear sense of who Hillary Clinton is, and those opinions may be difficult for either Clinton or her foes to change.
UPDATE: Nancy Mathiowetz, the president elect of the American Association for Public Opinion Research (AAPOR) just sent out the following release concerning the Zogby/Judicial Watch poll (interests disclosed - I serve on AAPOR's Executive Council):
It's always disappointing when pollsters who are internationally known and widely quoted engage in practices that are so clearly out of line with industry standards -- like using loaded and biased questions. There's no other way to describe the questions in the Zogby poll performed for Judicial Watch.
The good news is that it did not fly under the radar -- The Washington Post was quick to point out the flagrant disregard for accepted survey standardsin the poll - A number of blogs whose authors are well versed in industry best practices and standards also wrote about the poll.
Industry standards, including the American Association for Public Opinion Research's Best Practices , make it clear --the manner in which questions are asked as well as the response categories provided, can greatly affect the results of a survey.
That's why question wording and order are some of the toughest parts of designing a good survey or poll, and thoughtful practitioners will spend a significant amount of time trying to ensure that they are balanced, simple, direct and clear
Nancy Mathiowetz
President-elect,
American Association for Public Opinion Research
By Mark Blumenthal on April 6, 2007 2:08 PM | Permalink | Comments (13) | TrackBacks (0)
