Pollster.com

Articles and Analysis

 

Zogby, Hillary and the Judicial Watch Poll


Today brings another controversy involving pollster John Zogby with two potential lessons, first, about a set of transparently biased and leading questions and, second, on the limits of such efforts to manipulate opinions.

This morning, the Washington Post's Dana Milbank tells the story of a new poll conducted by Zogby Interactive and sponsored by Judicial Watch, a group that "back in the day filed drawers full of lawsuits alleging Clinton corruption." Milbank describes the poll as "rather loaded in its language:"

"Some people believe that the Bill Clinton administration was corrupt," one question begins. In another question about Hillary Clinton, every answer included the word "corrupt," and the question was not asked about other candidates so that a comparison could be made.

The pollster, John Zogby, defended the questions as "balanced" -- a label Fitton [president of Judicial Watch] made no attempt to earn. As he presented the results yesterday, he announced that Bill Clinton's financial conflicts of interest "make the issues of Halliburton and Dick Cheney . . . pale in comparison."

Let's take a look at the first two questions:

304. Some people believe that the Bill Clinton administration was corrupt. Whether or not you believe the Clinton administration was corrupt, how concerned are you that there will be high levels of corruption in the White House if Hillary Clinton is elected President in 2008?

26% Very concerned
19% Somewhat concerned
20% Not very concerned
33% Not at all concerned
1% Not sure

305. When thinking about Hillary Clinton as a politician, which of the following best describes her?

17% Very corrupt
25% Somewhat corrupt
21% Not very corrupt
30% 51% Not at all corrupt
7% Not sure

You can pretty much stop after the first sentence. The suggestion that "some believe the Clinton administration was corrupt" is an obvious effort to lead the respondents to the desired answer. The drumbeat of "corrupt" and "corruption" that follows - implying that the issue is not whether Clinton is corrupt but how much - makes the bias almost comic. MyDD's Jonathan Singer has it exactly right:

[T]he apparently unbalanced wording of the polling conducted by Zogby International belies the notion that the organization is serious about coming up with results that actually reflect the views of the American public rather than just the views of those who paid for its services. To harp on one example, beginning a question on the scruples of a politician by saying that some people believe his or her spouse was corrupt inserts such a bias to void the results of the question -- and perhaps even the questions that follow. Simply put, the questions in the poll were not, as Zogby insists, "balanced."

But this episode also raises a second issue. How effective were these leading questions in producing the desired response? Put another way, did Judicial Watch get their money's worth?

Putting aside the obvious - that a 53% majority is not concerned about corruption in a Hillary Clinton White House - consider how the Zogby results compare to a set of balanced (though somewhat dated) questions about honesty and trust (via Polling Report):

ABC News/Washington Post (May 11-15, 2006. n=1,103 adults) - Please tell me if the following statements apply to Hillary Clinton or not... She is honest and trustworthy

52% applies
42% does not apply
6% unsure

CNN/USA Today/Gallup (Aug. 5-7, 2005. n=1,004 adults) Thinking about the following characteristics and qualities, please say whether you think each applies or doesn't apply to Hillary Clinton. How about...Is honest and trustworthy?

53% applies
43% does not apply
4% unsure

So a year (or more) ago, roughly the same percentage of Americans considered Hillary Clinton "honest and trustworthy" as expressed little or no concern about Clinton corruption in the Zogby/Judicial Watch survey. While the comparison is obviously imperfect, the lesson here may be that well developed opinions tend to be more resistant to manipulation by leading questions. If you are convinced that Hillary Clinton is honest (or dishonest), the leading language is unlikely to alter your answer either way. 

Don't get me wrong. I am not defending the Zogby questions, which are obviously and comically biased. However, the similarity in results when compared to fairly worded questions about honesty and trust suggest that opinions toward Hillary Clinton are well developed and resist manipulation. Voters have a pretty clear sense of who Hillary Clinton is, and those opinions may be difficult for either Clinton or her foes to change.

UPDATE: Nancy Mathiowetz, the president elect of the American Association for Public Opinion Research (AAPOR) just sent out the following release concerning the Zogby/Judicial Watch poll (interests disclosed - I serve on AAPOR's Executive Council):

It's always disappointing when pollsters who are internationally known and widely quoted engage in practices that are so clearly out of line with industry standards -- like using loaded and biased questions. There's no other way to describe the questions in the Zogby poll performed for Judicial Watch.

The good news is that it did not fly under the radar -- The Washington Post was quick to point out the flagrant disregard for accepted survey standardsin the poll - A number of blogs whose authors are well versed in industry best practices and standards also wrote about the poll.

Industry standards, including the American Association for Public Opinion Research's Best Practices , make it clear --the manner in which questions are asked as well as the response categories provided, can greatly affect the results of a survey.

That's why question wording and order are some of the toughest parts of designing a good survey or poll, and thoughtful practitioners will spend a significant amount of time trying to ensure that they are balanced, simple, direct and clear


Nancy Mathiowetz
President-elect,
American Association for Public Opinion Research

 

Comments
DougH:

Actually I found the polls gave similar results and the Zogby poll provided a more realistic results.. instead of the simple yes/no, approve/do not approve approach of most polls.

Both the ABC and CNN pols gave similar results
Trustworthy 52
Does not apply 42

From the Zoby poll you get
Not at all cirrupt 31% (I assume the 51 you used is a typo)
Not very corrupt 21%

Amazingly comes to 52% , the same as the other polls.

As for claiming the Zogby poll uses a predertimed bias in there questionaires, is false.. and obviously had no impact on the results of the poll.

Other polls also use similar questions;
Newsweek Poll
"Since news reports last month about neglect and poor health care for military personnel returning from Iraq and Afghanistan at the Walter Reed Army Medical Center, do you think the Bush Administration has done a good job or
a bad job of dealing with this situation

"As you may know, several military officials have been fired or forced to resign as a result of the Walter Reed health care scandal. From what you've seen or heard, do you think enough has been done to hold those responsible accountable, OR do you think MORE people should lose their jobs over this?"

or you could use a CNN pol questionaire
Compared to the last Congress, do you think the new Congress will [see below]?"

"Be more corrupt, less corrupt, or won't there be any difference"

"Be more responsive to what the pubic wants, less responsive, or won't there be any difference"

"Get more done, get less done, or won't there be any difference"

The questions imply that the previous Congress was corrupt, did not get anything done and was not responsive to the public.

I have also noticed that where polls allow for various categories, that they lump "fair" in with negative totals, giving a false impression of the results.

As for quoting Jonathan Singer, the authour forgot to mention he is a popular left-wing political blog.

But I guess you didn't want his comments to come off as bias..

As for the response from Nancy Mathiowetz is concerned, if you read the best practices guidlines, you will find most poll would fail the "Ideally, multiple rather than single indicators or questions should be included for all key constructs"

As most polls simply ask - yes or no, for or opposed, good or bad...

All, which are singular indicators.

Zogby's questionaire may have included bias but the results are similar and Zogby poll provides a more realistic result...

Do't you agree?
Yes or No..... no you don't get any other option

____________________

Alan:

Mystery Pollster wrote:
"MyDD's Jonathan Singer has it exactly right:..."

Anonymous commenter wrote:
"As for quoting Jonathan Singer, the authour forgot to mention he is a popular left-wing political blog. But I guess you didn't want his comments to come off as bias."

Doug, you completely missed the point of the quotation. If you are citing someone as an expert or authority figure, then it makes sense to tell the reader what qualifies that person as an expert and whether they have any known affiliations of interests that could indicate bias.

In this case however, Mystery Pollster adopted Singer's words as his own opinion, and stated they were "exactly right". The reference to Singer is simply an acknowledgement that the words are his work. To not acknowledge them would (in some contexts) be plagiarism. However, Singer's qualifications are irrelevant. It doesn't matter who wrote the words--they could have been written by you, me, or my 4 year old niece. What is important is that the words accurately reflect Mystery Pollster's opinion, and that is the reason the words were included. In this case, the qualifications or affiliations of the person who wrote the word are irrelevant, because he he not cited as an expert or authority, only as a word-smith.

____________________

Goldenggreg:

Well now we know why Bushes numbers are so low, since most of the left org run the polls. Thanks for the insight!

____________________

Did anyone ever explain how she made $100,000 off a $1,000 cattle futures investment? If not, 9 months of GE campaigning are going to send that corruption number skyrocketing even with the MSM in the bag for her.

Anyway, the problems with this poll are nothing compared to the CNN/ABC/USA Today Iraq poll, whose sample was 35% Sunni Arabs (the group by far the most opposed to pretty much everything we're doing there), when most estimates of their actual proportion in the Iraqi population are around 15%. This wasn't insignificant, as answers varied by as much as 90% between sects. Every conclusion made on the total numbers has to be thrown out.

Polling is becoming more and more the science of eliciting the results the pollster is looking for -- and getting away with it.

____________________

BrianNYC:

I agree with TallDave's comment that polling = eliciting the results the pollster is looking for, but would add that the results that are sought are those that will bolster the news story that they want to be written.

In addition, did anyone ever explain how subpoenaed Rose Law Firm billing records turned up in Hillary's private residence at the White House with her fingerprints on them, after she testified that she neither came into contact with them, nor knew where they were?

____________________

Brett Bellmore:

"Did anyone ever explain how she made $100,000 off a $1,000 cattle futures investment?"

Yup, and enough members of Congress of both parties use that particular technique for money laundering, that it's too hot to make an issue of during the campaign.

Judicial Watch didn't merely "allege" Clinton corruption. They proved it, to anyone without a closed mind on the subject. Got an admission in court that the administration was refering it's enemies to the IRS for audits, for instance. Uncovered the way Clinton related cases were being diverted from random judge assignment, and all being heard by judges appointed by... Clinton.

____________________

Steevo:

I think those 2 questions are misleading.

I also question political and ideological affiliation in one's life which usually has proven a consistent influence when making judgment. So... when the editor and publisher of this site having 20 years time doing work for the Democratic party simply tells me his judgment won't be affected, I wonder how trustworthy he is to point out bias.

Recently Zogby did a poll of US military personnel and their overall moral concerning their participation in the Iraq war. It painted a much more negative outlook than all I've read. Actually, it wasn't even close to the good to high consistently given from the soldiers and others on the ground observing. Now as I remember there were some serious questions about who hired Zogby and his methods. Has Mark Blumenthal discussed this controversial poll?

____________________

Mark Blumenthal:

Steevo: Yes, here, here and here. [And forgot to add one more, here].

And when, exactly, did I "simply tell" readers that my "judgment won't be affected?"

____________________

We all have our leanings; Mark seems to have been pretty evenhanded, but nevertheless still honest enough to admit his prejudices might color his judgment.

Which is more than one can say for the MSM, who cling tenaciously to their pretense of objectivity. I think we'd all be better off if every byline contained some reference to the reporter's political leaning.

____________________

W Wilson:

The questions were obviously biased and leading. I'm surprised that Zogby would lend its name to such a hatchet job. Kudos to Dana Millbank for shortstopping it.

____________________

Heh.:

So, the trade association feels the need to issue a detailed press release when a push poll is slanted against Hillary Clinton. Remarkable dedciation to the science of polling.

One wonders how pollsters can see their desks under the stacks of press releases the AAPOR must have been releasing on the push polling pretty much all the media outlets have been running against the current Administration...

____________________

DougH:

Alan,,

I would agree with your assessment, except when the quote is from a person with direct links/affiliation to a particular political party.

I think it's imporatant to readers, who are unaware of where the quotations are taken from to receive full disclosure...

Especially when a person comments are preceded with "The drumbeat of "corrupt" and "corruption" that follows - implying that the issue is not whether Clinton is corrupt but how much - makes the bias almost comic. MyDD's Jonathan Singer has it exactly right: "

I had to actually do a search to find out who Jonathan Singer is ...

Today's medi and internet is already flooded with misinformation and poltical bias.... so it's important for people to start reporting the facts and supplying full details of their sources.....

____________________

Carmel Bailey:

One poll that I took on Hillary was acidentally deleted and I noted that the question: Who would I pick for President: My man was not in the poll. He is Ron Paul. So you people are totally biased. That's okay because everytime you leave him out it ticks off people and they send him money so keep it up.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR