Articles and Analysis


Order Effects in The Fox News Poll?

A series of unusual questions near the end of a Fox News/Opinion Dynamics survey released last week generated criticism from the left side of the blogosphere (see Jonathan Singer on MyDD, TPM Cafe, Carpetbagger Report, Keith Olbermann via Crooks and Liars). As Carpetbagger's Steve Benen puts it, "FNC [Fox News Channel] is practically offering professors of quantitative analysis case studies in what not to do in a poll." He has a point. The wording and ordering of some of the question in the Fox News survey are a lesson in things pollsters should avoid.

Let's start near the end of the survey with a question about the recent congressional vote on the Iraq War (the Fox poll was fielded March 27-28 among 900 registered voters nationwide):

40. Last week the U.S. House voted to remove U.S. troops from Iraq by no later than September 2008 -- would you describe this as a correct and good decision or a dangerous and bad decision?

44% Correct and good
45% Dangerous and bad
11% (Don't know)

Now compare these results to findings from four other recent surveys (each sampled all adults rather than registered voters - text and results via Polling Report):

Newsweek Poll, March 28-29, 2007, n=1,004 adults - Do you support or oppose the legislation passed this week by the U.S. Senate calling for the withdrawal of U.S. troops from Iraq by March 2008?

57% Support
36% Oppose
7% Unsure

CBS News Poll, March 26-27, 2006, n=831 adults - Do you think the United States should or should not set a timetable for the withdrawal of U.S. troops from Iraq that would have MOST troops out by September 2008?

59% Should
37% Should not
4% Unsure

USA Today/Gallup, March 23-25, 2007, n=1,007 adults - Would you favor or oppose Congress taking each of the following actions in regards to the war in Iraq? How about setting a time-table for withdrawing all U.S. troops from Iraq no later than the fall of 2008?

60% Favor
38% Oppose
2% Unsure

Pew Research Center, March 22-25, 2007, n=1,245 adults - And thinking about a specific proposal: The Congress is now debating future funding for the war in Iraq. Would you like to see your congressional representative vote FOR or AGAINST a bill that calls for a withdrawal of troops from Iraq to be completed by August of 2008? .

59% Vote For
33% Vote Against
8% Unsure

So we have four slightly different questions from four different survey organizations asked during roughly the same period that provide very similar results. Support for the congressional vote to withdraw troops from Iraq in 2008 varies between 57% and 60%. But the Fox News poll shows only 45% support. Why the difference?

One contributing factor may be the oddly double-barreled answer categories on the Fox question that imply "danger" in the congressional vote. Put another way, the Fox item asks two questions: Was the vote "good" or "bad," and was the vote "correct" or "dangerous?" Good pollsters avoid double barreled questions, but even then, the wording of this question is odd. "Dangerous" is not exactly the converse of "correct." Some Americans may see the vote is both good and dangerous.** How should they answer the question?

But the bigger potential problem has to do with question order. Consider the question that Fox asked just before the item on the Congressional vote:

39. Who do you trust more to decide when U.S. troops should leave Iraq -- U.S. military commanders or Members of Congress? (ROTATE)

69% Commanders
18% Congress
7% (Both)
3% (Neither)
3% (Don't know)

40. Last week the U.S. House voted to remove U.S. troops from Iraq by no later than September 2008 -- would you describe this as a correct and good decision or a dangerous and bad decision?

44% Correct and good
45% Dangerous and bad
11% (Don't know)

While I can only speculate, I suspect that the order of these two questions primed respondents with the notion that the House vote was at odds with the recommendations of military commanders (although the initial item certainly said nothing about the views of those commanders). The only way to know for certain would be to conduct a controlled experiment that divides respondents into two random samples, asking Q39 first for half the respondents and Q40 first for the other half. And even that sort of experiment would tell us nothing about the potential impact of the 38 questions that came first.

Academic methodologists have conducted many such experiments over the years showing that the order of questions can sometimes affect results, often in very subtle and unexpected ways. In fact, the academic evidence on these "order effects: tends to involve far more subtle examples of priming than the one theoretically at work above (for examples, see Chapter 2 of the seminal text by Howard Schuman and Stanley Presser, Questions and Answers in Attitude Surveys).

I have sympathy for the task media pollsters often face in ordering questions on long surveys covering a wide range of topics that change from month to month. They seem to struggle to resolve goals that often come into conflict: On the one hand, they want to preserve the order of questions asked previously. On the other hand, inserting new items can risk an order effect in which the older items asked earlier bias the new questions inserted later. So they often face no-win choices.

In this case, however, the intent of the author of these questions seems clear. Check the MoveOn.org question (Q36) that nearly the liberal bloggers cited as a self-evident example of partisan bias:

36. After the 2004 presidential election, the president of the left-wing Moveon.org political action committee made the following comment about the Democratic Party, "In the last year, grassroots contributors like us gave more than $300 million to the Kerry campaign and the DNC, and proved that the Party doesn't need corporate cash to be competitive. Now it's our Party: we bought it, we own it and we're going to take it back." Do you think the Democratic Party should allow a grassroots organization like Moveon.org to take it over or should it resist this type of takeover?

16% Yes, allow a grassroots organization to run the party
61% No, don't allow a grassroots organization to run the party
24% Don't know

Could this clearly partisan "message test" have had an order effect of its own on the subsequent questions on the congressional vote on Iraq? Your guess is as good as mine, but either way, the items at the end of this Fox News survey are not exactly "by the book."

**See for example, the last two questions on this same Fox poll. On Q41, 59% agree that the "killing or capturing of Usama bin Laden" would be a "huge" or "major" accomplishment. But on Q42, 47% go on to say that bin Laden's capture would also "encourage additional terrorist attacks," while only 24% say it would "discourage" further terrorism. Many Americans apparently consider that outcome both desirable and dangerous.