Articles and Analysis


"Pushing" the Ethics of Message Testing?

Topics: 2008 , Push "Polls" , The 2008 Race

Another day, another "push poll" story. This time, Politico's Jonathan Martin reports on an "apparent push poll" in Iowa involving a "research firm" that "called Iowa Republicans this week praising John McCain and criticizing Mitt Romney and his Mormon faith." AP's Phillip Elliot traced the calls to:

Western Wats, a Utah-based company, placed the calls that initially sound like a poll but then pose questions that cast Romney in a harsh light, according to those who received the calls.

Elliot then leaps to the same quick shortcut that tempts all to many reporters:

In politics, this type of phone surveying is called "push polling" - contacting potential voters and asking questions intended to plant a message in voters' minds, usually negative, rather than gauging peoples' attitudes.

No it's not. The information described in the reports by Martin and Elliot sounds more like a form of message testing done by a real pollster - not the classic "push poll" dirty trick, although that distinction does not absolve the pollster from ethical responsibility for the content of their questions.

In writing about this issue I have tried to distinguish between the classic so-called "push poll," which is not a poll at all. It has no "sample" (in any statistical sense), no data collected, no analysis. It just amounts to someone making phone calls to spread a nasty rumor under the guise of a survey.

What confuses everyone is that campaign pollsters routinely conduct surveys that test campaign messages and try to simulate the dialogue of a real campaign. That message testing can often involve negative information. As Guiliani pollster Ed Goeas told John Martin:

"When you're doing a research call you ask positive and negative questions on [your own candidate] and positive and negative questions on [your opponents]," he said. "You're trying to war-game."

In this case, the calls apparently came from a survey call center known as Western Wats that acts as a vendor for many legitimate pollsters and survey researchers. The calls reported were part of a longer interview. Elliot included this account from one respondent:

The first 15 or 20 questions were general questions about the leading candidates," she said. "Then he started asking me very, very negatively phrased questions about Romney. The first one was would you have a more favorable, less favorable, blah, blah, blah, impression of Mitt Romney if you knew that his five sons had never served in the military and that he considered working on a presidential campaign as public service or some such question.

Based on those descriptions, these calls sounds like some sort of "message testing." But tossing aside the "push poll" label does not absolve the pollster of ethical responsibility. At a minimum, as the statement by the American Association for Public Opinion Research (AAPOR) puts it:

[Message testing] surveys should be judged by the same ethical standards as any other poll of the public: Do they include any false or misleading statements? Do they treat the respondent with fairness and respect?

The respondents quoted in the two news stories were certainly disturbed and angered by the questions they heard. Consider also the details these respondents remembered. Martin passes on the report of one respondent:

"Statements were on baptizing the dead, the Book of Mormon being on the level of the Bible, and one about equating it to a cult," said the Iowan, deeming them "common criticisms of Mormonism."

AP's Elliot added:

Among the questions was whether a resident knew that Romney was a Mormon, that he received military deferments when he served as a Mormon missionary in France, that his five sons did not serve in the military, that Romney's faith did not accept blacks as bishops into the 1970s and that Mormons believe the Book of Mormon is superior to the Bible.

One thing we can say without jumping to any conclusions about who may be responsible: No campaign has made these sorts of statements or attacks openly, and the organization that paid Western Watts to make the calls has so far been unwilling to take responsibility for the survey. So even if these calls were part of real survey, even if the information was narrowly factual in some sense, the refusal of the sponsors to accept responsibility for testing it speaks volumes about the ethics of the test itself.

Yes, campaigns (and independent groups) have the right to privately consider strategies they ultimately decide not to pursue. But when the market research for those potential strategies touches hundreds (or thousands) of volunteer respondents with a message that deeply offends, and when the organizations that sponsor the research hide behind a cloak of secrecy, something is very wrong regardless of the label we use to describe it.



wouldn't a legitimate "message testing call phrase a question something in terms of "how important is it to you" rather than "would you be more or less favorable"?

there's a subtle difference, that directly reflects how this information is being used.


"Would you be more favorable or less favorable towards John Edwards if you knew he paid $400 dollars for a haircut?".


"Would knowing that John Edwards paid $400 dollars for a haircut be important or unimportant to you when deciding to vote for him?"

the second version is more honest. in the first version, what are you learning by asking a question that probably not many would respond to as "favorable"?



Adam, not necessarily. Legitimate polls will likely include an "out" option.

In other words, "If you knew it were true that John Edwards paid $400 for a haircut, would it make you feel more favorably or less favorably toward him? If it would make no difference, just say so."

That last part would relieve an Edwards fan of the possibility of having no choice but to say getting a $400 haircut would actually make Edwards sound BETTER to him.



Even if the message testing was commissioned by a 527 or campaign, Western Wats is still responsible for developing this blatantly bigoted call. And why is it taking so long to find out who is actually behind this?!?



The problem is that it's still negative advertising. whatever the intent, you are still spreading a negative message about an opponent, and doing it a disguised fashion.



Thanks Adam, one of the best answers to the common "push poll" misconception I've seen. If it were a real "push poll" it wouldn't happen until the weekend before the election and everyone in the state would be complaining. If a campaign were stupid enough to resort to one, this far out and so few calls would be a waste of their money.



Adam makes a good point, but a key thing to keep in mind is that the pollster and whoever paid for the poll are hurting themselves by asking bad questions--bad questions equal bad or useless answers.

I definitely agree that a more honest question, such as one that allows a choice of "makes no difference" would be better in assessing reality... therefore it would be more instructive to the campaign about what to do with negative information. So it sounds like they hired bad pollsters.

But the intent of these things in never to spread negative or false information--it's to test the information to see if it would be useful.

A normal sample for a state survey is 500-600 people. They are anywhere from 10-20+ minutes long and cost tens of thousands of dollars. Forty grand for 500 people at a time is a terribly inefficient way to spread rumor.


John Foster:

The goal was to make Romney look bad. It's very simple. They basically spread true information that people may or may not have been aware of. It's free speech like it or not.

Romney calling it un-American shows what he'd be like as a president. If he can't handle criticism he should drop out now.


Post a comment

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.