Pollster.com

Articles and Analysis

 

Message Testing: Hear It For Yourself

Topics: 2008 , AAPOR , Barack Obama , Ben Smith , Garin-Hart-Young , George Bush , Hillary Clinton , Jake Tapper , Pollster , Push "Polls" , Roll Call , Stu Rothenberg

Late last week, a North Carolina musician named David LaMotte received a survey call from Garin-Hart-Yang, the firm of Clinton pollster Geoff Garin. The call, as he reported to HuffingtonPost blogger and DailyKos diarist Paul Loeb, "started out normal enough" but soon "turned to long Hillary-praising and Barack bashing policy statements" with response options that asked him to evaluate each statement. At the end of the call, they asked, "now based on everything we've discussed, who would you vote for?" LaMotte used his telephone answering machine to record the latter half of the call, and as a result was the transcript that Loeb posted at DailyKos and later as streaming audio posted by Loeb, Politico's Ben Smith and ABC's Jake Tapper.

Not surprisingly, much of the commentary about this call focuses on whether the Garin survey meets the classic definition of a "push poll." It does not, at least as far as I can tell.

The call in question was long, included dozens of question that seemed "normal enough" to LaMotte and, as he confirmed to me via email, concluded with a set of demographic items that LaMotte deleted from the audio recording in order to protect his own privacy. This call has none of the hallmarks of the classic, so-called "push poll" intended only to spread a negative message under the false guise of a survey.

It was, rather, a "message testing" survey, albeit one that tested a highly negative and -- to many -- objectionable message. It was not measuring "public opinion" as it exists now but rather voter reactions to a series of positive statements about Hillary Clinton and negative attacks directed at Barack Obama. Garin asked respondents to react to each statement, and subsequently asked a second vote question ("Now based on everything we've discussed, who would you vote for?"), in order to identify the most effective attack and the voters most likely to be swayed by it.

Like it or not, this sort of testing is common in most campaigns, and almost none of the results ever see the light of day. Full disclosure: As a campaign pollster, I helped design hundreds of surveys with similar tests of messages. (I have written previously about the differences between message testing and "push polls,' see also the commentary by Roll Call's Stu Rothenberg and the recent statement on "push polls" and message testing by the American Association for Public Opinion Research-AAPOR).

Of course, simply labeling this survey as "message testing" does not absolve the pollster of ethical constraints. The pollster still has an obligation to tell the truth and treat respondents with fairness and respect. Did this survey do that? LaMotte's audio has the interviewer reading five statements that he describes as "criticisms that opponents might make about Barack Obama." After each of the statements below, the interviewer asks "if they would give you very major doubts, some doubts or no real doubts about supporting Obama."

At a time when we need leaders who are clear, strong and decisive, Obama has been inconsistent, saying he would remove all troops, but then indicating that he might not, and pledging to renegotiate NAFTA, but then sending signals that he would not actually do so as president.

He supported George W. Bush's 2005 energy bill which payed six billion dollars in subsidies to the oil and gas industry, nine billion dollars in subsidies to the coal industry and twelve billion dollars in subsidies to the nuclear power industry. It was called 'a piñata of perks' and 'the best energy bill corporations could buy.

He leads the committee with oversight on Afghanistan but failed to hold a single committee meeting or hearing on Al Qaeda in Afghanistan or anything else.

He sided with the credit card companies voting against the bill that would cap interest rates at 30 percent.

While he talks about universal health care he has failed to make the hard choices that would truly get us to universal coverage and lower health care costs for all. His plan would leave 15 million Americans uninsured.

Let's stipulate up-front that the Obama camp vigorously contests these arguments, with some support from journalists. While by no means a complete listing, here are links to reports that provide more context on the NAFTA, energy, credit card, health care and Afghanistan issues. Readers are encouraged to add more in the comments, if warranted.

However, here is the non-rhetorical question that interests me most: How much do these statements differ from those included in Clinton mailers on NAFTA, the energy bill, the credit card bill, health care or Hillary Clinton's statements on the stump about the Afghanistan oversight committee? And if they are essentially the same, why would testing these assertions in the context of a survey be any more or less objectionable than making the same assertions in a debate, a speech, a television ad or a campaign mailer?

[The original version of this post included some extraneous verbiage in the third paragraph that I've cleaned up]

 

Comments
jsh1120:

Mark,

Another great post. You're outdoing yourself in terms of clarifying distinctions (e.g. Bradley/Wilder) that need clarifying.

Of course the distinction you draw between message testing and push "polls" is valid. I'd only add one other consideration. Message testing is effective only with a competently drawn sample of respondents. "Push polls," on the other hand are only likely to be effective if directed toward a relatively large segment of the public, like any advertising. And since the economics of such "narrowcasting" necessitate a short call, it should be fairly obvious which is which.

By the way, I haven't noticed the phenomenon recently, but about a year ago I noticed a Fox News Poll (conducted I believe by Opinion Research, but don't hold me to it)included a number of "message testing" questions that were clearly directed at evaluating the effectiveness of words and phrases used by Fox News. The items were sprinkled in a more comprehensive political poll and, at least in my mind, were clear examples of potential contamination of results.

As I said, I haven't noticed such items in recent polls from Fox (or anyone else), but I am somewhat suspicious that they may be included in some media-financed polls and simply not reported along with other results.

____________________

Patrick:

And how do they differ from all the misleading mailings the Obama campaign has sent out about Clinton? All campaigns do this and have been doing it for many, many years. In fact, it used to be much worse. Presidential candidates used to attack each other (and their family members) very viciously and personally. Of course, they can't do that any more because of our mass media). And I'm sorry to break this to some, but ALL people running for national office are 'typical' politicians. The one thing they ALL have in common is that regardless of the "brand" they are marketing, they ALL pander (and tell people what they want to hear) and they ALL lie. And there's nothing different about this year except the players are different.

____________________

Chris G:

agreed with jsh1120. if a campaign's aim is to spread a message to a significant audience under the guise of a push-poll, they're not gonna use Garin-Hart-Yang, would be a phenomenal waste of cash (ethics aside).

but now look at it from the respondent's perspective. should pollsters at least be required to tell them who they work for, or provide some caveat about the "information" they're being given?

____________________

Ciccina:

God bless, Mark. You really do great work.

I have to agree that this sounds like a garden variety message testing survey. My first contact with polling, ever, was when a DC pollster walked my internship group step by step through a message testing survey designed for Wellstone's first Senate race. Watching the numbers shift as the respondents progressed through the survey seemed like magic, and I been fascinated by this stuff ever since.

This sounds quite similar. If blessed Sen. Wellstone could do it, I'm pretty sure its okay for Hillary too.

Frankly, from the run-up to the five statements, I expected something gratuitous... but to me, someone who made up her mind about the candidates a long time ago, the five statements sound simply factual.

I know, as you said, there are arguments to the contrary but *to my ear* they sound like statements of fact rather than "bashing" (i.e. per the first part of the first statement, I watched the segment of Samantha Power's BBC television interview where she said exactly this).

If this is where their "negative" messaging is going, well, I'm surprised that its so timid. It would seem that there are worse things one could say.

Nevertheless I'm sure this will play into the media's image of a shrill and vindictive Hillary.

____________________

americaferreraisafatho:


Hmm, that's funny Ciccina, I don't recall reports of anyone receiving calls like these from the Obama campaign.


Once again Hillary proves that she will say anything (literally) and do anything to be president. What a sad, sad, person she has become.

And what a sad, sad day it is when the perpetuation of obvious lies are not seen as push-polling. And yes, the omission of the whole truth is, in fact, a lie.

But what else do we expect from Bill "I did not have sexual realtions with that woman" Clinton or Hillary "I was running from sniper fire" Clinton ??


____________________

jsh1120:

If, in fact, the Obama campaign is NOT engaged in message testing, I'd be both surprised and (as an Obama supporter) very disappointed. On the other hand, the tin ear the Obama campaign has demonstrated at several points in the campaign may suggest they aren't.

____________________

David LaMotte here, recorder of the call above.

It's for pollsters to decide, I suppose, between the definitions of "message testing" and "push polling." Industry jargon should be defined by the industry. Honestly, the nomenclature doesn't concern me much.

What is troubling to me is not whether it's called 'message testing' or 'push polling,' but whether or not I'm being directly lied to and manipulated in my own home on my own telephone by a candidate for president.

To weigh in, though, on the push poll question: Mark generally defines a push poll as "intended only to spread a negative message under the false guise of a survey."

As part of one of the questions, the pollster stated "He leads the committee with oversight on Afghanistan but failed to hold a single committee meeting or hearing on Al Qaeda in Afghanistan or anything else."

A commenter above states that these questions seem to include only statements of fact, but it's easily researched and confirmed that Obama chairs the subcommittee on *European* affairs. Afghanistan, last I checked, isn't in Europe.

Is this a mistake or a willful deception? It's hard to swallow either possibility, but it has to be one or the other, doesn't it?

My options for answering that question were limited to points on a continuum of how uncomfortable these 'facts' made me with Obama's candidacy. How could I answer with any degree of accuracy or integrity? Which of the options could I truthfully choose? None of the above.

Given that I had no option for an accurate answer, how could this kind of question yield any meaningful data? It can't. All it can do is promote the lie that Obama has neglected responsibility in dealing with Al Qaeda in Afghanistan.

Setting aside all other discomforts with the call, of which there are several, this one question alone seems to place it squarely in the middle of the definition offered, doesn't it?

Thanks, Mark, for examining the question, and to the rest of you for caring about it, whatever your views.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR