Pollster.com

Articles and Analysis

 

Novak Calls It a Push Poll

Topics: Push "Polls"

Before I leave the topic of "push polls" and the indiscriminate way reporters and pundits use that label, I want to mention a prominent example that appeared earlier this month. Conservative commentator Robert Novak led off his March 1 column with this reference to a survey of Republicans in Iowa:

New York-based political consultant Kieran Mahoney's statewide survey of probable Republican participants in the 2008 Iowa presidential caucuses shows this support for the "big three" GOP candidates: John McCain, 20.5 percent; Rudy Giuliani, 16.3 percent; Mitt Romney, 3.5 percent. Astonishingly, they all trail James Gilmore, the former governor of Virginia, with 31 percent.

How could that be? Because it was not a legitimate survey, but a "push poll." That normally is a clandestine effort to rig a poll by telling respondents negative things about various candidates. Mahoney makes no secret that his voter sample was told of liberal deviations by McCain, Giuliani and Romney, and of true-blue conservatism by Gilmore (Mahoney's client)

"Illegitimate" would be a clearly appropriate term had the sponsors presented their results as a fair reading of current caucus preferences in Iowa. However, Novak tells us that Gilmore's consultant "made no secret" that the survey question relayed information about the "liberal deviations" of the other Republicans and "true blue conservatism" of Gilmore. So while questions may remain about the poll's fairness, the release itself is not quite as deceptive as Novak seems to imply.

Either way, the results as described do not add to a "push poll." That more nefarious dirty trick is a truly "clandestine effort" to communicate to a mass electorate with telephone calls made under the guise of a public opinion poll (columnist Stu Rothernberg prefers the term "advocacy call"). Rather, in this case, Gilmore's consultant apparently conducted a legitimate poll that tested how a random sample of likely Iowa caucus goers would react to a set of messages that were (apparently) highly favorable to Jim Gilmore.

Again, even if his terminology was sloppy, Novak was right to distinguish this result from an attempt to measure current vote preference. In this case, the Gilmore survey used what some call a "push question" (others an "informed vote") to see how specific messages would move the Iowa Republican electorate. Such exercises are common, legitimate tools used by campaign pollsters to gauge the way voters will react to new information received during the campaign. You can see many examples in the surveys regularly released by the Democracy Corps project of Democratic pollster Stan Greenberg.

Now, describing a poll as "message testing" does not let the pollster off the hook. In some cases, pollsters work to make these questions as fair and even-handed as possible. In others, their efforts may be fairly characterized as "rigged" to produce results that work in one candidate's favor. And sometimes - in a circumstance that brings joy to every campaign pollster - they can do both at the same time.

So was the Gilmore question fair? It is hard to tell in this case,** because neither Kieran Mahoney nor the Gilmore campaign are willing to share the full text of the question with anyone other than Robert Novak. I spoke with Kieran Mohoney today, and he explained that he let Novak see the results and the verbatim text because he believed the columnist would see his characterizations of the candidates as fair. Mohoney believes that Novak's review "constitutes as much public validation as I'm interested in at this time," and politely declined to release the text.

All of which brings me to three pieces of advice:

1) Be highly skeptical of results from an "informed vote" (or any other form of "message testing") that does not include the full, verbatim text of the questions. We have no way to judge the fairness of the questions without reading their text. If you are a reporter, do not even think about reporting such results unless you can see the full text, and (ideally) point your readers to it as well.

2) A skeptical and critical read remains in order even when you do have access to the full text. Ask yourself, are the candidate descriptions balanced? Do they provide only positive information about one candidate and negative facts about the others? Do the descriptions leave out important points that candidates will emphasize? Do they give some issues an unlikely prominence? Remember, even a perfectly fair "push question" attempts to predict the information flow of a real campaign, and that is not an easy thing to do.

3) If you write about an unfair or biased result from a message testing poll, please, please refrain from calling it a "push poll." The English language leaves you many fine terms - negative, unfair, untrue, distorted, biased, slanted and, yes, even "rigged" - that will describe an offensive poll with far more accuracy.

**We can evaluate one aspect of this result, by reading between the lines a bit. Novak's description implies a skew against other conservative alternatives to Giuliani, McCain and Romney, such as Sam Brownback and Mike Huckabee, because their credentials were apparently not described.

 

Comments
Tom Guterbock:

Well said, Mark. The distinction between 'push polls' and real political message testing surveys is an important one to make. The former are deceptive advocacy calls that AAPOR, AAPC, and NCPP have each condemned. The latter are real surveys undertaken for legitimate research purposes by candidate campaigns. Mr. Novak's column seems to fog over that important distinction.
BUT, just because the purposes are legitimate doesn't mean that the methods used in any message-testing study are all OK. Respondents are too often left feeling exploited, confused and/or just plain pissed when they hear the interviewer starting to read out political attacks on one candidate or another, while praising another candidate to the skies. I'm concerned that this kind of experience tends to diminish the public's support for polling in general. I also think that these negative effects could be mitigated if the surveys were carried out with more carefully designed techniques. I think it's time that the polling profession started to take a serious look at the issues involved in some message testing polls.
Thanks for helping to bring these issues to light.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR