Mark Blumenthal | June 18, 2007
Last Friday, Mickey Kaus noticed a "stark conflict" between the recent NBC/Wall Street Journal, Los Angeles Times and CBS/New York Times polls on immigration that "demands a Mystery Pollster explanation." Lets see what we can do.
The questions concern what Kaus calls the "legalization plank" of the immigration legislation before Congress. The challenge in interpreting any of these results is to remember that most Americans have, at best, only a vague sense of what the immigration bill does, much less which provisions they favor. As I wrote previously (especially here and here), survey questions about specific proposals largely measure reactions, not preexisting opinions about the proposals. Respondents tend to listen to the text of the question and form opinions on the spot. As such, the results can vary greatly depending on the way the pollster asks the question.
One particular contrast illustrates how minor differences in wording can produce dramatic differences in the results: Consider these two questions
NBC/WSJ (June 8-11, n=1,008 adults) - Allowing illegal workers who arrived in the U.S. before January first of this year to receive an automatic work visa if they pay a fine of around five thousand dollars.
- 30% Strongly/somewhat favor
- 64% Strongly/somewhat oppose
- 6% Not sure
CBS/NYT (May 18-23, n=1,125 adults) - Would you favor or oppose allowing illegal immigrants who came into the country before January to apply for a four-year visa that could be renewed, as long as they pay a $5,000 fine, a fee, show a clean work record and pass a criminal background check.
- 67% Favor
- 28% Oppose
- 5% DK/NA
Both ask about essentially the same provision in the bill, and include many of the same elements of that proposal, yet show diametrically opposite results. Look more closely, however, and the questions differ in potentially crucial ways:
- The NBC/WSJ question describes the "work visa" as "automatic," while the CBS/NYT question says it is "a four year visa that could be renewed." I have no idea whether the every-four-year renewal procedure is "automatic" or not, but I would wager that few CBS/NYT respondents heard it that way.
- The CBS/NYT question describes requirements for a "clean work record" and "criminal background check," elements not mentioned in the NBC/WSJ question.
- Do respondents assume an "illegal worker" is the same thing as an "illegal immigrant?" Presumably, "illegal" is the key word in both, but perhaps the two phrases conjure different images.
Unfortunately, media and campaign pollsters know little about how respondents hear these sorts of questions or what pieces of information they weigh most heavily in their answers. We often refer to these sorts of differences as "dog whistle effects" -- the respondents seem to hear things we miss. There is a way to "debug" this sort of question (hint: Google "cognitive pretesting"), but it requires far too much lead time and costs far, far too much to be practical for media and campaign pollsters. So we are left to speculate about how to interpret the results.
However, let me suggest this rule of thumb for interpreting results of different survey questions that ask about essentially the same proposal: The more consistent the results, the more likely that we are measuring true, pre-existing opinions about the proposal itself. The more results tend to diverge, the greater the odds that respondents are confronting the proposal for the first time and are simply reacting, drawing upon real (and sometimes conflicting) attitudes triggered by the information provided in each question.
What's the difference? I suspect that in this case, we will hear from partisans on both sides of the immigration debate as to which language is best and which "flawed." But in that case, we are no longer debating public opinion, but rather the most accurate way to characterize the bill. So in this case, the debate about the polls is mostly a debate about the bill itself.