Mark Blumenthal | May 30, 2007
Last week's CBS News/New York Times survey (CBS story, results; Times story, results) is a classic example of the way public opinion pollsters will try to learn all they can about one complex but critical public policy issue. The survey focused special attention on immigration, asking 38 questions that probed the issue from almost every angle. Not surprisingly, the results to some of those questions have generated controversy. While some of my pollster colleagues will be quick to dismiss the complaints as partisan grousing, this survey provides some helpful lessons about the limitations of what polls can tell us when asking about subjects with which most Americans are unfamiliar.
I find it helpful to divide the immigration questions on the CBS/New York Times poll into two sections. The first twenty or so items (beginning with #41 in either the CBS or the NYTimes releases) use simple neutral questions to probe the presumably pre-existing attitudes that most Americans have about immigration, legal and illegal. We learn, for example (and quoting from the CBS release):
- "Americans have a positive impression of immigrants generally," yet "are highly critical of current U.S. immigration policy, especially when it comes to illegal immigration."
- 75% believe most immigrants are "here illegally"
- 82% "think the U.S. could be doing more along its borders to keep illegal immigrants from crossing over into the U.S.; just 14% think the U.S. is now doing all it can."
These initial results have stirred little controversy, perhaps because they are not particularly new (in fact, as the Times release shows, most have been asked before), and thus, not particularly newsworthy.
Needless to say, what interests everyone -- including those most critical of the CBS/NY Times poll -- is what Americans think of the immigration reform bill now before Congress. So starting at about item #60, the poll asks for reactions to a series of policy proposals. These questions introduce new and sometimes complex information about the various potential immigration policy alternatives. The results, which lead the coverage by both organizations, show "broad support" for "the major provisions" in the immigration bill now before Congress" (quoting the Times story).
These results -- and especially the wording of the questions involved -- have drawn intense criticism from opponents of the immigration bill (see Kaus, Allahpundit, Mark Krikorian). I will not try to referee the various critiques here, except to point out that the results from the second set of questions on the CBS/New York Times poll are better characterized as a measurement of how Americans react to descriptions of the immigration proposals than a measurement of Americans' current opinions on those proposals.
How much do Americans really know about the immigration reforms currently being debated in Congress? One question from the CBS/NYTimes poll suggests not very much: Only 26 % say they have read or heard "a lot" about "changing the laws regarding immigration in the United States;" 51% have heard "some," 18% not much, 4% nothing at all.
Moreover, when other pollsters have asked about the immigration bill in general terms (without providing specific descriptions), reactions are more negative than they are to the provisions tested on the CBS/NYTimes poll. For example, the Rasmussen Reports automated survey asked: "From what you know about the agreement, do you favor or oppose the immigration reform proposal agreed to last Week?" Roughly half (48%) opposed the agreement, about a quarter (26%) favored it with just as many (26%) unable to answer.
Survey researchers have long understood that respondents will readily answer questions about unfamiliar issues, typically drawing on their real attitudes or values concerning the subject of the question. Americans do have a sense of how their government is handling immigration (badly - see above), and so it is not surprising to see skepticism about whatever Congress is up to from those not following the immigration debate closely (see the analysis by Scott Rasmussen).
Two questions from a recent automated survey conducted in the Charleston, South Carolina media market by SurveyUSA make this point more vividly:
- Asked if they "understand the provisions" of a "proposed immigration reform bill" resulting from a "deal" reached by "the Senate and the White House," only 10% say they understand it very well, 34% somewhat well, 38% not very well and 17% not at all.
- 47% say they oppose the "immigration reform bill," 36% support it and 18% are unsure.
Put those two questions together and we see that the less informed respondents are less supportive of the bill. Three out of four who understand little or nothing about the bill offer an opinion, and they oppose the immigration bill by a margin of roughly two to one (49% to 25%). The better informed respondents divide evenly (46% favor, 44% oppose).
So back to the CBS/ New York Times results showing positive reactions to the various specific "provisions" of the immigration reform bill. What should we make of those? Although many respondents are probably forming opinions on the fly in reaction to the text of the questions, their reactions have meaning. They can tell us a lot about how Americans may react, should the news media or the campaigns focus far more attention on the immigration debate than they have thus far. But I would urge a few cautions:
First, it is entirely appropriate to scrutinize the language of the questions as the critics have, and compare and contrast results when pollsters use different language to ask similar questions. Follow the advice of from Frank Newport and my old anonymous friend, Professor M, and consider the various results as the best measure of potential range of support for the legislation.
Second, be realistic about how much more closely Americans will follow the immigration issue and what new information they are most likely to receive. We will certainly hear arguments about whether the immigration bill amounts to "amnesty," but will Americans ever consider it in the level of detail provided by the questions on the CBS/New York Times poll?
Put another way, we might want to think about what similar survey results foretold about previous legislative debates. Consider, for example, the Clinton health care proposal of the early 1990s. In September 1993, on the eve of President Clinton's address to Congress describing his plan, the CBS/New York Times poll tested its widely anticipated elements. Here is how the Robin Toner described the poll results in the New York Times poll story on September 23, 1993 (sub. req.):
The survey shows overwhelming majorities support the idea of assuring health coverage to all Americans and guaranteeing that no one ever loses their insurance when they switch jobs or suffer a medical catastrophe. Sixty-one percent said they were willing to pay higher taxes to achieve those goals, and more than half said they were willing to have the Government require employers to pay most of the health insurance premiums to cover their workers -- a centerpiece of the Clinton plan.
Of course, despite these favorable reactions, the Clinton plan collapsed under an assault by opponents who played to cynicism about government programs, fears of health care rationing and disapproval of the Clinton administration's early missteps. Those underlying, pre-existing attitudes, which were also in evidence in that September 1993 poll, turned out to be the ones that mattered most.
UPDATE: Rasmussen Reports published updated results on immigration this morning (via Kaus). These latest results show a different pattern among the most attentive respondents than what SurveyUSA showed in Charleston:
Overall, despite a major push by the President and others over the past week, support for the Senate bill has not increased at all. In polling conducted last night (Tuesday, May 29), 26% of voters favor passage of the bill. That’s unchanged from the 26% support found in polling conducted the previous Monday and Tuesday. Forty-eight percent (48%) of voters remain opposed.
Eighty-one percent (81%) of American voters are closely following news stories about the issue, including 37% who are following it Very Closely. Those with the highest interest in the issue oppose the legislation by a 3-to-1 margin (69% to 23%). By a 55% to 15% margin, those following the story Very Closely believe the bill will lead to increased levels of illegal immigration.