Pollster.com

Articles and Analysis

 

Bad CBS/NYT poll question on AZ profiling debate

Topics: Arizona , immigration , Race

A New York Times article on the latest CBS/NYT poll (PDF) suggests that a majority of Americans believe the new immigration law in Arizona "would result in racial profiling":

[D]espite protests against Arizona's stringent new immigration enforcement law, a majority of Americans support it, even though they say it may lead to racial profiling...

[T]the respondents broadly agreed that the Arizona law would result in racial profiling...

However, as a reader noted, the poll question featured in a sidebar to the article doesn't ask about racial profiling, which is typically defined as targeting individuals solely based on their racial or ethnic background. Instead, CBS and the Times asked the following:

How likely do you think it is that the new law in Arizona will lead to police officers detaining people of certain racial or ethnic groups more frequently than other racial or ethnic groups? Do you think that is very likely to happen, somewhat likely, not too likely or not at all likely to happen?

Given the composition of the illegal immigrant population, Latinos will almost certainly be detained more frequently than other racial or ethnic groups under any enforcement regime in Arizona or any other state (particularly in comparison with their representation in the population). The relevant policy issue is whether the Arizona law will lead to detentions of Latinos based solely on their ethnic background. The Times article vaguely notes that "the Arizona Legislature and Gov. Jan Brewer made changes to the law on Friday that they say explicitly ban the police from racial profiling," but doesn't specify that the changes bar consideration of race or ethnicity in enforcement "except to the extent permitted by the United States or Arizona Constitution" by removing the word "solely" from the following provision:

A law enforcement official or agency of this state or a county, city, town or other political subdivision of this state may not solely consider race, color or national origin in implementing the requirements of this subsection except to the extent permitted by the United States or Arizona Constitution.

The poll question should have asked if people believe that this provision will be upheld in practice -- that's the question in dispute right now.

[Cross-posted to brendan-nyhan.com]

 

Comments
Marcello Mastroianni:

Are there any polls that show the ethnic background of those who support the new immigration law in Arizona? I've heard that support in the state is between 60 and 70 percent, and Wikipedia states that 60 percent of Arizona is white and non-Hispanic. That suggests that the bill must be getting a fair amount of non-white -- and perhaps even Hispanic -- support.

The national discourse strongly implies that this is a purely racial issue, but the numbers I'm seeing don't seem support that thesis.

____________________

Rob:

I'm the reader who initially called this poll question to Brendan's attention, which is not to say that he wouldn't have found it on his own or that he necessarily shares all my views about it. I'd like to use this question as the jumping off point for a couple of observations about opinion polls and the way in which they're reported. (This comment was originally posted on Brendan's site, but in view of its relevance to the specialized nature of this pollster site, I'm also posting it here.)

The Times's reporting about this question is clearly deficient, reflecting either a sad misunderstanding of what profiling means or a mindset so wedded to the chosen narrative that the reporters and editors are unable to interpret the poll question correctly. One wonders whether if it turns out that the Arizona law does in fact net 20 illegal Latino immigrants for every illegal immigrant of another ethnicity, the Times will regard that as confirmation that profiling has occurred.

But looking beyond the way in which the poll is reported, let's consider the poll question itself. If the pollsters were trying to get a response about the issue of profiling, they obviously did a poor job framing the question. On the other hand, in light of the demographics of illegal immigration, the wonder is that anyone could answer the question other than that people of certain ethnic groups are likely to be detained more frequently than others. My suspicion is that a large number of the people answering "not too likely" or "not at all likely" didn't consider the question any more carefully than the Times's reporters did. They understood that the question was aimed at profiling, and they wanted to express their disagreement that profiling would occur.

When questions are as intricately phrased as this one, or as the question Brendan discussed last month about tax burden (a question I also first called to Brendan's attention), how really meaningful are the answers of respondents who are asked the question over the phone, with no chance to read it over or parse it and without the training that some of us have had in textual analysis? If the goal of the present question is to ask whether racial or ethnic profiling will take place, maybe the question should be asked in just that simple a manner. Sure, there will be differences in how people interpret "profiling," but the result of such a question is almost certainly more useful than the result for the intricate and off-the-topic question that was asked.

Some bad questions can be avoided by thoughtful survey design. Others can be discovered if the pollsters go through an additional step of pre-qualification of questions, asking potential questions of a small group of respondents (not tabulated in the final survey) and inquiring about why they answered as they did, in order to discover flaws in the question not originally intuited by the survey designers.

But the truth is that sometimes the public's opinions on issues can be best understood not by survey questions but by using focus groups or even by reading comments on newspapers like the Washington Post (which unlike the NYT allows comments) or by reading blogs. The problem is that these methods aren't scientific and don't yield statistics that can be subjected to quantitative analysis. Yet all the highfalutin quantitative analysis in the world can't cure the inability of respondents to parse and give a thoughtful response to an intricate and difficult question.

So maybe we should consider poll questions as useful when they present easily understood questions (which candidate one supports, right track/wrong track, which issues are of concern, etc.), but not when it takes an advanced degree to figure them out. And maybe those political scientists who revel in statistical analysis should stop pretending that the answers to complicated and easily misunderstood questions, no matter how thoroughly t-tested and cross-tabulated, yield any truly valid insights.

____________________

Aaron_in_TX:

"But the truth is that sometimes the public's opinions on issues can be best understood not by survey questions but by using focus groups or even by reading comments on newspapers like the Washington Post (which unlike the NYT allows comments) or by reading blogs."

I don't think online comments offer any insight whatsoever on true public opinion. They're a self-selecting and non-representative sample. Just look at the comments on any yahoo article - hundreds of the most vile, bigoted comments you've ever seen.

Focus groups maybe, but still subject to error. There can be heavy bias depending on how the participants are chosen.

I think what we forget is that the majority of Americans don't pay as close attention, and don't really care to "parse" the issues as you suggest. Asking questions that do so only leads the participants in one direction or another. Best way to ask questions is to keep them as short and straightforward as possible.

Gallup's questioning on this issue was straightforward. Based on what you have heard about the AZ law, do you approve or disapprove? Since the margin was 11 points in favor of those who had heard a lot about the law and 9 points in favor among those who hadn't heard much about it, my conclusion is that Americans are generally going to support ANY immigration law that curbs illegal immigration activity by about 10 points.

People's opinions on this issue are pretty much set, the particulars don't matter too much. There is an upside for democrats in that hispanics nationally may support them somewhat more, although they can't do too much better than they did in 2008.

Republicans don't have much to gain besides shoring up the people they've already got.

____________________

GARY WAGNER:

This is the second time in a week that CBS has been caught not just misrepresenting the findings of a poll but outright lying about those results.

Last week they asked about taxes going up with Obama and claimed that anyone who answered that they had gone up were guilty of believing "misperceptions" - even though Obama has raised taxes much more than once since he took office. The results they claimed were completely false.

Now, just a week later they ask one question and attribute the response to a different question.

Is it a coincidence that CSB has been caught twice in a week using a vague and poorly worded question to present a completely false interpretation?

CBS is on the attack. They are now completely making things up to present their prejudiced point of view. They have gone far beyond just having left-wing journalist to full scale propoganda and lies on belhalf of Obama and the democrats.

This is why politically biased news organizations shouldn't do their own polls and shouldn't be paying a firm to do them. Case in point is USA today. When USA Today comissions Gallup to do a poll for them, that poll always leans at least 4 points to the left of the same poll Gallup does for themselves.

And yes, that applies to Fox News. They shouldn't be doing their own polling. Not that it matters - half of the people instantly dismiss them anyway. Now with evidence of CBS pushing a leftist agenda - people should ignore their polls too.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR