Pollster.com

Articles and Analysis

 

The Harris "Wingnut" Poll: Could We Do Better?

Topics: aquiescence bias , Harris Interactive , Harris Poll , Humphrey Taylor , Markos Moulitsas , Measurement , straightlining , Wingnut poll

My column reviews the criticism of the Harris Interactive "wingnut" survey released last week and suggests some ways to better measure some of the beliefs it tested (taking up the dare issued via Twitter by Markos Moulitsas). Please click through and read it all.

One issue I did not mention in the column, and have not yet seen raised elsewhere, is that the Harris survey asked the questions using a "grid" or "matrix" format, something Harris Poll Chairman Humphrey Taylor confirmed via email. That means that the questions and answers were presented in a table (like the examples shown here).

While many online surveys rely on the grid format, it as an critical shortcoming: It makes it easy for respondents who prefer not to think hard about the questions to "straightline"" their answers. As such, this aspect of the design may have increased the potential for the "aquiescence bias" to have increased the percentage who said that some of the statements were true.

For more on the shortcomings of grids, see this report from Survey Sampling International and the ongoing criticism from the anonymous market research blogger Research Rants.

 

Comments
Mark Lindeman:

Your column asks how extreme it is for many ordinary Americans to use words like "socialism" to describe Obama's policies, when prominent Republicans have done the same.

I'd venture that it is extremely wrong, and that extremism in the wake of demagoguery is still vice. At the same time, the character of the "extremism" is ambiguous; we shouldn't assume that people who acquiesce in such labels in an online poll have been whipped into an anti-Obama frenzy.

____________________

Theo Kozel:

"One valid concern about the Harris survey is that many Obama supporters may have taken offense to the statements and stopped completing the survey. Keeping a politically representative sample on the phone would require both balance in the questions and a careful introduction to explain the purpose of the unusual questions to follow."

Just out of curiousity, since the poll breaks down respondents by political affiliation, does this at all impact the Republican responses?

"Finally, we all need to take seriously Langer's admonition about "over-literalism" in interpreting this sort of survey. Consider the context of labels tested on the Harris poll like "socialism" or doing "many things that are unconstitutional": During the 2008 campaign, John McCain and prominent Republicans used the word "socialist" to describe Obama's policies; and right now, more than a dozen state attorneys general are challenging the constitutionality of the health care law. So how extreme is it for many ordinary Americans to apply these labels to Obama?"

This I just plain don't understand. Because mainstream politicians are peddling extremism for electoral advantage, the people who end up taking these extremist beliefs don't really have extremist beliefs?

Finally, the best rebuttal is an actual well-made poll that we can compare/contrast with the allegedly poorly done polls. My suspicion is that whereas things may not appear so terrible as they do in the Research 2000 and Harris polls, there will still be plenty of embarrassment to go around.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR