Pollster.com

Articles and Analysis

 

Acquiescence Bias: More on the Daily Kos Poll of Republicans

Topics: Acquiescence Bias , Daily Kos , Measurement , Research2000

I want to add a few thoughts to Emily's post earlier today on the DailyKos/Research 2000 poll of Republicans and how they might check for a skew in the sample that some argue would result from "sane Republicans" hanging up after taking offense to the questions. Another potential problem, called out today by Republican pollster Alex Lundry, is not as easy to check: The possibility of a skew in respondents' answers caused by what pollsters call "acquiescence bias."

Acquiescence bias is the tendency of some respondents to select affirmative answers where the choice is whether to affirm or reject the statement presented (including "agree or disagree," "favor or oppose" and "yes or no" formats). This topic has been the subject of decades of study and debate among social scientists, and even though pollsters continue to rely on agree-disagree questions, academic survey researchers mostly agree that this format tends to produce more apparent agreement than questions offer a choice between two competing statements.

Here is an example from Schuman and Presser's classic text, Questions and Answers in Surveys (p. 221), based on an experiment first conducted by the NORC General Social Survey in 1974: They asked a random half sample to agree or disagree with this statement: "Most men are better suited emotionally for politics than women." Slightly less than half (47.0%) agreed, 53.0% disagreed.

They asked the other random half-sample to choose between two statements (and included a middle choice):

Would you say that most men are better suited emotionally for politics than are most women, that men and women are equally suited, or that women are better suited than men in this area?

Fewer (33.1%) agreed that men were better, 4.3% said women were better suited than men, and 62.6% said they were both equally suited. Researchers at the University of Michigan's Survey Research Center replicated the experiment three times between 1974 and 1976, producing similar results. They produced consistently greater agreement that "men are better" using the agree/disagree format (ranging from 44.3% to 45.5%) than when using forced-choice format (ranging from 32.5% to 38.3%).

Another strategy to reduce this bias is to try to balance the direction of the statements, as recommended in Presser, et. al, Methods for Testing and Evaluating Survey Questions (p. 440):

Acquiescence bias can be reduced by balancing scales so that the affirming response half the time is in the direction of the construct and half the time is in the opposite direction (e.g. six agree/disagree items on national pride, with the patriotic response matching three agree and three disagree responses).

With those recommendations in mind, consider the questions asked on the DailyKos/Research2000 survey in the order in which they presented the results. The first eight present all of the more sensational, ludicrous assertions (most of which pertain to President Obama).  Seven of eight ask respondents to affirm or reject the extreme statement:

  • Should Barack Obama be impeached, or not?
  • Do you believe Barack Obama was born in the United States, or not?
  • Do you think Barack Obama is a socialist?
  • Do you believe Barack Obama wants the terrorists to win?
  • Do you believe ACORN stole the 2008 election?
  • Do you believe Sarah Palin is more qualified to be President than Barack Obama?
  • Do you believe Barack Obama is a racist who hates White people?
  • Do you believe your state should secede from the United States?

They then ask 15 issue questions that do mix up the order somewhat. Eight questions -- ask respondents if they agree with a liberal policy position, five ask about a conservative policy position, and two (the questions about Christ and marriage as a partnership) force choices between two statements:

  • Should Congress make it easier for workers to form and join labor unions?
  • Would you favor or oppose giving illegal immigrants now living in the United States the right to live here legally if they pay a fine and learn English?
  • Should openly gay men and women be allowed to serve in the military?
  • Should same sex couples be allowed to marry?
  • Should gay couples receive any state or federal benefits?
  • Should openly gay men and women be allowed to teach in public schools?
  • Should sex education be taught in the public schools?
  • Should public school students be taught that the book of Genesis in the Bible explains how God created the world?
  • Are marriages equal partnerships, or are men the leaders of their households?
  • Should contraceptive use be outlawed?
  • Do you believe the birth control pill is abortion?
  • Do you consider abortion to be murder?
  • Do you support the death penalty?
  • Should women work outside the home?
  • Do you believe that the only way for an individual to go to heaven is though Jesus Christ, or can one make it to heaven through another faith?

I don't want to overstate the consensus of pollsters -- academic or otherwise -- on this issue. Many highly regarded survey researchers continue to rely on agree/disagree questions, often because of their simplicity and brevity or because such questions are part of a long-standing time series that the pollster would rather not disrupt (good example of the latter here; for more discussion see Javeline, 1999).

So while it would be a bit unfair to condemn Research 2000 for relying on question formats that pollsters and academics continue to rely on, Lundry has a point. Acquiescence bias probably exaggerates the amount of agreement measured for some of the more ludicrous assertions about Barack Obama tested on the Kos poll.

Update: As Alex Lundry notes below, his comments about acquiescence bias earlier today came after reading a message sent by Stanford graduate student Josh Pasek to AAPOR's members only listserv.  With Josh's permission, here is a portion of that message:

Given that 10-20% of respondents tend agree with any statement (likely due to social norms), I went through the survey mentally subtracting 15 percentage points from every "yes" answer.  That does leave some shocking numbers -- particularly as acquiescence tended to indicate support for gay rights, sex education, etc. -- but suggests that Birthers, for instance, may be outnumbered in the party (a slight consolation at best).  I'm not saying this to suggest that the opinions being expressed even with a correction are reasonable, but I worry that not addressing this kind of issue is the reason so many people out there are skeptical of survey results in the first place.

 

Comments

Just want to be sure that full credit goes to the AAPOR listserve, specifically Josh Pasek from Stanford University, for first pointing out this issue of acquiescence bias.

____________________

If there is an acquiescence bias, that presumably makes some of the answers even less sane. It would mean, for example, that *less* than 8% of Republicans think that gays should be allowed to be school teachers. It's true that reducing the number of 'yes' answers would make the population being sampled look less crazy on the whole, but it wouldn't be all one-way traffic.

____________________

B:

Minimal balancing using "or not" can be just as effective as "full balancing." See:

Shaeffer, Krosnick, Langer, and Merkle (2005). "Comparing the Quality of Data Obtained by Minimally Balanced and Fully Balanced Attitude Questions." Public Opinion Quarterly, 69, 417-28.

____________________

GARY WAGNER:

None of these percentages are based on anything other than my opinion.

If the interviewers told the person on the phone that this was in any way related to Daily Kos, 75% of them would have hung up. At the same time, I think there would be a significant number of big fans of Daily Kos that would have identified themselves as Republicans just to give outrageous answers.

75% of real Republicans would have hung up after the second question about Obama's citizenship. It is a dead giveaway about where the poll is headed and what they are trying to accomplish.

By the time you get down to 5th question it is so obvious to anyone except complete morons what this poll is trying to do. Say what you will but most Republicans are not complete morons.

This is possibly the most ridiculous poll I have seen yet. It is designed to eliminate all respondents except for the most extreme fringe of people who claim they are part of a political party. Plus it was commissioned and published by Daily Kos.

The left wing hate-sites will eat this up. Olberman and Maddow are probably salivating and covering the camera with spittle. People with a bit of reason see this for what it is - a publicity stunt.

____________________

Ken Sandale:

I think some of the posters here are so skeptical of how extreme the Republicans have become because the Republicans they personally know are not that way. But that is not a random sample. Extremists do not tend to be friends with non-extremists. If you read internet political message boards or listen to talk radio, almost all the Republicans hold most of the extreme positions in the poll, although granted that is not a random sample either.

If I remember correctly there has been polling about Rush Limbaugh's popularity, and a large fraction of Republicans view him favorably. Limbaugh constantly preaches most of the extreme stuff, and so his popularity among Republicans is pretty strong evidence that most really do hold those views that many of the posters here cannot believe they really hold.

____________________

JamesF:

On the other hand, just about all of the Republicans that I know, especially the Fox-watchers as opposed to the newspaper readers, believe this stuff without question.


In fact, many of them are walking around in a state of near panic due to the impending socialist takeover of the country and the resultant high taxes and re-education farms. I don't know where all of these "sane" Republicans have migrated to, but it sure ain't California. You try to point out how untrue and preposterous those extremist ideas are and you find they are residing in some kind of alternate place without facts or reason. And there is no bringing them back.

These findings from Research 2000 are consistent with polling by other outfits such as PPP who have asked similar questions in August and November:
Public Policy Polling: ACORN
and
Public Policy Polling: A deeper look at the birthers

There are links there to pdfs with the cross tabs. So you can try to discredit the results all you want, but they are consistent with other polling. All that energy expended railing about Daily Kos might be better spent trying to bring them back from the edge of their insanity.

____________________

GARY WAGNER:

Sorry to burst your bubble, Ken, but my opinion is that some of the posters here are skeptical because Republicans have not "become" anything. The liberal media has gone even further in pushing out their idea of the stereotypical Republican. They are much more direct about it.

Then when you talk about Daily Kos, this is not even a liberal media site. It is nothing but pure hatred. I can't even get to the site through my employers webwasher proxy server because it is classified as a hate site.

Referencing Limbaugh shows that you have bought into the image being presented by those who hate him rather than Limbaugh himself. I don't listen to him because he is boring and repetitive. But I know that what he says and what the liberals say he says are two different things. He makes outrageous remarks but if you label them as hateful then what do you say about what comes out of Olberman, Maddow, Daily Kos, or Huffington Post?

____________________

Proud Liberal:

So sorry to be you Gary Wagner. DKos a 'hate' site? You mean like the KKK? While I don't care that your work won't let you on their site, that doesn't mean it's a 'hate' site. More commonly it means the folks that run your network/company are more Fox News friendly. Can you log onto Fox News? That might tell you something.

As to the skewing of the survey's results. No one likes to admit to what extent their peers are ill informed. There is nothing remotely scientific in picking a number (say 15%) and then using that number to subtract from the results you might not like so as to make one side look better. Sadly, the results of the survey show there are a great many people who literally believe what Fox news & Glenn Beck & Limbaugh tell them on a daily basis. Nice try though.

____________________

Farleftandproud:

I think a good suggestion for Democrats which has to be done soon to put more preasure on lawmakers, is will you vote in the midterms if Democrats can't get a comprehensive health care package done? It has to be conducted soon in the next month, otherwise the GOP will try even harder to kill any bill the Democrats try. My argument is Dems will lose fewer seats if they can get back to work on health care now instead of waiting.

Thanks, Dave

____________________

GARY WAGNER:

There were polls released in 2007 that said that 42% of all democrats think that Bush was behind the 9/11 attacks. If I recall correctly, the daily kos attacked those polls with their normal hatred.

Is that really what almost half of all democrats think? Do half of you defenders of this silly daily kos poll believe that? That's pitiful if true.

____________________

JamesF:

Please link to that poll that you claim "said that 42% of all democrats think that Bush was behind the 9/11 attacks." I'm calling bs.

____________________

Proud Liberal:

Keep digging Gary.

____________________

Farleftandproud:

There was some merit though for the fact that Bush could have known about the 9/11 attacks. I had my suspicions at the time. There were some theories that Bush waited for us to be attacked, and decided it was his take charge moment to raise his approval rating. I think Karl Rove could have pulled something like that off being the sleazebag he is. At least if Republicans want to go to the library in Hawaii to see Obama's birth certificate at the city hall, they can at least go and see it for themselves. Why doesn't Hannity go there for Fox and the librarian can let him see it for himself in front of the cameras.

____________________

Farleftandproud:

The Tea Baggers are truly setting forth a message of hate that is not going to ultimately help the GOP in much of the country. They want government off their backs, yet encourage a lot of pro life and anti gay extremists. "abort Acorn not babies". That isn't going to win independents in heavily blue states. I mean, they have had Obama with horns, Obama as a witch doctor, Obama wearing lipstick, Obama being hung in efigy. This is sick. Do progressives ever want to have these angry rallies against right wing extremists? Stooping to their level is not the course Obama wants to go. Of course we would love to take them on, but it just isn't the progressive style. Progressives will likely put their capital in protesting the next Republican convention and speaking out. The left wing at least gives conservatives more than 5 months before they start their angry protests.

I honestly think that Reagan if he were alive today wouldn't like the attitude of the tea baggers. He tried to reach out to his adversaries, and promote conservatism in a intellectual manner instead of acting like lunatics from the asylum.

____________________

Aaron_in_TX:

"I honestly think that Reagan if he were alive today wouldn't like the attitude of the tea baggers."

He probably wouldn't like it, but he wouldn't divorce himself from them either. In fact, he would probably quietly court their support. He started his 1980 campaign in Philadelphia, MS declaring that he believed in "states rights," a subtle play to nativists & racists. When running for CA governor in 1966, he refused to reject the support of the John Birch Society. Later on he had some former Birchers in his administration.

He was a politician; he took the support of anyone who was willing.

____________________

GARY WAGNER:

Here is a link to the Zogby Poll: http://www.911truth.org/images/ZogbyPoll2007.pdf

6.3% of democrats said that GWB directly caused 9/11 and 36.3% think he knew about it and did nothing to stop it. That's 42.6%.

You'll find that question on page 8.

On Page 17 you will find that 49% of democrats believe that more investigation into administration involvement in 9/11 is needed.

The Rasmussen poll in May, 2007 shows that 35% of democrats think Bush was involved in 9/11. http://www.rasmussenreports.com/public_content/politics/current_events/bush_administration/22_believe_bush_knew_about_9_11_attacks_in_advance

I'll accept your apologies now James F and Proud Liberal. I forgive you. You problably don't read any real news.

____________________

GARY WAGNER:

FarLeftAndProud:

You said, "Do progressives ever want to have these angry rallies against right wing extremists? "

On what planet did you live from 2001 through 2008? I live right down the street from the small college Bush spoke at in 2006. Along that route were people holding up mannequins of bush hanging in effigy, pictures of him in a nazi uniform, posters with pictures of his daughters saying he should send them off to die before he sends any more soldiers. It was vile. It was 10 times worse than anything the liberal media has pretended to see at tea party rallies.

____________________

Aaron_in_TX:

"Along that route were people holding up mannequins of bush hanging in effigy, pictures of him in a nazi uniform, posters with pictures of his daughters saying he should send them off to die before he sends any more soldiers."

I've seen these things too. How many people were there? A few dozen maybe? The tea parties attracted a lot more people.

What's clear is that the number of people that are extremely intolerant of a democratic administration far outnumber those who are intolerant of a republican administration.

____________________

Aaron_in_TX:

OK, here is what you said at 9:13 am:

"6.3% of democrats said that GWB directly caused 9/11 and 36.3% think he knew about it and did nothing to stop it. That's 42.6%.

You'll find that question on page 8.

On Page 17 you will find that 49% of democrats believe that more investigation into administration involvement in 9/11 is needed."

And here is what you said the day before at 4:15 pm: "42% of all democrats think that Bush was behind the 9/11 attacks."

According to the poll you cited, only 6.3% of democrats believe that Bush "was behind" the attacks. The other 36% felt that he knew something and didn't take action to stop it. The poll doesn't specify if the people thought this was willful inaction or just negligence. So the poll doesn't say what you want it to. You're projecting what YOU THINK the 36% thought.

____________________

Ken Sandale:

Gary Wagner, you help prove my point.

You screamed at me about a liberal media conspricy. That is one of the very things that Republicans do that is over the top. I read the media, and the liberal bias you guys are so certain about just is not there. Yelling it over and over does not make it true.

A similar example is that you guys think there is some conspiracy, led by Al Gore to brainwash almost every climatologist into falsely believing that global warming is occurring, and that there are emails instructing collegues to use tricks to hide the decline in temperatures. (The decline referred to in the emails was NOT in temperature) You guys really believe these things!!!

____________________

GARY WAGNER:

Aaron: your little game of semantics proves my point about dealing with misleading polls. Thanks for playing along.

Ken: Your embrace of stereotypes is breathtaking. I much appreciate your hypocrisy. It confirms everything I said.

____________________

Aaron_in_TX:

"your little game of semantics proves my point about dealing with misleading polls."

It's a shame that you don't realize the inconsistency in your own statements. Somehow you think that pointing out that inconsistency proves your point.

If you believe that the media narrative presents an unfair picture of a "stereotypical republican" but accept the narrative that there is a "liberal media" conspiracy, there are serious flaws in your thinking, or at least a bias that you don't challenge at all.

____________________

GARY WAGNER:

Aaron: I believe I said either here or somewhere (I'm too tired to look right now) that all news, and all polls, and all people are biased. Everything is political these days. There is no such thing as dead center. I think MSNBC is so liberal that they are to the left of Karl Marx. You think Fox News is so conservative that it makes Ronald Reagan look like Teddy Kennedy's clone. We're both right and we're both wrong.

This country is so polarized that what kind of coffee we drink shows a right or a left slant.

The Daily Kos poll was as stupid as the Zogby poll of 2007. That's my story and I'm sticking to it.

____________________

Ken Sandale:

Gary Wagner rights: >

Some stereotypes are correct. There are certain characteristics that most liberals have, and certain characteristics that most conservatives have.

What is troubling to me is that a lot of the "characteristics" that many conservatives have are not simply political conservatice ideology, but stuff like thinking Obama might not really be an American. Sane people, regardless of ideology, should think Obama is an American. Likwise, global warming is not really controversial scientifically, but many conservatives actually pervert science because of an abnormal type of hostility against Al Gore.

____________________

GARY WAGNER:

This poll was not taken of self identified republicans - not conservatives - don't confuse the two.

The purpose of this Daily Kos poll was to make it look like a majority of republicans believe what only a fringe of republicans believe. This wasn't a poll designed to find something out. This was a poll to push results in one single desired direction.

There are republicans who don't believe Obama was born in the US just as their are democrats that believe that Bush had explosives planted in the world trade center towers. There are fringe loons in both parties. The entire party doesn't need to be stereotyped with them.

What would be a really interesting poll is to ask avid Daily Kos fans if they think Bush blew up the twin towers. I'd be willing to bet a majority do. That would tell us something about what kind of validity a poll from Daily Kos has.

____________________

GARY WAGNER:

I meant to say, "This poll was taken of self identified republicans - not conservatives - don't confuse the two."

____________________

Ken Sandale:

Gary writes: "This poll was taken of self identified republicans - not conservatives - don't confuse the two."

I would expect the conservatives to be MORE likely to claim Obama is a socialist and other bizarre things.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR