Emily Swanson | February 3, 2010
Topics: Daily Kos , Interpreting polls , Poll Accuracy , Sampling
Yesterday, Daily Kos released a poll of self-identified Republicans which showed surprisingly high agreement on a series of questions asking whether Barack Obama should be impeached, is a socialist, is a racist, and if he was born in the United States, among other things. The poll has created much debate in the blogosphere about the extremism of Republicans in the U.S.
While the results of the survey may be troubling, some have asked whether they are really representative of all Republicans or only the most extreme. Newsweek's Katie Connolly speculates, for example, that the potentially loaded questions that follow two more conventional probes of vote likelihood and 2012 vote preference may have caused some moderate respondents to discontinue the survey:
It's worth noting that those who completed the poll are also a self-selecting group. It's pretty commonplace in phone polls for respondents to simply hang up when asked loopy questions, and many of these questions qualify as loaded. As this poll progressed and questions like "do you think your state should secede from the union" came up, I'm willing to bet that the more rational Republicans just hung up, leaving the poll to be completed by those more divorced from reality.
Markos ("Kos") Moulitsas responded to such concerns in via twitter:
Cons[ervatives] arguing R2K poll might be skewed by "sane" Republicans hanging up b/c of crazy questions. R2K says no diff in response rates.
This may be true, but straight response rates might not tell the whole story here. A more important question is whether those respondents who started but failed to complete the survey were any different from those who did complete it.
Typically, the vast majority of those who do not respond are either not available when the pollster calls or refuse to participate immediately after answering the phone Those who hang up during the interview are a small component. For example, a report on the response rates for Marist's final 2008 New Hampshire primary poll includes only 70 incomplete interviews, compared to 2,990 dialed numbers that resulted in refusals or where the household could not be contacted. The cases that pollsters call "mid-interview terminates" could increase significantly without making a big dent in the overall response rate.
Moulitsas attempts to address that question with the following tweet:
In fact, R2K says that more than usual, "most of respondents enjoyed answering the questions once they agreed to participate."
Having been employed as a survey interviewer before, I know interviewers can get a feel for the reactions of respondents that might provide at least anecdotal evidence of whether it was difficult to keep some survey respondents on the line or whether respondents were offended by any of the questions. However, Daily Kos and Research 2000 could provide harder evidence that their results were not skewed toward the most extreme Republicans by providing the following information from their data on respondents who began but did not complete the survey:
- What were their responses to the second question, about the 2012 presidential primary? This question comes before any of the potentially tainted questions and could therefore give some indication as to whether those who hung up the phone mid-survey were any different from those who did not. For example, were the respondents who completed the survey more likely to support Sarah Palin or Ron Paul than those who did not?
- How did they respond to the third question, about Obama's impeachment? If some respondents continued part way through the survey after the more loaded questions began before breaking off, this is the question for which Daily Kos should have the most responses from those who stopped the survey later. If those who broke off after this point answered similarly to those who completed the survey, it would lend credence to the rest of the data and suggest that the final sample was indeed representative of Republicans generally. If fewer of those who refused to continue agreed that Obama should be impeached, that would suggest that the final sample was skewed toward those who agreed with the questions posed.
- How many incomplete interviews were there? If the incomplete interviews were different from the interviews included in the final sample, but there were very few of them, their inclusion might not significantly alter the final results.
By answering these questions, Daily Kos and Research 2000 would provide important context and help to address valid questions about whether their final sample was skewed in any way.
Thanks to Mark for contributing to this post