Obama Overextended...Or Not
Mark Blumenthal | March 18, 2009
Topics: Barack Obama , Cognitive Pretesting , Measurement
The polling head-scratcher of the day: In a survey conducted March 12-15 (among 1,019 adults, margin of sampling error +/-3%), CNN/ORC obtained these results, producing the headline "Obama's Taken on Too Much":
Since he became president, do you think Barack Obama has tried to handle more issues than he should have, or hasn't he done that?
55% - More issues than he should have
43% - Has not done that
2% - No opinion
But just yesterday, the Pew Research Center released a survey conducted March 9-12 (among 1,308 adults, margin of error +/- 3%) with this question, which supported the sub-heading "Obama Not Seen as Overextended" within their report:
So far, do you think Barack Obama is [read and randomize response options, with option 3 always last]
35% - Trying to address too many issues at once
4% - Focusing on too few issues [OR]
56% - Doing about right
5% - Don’t know/Refused
Huh?
Let's stipulate that the following is speculation, but the usual culprit in these sorts of discrepancies is that pollsters are asking about something that a lot of respondents have not really considered before. Some call these "non-attitudes." As a general rule, it is hard to understate how often ordinary Americans are oblivious to the controversies that seem oh-so important inside the beltway, on cable news or the blogosphere. I would wager that the is-Obama-overextended meme is probably one of them (same goes for the word "meme").
One big clue is that although both questions ostensibly ask about the same idea, they use different wording. In particular, the Pew question is far more explicit about labeling one option as approving of Obama's performance ("Obama is...doing about right"), while the CNN equivalent is a little more vague ("Obama has tried to handle more issues than he should have"). Perhaps some respondents interpret the words "should have" differently, agreeing that Obama is having to handle more issues than he "should have" were these ordinary times. Or perhaps some are not hearing the words "than he should have" at all, and instead interpret the question as asking whether Obama is trying to handle many issues or few issues.
Notice also that the CNN question offers only two options, while the Pew question offers three. Most important, the "doing about right" response is the middle option on the Pew question, and survey methodologists know that when respondents are confused or uncertain, they often flock to the middle choice.
In the course work I did a dozen years ago through the University of Maryland's Joint Program in Survey Methodology (JPSM), one of the most valuable experiences was a class project that conducted "cognitive pre-testing" of survey questions . Each student had to find 10 subjects -- at least half without a college education -- willing to "think out loud" while they answered 20 minutes worth of survey questions. We did the interviews ourselves, and then compared notes on problems we discovered (for more info, see also Methods for Testing and Evaluating Survey Questionnaires).
Even though I had written survey questions for many years, the experience was an eye-opener. It was stunning to see how respondents failed to hear key portions of a question, ignoring introductory text altogether or hearing something or answering in a way completely different from what the authors had intended. Some stock questions formats I had used for years failed miserably. Unfortunately in-depth cognitive pre-testing is too costly and time consuming to be used on most political and media surveys. A true cognitive pretest of the CNN or Pew questionnaires would take weeks and might end up costing more than the surveys themselves.
But back to the questions about Obama doing too much. While I cannot prove that question wording and structure is the culprit here, it is worth restating a favorite rule of thumb: When small differences in question wording, structure or order produce wildly different results, the thing we are trying to measure is probably not a fully formed, pre-existing opinion for a significant number of respondents.
PS: Thanks for the Washington Post's Dan Froomkin for suggesting this topic.
By Mark Blumenthal | March 18, 2009 5:18 PM | Permalink | Comments (3) | TrackBacks (0)
Comments
Good topic indeed
Posted on March 19, 2009 1:22 PM
It appears to me that the discrepancy is the result of CNN's poorly worded questions. While the tendency of respondents to gravitate towards a neutral response may account for a small fraction of the majority who selected "doing about right " in the Pew poll, there are several factors that lead me to believe that a majority of people do actually feel this way.
The first is Obama's approval/disapproval ratings, which are nearly in alignment with the results. The second is that, historically, polls have shown that most people believe, generally, that office holders don't do enough. On an anecdotal level, people from outside the beltway I've talked to, both liberal and conservative, are impressed by Obama's willingness to tackle tough issues, even when they don't agree with his prescriptions, but none have expressed any concern that he is biting off more than he can chew.
In any event, I agree that this is probably a non-issue for most people, and that we're witnessing yet another example of pundits and pollsters fabricating and disseminating memes (a word which, by the way, first began taking popular root in the non-political blogosphere) as you suggest in your opening paragraph.
Posted on March 20, 2009 2:51 PM
Who wants to answer "he hasn't done that?" It's such weak wording, and sounds defensive. Clearly the poll leads you to a certain "right" conclusion.
Posted on March 28, 2009 12:40 PM
Post a comment