Pollster.com

Articles and Analysis

 

SurveyUSA's Post Debate "Panel Back"

Topics: 2008 , Debates , The 2008 Race

Tonight, partly in response to a critique offered here in May, SurveyUSA fielded an automated "panel-back" survey among 717 Americans who said they watched tonight's CNN/YouTube debate. The idea of a panel-back survey is to interview the same sample before and after some event, allowing for an individual level measurement of change. With a panel back survey, a pollster can determine how many respondents shifted their position

Here is the key question asked of these respondents before and after the debate:

Regardless of who you may vote for, and regardless of whether or not you would vote for a Democratic candidate for President, which Democratic candidate would make the best President of the United States?

The result:

Americans who watched the YouTube Democratic Debate 7/23/07 on CNN went into the debate thinking Hillary Clinton would make the best President of the 8 Democrats on stage, and came out of the debate even more convinced, according to an exclusive SurveyUSA poll. Before the debate, 40% saw Clinton as the most Presidential Democrat; after the debate, when the same respondents were re-interviewed, 43% saw her as the most Presidential Democrat

It is not obvious from the before and after snapshot, but it appears that most of those who responded to both calls were consistent in their answer to this question, before and after. To the extent that there was a shift, however, it worked slightly to the benefit of Clinton and Biden, slightly against Barack Obama.

They also asked if respondents "viewed each Democrat positively or negatively" before and after the debate. The results indicate that exposure to the debate improved perceptions of all the candidates except for Gravel. The summary reported by SurveyUSA (with bullets added):

  • Biden went up 38 points, from Minus 6 to Plus 32.
  • Obama went up 17 points, from Plus 24 to Plus 41.
  • Clinton went up 16 points, from Plus 34 to Plus 50.
  • Edwards went up 16, from Plus 22 to Plus 38.
  • Dodd went up 15, from Minus 21 to Minus 6.
  • Richardson went up 14, from Minus 1 to Plus 13.
  • Kucinich went up 7, from Minus 21 to Minus 14.
  • Gravel went down 3, from Minus 29 to Minus 32

The SurveyUSA summary does include some important caveats worth noting. The most important is that this sample of 717 is not a random sample of all Americans (or likely voters), or even a random sample of all Americans who happened to be at home this evening. They started with a first round of calls to a random sampling of households, but this final pool of 717 reflects only those who responded to the first survey, reported that they would watch the debate and take a second call and then actually completed a second call. The summary also notes that "panel back is not universally sanctioned" as a tool to measure debate reaction, pointing to the unsparing criticism by pollster David Hill. Writing for The Hill in 2004, he described the panel-back as "worthless:"

Considerable scholarly research demonstrates that simply being interviewed renders an otherwise normal voter abnormal. After being polled, voters are much more likely to seek out political information through the media, discuss politics with others and eventually to vote. The known effects are so great that in the earliest days of polling, voters would be screened at the outset of an interview to ascertain if they had ever been interviewed before.

I don't share Hill's blanket aversion to panel-back surveys, but he is right to warn that the first interview can add some artificiality to either the nature of the sample or the way those surveyed watch and react to the debate. For all its drawbacks, however, this approach is a great improvement to simply calling a fresh sample of those at home and asking "who won" without any data on which candidate they supported beforehand.

In the spirit of better understanding the data, I do have a few questions for the analysts at SurveyUSA:

First, what percentage of the 717 debate watchers provided the same answer to the first question before and after the debate?

Second, the summary reports:

SurveyUSA dialed at random into all 50 states. Respondents were asked if they planned on Monday night, 7/23/07, to watch the entire YouTube debate on CNN. Those respondents who said "yes" were then asked if SurveyUSA could call them back, immediately after the debate

What percentage of those adults interviewed on the first call agreed to watch the debate and be interviewed a second time?

Finally, did SurveyUSA have any way to confirm that they interviewed the same individuals within each household before and after the debate?

 

Comments
Andrew:

Let's not underestimate the high number of stupid Americans who cannot think by themselves, and would rather wait until the media tells them who "won" the debate before making a decision.

____________________

I actually happened to be a participant in this survey. As a public opinion scholar, it was a nice opportunity to hear exactly how the Survey USA interface works. The experience got me thinking about some other problems that the IVR technology may run up against. I have posted about those problems here:

http://ccpsblog.blogspot.com/2007/07/my-conversation-with-survey-usa.html

____________________

Jonesy:

I agree with Andrew. I dont have a favorite in this race, but I get the impression that right after the debate most people thought Obama won, UNTIL they heard what the pundits said, then they changed their opinion to Hillary.

There was no better example of this than on CNNs own website, where the majority in the focus group chose Obama while those answering on the CNN Website (having read three opinions by pundits) picked Hillary.

____________________

Kristoph:

@Jonesy

Actually this is not correct.

The CNN group chose Clinton as the winner immediately after the debate (without other input).

What is in interesting is that during the debate Edwards scored the highest, followed by Obama, and then Clinton and yet when asked 'who won' the group said Clinton, then Obama, then Edwards.

]{

____________________

do not deny it any more!
It is clear what should be done about this survey.
Now we are aware of results of "panel-back" survey

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR