Mark Blumenthal | July 24, 2007
Topics: 2008 , Debates , The 2008 Race
Tonight, partly in response to a critique offered here in May, SurveyUSA fielded an automated "panel-back" survey among 717 Americans who said they watched tonight's CNN/YouTube debate. The idea of a panel-back survey is to interview the same sample before and after some event, allowing for an individual level measurement of change. With a panel back survey, a pollster can determine how many respondents shifted their position
Here is the key question asked of these respondents before and after the debate:
Regardless of who you may vote for, and regardless of whether or not you would vote for a Democratic candidate for President, which Democratic candidate would make the best President of the United States?
Americans who watched the YouTube Democratic Debate 7/23/07 on CNN went into the debate thinking Hillary Clinton would make the best President of the 8 Democrats on stage, and came out of the debate even more convinced, according to an exclusive SurveyUSA poll. Before the debate, 40% saw Clinton as the most Presidential Democrat; after the debate, when the same respondents were re-interviewed, 43% saw her as the most Presidential Democrat
It is not obvious from the before and after snapshot, but it appears that most of those who responded to both calls were consistent in their answer to this question, before and after. To the extent that there was a shift, however, it worked slightly to the benefit of Clinton and Biden, slightly against Barack Obama.
They also asked if respondents "viewed each Democrat positively or negatively" before and after the debate. The results indicate that exposure to the debate improved perceptions of all the candidates except for Gravel. The summary reported by SurveyUSA (with bullets added):
- Biden went up 38 points, from Minus 6 to Plus 32.
- Obama went up 17 points, from Plus 24 to Plus 41.
- Clinton went up 16 points, from Plus 34 to Plus 50.
- Edwards went up 16, from Plus 22 to Plus 38.
- Dodd went up 15, from Minus 21 to Minus 6.
- Richardson went up 14, from Minus 1 to Plus 13.
- Kucinich went up 7, from Minus 21 to Minus 14.
- Gravel went down 3, from Minus 29 to Minus 32
The SurveyUSA summary does include some important caveats worth noting. The most important is that this sample of 717 is not a random sample of all Americans (or likely voters), or even a random sample of all Americans who happened to be at home this evening. They started with a first round of calls to a random sampling of households, but this final pool of 717 reflects only those who responded to the first survey, reported that they would watch the debate and take a second call and then actually completed a second call. The summary also notes that "panel back is not universally sanctioned" as a tool to measure debate reaction, pointing to the unsparing criticism by pollster David Hill. Writing for The Hill in 2004, he described the panel-back as "worthless:"
Considerable scholarly research demonstrates that simply being interviewed renders an otherwise normal voter abnormal. After being polled, voters are much more likely to seek out political information through the media, discuss politics with others and eventually to vote. The known effects are so great that in the earliest days of polling, voters would be screened at the outset of an interview to ascertain if they had ever been interviewed before.
I don't share Hill's blanket aversion to panel-back surveys, but he is right to warn that the first interview can add some artificiality to either the nature of the sample or the way those surveyed watch and react to the debate. For all its drawbacks, however, this approach is a great improvement to simply calling a fresh sample of those at home and asking "who won" without any data on which candidate they supported beforehand.
In the spirit of better understanding the data, I do have a few questions for the analysts at SurveyUSA:
First, what percentage of the 717 debate watchers provided the same answer to the first question before and after the debate?
Second, the summary reports:
SurveyUSA dialed at random into all 50 states. Respondents were asked if they planned on Monday night, 7/23/07, to watch the entire YouTube debate on CNN. Those respondents who said "yes" were then asked if SurveyUSA could call them back, immediately after the debate
What percentage of those adults interviewed on the first call agreed to watch the debate and be interviewed a second time?
Finally, did SurveyUSA have any way to confirm that they interviewed the same individuals within each household before and after the debate?