Mark Blumenthal | July 19, 2007
Topics: IVR , IVR Polls , Measurement
The latest automated SurveyUSA poll in the Kentucky Governor's race provides us with one of those classic conflicting poll stories that we just love here at Pollster.com, because it illustrates how small differences in methodology can have a profound effects on the results. In this case, SurveyUSA shows Democrat Steve Beshear leading incumbent Republican Ernie Fletcher by a 23 point margin (59% to 36%) with only 5% undecided. Meanwhile, an InsiderAdvantage poll conducted a week earlier shows Beshear leading by just three points (41% to 38%) with a much larger number (21%) in the undecided category
What explains the difference? Continue after the jump for more explanation, but my best guess is that the solution can be found in this conundrum: On a poll, "undecided" means something different than "still trying to decide."
The results shown below add a little more context. Back in May, SurveyUSA showed Beshear leading by an even wider (62% to 34%) margin at about the same time that another automated survey by Rasmussen Reports showed Beshear 16 points ahead (51% to 35%).
One obvious difference is that SurveyUSA and Rasmussen use an automated, recorded voice that asks respondents to answer by pressing numbers on their touch tone phones. InsiderAdvantage uses live interviewers.
As part of a presentation I did with Charles Franklin at the AAPOR conference in May, I took the statewide surveys we gathered during 2006 and calculated the average undecided/other response for automated and conventional (interviewer) phone polls during each month (technically, I averaged results within states, and then averaged values for all states, looking at what was left after summing results for each reported candidate). The bottom line is that the undecided percentage was typically 3 to 5 percentage points lower on automated polls than those using live interviewers.
The key question about this pattern, as suggested by the chart's subtitle, is whether it results from IVR sampling different kinds of people or from respondents giving different answers on an automated survey.
It is certainly possible, of course, that kinds of people willing to participate in an automated sample might be more opinionated than those who needed extra prodding from a live human being. Critics of the automated polls frequently make this argument, but this is a tough claim to evaluate given the data we have available.
It is also possible that other sampling differences - such as the sample "frame" and the kinds of voters that qualified as "likely" - helped produce the different result in this example from Kentucky. SurveyUSA reports, for example, that they started with a random digit dial (RDD) sample of 1,000 adults and screened down to the 560 they deemed most likely to vote.
InsiderAdvantage reports that they interviewed "693 registered voters." Theoretically, those who say they are likely to vote should be a bit more opinionated than those who are just registered, a factor that may contribute to the difference in the undecided percentage. InsiderAdvantage provides no further information about their sample, although they did tell me in October 2004 that all of their surveys that year sampled from registered voter lists rather than RDD.
In this case, however, my guess is that the most likely culprit is not the sample but rather the way the two polls measured vote preference, and more specifically, how hard they pushed voters to express a preference.
Automated pollsters argue that they get a lower undecided partly because the Interactive Voice Response (IVR) methodology better simulates the secret ballot. Respondents are sometimes reluctant to reveal their preference to a stranger on the telephone, hesitating to voice an opinion might introduce some "social discomfort" into the conversation (for either the respondent or the interviewer). Conventional pollsters have long appreciated this challenge, which is why very few provide "undecided" as an explicit answer category. Interviewers are usually trained to record "undecided" as an answer only when respondents cannot provide an answer.
Most political pollsters also include a follow-up question that presses the initially reluctant respondents about which candidate they "lean" toward supporting. SurveyUSA adapts this practice to their automated methodology by offering "undecided" as the final answer option, but only after a pause of several seconds that essentially pushes respondents to make a choice.
All of this makes the following webcast comments of InsiderAdvantage CEO Matt Towery especially telling:
When you are this far out from an election, November is the election day, you have to make sure that the individuals who are taking the poll understand that they have the option to say "undecided" and that's very important when you survey this far out in a gubernatorial race, particularly one that's in flux.
If InsiderAdvantage explicitly prompted for undecided, it is not at all surprising that their survey produced a much higher than average undecided percentage than the SurveyUSA poll.
So which approach is right? Virtually all pollsters agree that it is poor practice to suggest "undecided" as an answer a few days before an election, a practice that will likely result in a less accurate forecast. However, this issue is not as clear four months before an election when relatively few voters have made a final decision. Towery is right to argue in his webcast that "large numbers of people . . . don't care about politics until the bitter end." This far out, voters may express a preference when asked to choose, but that preference is still subject to change.
As evidence, consider the results from this week's WMUR/CNN/UNH New Hampshire primary poll. Roughly nine out of ten Republicans (88%) express support for one of the candidates on the trial heat question, but only 7% have "definitely decided who they will vote for," and 71% of New Hampshire Republicans are "still trying to decide." Similarly, 91% of Democrats in New Hampshire express a preference, but only 10% are certain and 64% are still trying to decide.
This is the key point: Expressing a preference on a pre-election poll months before an election is not the same as making a final decision. If you assume that the "undecided" category captures everyone still weighing their choices, you will badly misinterpret the results whether you think the better "undecided" number is 5% or 21%.
The most striking thing to me in the Kentucky results is the relative consistency in support for Fletcher across all four polls. Fletcher's support varies within a four point range (34% to 38%), while support for Beshear varies widely (from 41% to 62%). That pattern is not unusual in a race involving a well known incumbent, particularly when the incumbent is embattled by various scandals that have lowered his job performance rating to just 39% of adults. Attitudes about the incumbent typically drive voter preference at this stage of the race, lesser known challengers typically gain support as the campaign progresses.
Collectively, these results suggest trouble ahead for Governor Fletcher unless he can convince a big chunk of Kentucky voters to either reconsider his performance as governor or rule out Steve Beshear as a credible alternative.
Finally, for what it's worth, my preference even at this stage in a race is to push initially hesitant respondents hard to express a preference, but then to immediately follow-up with a question (like the one asked by WMUR/CNN/UNH) about how certain they are to actually support their preferred candidate. Every campaign pollster I know asks a variant of the "certainty" question -- I wish more media pollsters would do the same.
PS: One more thing. The SurveyUSA question includes the names of the two running mates, while InsiderAdvantage appears to name just Fletcher and Beshear. While Steve Beshear held statewide office in the 1980s and was the Democratic nominee for U.S. Senate in 1996, his running mate Dan Mongiardo lost an even closer Senate contest just three years ago. So the ticket of Beshear and Mongiardo may sound like a more compelling alternative to uncertain voters than Beshear alone.