Articles and Analysis


South Carolina: Why So Much Variation?

Topics: 2008 , ARG , Bradley0/Wilder , IVR , Likely Voters , Mason-Dixon , Pollsters , PPP , Rasmussen , SurveyUSA , The 2008 Race

We've had quite a bit of discussion today in the comments section about the wide variation in results from the South Carolina polls. Reader Ciccina noticed some "fascinating" differences in the percentages reported as undecided, differences that lead reader Joshua Bradshaw to ask, "how is it possible to have so widely different poll numbers from the same time period?" There are many important technical reasons for the variation, but they all stem from the same underlying cause: Many South Carolina voters are still uncertain, both about their choices and about whether they will vote (my colleague Charles Franklin has a separate post up this afternoon looking at South Carolina's "endgame" trends).

Take a look at the results of eight different polls released in the last few days. As Ciccina noticed, the biggest differences are in the "undecided" percentage, which varies from 1% to 36%:

01-24 SC polls.png

1) "Undecided" voters -- Obviously, the differences in the undecided percentage are about much more than the random sampling variation that gives us the so-called "margin of error," but they are surprisingly common. Differences in question wording, context, survey mode and interviewer technique can explain much of the difference. In fact, variations in the undecided percentage are usually the main sources of "house effect" differences among pollsters.

The key issue is that many voters are less than completely certain about how they will vote and will hesitate when confronted by a pollster's trial heat question. How the pollster handles that hesitation determines the percentage that ultimately get recorded as undecided.

On one extreme, is the approach taken by the Clemson University Palmetto Poll. First, their trial-heat question, as reproduced in an online report, appears to prompt for "undecided" as one of the four choices. And just before the vote question, they asked another question which probably suggests to respondents that "undecided" is a common response:

Q1. Thinking about the 2008 presidential election, which of the following best describes your thoughts on this contest?

1. You have a good idea about who you will support
2. You are following the news, but have not decided
3. You are not paying much attention to the news about it
4. Don’t know, no answer

So two of the categories prime respondents with the idea that other South Carolina voters either "have not decided" or are "not paying much attention."

Most pollsters take the opposite approach. They try to word their questions, train their interviewers or structure their automated calls in a way to push voters toward expressing a preference. Most pollsters include an explicit follow-up to those who say they are uncertain, asking which way they "lean." The pollsters that typically report the lowest undecided percentages have probably trained their interviewers to push especially hard for an answer. And SurveyUSA, the pollster with the smallest undecided in South Carolina (1%), typically inserts a pause in their automated script, so that respondents have to wait several seconds before hearing they can "press 9 for undecided."

But it is probably best to focus on the underlying cause of all this variation: South Carolina voters feel a lot of uncertainty about their choice. Four of the pollsters followed up with a question about whether voters might still change their minds, and 18% to 26% said that they might. So many South Carolina Democrats -- like those in Iowa, New Hampshire before them -- are feeling uncertain about their decision. Thus, as reader Russ points out, "the last 24 hours" may count as much in South Carolina as elsewhere.

2) Interviewer or automated? - A related issue is what pollsters call the survey "mode." Do they conduct interviews with live interviewers or with an automated methodology (usually called "interactive voice response" or IVR) that uses a recording and asks respondents to answer by pressing keys on their touch-tone phones.

Three of the pollsters that released surveys over the last week (SurveyUSA, Rasmussen and PPP) use the IVR method (as does InsiderAdvantage), while the others use live interviewers. One thing to note is that the so-called "Bradley/Wilder effect" (or a the "reverse" Bradley/Wilder effect - via Kaus) assumes that respondents alter or hide their preferences to avoid a sense of "social discomfort" with the interviewer. Without an interviewer, there should be little or no effect.

In this case the difference seems to be mostly about the undecided percentage, which is lower for the IVR surveys. In the most recent surveys, the three IVR pollsters report a smaller undecided percentage (7%) than the live interviewer pollsters (17%). That pattern is typical, although pollsters disagree about the reasons. Some say voters are more willing to cast a "secret ballot" without an interviewer involved, while others argue that those willing to participate in IVR polls tend to be more opinionated.

If the Bradley/Wilder effect is operating, we would expect to see it on surveys that use live interviewers, but in this case, the lack of an interviewer seems to work in Obama's favor. He leads Clinton by an average of 17 points on the IVR polls (44% to 27%, with 19% for Edwards), but by only 9 points on the interviewer surveys (37% to 28%, with 17% for Edwards).

3) What Percentage of Adults? -- Four years ago, the turnout of 289,856 573,101 South Carolina Democrats amounted to roughly 9% 20% of the eligible adults in the state.* Turnout tomorrow will likely be higher, but how much higher is anyone's guess. Thus, selecting "likely voters" in South Carolina may not be as challenging as the Iowa or Nevada caucuses, but it comes close.

For Iowa, I spent several months requesting the information necessary to try to calculate the percentage of adults represented by each pollster. With the exception of SurveyUSA (who tell us their unweighted Democratic likely voter sample amounted to 33% the adults they interviewed), none of the pollsters have reported incidence data.

So some of the variation in results may come from the tightness of the screen, but we have no way to know for certain.

4) List or RDD? One important related issue is the "sample frame." Three of the South Carolina pollsters (SurveyUSA, ARG and Rasmussen) typically use a random-digit dial (RDD) technique that samples from all landline phones. They have to use screen questions to select likely Democratic primary voters.

As least two (PPP and Clemson) drew samples from lists of registered votes and used the records on the lists to narrow their sampled universe to those they knew had a past history of participating in primaries.

These two methods may also contribute to different results, and pollsters debate the merits of each approach.

5) Demographics? Differences in the likely voter selection methods mean that the South Carolina polls have differences in the kinds of people sampled for each poll. One of the most important characteristics is the percentage of African-Americans. It varies from 42% to 55% among the five pollsters that reported it (I extrapolated an approximate value for Rasmussen from their results by race crosstab).

01-24 SC AA.png

Another important difference largely hidden from view is the age composition of each sample. Only three pollsters reported an age breakdown. SurveyUSA reports 50% under the age of 50, compared to 43% on the McClatchy/MSNBC/Mason-Dixon survey. PPP had an older sample, with only 23% under the age of 45.

So the bottom line? All of these surveys indicate quite a bit of uncertainty, both about who will vote and about the preferences that their "likely voters" express. Obama appears to have an advantage, but we will not know how large until the votes are counted.

*Kevin: thank you for the edit.



I have heard different reports that those Republicans that did not vote last week in the R primary can actually vote in the Democratic primary tomorrow. Is that true and if so, is anyone including them in their polls?



I have heard different reports that those Republicans that did not vote last week in the R primary can actually vote in the Democratic primary tomorrow. Is that true and if so, is anyone including them in their polls?

Yes and Yes.

In fact, South Carolina does not even keep track of party affiliation. And all the polls I've seen have included a small percentage of Republicans. (Just as the exit polls from the GOP primary included a small percentage of Democrats).


Kevin Jochems:

Percentage of voters? In 2004, only 10% of eligible adults in SC voted in the Democratic primary or 290,000 people, not the higher number stated. This makes me wonder about all of these polls.


Kevin Jochems:

You are welcome.

I just read the latest Survey USA poll that came out tonight. Obama 43% Clinton 30% and Edwards 24%. They continue with the 33% of eligible adults. I worked in SC for the week of the 2004 primary race and was amazed by the weak turnout. Even with all the attention and focus this year, I don't see more than 20% of eligible adults voting and that number is more likely to be 15%. Using samples that don't correlate with that gives us nothing more than movement trends. Factor in the other elements like Independents, Demographics and RDD- I don't know who to believe. Guess we'll find out tomorrow.


Based on your analysis, it seems that

1)IVR-based polls (SurveyUSA, Rasmussen, and PPP)coincide with the secret ballot primary but the interviewer-based polls (Reuters, MSNBC, ARG, and Clemson) coincide with the open ballot caucus results.

2) In case of SurveyUSA Clinton/Edwards/Obama share, Two-day rolling average for Jan 22-23 = 29/22/45 and for Jan 23-24 = 30/24/43 yields average for Jan 24 = 31/26/41 with margin of error ~ 7%.



Great stuff. Every political reporter/pundit should be required to read this blog everyday as a condition of employment.



1) Being a Democrat in SC these days is akin to being a lesbian. It is not something you admit to except amongst close friends. It doesn't surprise me to see those numbers.

2)When I went and voted this morning in the primary, you are asked by the poll worker if you are a Democrat or Republican. Then you sign a piece of paper affirming you are a Democrat. But it is very informal.


Why so much variation? Wrong question.

A better question: Why does pollster.com even include sub-standard samplings in its compilations? Several companies listed here have quite poor track records, largely because they cut corners on cost.

No need to name names. We can all see who they are. Instead, let's just fast foward to a proposed solution: Any (any) polling organization that polls within 48 hours of voting -- yet reports a variance of PREDICTED (once undecideds are factored in) vs. ACTUAL results of greater than 15-20% -- is either excluded from pollster.com or else put into a second table labeled ~ "probationary polling firms." If they improve performance, great. Then restore them with honors. If they don't, stop rewarding lousy work. Because too many reporters do not (do NOT) see the difference. Too, it's time we all move on to the next stage of industry development and begin to create standards and judge outputs, not just dutifully report EVERY reckless input. Isn't that common sense?


In case of Reuters Clinton/Edwards/Obama share, Three-day rolling average for Jan 22-24 = 25/21/38 and for Jan 23-25 = 26/19/41 yields average for Jan 25 = 28/15/47 with margin of error ~ 7% (Interviewer-based poll).

In case of SurveyUSA Clinton/Edwards/Obama share, average for Jan 24 = 31/26/41 with margin of error ~ 7% (IVR-based poll.

Because SC is closed ballet primary I would favor SurveyUSA over Reuters.


Ray Lee:

This is a question from Florida. Why have there been no discussions about the "unsurveyed" and "unpollable" other than young adults with no landlines. In Florida, caller ID is ubiquitous and most people I know do not answer the phone if the number is not recognized. Some of my calls lately have obviously been either from polling firms or candidates; I answer none of them. To my statistically unsophisticated mind, this has to skew the results if one is sampling in an area where caller ID is used extensively. Specifically, higher income, tech savvy households are most likely to screen calls and use answering machines to ascertain if they wish to respond. SO, it seems as if a whole swath of the electorate could be missed. I understand that the polls are "corrected" for categories like gender, race, income levels etc. but how do you correct for excluding more sophisticated, busy adults who do not wish to be bothered by strangers? Believe me, we often vote more reliably than other "likely voters."


Manoa Kahuna:

Rarely have I read such
sophisticated comments. Thanks
everyone, I learned something.

I can't wait for the real
vote to come in to compare the
results with the polls.


Post a comment

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.