Pollster.com

Articles and Analysis

 

Rasmussen's Democrats -- Don't Tell Mama?

Topics: 2008 , IVR , IVR Polls , The 2008 Race

Both Mickey Kaus and Chris Bowers at MyDD noticed that Rasmussen Reports has been showing a much closer race on their automated national tracking of the 2008 Democratic presidential primary contest. Both floated different theories for that difference that imply that the Rasmussen's numbers are a more accurate read. This post takes a closer look at those arguments, although the bottom line is that hard answers are elusive.

The chart below shows how the recent Rasmussen surveys compare to the trend for all other conventional polls as tracked by Professor Franklin here at Pollster. The bolder line represents the average trend across all conventional surveys, while the shorter narrow lines connect the recent Rasmussen surveys. Click the image to enlarge it, and you will see that all but one of the Rasmussen surveys shows Barack Obama running better than the overall trend. The Rasmussen results for Clinton show far more variability, especially during the first four weeks of Rasmussen's tracking program. They show Clinton running worse than other polls over the last three weeks. Note that a new survey released overnight by Gallup (that shows Clinton's lead "tightening") has not altered the overall trend.

RasVsAllPolls%20legend_sml.png

Of course the graphic above includes survey questions that continue to include Al Gore on the list of candidates. In order to reduce the random variability and make the numbers as comparable as possible, I created the following table. It shows that Clinton leading by an average of roughly 15 points (38.6% to 23.8%) on the three most recent conventional telephone surveys, but by just 5 points (33.0% to 28.3%) on the three most recent Rasmussen automated surveys (surveys that use a recorded voice and ask respondents to answer by pressing buttons on their touch tone phones). Given the number of interviews involved, we can assume that these differences are not about random sampling error. Something is systematically different about the Rasmussen surveys that has been showing a tighter Democratic race over the last three weeks.

04-18%20recent%20dem%20results.png

But what is that difference? That's a tougher question to answer. Here are some theories, including those suggested by Bowers and Kaus:

1) The automated methodology yields more honest answers about vote choice (and thus, a more accurate result). The theory is that some people will hesitate to reveal certain opinions to another human being, particularly those that might create some "social discomfort" for the respondent. Thus, Kaus provides his "Don't Tell Mama" theory: "men don't like Hillary but are reluctant to say so in public" or to "tell a human interviewer -- especially, maybe, a female interviewer."

2) The people sampled by Rasmussen's surveys are more representative of likely Democratic primary voters because it uses a tighter screen. Chris Bowers makes that point by arguing that the Rasmussen screen looks slightly tighter than those used by other pollsters - "38-39% of the likely voter population" rather than the "40-50% of all registered voters [sampled by] the vast majority of national Democratic primary polls."

3) The people sampled by automated surveys are more representative of likely primary voters because they give more honest answers about whether they will vote. We know from at least 40 years of validation studies that many respondents will say they voted when they did not, due to the same sort of "social discomfort" mentioned above. Voting is something we are supposed to do, and a small portion of adults is reluctant to admit to non-voting to a stranger on the telephone. In theory, an automated survey would reduce such false reports.

4) The people sampled by automated surveys are less representative of likely primary voters because they capture exceptionally well informed respondent. This theory is one I hear often from conventional pollsters. They argue that only the most politically interested are willing to stay on the phone with a computer, and so automated surveys tend to sample individuals who are much more opinionated and better informed than the full pool of genuinely likely voters.

Lets take a closer look at the arguments from Kaus and Bowers.

Kaus makes much of the fact that the Rasmussen poll shows a big gender gap, with Clinton showing a "solid lead" (according to Rasmussen) among men, but trailing 11 points behind Obama among men. He wonders if other polls show the same gender gap. While precise comparisons are impossible, all the other polls I found that reported demographics results also show Clinton doing significantly better among women then men (Cook/RT Strategies, CBS News, Time and the Pew Research Center). Rasmussen certainly shows Obama doing better among men than the other surveys, but then, Rasmussen shows Obama doing better generally than the other surveys.

Kaus also offers a "backup" theory:

Of course (if it turns out the gender gap in the two polls is roughly comparable) it could be that many men and many women don't like Hillary but are reluctant to say so in public. (if it turns out the gender gap in the two polls is roughly comparable) it could be that many men and many women don't like Hillary but are reluctant to say so in public.

His backup may be plausible, especially when interviews are conducted by women, although we obviously have no hard evidence either way.

Bowers' theory feels like a better fit to me, especially if we also consider the possibility that the absence of an interviewer may reduce the "measurement error" in the selection of likely voters. The bottom line, however, is that we really have no way to know for sure. It is certainly possible, of course, that the Rasmussen's sampling is less accurate. All of these theories are plausible, and without some objective reality to use as a benchmark, we can only speculate about which set of polls is the most valid.

What strikes me most, as I go through this exercise, is how little we know about some important methodological details. What are the response rates? Are Rasmussen's higher or lower than conventional polls? How many respondents answered the primary vote questions on recent surveys conducted by ABC News/Washington Post, NBC/Wall Street Journal and Fox News and the most recent CNN survey? Many pollsters provide results for subgroups of primary voters, yet virtually none tell us about the number of interviews behind such findings. We also know nothing of the demographic composition of their primary voter subgroups, including gender, age or the percentage that initially identify as independent.

And how exactly do those pollsters that currently report on "likely voters" select primary voters? How tight are their screens? Very little of information is in the public domain (and given that these numbers involve primary results, my likely voter guide from 2004 is of little help).

I emailed Scott Rasmussen to ask about their likely voter procedure for primary voters. His response:

We start with the tightest segment from our pool of Likely Voters... Dems are asked about how likely they are to vote in Primary... Unaffiliateds are asked if they had the chance, would they vote in a primary... if so, which one...

I am not completely sure what the "tightest segment" is, but I my guess is that they take those who say they will definitely or certainly vote in the Democratic primary. He also confirmed that the 774 likely Democratic primary voters came from a pool of 2,000 likely voters. So last night I asked what portion of adults qualified as likelyvoters so we might do an apples-to-apples comparison of the relative "tightness" of survey screens. As of this writing, I have not received an answer. UPDATE: Via email, Scott Rasmussen tells me that while he did not have numbers for that specific survey readily available, the percentage of adults that qualify as likely general election is typically "65% to 70%...for that series." He promised to check and report back if the number for this latest survey are any different.

But with respect to all pollsters again, and not just Mr. Rasmussen, why is so little of this sort of information in the public domain? Most media pollsters pledge to abide by professional codes of conduct that require disclosure of basic methodological details on request. Maybe it's time we start asking for that information for every survey, and not just those that produce quirky results.

 

Comments
Chris S.:

Is Rasmussen the only outlier though? I've been following the list of polls at RCP:

http://www.realclearpolitics.com/epolls/2008/president/us/democratic_presidential_nomination-191.html

for the last several months, and there are three pollsters that they indicate are sampling "likely voters" rather than "registered voters": Rasmussen, ARG, and Zogby. Granted, ARG and Zogby don't have nearly as many polls as Rasmussen. But of the ones they do have, they also seem to show a somewhat tighter race than the pollsters who use "registered voters" (the latest ARG poll being an exception).

Or maybe it's just my imagination. I haven't plotted all the polls myself. But maybe you could try combining all of the LV polls from this year, and plotting that against all of the RV polls from this year. Is the difference as significant as when you're only separating out Rasmussen?

____________________

Jane Sobola:

Man, you people need to get ****ing lives.

____________________

Thom:

This is our life, Jane.

____________________

Dickie Flatts :

Sounds like a some folks making excuses for why Hillary is not leading in the polls and Obama is.

As '04, '05, and '05 pointed out, national polls do a pretty good job.

____________________

JoeCHI:

Clinton is...smart, tough, street-wise, calculated, wonkish, worldly, ambitious, triangulating? All admirable qualities in my opinion!

In fact, Love it! LOVE IT! Thank you, I'll have another!

____________________

matt:

I second the call to release response rates for every poll, but I doubt this will actually occur. Response rate is the Achilles heel of all public polling. All the statistics that allow for polling assume random samples, but if the sample is skewed (be it demographically, politically, etc.) then who can tell what the "correct" result is? To acknowledge that polling has some random error is one thing, but if it has some unknown systematic error then who would listen?

____________________

Sara:

Dickie, your comment makes no sense. I have yet to see a national poll, regardless of the screeners used, where Obama was beating Hillary.

____________________

Justin:

At Public Policy Polling, an IVR company in North Carolina, our samples are based on the voter registration database. Therefore, when we conduct primary polls we are only calling households that have a history of voting in at least one party primary.

I think that's the tightest screen.

Unfortunately, those databases are not available nationally. They are only available in about 3/4ths of the states.

http://publicpolicypolling.blogspot.com/2007/04/do-national-polls-inflate-hillary.html

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR