Pollster.com

Articles and Analysis

 

Iowa Caucus Polling: Only the Beginning

Topics: 2008 , The 2008 Race

It's probably the blogger variant of Murphy's Law, but the most interesting topics often bubble up whenever I take time off. The last few weeks were no exception as several new polls were released on the 2008 primary contests, particularly in Iowa and New Hampshire. As often happens this early in the process, some produced contradictory results, especially in Iowa.

The most puzzling - as noted by our friend Mickey Kaus - involves the performance of Hillary Clinton and Barack Obama in two polls of likely Democratic caucus goers conducted in Iowa in late December by Research2000 and the American Research Group (ARG). Both showed John Edwards with roughly the same support (20-22%). ARG Research 2000 showed Clinton leading with 31% and Obama running distant forth (at 10%) behind outgoing Iowa governor Tom Vilsack (17%). Research 2000 ARG showed Obama and Edwards tied for first (22%), with Clinton running forth (10%) behind Vilsack (17%).

So...Hillary Clinton is either their clear front runner in Iowa (with 31%) or running a distant fourth (with 10%).

Ladies and gentlemen, welcome to the pollster's nightmare: The Iowa Caucuses.

I have written before about the challenge of polling the caucuses before and will certainly do so again, but the numbers behind the challenge remain the same. Here is the way I put it, when the Des Moines Register released its first 2008 caucus poll last June:

The big challenge for polling this contest, of course, is that turnout for the Democratic caucuses is typically a small percentage of eligible voters. Iowa had roughly 2.2 million voting eligible adults in 2004, of whom (as of last month) approximately 1.9 million are considered "active" registered voters by the Iowa Secretary of State. But only 124,331 participated in the 2004 Democratic Caucuses for President (according to the subscription only Hotline). That number amounts to roughly 6% of all registered voters, so selecting "likely caucus goers" is no easy task.

When I first saw the conflicting results, I assumed something obvious about the survey design or field dates might explain the difference. For example, some pollsters sample likely caucus-goers by calling a random digit dial (RDD) sample of all telephone households and will then screen for likely voters. Some will sample from the lists of registered voters (with many unlisted numbers missing) and select using a combination of screening and various "vote history" criteria, including participation in past caucuses. In past elections, Iowa caucus surveys drawn exclusively from lists of past caucus-goers have differed from those based on RDD methods.

I spoke earlier today to both Dick Bennett of ARG and Del Ali of Research 2000, and in this case the sample procedures and field dates were more similar than different:

  • Research 2000 conducted a survey among 400 Democratic "caucus goers" December 18 through December 20. They started with a random digit dial (RDD) sample of Iowa households and screened for those who (a) say they frequently vote in statewide general election[s] and (b) report having participated in the 2004 Democratic caucuses.
  • The American Research Group conducted a survey among 600 "likely Democratic caucus goers" between December 19 and December 23. They too started with a random digit dial (RDD) sample of Iowa households and screened for those who were (a) registered vote as either Democrats or with no party affiliation who also said they (b) "definitely plan to participate in the 2008 Democratic presidential caucus" (those screened out those who said they "might" participate or who "probably will not").

The field dates overlap, so timing seems unlikely to explain much of the difference, particularly if the theory is that Obama's support has been rising of late. Keep in mind that the ARG poll, which finished later, showed Obama doing worse.

The biggest difference that we are aware of is that the Research2000 screens were based on self-reported past participation, while ARG screens rely on a question of prospective intent to participate. We can debate the relative merits of each approach (and no doubt will in the coming months), but it seems unlikely that this particular difference produced a 21-point shift in support for Hillary Clinton.**

Of course, it may be that one approach was significantly "tighter" than the other. That is, did one capture a much narrower slice of Iowa voters than the other? Unfortunately, neither pollster has released data on how many otherwise qualified respondents they screened out in order to select their final sample (as the Des Moines Register did last June).

It is also important to focus on the questions asked. Both pollsters asked respondents to choose from a list of eleven potential candidates, both "rotated" (or randomized) the order of names as read by interviewers and both reported relatively few in the completely undecided category (11% for Research 2000 and 8% for ARG). But the candidate listings were not identical. ARG included the names of two potential candidates -- Connecticut Senator Chris Dodd and former Alaska Senator Mike Gravel - that Research 2000 omitted. Likewise, Research 2000 included Al Gore and Evan Bayh, while ARG did not. Of these, Gore had the most support (7%), Dodd had 2% and the rest just 1%. Given the numbers involved, it is hard to see how these minor differences contributed much to the Clinton-Obama discrepancy.

All of which leaves me scratching my head, except to say this: Whenever very small differences in methodology make for huge differences in results, it suggests that voters are not yet engaged in the race enough to have strong allegiances. Put another way, while each poll may have a candidate running in front, in Iowa at least, there is not yet a true "front runner."

**UPDATE: Ok, make that coming hours. Mickey Kaus considers the prospective / retrospective difference in the survey screens a more likely explanation than I did:

There's a big difference between 1) asking voters if they "definitely plan" to go to the caucuses, and 2) asking voters if they actually participated in the 2004 caucuses. Lots of people say they "plan" to attend. That's normal! But those who have attended are the sort of pathetically unrepresentative hard core activists ...sorry, committed citizens who make up the tiny sliver (6%) of Iowa voters who actually show up and choose the winner: ... In this case, the merely aspirational caucusgoers pick Clinton, while the hard core goes for Obama--a result consistent with the idea that Obama is capturing those who think a lot about politics, while those who don't think as much about politics haven't yet been hit by the wave.

That's a plausible theory, particularly if the retrospective caucus participation question successfully identified actual past caucus goers. Retrospective vote questions typically over-report past voting behavior, but in this case the Research 2000 question may have produced an appropriately tighter screen. Of course, without the ability to compare the relative incidence of each survey, we are just speculating.

In 1988, I worked for Paul Maslin, the pollster for Democratic Senator Paul Simon. Simon always did better on the samples we drew from lists of actual past caucus-goers, while Congressman Dick Gephardt did consistently better when when we included registered voters that had not participated in the previous caucuses in 1984. Gephardt also did consistently better on the RDD surveys in the public domain. As I recall, those differences persisted through the final round of polling, though they probably narrowed a bit toward the end. Of course, the challenge is that every election year, the caucuses attract large numbers of voters who did not participate in the prior election cycle. And true junkies will remember that Gephardt ultimately won the Caucuses, although as I recall, the actual result fell somewhere between the two methodologies.

 

Comments

Good post. Excellent point by Mickey Kaus, I was so glad to read that, as I had the exact same thought the moment you mentioned that one poll was asking about "planning on going" to the caucuses and one poll was asking about having actually attended last time. Although, as you note, it's no perfect science, I'd bet on the latter group being a more accurate sample over the former in a second if I had to. As Kaus points out, not all those those planning on going will go, and those people are more likely to base their decision on name recognition than the hardcore voters who actually have a reputation of showing up for caucuses.

On a different note, these ARG numbers for all four states were a bit suspect to begin with anyway; the numbers look like they were tailor made for the Clinton campaign. We're supposed to believe not only that Obama is down to 10% in Iowa, but that Edwards is in single digits in Nevada? That's hardly plausible.

At best, this suggests that the ARG numbers are MUCH more based on name recognition than those surveys conducted by local newspapers in Iowa and New Hampshire, which show Obama starting to overtake Clinton in NH and Edwards not too far behind them, and Obama and Edwards leading Iowa with Hillary in their dust.

____________________

TarHeel:

the research 2000 poll is much more likely to be true.

A- no other poll has shown hillary with that much support in Iowa EVER

B - the Res2000 threshold of did you caucass in 2004 and do you vote in statewide elections is MUCH higher than ARG's which says are you a registered voter and do you plan to caucass for president for 2008

the ARG poll in a caucass state is crap.

____________________

TarHeel:

the CRAZY ARG poll also had NEWT GINGRICH 3rd in EVERY state.. even though he says he won't even decide about running until 9-2007.. While every other poll never has him in the top 3.

The ARG poll is a name recognition poll for partisans whether it be Hillary or Newt Gingrich.. period...

____________________

Dave Michaels:

Hotline mentions that:

FWIW, there's not an Iowa Dem Caucus poll we have heard about, public or private, that has had Clinton in first. (bold added)

It looks like ARG's screen for likely caucus-goers was not tight enough, which would explain the numbers in IA and NV. The primary numbers in SC and NH look more realistic.

____________________

The Iowa poll of 2005 gave Condi Rice as 30% rating and was reported in all the Iowa newspapers. The buzz might have been created with the efforts to run ads in the state on radio and TV, helping to create a vision of our Secretary of State at the helm. With the growing competition of Rudy, McCain, Romney and others, Condi Rice is lower in the polls, yet is still seen as a contender.
The Marist poll of November 2006 asked which person they WANTED to run, and Condi got 45% of the support to run. That is newsworthy.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR