Mark Blumenthal | August 31, 2007
Topics: 2008 , Disclosure , Likely Voters , The 2008 Race
Following up on yesterday's post, in which I speculated - wrongly, as it turns out -- about the incidence of eligible adults selected by the American Research Group (ARG) as likely caucus goers for their most recent surveys of Democrats and Republicans in Iowa. I emailed Dick Bennett, and can now report on how their surveys compare to the others that have provided us with similar details.
First, according to Bennett, I was incorrect in speculating that they use only one question to screen for "likely caucus goers." They start with a random digit dial (RDD) sample of adults in Iowa in households with a working telephone and then ask four different questions (although they provide only the last question on the page reporting Iowa results):
- They ask whether respondents are registered to vote, and whether they are registered as Democrats or Republicans. Non-registrants are terminated and not interviewed.
- They ask registrants how likely they are to participate in the Caucus "a 1-to-10 scale with 1 meaning definitely not participating and 10 meaning definitely participating." Those who answer 1 through 6 are terminated and not interviewed.
- They ask unaffiliated registrants ("independents" registered as neither Democrats nor Republicans) whether they plan to participate in the Democratic or Republican caucus. Registered Democrats and independents who plan to caucus with the Democrats get the Democratic vote question; Registered Republicans and independents who plan to caucus with the Republicans answer the Republican question.
- After asking vote question, they asks the question that appears on the web site: "Would you say that you definitely plan to participate in the 2008 Democratic presidential caucus, that you might participate in the 2008 Democratic presidential caucus, or that you will probably not participate in the 2008 Democratic presidential caucus?" Only the definite are included in the final sample of likely caucus voters.
So the process involves calling a random sample of adults until they reach a quota of 600 interviews for voters of one of the parties. In their most recent Iowa survey, they were able to fill the quota for Democrats first, so they continued dialing the random sample until they had interviewed 600 Republicans, terminating 155 Democrats in the process. Bennett reports that they also terminated another 4,842 adults on their various screen questions (740 who say they were not registered to vote, 3,598 who rated their likelihood of participating as 6 or lower and 504 who were less than "definite" about participating on the final question).
So, the "back of the envelope" calculation for ARG is that their most recent sample of Democrats represents 12% of Iowa adults (755 Democrats divided by 755+600+4,842). Their most recent sample of Republicans represents roughly 10% of Iowa adults (600 Republicans divided by 755+600+4,842). We can compare the Democratic statistic to those provided by other Iowa pollsters:
- 12% - ARG (August)
- 12% - Time/SRBI (August)
- 12% - ABC/Washington Post (August)
- 29% - University of Iowa (August)
- 9% - Des Moines Register/Selzer & Co. (June 2006)
And again, for those just joining this discussion, the 2004 Democratic caucus turnout was reported as 122,200, which represented 5.4% of the voting age population and 5.6 of eligible adults.
So, if we take all of these pollsters at their word, my "blogger speculation" yesterday was off-base: ARG's incidence of Democratic likely voters as a percentage of eligible adults is very close to the surveys done by Time and ABC/Washington Post. Apologies to Bennett.
But we still have a mystery. Why the consistent difference between the result from ARG and other surveys that appears to favor Clinton? Professor Franklin is working on a post as I speak that will chart the difference, but when we exclude the ARG's surveys from our estimate for Iowa, Clinton's current 2 point margin over Edwards (26.2% to 24.2%) becomes a 1.3 point deficit (24.6% to 25.9%). [See Franklin's in-depth discussion, now posted here].
I asked Bennett whether he had any theories that might explain the difference. Here is his response:
Our sample size is larger and our likely voter screen is more difficult to pass. As you have pointed out, many surveys (although they are not designed to project participation) project unrealistic levels of participation. A likely voter/participant does not need to vote/participate to represent the pool of likely voters/participants, but the likely voter/participant pool is not much larger than the actual turnout.
Our results in Iowa show that John Edwards has a slight lead over Hillary Clinton among those voters saying they have attended a caucus in the past. Hillary Clinton has a greater lead among those saying this will be their first caucus. Hillary Clinton also has very strong support among women who say they usually do not vote/participate in primary/caucus races - this is true in Iowa and the other early states
Sample size is largely irrelevant to the pattern in our chart. Smaller samples would explain greater variability, but not a consistent difference across a large number of samples. The observation in his second paragraph is much more important. Since ARG's previous releases did not mention these results, I asked for the question about past caucus participation and the associated results. His response:
The question is: Will this be the first Democratic caucus you have attended, or have you attended a Democratic caucus in the past?
We first asked this in Feb:
Feb - 41% first, 59% past
Mar - 44% first, 55% past
Apr - 39% first, 60% past
May - 45% first, 55% past
Jun - 42% first, 57% past
Jul - 40% first, 60% past
Aug - 43% first, 57% past
We can compare this result to similar questions or reports from other recent surveys and they show a clear pattern. The differences among the four pollsters are huge and show a clear pattern, consistent with the differences Bennett reports in his own surveys: John Edwards does better against Clinton as the percentage of past caucus goers increases.
So what is the right number of past caucus goers? Bennett can certainly argue that the entrance polls from the 2000 and 2004 Caucuses are on his side. Bennett used exactly the same question as the network entrance poll, which reported the percentage of first-time Democratic caucus goers as 53% in 2004 and 47% in 2000. Of course, as we learned three years ago, exit polls have their own problems, and I am guessing that other pollsters will debate what past-caucus goer number is correct. We will pursue this point further.
Finally, it is worth saying that this exchange and my arguably unfair "blogger speculation" yesterday makes one thing clear: If we are going to dig deeper into these issues, we have an obligation to ask these questions (about incidence and sample characteristics) about all polls, not just those from ARG, Time and a handful of others.