Mark Blumenthal | August 30, 2007
Topics: Disclosure , Iowa , Likely Voters , The 2004 Race
So today we have another installment in that pollster's nightmare known as the Iowa caucuses: Two new polls of "likely Democratic caucus goers" conducted over the last ten days that show very different results. The American Research Group (ARG) survey (conducted 8/26-29, n=600) shows Hillary Clinton (with 28%) leading Barack Obama (23%) and John Edwards (20%). And a new survey from Time/SRBI (conducted 8/22-26, n=519, Time story, SRBI results) shows essentially the opposite, Edwards (with 29%) leading Clinton (24%) and Obama (22%).
Is one result more trustworthy than the other? That is always a tough question to answer, but one of these polls is considerably more transparent about its methods. And that should tell us something.
While I have been opining lately about both the difficulty in polling the Iowa Caucuses and the remarkable lack of disclosure of methodology in the early states (especially here and here and all the posts here), the new Time survey stands out as a model of transparency:
The sample source was a list of registered Democratic and Independent voters in Iowa provided by Voter Contact Services. These registered voters were screened to determine their likelihood of attending the 2008 Iowa Democratic caucuses.
Likely voters included in the sample included those who said they were
- 100% certain that they would attend the Iowa caucuses, OR
- probably going to attend and reported that they had attended a previous Iowa caucus.
The margin of error for the entire sample is approximately +/- 5 percentage points. The margin of error is higher for subgroups. Surveys are subject to other error sources as well, including sampling coverage error, recording error, and respondent error.
Data were weighted to approximate the 2004 Iowa Democratic Caucus "Entrance Polls," conducted January 19, 2004.
Turnout in primary elections and caucuses tends to be low, with polls at this early stage generally overestimating attendance.
The sample included cell phone numbers, which, to the extent SRBI was able to identify them, were dialed manually.
I emailed Schulman to ask about the incidence and he quickly replied with a "back of the envelope" calculation: Their sample of 519 likely caucus goers represents roughly 12% of eligible adults in Iowa (details on the jump), exactly the same percentage as obtained by the recent ABC News/Washington Post poll, but higher than the reported 2004 Democratic caucus turnout (5.5% of eligible adults). Keep in mind, however, that the ABC/Post poll used a random digit dial methodology and screened from the population of all Iowa adults.
The Time/SRBI survey started with a list of registered Democrats and independents - so theoretically did a better job screening out non-registrants and Republicans. On the Time survey, 92% of respondents report having "ever attended" Iowa precinct caucuses (see Q2)." On the Post/ABC survey, 68% report having "attended any previous Iowa caucuses" (see Q12). Readers will notice that on the 2004 entrance poll, 55% of the caucus-goers said they had participated before.
What is the American Research Group Methodology? All they tell us on the website is that they completed 600 interviews and that respondents were asked:
Would you say that you definitely plan to participate in the 2008 Democratic presidential caucus, that you might participate in the 2008 Democratic presidential caucus, or that you will probably not participate in the 2008 Democratic presidential caucus?
Blogger speculation alert: If this was the only question used to screen, it is likely that ARG's incidence of eligible adults was much higher. Such a difference likely explains why they show Clinton doing consistently better in Iowa than other pollsters, but that is just an educated guess. [Update: A guess that turns out to be wrong....]. We owe Dick Bennett the opportunity to respond with more details. I have emailed him with questions and will post a response when I get it. [Update: Details of Bennett's response here. They ask four questions to screen for likely voters and their Democratic sample in this case represented roughly 12% of adults in Iowa. Apologies to ARG].
I suspect that if we could know all about every pollsters' methods in Iowa, we would see evidence of a disagreement about how tightly to screen and about what percentage of the completed sample should report having participated in a prior caucus.
The resolution of that argument is neither simple nor obvious, but seems to have a profound impact on the results. Surveys that appear to include more past caucus goers (Time, Des Moines Register and One Campaign survey -- see our Iowa compilation) tend to favor John Edwards, while Hillary Clinton does better on surveys that define the likely caucus-goer universe more broadly. [Update: The disagreement may have more to do with the appropriate number of self-reported past caucus goers].
Details on Time's "back of the envelope" incidence calculation after the jump...
- According to the Iowa Secretary of State, Iowa has 596,602 registered Democrats and 727,480 with no party affiliation (596,602 + 727,480 = 1,324,082). That is the universe initially sampled by Time/SRBI.
- 1,324,082 divided by 2,153,200 eligible adults = 61% of adults in the population sampled by Time/SRBI.
- Mark Schulman reports that 20% of the sampled adults identified as "likely caucus goers" on their screen questions
- So, Time's sample of likely Democratic caucus goers projects to 12% of the eligible adults in Iowa (61% * 20% = 12%).