Pollster.com

Articles and Analysis

 

Iowa: A Tale of Two New Polls

Topics: Disclosure , Iowa , Likely Voters , The 2004 Race

So today we have another installment in that pollster's nightmare known as the Iowa caucuses: Two new polls of "likely Democratic caucus goers" conducted over the last ten days that show very different results. The American Research Group (ARG) survey (conducted 8/26-29, n=600) shows Hillary Clinton (with 28%) leading Barack Obama (23%) and John Edwards (20%). And a new survey from Time/SRBI (conducted 8/22-26, n=519, Time story, SRBI results) shows essentially the opposite, Edwards (with 29%) leading Clinton (24%) and Obama (22%).

Is one result more trustworthy than the other? That is always a tough question to answer, but one of these polls is considerably more transparent about its methods. And that should tell us something.

While I have been opining lately about both the difficulty in polling the Iowa Caucuses and the remarkable lack of disclosure of methodology in the early states (especially here and here and all the posts here), the new Time survey stands out as a model of transparency:

The sample source was a list of registered Democratic and Independent voters in Iowa provided by Voter Contact Services. These registered voters were screened to determine their likelihood of attending the 2008 Iowa Democratic caucuses.

Likely voters included in the sample included those who said they were

  • 100% certain that they would attend the Iowa caucuses, OR
  • probably going to attend and reported that they had attended a previous Iowa caucus.

The margin of error for the entire sample is approximately +/- 5 percentage points. The margin of error is higher for subgroups. Surveys are subject to other error sources as well, including sampling coverage error, recording error, and respondent error.

Data were weighted to approximate the 2004 Iowa Democratic Caucus "Entrance Polls," conducted January 19, 2004.

Turnout in primary elections and caucuses tends to be low, with polls at this early stage generally overestimating attendance.

The sample included cell phone numbers, which, to the extent SRBI was able to identify them, were dialed manually.

I emailed Schulman to ask about the incidence and he quickly replied with a "back of the envelope" calculation: Their sample of 519 likely caucus goers represents roughly 12% of eligible adults in Iowa (details on the jump), exactly the same percentage as obtained by the recent ABC News/Washington Post poll, but higher than the reported 2004 Democratic caucus turnout (5.5% of eligible adults). Keep in mind, however, that the ABC/Post poll used a random digit dial methodology and screened from the population of all Iowa adults.

The Time/SRBI survey started with a list of registered Democrats and independents - so theoretically did a better job screening out non-registrants and Republicans. On the Time survey, 92% of respondents report having "ever attended" Iowa precinct caucuses (see Q2)." On the Post/ABC survey, 68% report having "attended any previous Iowa caucuses" (see Q12). Readers will notice that on the 2004 entrance poll, 55% of the caucus-goers said they had participated before.

What is the American Research Group Methodology? All they tell us on the website is that they completed 600 interviews and that respondents were asked:

Would you say that you definitely plan to participate in the 2008 Democratic presidential caucus, that you might participate in the 2008 Democratic presidential caucus, or that you will probably not participate in the 2008 Democratic presidential caucus?

Blogger speculation alert: If this was the only question used to screen, it is likely that ARG's incidence of eligible adults was much higher. Such a difference likely explains why they show Clinton doing consistently better in Iowa than other pollsters, but that is just an educated guess. [Update: A guess that turns out to be wrong....]. We owe Dick Bennett the opportunity to respond with more details. I have emailed him with questions and will post a response when I get it. [Update: Details of Bennett's response here. They ask four questions to screen for likely voters and their Democratic sample in this case represented roughly 12% of adults in Iowa. Apologies to ARG].

I suspect that if we could know all about every pollsters' methods in Iowa, we would see evidence of a disagreement about how tightly to screen and about what percentage of the completed sample should report having participated in a prior caucus.

The resolution of that argument is neither simple nor obvious, but seems to have a profound impact on the results. Surveys that appear to include more past caucus goers (Time, Des Moines Register and One Campaign survey -- see our Iowa compilation) tend to favor John Edwards, while Hillary Clinton does better on surveys that define the likely caucus-goer universe more broadly. [Update: The disagreement may have more to do with the appropriate number of self-reported past caucus goers].

Details on Time's "back of the envelope" incidence calculation after the jump...

  • According to the Iowa Secretary of State, Iowa has 596,602 registered Democrats and 727,480 with no party affiliation (596,602 + 727,480 = 1,324,082). That is the universe initially sampled by Time/SRBI.
  • 1,324,082 divided by 2,153,200 eligible adults = 61% of adults in the population sampled by Time/SRBI.
  • Mark Schulman reports that 20% of the sampled adults identified as "likely caucus goers" on their screen questions
  • So, Time's sample of likely Democratic caucus goers projects to 12% of the eligible adults in Iowa (61% * 20% = 12%).

 

Comments
hwc:

It seems fairly obvious why screening for previous caucus voters would favor the one candidate in the field who was also a significant player in the last Iowa caucus.

32% of the votes in the last Iowa caucus were from Edwards supporters. No votes were cast for Clinton, Obama, Richardson, Biden, or Dodd in the most recent caucus.

This is further compounded by the signficant support for the lefty netroots Howard Dean in the most recent caucus, another group that is atypical in their support of Edwards (see DKos staw polling).

The final issue that is going to plague pollsters in this cycle is adjusting polls to reflect historical turnout by women voters. This adjustment will nullify increased turnout of women (if any) by the first legitimate female candidate for President. It seems illogical to predict that Clinton's candidacy will have no impact on womens turnout, which is the assumption the Time polling is making.

____________________

mop:

Of course, on the other hand hwc, Time and the others are already screening far too widely. EVen if turnout as a % of eligible adults doubled, they'd still have a voter screen that is too inclusive. So the fact that ARG is going even more inclusive than Time is no virtue. Simply put, it's ridiculous to expect a turn out of 15-20% of eligible adults, and any survey defining "likely caucus-goers" in a manner that gives you those numbers is going to sampling from a crowd that is too big and, as such, most likely non-representative.

____________________

zogby:

zogby,

Can you explain the metholody of Zogby IA poll. Thanks.

____________________

OEST:

There is no tale of two polls.

ARG polls are a joke. They are inconsistent, even with themselves. The results are all over the place for every candidate buy Clinton.

Even when ever other poll showed John Edwards winning by like 10 in Iowa, ARG still showed Clinton winning their every time.

ARG is a fluke.

____________________

Matt:

Based on this, it would be very interesting to see what your Iowa (and NH) trendlines would look like if all the data from ARG were excluded. Even better if the trends with and without ARG could be compared side by side. Any chance of this happening, just as an experiment?

____________________

Nixxer:

ARG is weird. They've always shown Clinton winning Iowa by a comfortable margin.

Someone stated that women turn out may increase a bit because of Hillary but i think you could make a similar case with Obama and his youth movement.

Obama should work hard to maximize turn out at every campuses across Iowa.

____________________

hrbob:

SRBI hosed you, Mark. Look at Q2 at http://www.srbi.com/TimePoll4197-Final%20Report-2007-08-27-2.10pm.pdf 92% of their sample has attended a caucus.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR