Pollster.com

Articles and Analysis

 

Screens & RDD: The ABC/Post Survey

Topics: 2008 , Disclosure , Likely Voters , Sampling , The 2008 Race

It was probably Murphy's Law. Within hours of my posting a review of the sorry state of disclosure of early primary poll methodology, ABC News and The Washington Post released a new survey of likely caucus goers in Iowa that disclosed the two critical pieces of information I had searched for elsewhere. The two ABC News releases posted on the web (on Democratic and Republican caucus results) disclosed both the sample frame and the share of the voting age population represented by each survey. ABC News polling director Gary Langer also devoted his online column last Friday to a defense of his use of the random digit dial (RDD) methodology to sample the Iowa caucuses.

Let's take a closer look.

Langer concluded his column with a note on "likely voter screening," a subject I have been posting on lately. He writes:

Some polls of likely caucus-goers, or likely voters elsewhere, may include lots of people who aren't really likely to vote at all. Drilling down, again, is more difficult and more expensive. But if you're claiming to home in on likely voters, you want to do it seriously. Anyone producing a poll of "likely voters" should be prepared to answer this question: What share of the voting-age population do they represent?

Amen.

The good news is that Langer and ABC News also provided an answer. For the Democratic sample:

This survey was conducted by telephone calls to a random sample of Iowa homes with landline phone service. Adults identified as likely Democratic caucus goers accounted for 12 percent of respondents; with an adult population of 2.2 million in Iowa, that projects to caucus turnout of 260,000.

In 2004, by comparison, just over 122,000 Democrats (5.5% of the voting age population) turned out for the caucuses.

And for the Republicans:

Adults identified as likely Republican caucus-goers accounted for seven percent of respondents; with an adult population of 2.2 million in Iowa, that projects to caucus turnout of 150,000. That's within sight of the highest previous turnout for a Republican caucus, 109,000 in 1988.

The estimated turnout for the 2000 Republican caucuses was lower (approximately 86,000), partly because John McCain focused his campaign on the New Hampshire primary. Thus, Republican turnout amounted to 4% to 5% of the voting age population in the last two contested Iowa caucuses.

So first, let's give credit where it is due. Of the thirteen organizations that have released surveys in Iowa so far this year, only ABC News has published full information about how tightly they screened likely caucus voters.

Having said that, two questions remain: First, is the screen used by the ABC/Washington Post poll screen tight enough? After all, their screen of Democrats projects to "likely voter" population of 260,000, a number more than double both the 2004 turnout (122,000) and the all-time record for Democrats set in 1988 (125,000). The ABC release seems to anticipate that question with the following passage:

A more restrictive likely voter definition, winnowing down to half that turnout, or about what it was in 2004, does not make a statistically significant difference in the estimate -- Edwards, 28 percent; Obama, 27 percent; and Clinton, 23 percent, all within sampling tolerances given the relatively small sample size. The more inclusive definition was used for more reliable subgroup analysis.

The full sample had Obama at 27% and Edwards and Clinton at 26% each. While the release does not specify the "more restrictive" definition they used, The Washington Post's version of the results indicates that exactly half (50%) of the likely Democratic caucus goers indicated that they are "absolutely certain" they will attend.

The Republican release makes essentially the same assertion: "A more restrictive likely voter definition, winnowing down to lower turnout, makes no substantive difference in the results."

So ABC's answer is: We could have used a tighter screen but it would have made no significant difference in the results.

Their decision is reasonable considering that the Des Moines Register poll used essentially the same degree of screening for their first poll of Democrats in 2006, using a list based methodology that nailed the final result in 2004. Also keep in mind that no screen based on self-reports of past behavior or future intent can identify the ultimate electorate with anything close to 100% accuracy. Pollsters know that some respondents will falsely report having voted in the past, and that respondents often provide wildly optimistic reports about their future vote intent that typically bare little resemblance to what they actually do on Election Day. And while we know what turnout has been in the past, we can only guess as to the Iowa Caucus turnout this coming January (or, perhaps even December). The ideal methodology defines turnout a bit more broadly than expected...[Oops, forgot to finish that sentence: An ideal method defines turnout a bit too broadly but also looks at narrower narrower turnout groups within the sample as this survey did].

The second and more complex question involves the ABC/Washington Post to use a random digit dial (RDD) sample frame rather than a sample drawn from a list of registered voters.

Langer makes the classic case for RDD, by pointing out the potential flaws in samples drawn from the list of registered voters provided by the Iowa secretary of state. Roughly 15% of the voters on the Secretary of State's list lack a telephone number and about as many will turn out to be non-working or business numbers (according to data he cites from a Pew Research Center Iowa poll conducted in 2003). Include the traditionally small number of Iowans that may still register to vote (or participate after having been inactive for many years), and we have, he writes, "a lot of noncoverage - certainly enough, potentially, to affect estimates." Langer acknowledges that RDD samples now face their own non-coverage problem due to the growth of cell phone only households (12-15% now lack landline phone service), but concludes that RDD "produces far less noncoverage than in list-based sampling."

True enough. But Langer leaves out some pertinent information. First, campaign pollsters that make use of registered voter lists typically use a vendor that attempts to match the names and the addresses on the list to telephone listings. Two vendors I spoke with today tell me that they are able to use such a process to increase the "match rate" to over 90%, a level that makes Iowa's lists among the best in the nation for polling.

Second - and this is a more complicated issue that really demands another post - the potential value of sampling from a registered voter list is not the ability to call only registered voters with the confidence that "people are reporting their registration accurately." It also allows pollsters to use the rich past vote history data available on the list for individual voters to inform their decisions about which voters to sample and interview. Pollsters can also make use of data providing the precise geographic location, party registration, gender and age of each sampled voter provided on the list to correct for non-response bias.

Finally, the campaign pollsters on the Democratic side that shell out "up to $100,000" to the Iowa Democratic Party for access to the list do not conduct polls that "entirely exclude" first time caucus goers (as Langer suggests). The Iowa party appends past caucus vote history to the full list of registered voters, and pollsters can use the additional data to greatly inform their sample selection methodology (Democrat Mark Mellman gives a hint of how this works here; Mellman's complete procedure probably resembles the methodology proposed by Yale political scientists Donald Green and Alan Gerber here and here).

Ultimately, the decision about what sample frame to use involves a trade-off between the potential for greater coverage error (when using a list) and greater measurement error in identifying true likely voters (when using RDD). The decision between the two is ultimately a judgment call for the pollster. Those of us who have grown comfortable with list samples believe that the increased accuracy in sampling true likely voters offsets the risk of missing those without accurate phone numbers on the lists. But the choice is not obvious. The fact that ABC and the Post have gone in a different direction -- and have disclosed the pertinent details -- will ultimately enrich our understanding of both the poll methodology and the Iowa campaign.

 

Comments
cms:

You guys are ridiculously thorough, professional, and even-handed. I don't post comments here often (this may be my first), but I read you all the time and truly appreciate your perspective. And I'm glad you're on the right team.

Thanks, once again, for making sense of the chaos of polling.

____________________

Andrew:

I have little doubt that turnout will be significantly higher in the '08 than in '04 primaries. A chart was shown here proving that pollsters have conducted many more polls now than at a similar timeframe before the '04 primaries.
These polls show up in the news and blogs, creating interest in the readers' minds.

____________________

debcoop:

I am hoping you will get to the latest Rasmussen poll

http://www.rasmussenreports.com/public_content/politics/democratic_presidential_candidates_have_no_advantage_over_gop_hopefuls_on_iraq_the_economy_or_restoring_the_nation_s_optimism

entitled
Democratic Presidential Candidates Have No Advantage Over GOP Hopefuls on Iraq, the Economy, or Restoring the Nation?s Optimism

I first saw it on MyDD and wrote the following comment. Am I off base...does this format have any credibilily and wouldn't head to head matchups have yielded meaningful data?

MyDD link
http://www.mydd.com/story/2007/8/8/23223/77033#9

Below is my comment on MyDD

The format is screwy and proves nothing (none / 0)

At first I was dismayed, and puzzled....it just did not compute. But then I went to look at the breakdown of the numbers. What can I say? The format is idiotic, idiotic enough to prove that Rasmussen as usual was casting about for a way to engender just such a result.

I tried to copy it but it won't line up enough to display well enough to show why this is sheer idiocy. Below is the link.

It reminds me of the Math comedians , who would come onto Ed Sullivan and in the most laughable fashion show that 43-39 is 11, not 4. They did this by subtracting the smaller numbers from the bigger no matter in what place they were. this shows the same lack of rhyme or reason.

http://www.rasmussenreports.com/public_c ontent/politics/election_2008__1/electio n_2008_best_candidate_to_handle_key_issu es

They list all the candidates, Democratic and Republican, ask people to choose one and only one and then based on no rationale whatever add up the numbers for each candidate to get the total party numbers. The sheer number of candidates alone almost guarentee this kind of result.

What they should have done, but didn't, was to do head to head match ups, candidate to candidate....Dem to Republican. This of course might replicate the general presidential matchups anyway, but at least that would be legitimate and have some basis in prior polling history and procedures..


____________________

DTM:

I understand that one could not reliably draw any conclusions from the relatively small differences resulting from the tighter and looser screens in this one poll. Of course, hopefully we will continue to get this data and we may eventually be able to develop a decent theory about screen effects (in Iowa and elsewhere).

With that caveat in mind, I feel compelled to note that with respect to the Democratic candidates, the tighter screen seemed to further extrapolate the difference between the national polls and Iowa polls: Edwards did significantly better in the looser version of this Iowa poll than he has been doing in national polls and he did even better in the tighter version; it was vice-versa for Clinton; and for Obama, he did a bit better in Iowa than nationally but the delta was not nearly as significant in his case, and the tighter screen apparently made no further difference.

____________________

Two points:

1. If you're interested in more of the issues of working with registration lists (and as a by product, what they tell us about the CPS and exit poll demographics), see a conference paper
I presented at the 2006 American Association of Public Opinion Researchers conference, a revised version is forthcoming in Public Opinion Quarterly. Among the many tidbits in the paper is that 1.6% of Iowa's registred voters with a 2004 vote history were purged from the rolls by the first update to the registration file in 2005. Presumably, these are people who moved or otherwise became ineligible. Follow up research will examine who is purged in more detail.

2. The big unknown is if the extremely high interest in the election will persist or is an artifact of the accelerated calendar and will revert back to typical levels. I would also not be surprised if interest is higher among Democrats than Republicans. This all is relevant to how loose the screen really should be.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR