Mark Blumenthal | September 11, 2006
Topics: Divergent Polls , Likely Voters
My last post involved what appears to be an example of true push polling in tomorrow's Republican Senate primary in Rhode Island. Although I nearly missed it in the chaos of Pollster's debut, two true surveys in that contest also provide a classic story of oddly conflicting poll results. In the contest between incumbent Lincoln Chafee and his challenger Steve Laffey, one poll had Laffey ahead by seventeen percentage points, another conducted at exactly the same time Chafee ahead by fourteen. How can that be? The most probable explanation is the difference in the way each pollster selected their sample of "likely voters."
The first survey was conducted by the Bureau of Government Research and Services at Rhode Island College (RIC) from August 28 to August 30 (release & results). It showed Chafee trailing Laffey, 34% to 51%. The survey sampled 363 likely primary voters and reported margin of sampling error of 5%.
"A few hours later," according to the Political Wire of the Wall Street Journal Online, "the National Republican Senatorial Committee [NRSC], which supports Mr. Chafee, released its own internal poll done by Public Opinion Strategies [POS] showing the incumbent leading 53% to 39%." The POS survey was fielded over exactly the same evenings (August 28-30). It had a sample size of 400 likely voters and a margin of sampling error of 5%.
As should be obvious, random sampling error alone does not explain the difference. The conflicting results, combined with the high profile push on Chafee's behalf by the NRSC, has raised a few eyebrows. While conspiracy theories always seem to find a receptive audience in the Internet, I am dubious, here's why:
The methodology of the Rhode Island College poll is explained in their release, which explains that they drew their sample from "the most recent updated voting lists provided by the Office of the Secretary of State," limiting selections to registered Republicans and unaffiliated voters. According to their questionnaire, they asked respondents if they were "likely to vote in the Democratic Primary, the Republican Primary, or will you wait to vote in the November General Election?" Those who said they would vote in the Republican primary were then defined as likely voters and interviewed.
This one question approach to screening for likely voters is a bit unusual, at least in my experience. Most pre-election polls - especially those in low turnout primaries - typically take a few extra steps to try to narrow the respondent pool given the well documented tendency of respondents to over-report their intent to vote. These steps might include using questions about past vote experience or interest in the campaign to further screen out likely non-voters voters or perhaps using the actual past vote history on the voter file to narrow the sample.
I spoke briefly to Dr. Victor Profughi, the director of the Rhode Island College poll, about their likely voter model. He described it as the "same process we have been using to screen primary voters for a number of years." He also estimated that roughly two thirds of contacted respondents were screened out by their question about primary voting.
If the NRSC or Public Opinion Strategies has released information about their sampling methodology, I have been unable to find it. However, the WSJ Washington Wire item included an "intriguing nugget" that provides a pretty good clue about the discrepancy between the surveys: "Of the 53% of respondents who could actually name the primary election date, 58% support Mr. Chafee compared to 37% who back Mr. Laffey."
So Chafee's lead was wider on the POS poll among those who knew the primary date (+21 points) than among those who did not (+6 points by my calculation). True likely voters tend to be more knowledgeable about elections (one reason the Gallup likely voter model includes similar measures of knowledge). This result suggests that Chafee does better among the most likely of likely voters, a difference that may help explain the gap between the two polls.
Primary elections often present pollsters with a huge challenge because the proportion of eligible voters that turn out is typically very low. The Providence Journal reported over the weekend that "the previous modern GOP record primary turnout came in 1994, when 45,023 voters participated in the gubernatorial primary." That turnout was roughly 9% of the pool of registered voters in 1994. A comparable turnout tomorrow would amount to roughly 13% of the eligible Republicans and unaffiliated registrants.
What this tells us about tomorrow's results is anyone's guess, but I'm sure a lot of pollsters will be watching closely.