[And more...] Eric Kleefeld reviews the Nini-Hurwitz analysis to question whether the Florida and Michigan resuls reflected the"will of their states' Democratic voters [via alert Pollster reader RS and Josh Marshall].
By now your favoritepoliticalblog has probably informed you of David Runciman's essay in the London Review of Books in which he reviews the Obama-Clinton race from a British perspective and includes this broadside against American polling:
Yet if the voting patterns have been so predictable, why have the polls been so volatile? One of the amazing things about the business of American politics is that its polling industry is so primitive. Each primary has been preceded by a few wildly varying polls, some picking up big movement for Clinton, some for Obama, each able to feed the narrative of a contest that could swing decisively at any moment. All of these polls come with warnings about their margins of error (usually +/–4 per cent), but often they have been so far outside their own margins as to make the phrase ridiculous. A day before the California primary in February, the Zogby organisation had Obama ahead by 6 per cent – he ended up losing by 9 per cent. In Ohio, the same firm put Obama ahead by 2 per cent just before the actual vote – this time he lost by 10 per cent. The sampling of national opinion is even worse. Before the Indiana primary, two national polls released at the same time claimed to track the fallout from the appearance of Obama’s former pastor Jeremiah Wright on the political stage. One, for the New York Times, had Obama up by 14 per cent, and enabled the Times to run a story saying that the candidate had been undamaged. The other, for USA Today, had Clinton up by 7 per cent, leading the paper to conclude that Obama was paying a heavy price.
The reason for the differences is not hard to find. American polling organisations tend to rely on relatively small samples (certainly judged by British standards) for their results, often somewhere between 500 and 700 likely voters, compared to the more usual 1000-2000-plus for British national polls. The recent New York Times poll that gave Obama a 12 per cent lead was based on interviews with just 283 people. For a country the size of the United States, this is the equivalent to stopping a few people at random in the street, or throwing darts at a board. Given that American political life is generally so cut-throat, you might think there was room for a polling organisation that sought a competitive advantage by using the sort of sample sizes that produce relatively accurate results. Why on earth does anyone pay for this rubbish?
The polling misfires of the 2008 primary season are certainly a fair target for criticism and debate, but Runciman's diagnosis of the problem is both misleading and flawed.
First, Runciman does not compare "apples to apples," as the British polling blogger Anthony Wells puts it:
American polls normally quote as their sample size the number of likely voters, it is typical to see a poll reported as being amongst 600 “likely voters”, with the number of “unlikely voters” screened out to reach that eventual figures not made clear. In contrast, British polling companies normally quote as their sample size the number of interviews they conducted, regardless of whether those people were filtered out of voting intention questions. So, voting intentions in a UK poll with a quoted sample size of 1000, may actually be based upon 700 or so “likely voters”.
To give a couple of examples, here’s ICM’s latest poll for the Guardian. In the bumpf at the top the sample size is given as 1,008. Scroll down to page 7 though and you’ll find the voting intention figures were based on only 755 people. Here’s Ipsos-MORI’s April poll - the quoted sample size is 1,059, but the number of people involved in calculating their topline voting intention once all the unlikelies have been filtered out was only 582.
Let's also consider a few recent U.S. national polls. This week's Pew Research survey sampled 1,505 adults, 1,242 registered voters and 618 Democratic and Democratic-leaning registered voters. The Gallup Daily tracking survey typically reports on more than 4,000 registered voters and more than 1,200 Democratic and Democratic leaning "voters." Last week's Newsweek survey screened 1,205 registered voters from 1,399 adults, and in the process interviewed 608 "registered Democrats and Democratic leaners." Some pollsters use smaller samples, some bigger, but when it comes to national surveys of general election voters, American surveys are at least as large if not larger than their British counterparts.
Runciman confuses things further by comparing national British surveys to the U.S. polling in low turnout, statewideprimary elections. In 2004, 61% of eligible adults voted in the U.S. presidential election, but during the 2008 primary season the typical turnout -- while higher than usual -- typically ranged from 25% to 35% (including both Republican and Democratic primaires). "The challenge for U.S. pollsters," as Wells puts it,
is filtering out all those people who won’t actually take part. Getting lots of people per se can be a bad thing if those people won’t actually vote, the aim is getting the right people. Considering the rather shaky record of most British pollsters in some low turnout elections like by-elections, Scottish elections, the London mayoralty and so on, we really aren’t the experts on that front.
Setting aside Runciman's fallacious "our polls are bigger than yours" theme, the biggest problem with his overall argument is the assumption that larger samples would solve all problems. If only that were true. Runciman notices that actual election returns in the U.S. primaries have often" been so far outside their own margins as to make the phrase ridiculous." That's right. If the poll has a statistical bias (in sampling or in the way it selects likely voters), doubling or tripling the sample size will not solve the problem. Remember: the "margin of error" only covers the random variation that results from drawing a sample rather than trying to call all voters. It tells us nothing about other potential survey errors.
Here is one obvious example. Take a look at the final round of polls before the Democratic primary in Pennsylvania. Which pollster had the largest sample size? The winner on that score, by far, was Public Policy Polling (PPP) with 2,338 interviews of "likely voters" conducted Sunday and Monday before the election. And which pollster had the biggest error? The same pollster, the one with the biggest sample (and this example may be unfair to PPP -- they had better luck elsewhere this year).
Obama 54, Clinton 41
Obama 47, McCain 44... Clinton 48, McCain 44
"Favorable opinions of Obama among independent voters, who have provided him strong support in several of his primary election victories, also have declined over the course of the campaign. Obama's favorable ratings among this pivotal group have fallen from 62% in late February to just 49% in the current poll."
"Obama has a clear advantage over McCain on several major issues. In particular, voters say the Illinois Democrat could do better in improving economic conditions, dealing with the nation's energy problems, and improving the healthcare system."
"However, more voters continue to say that McCain is about right in his approach to foreign policy and national security issues than say that about Obama (51% vs. 43%). The view that Obama is not tough enough on foreign policy has not receded since earlier in the year."
Ruy Teixeira says Obama is "where he needs to be" with white, working class voters (via Smith).
Peter Hart's Virginia focus group finds much voter ignorance (via Sullivan). [Update: More coverage of the Hart's Charlottesville Virginia focus group here, here, here and here. According to Polman it included "12 independent voters" in who "are not close followers of politics" and did not vote in Virginia's Democratic or Republican primaries).
John Sides reviews historical evidence of partisans unifying during the fall campaign (with sources and more details here).
Jay Cost continues his profile of Obama's primary voting coalition.
Ben Smith reminds reminds us of various poll results showing "out of the mainstream" views held by many Americans.
Markos considers the IVR poll methodology and its critics (via PPP's Jensen).
Today's Guest Pollster's column comes from Robert M. Eisinger, a political science professor at Lewis & Clark College and the author of The Evolution of Presidential Polling (Cambridge U. Press).
The Obama "phenomenon" is a product of many things, most notably a superbly smart candidate and a sharp, disciplined campaign team, both of whom clearly articulated a resonating message that mobilized voters. What we don't know if how many of those supporters galvanized around Senator Obama simply because he is not Senator Clinton. This is not to say that Senator Clinton is without fans. To the contrary - she has many. They are devoted and dedicated, imparting the kind of loyalty that any political candidate would desire.
But as evidenced by the reaction to her recent mentioning of Robert Kennedy's assassination, Senator Clinton appears to be a lightning rod - people are either repelled or attracted to her.
Arguments for an Obama-Clinton dream ticket suggest that Senator Clinton's keen intelligence, legislative acumen and support among Democrats outweigh her negatives. However reasoned this claim is, it is potentially flawed in a critical way currently understudied by the public opinion and political cognoscenti. The problem is that we do not know enough about her positives and negatives, especially among voting Democrats and swing voters.
By itself, the "favorability question" is crude and insufficient indicator of likeability. It does not claim to measure the intensity underlying that favorability or lack of favorability. For example, a May 23, 2008 Newsweek poll asks, "Who would you MOST like to be nominated as the Democratic Party's presidential candidate this year...Hillary Clinton (or) Barack Obama (choices rotated)?" "Do you support (INSERT CHOICE) strongly or only moderately?" The poll then asks, "We'd like your overall opinion of the presidential candidates. As I read each name, please tell me if you have a favorable or unfavorable opinion of this person - or if you have never heard of them before this interview. What about (INSERT - READ AND RANDOMIZE). Do you have a favorable or unfavorable opinion of him (or her)?"
Favorability questions specifically asked about Hillary Clinton in the past have been worded in numerous, thoughtful ways, including the following: 1
CBS News/New York Times: Is your opinion of Hillary Clinton favorable, not favorable, undecided, or haven't you heard enough about Hillary Clinton yet to have an opinion?
Gallup/USA Today/CNN: I'd like some overall opinion of some people in the news. In general, do you have a favorable or unfavorable opinion of Hillary Clinton?
Yankelovich/Time/CNN: Please tell me whether you have generally favorable or generally unfavorable impressions of [Hillary Clinton], or whether or not you are familiar enough with [Hillary Clinton] to say one way or another.
Each of these questions is carefully written, but they do not capture the intensity that may lie beneath the answer. In fact the Yankelovich/Time/CNN question employs the phrases "generally favorable" and "generally unfavorable", allowing the respondent to articulate her overall impression, but in doing so, diffuses the potential passion or force embedded within that answer.
If respondents were asked to place their favorability/un-favorability on a seven point scale, then one might get a better sense of the potential polarizing nature of the answer, or put another way, the strength of that favorability and its opposite. Such an option is costly in that it requires an additional question to be asked, and more nuanced data analysis.
Anecdotal conversations in the blogosphere, in the taxi cab and the around the water cooler - reveal that many citizens - men and women, Democrats, Republicans and Independents - have a palpable and deep disdain for Senator Clinton. Different blog, cab and water cooler discourses tell us that Senator Clinton is revered. Scholars of public opinion and savvy journalists are appropriately suspicious of these unrepresentative remarks. Sure, the plural of anecdote is data, but we wonder if selection bias (i.e., we surround myself with like-minded folk; we listen more carefully only to extreme answers) taints our perspective and analysis.
The Obama campaign should be privately measuring the favorability intensity for all prospective Vice Presidential nominees. If not, then they are avoiding a datum that may be critical to their electoral success. Media polls should explore this question as well; given the dearth of the intensity question, the answers will undoubtedly surprise us. //END
1 From Barry C. Burden and Anthony Mughan, "Public Opinion and Hillary Rodham Clinton," Public Opinion Quarterly, Vol. 63, No. 2 (Summer 1999), 237-250.
"In the 20 states where Hillary Clinton has claimed victory in the 2008 Democratic primary and caucus elections (winning the popular vote), she has led John McCain in Gallup Poll Daily trial heats for the general election over the past two weeks of Gallup Poll Daily tracking by 50% to 43%. In those same states, Barack Obama is about tied with McCain among national registered voters, 45% to 46%.
"In contrast, in the 28 states and the District of Columbia where Obama has won a higher share of the popular vote against Clinton in the 2008 Democratic primaries and caucuses, there is essentially no difference in how Obama and Clinton each fare against McCain. Both Democrats are statistically tied with him for the fall election."
A reader sent the following email over the weekend:
We have consistently seen a flurry of polls ahead of every democratic primary contest -- however, since Oregon and Kentucky polls have all but gone silent... the upcoming states, Puerto Rico, Montana, and South Dakota have been virtually unreported in the public. What do you attribute this to? It seems almost like some polls have always had a second agenda, and with the coordinated silencing of polls at a time when there is also a coordinated push to reshape the media focus on the contest as being over, seems very suspect. Given your excellent nonpartisan analysis of the numbers and meaning behind the polls, I wonder if you could shed some light on the lack of polling at such a crucial time in the nominating process?
"Coordinated silencing?" I don't see it.
The following table shows the number of polls we logged for both the the final week before each primary (the last column on the right) and the full month before the final week (second column from the right). Obviously, there has been only one poll in the last month in any of the three remaining primary states, but that represents a considerable tapering off since the Pennsylvania primary.
Here are four reasons for the fall-off:
1) The number of delegates at stake in each state has also trended down considerably since Pennsylvania, and with it the budget priorities of pollsters and the national news media. Notice that the number of delegates up for grabs in Montana and South Dakota combined (47) is a fraction of those that were at stake in either Pennsylvania (188) or North Carolina (134). Also, national media outlets sponsored just 9 of the 124 polls logged in the table above (all from PA, NC and IN).
2) Puerto Rico is an obvious "outlier" of sorts, given that polling there requires Spanish speaking interviewers and (hopefully) a knowledge of the island's culture with respect to both politics and telephone surveys. Mainland pollsters with a lack of experience in Puerto Rico are understandably reluctant to field surveys there.
3) Many of the polls we have seen for previous primaries are sponsored by local newspapers or television stations. Larger states have more media markets and thus more media outlets and more competition among them. Those factors tend to translate into more polling in bigger states.
4) The remaining polls without media sponsors come mostly from pollsters or public relations firms that release polls for their "marketing" value, to generate traffic to their websites, or both. Many of these outlets are now shifting their resources to general election polling. Many have already spent more than they ever anticipated on primary polls. Most presumably see less of a return (in terms of marketing or traffic) from polls in those final states.
Altogether, the fall-off seems mostly evidence of a shifting market forces, not any coordinated effort.
Today's Guest Pollster article comes from David W. Moore, a senior fellow with the Carsey Institute at the University of New Hampshire. He is a former vice president and senior editor with the Gallup Poll, where he worked for 13 years, and is the founder and former director of the UNH Survey Center. He manages the blogsite, Skeptical Pollster.com.
This month, the American Association for Public Opinion Research (AAPOR) gave its most coveted honor, the AAPOR Lifetime Achievement Award, to Kathleen A. Frankovic, director the CBS News polls and a former AAPOR president. In her acceptance speech, she referred to her presidential address of a decade and a half ago, when she leveled several incisive criticisms at the media polls - criticisms that deserve to be re-examined today.
Since joining CBS News over thirty years ago, Frankovic has amassed an impressive set of accomplishments, including being president of both AAPOR and its sister organization, the World Association for Public Opinion Research (WAPOR); a member of the Market Research Council; a trustee both of The Roper Center for Public Opinion Research and, separately, the National Council on Public Polls; and a former chair of the Research Industry Coalition. She is also the author of numerous published articles and book chapters on public opinion. If anyone could be considered a pillar of the polling industry establishment, she would be it.
Yet, her writings on polls have not been effusive encomiums to the presumed benefits they bring to society. While always attributing much importance to the role of media polls in American politics, she has also expressed concerns about them, nowhere more evident than in her 1993 AAPOR presidential address, "Noise and Clamor: The Unintended Consequences of Success."1
Her theme is reflected in the title, as she raised questions about the increased frequency of polls and the lack of thought that goes into many poll questions - "Immediate response is more important than what the response is or what it really means. In other words, we may no longer have to think." She also worried about the decreased value and import of polls - "It's so easy to conduct polls now that it may actually cheapen the value of each one we do. Instead of meaning, we may just be getting noise - noise and clamor."
She noted that with the advent of scientific polls, we now have a "continuous ballot box," the dream of early democratic idealists. But is that good? Not always, apparently. In the several months prior to her presidential address, polls taken at two-week intervals had showed President Bill Clinton's approval rating bouncing all over the map, "from 58 percent to 53 percent to 59 percent to 53 percent to 57 percent to 49 percent to 57 percent to 45 percent." She lamented, "This is information, but how informative is it? It's almost like what Truman Capote once remarked about Jack Kerouac's novel, On The Road: 'That isn't writing-it's typing.' Continuing ballot boxes shouldn't bounce around so much." Indeed.
Today, the uninformative information provided by polls is even more acute, obvious to anyone who has followed the pollsters' fascination with the 2008 national Democratic electorate. It isn't the results at two-week intervals, but contemporaneous results that bounce all over the place these days. One has to look only at pollster.com from April 30 to May 4 to find five polls with three different results: Gallup by itself, reporting a dead heat (Obama up by two points); Gallup with USA Today and, separately, AP/Ipsos each reporting Clinton leading by 7 points; while CBS/NYT and Diageo/Hotline each reporting double digit leads (12 and 11 points respectively) for Obama. And this isn't the only time Gallup has contradicted itself this campaign season, or that different polling organizations have come up with contrasting results when interviewing in the same time period. (See comments by ABC's Gary Langer, Dec. 12, 2007 and Feb. 26, 2008; and Mark Blumenthal's "Dueling Gallups.")
Despite her criticisms, Frankovic proposed no remedies, nor special panels to investigate the problems, perhaps in recognition that they might entail a fundamental change in the way that polls are currently conducted. In the wake of the miscalls in the New Hampshire Democratic Primary, AAPOR's president, Nancy Mathieowetz, did in fact establish a special panel "to examine what occurred, provide a timely report of our findings, and promote future research on pre-election primary polls." No such panel has been established to examine all the subsequent conflicting polls, though the New Hampshire panel might want to consider broadening its scope. The "continuing ballot boxes" are not just bouncing around, they're running into each other going in opposite directions.
Frankovic concluded in her presidential address in 1993 that "We have achieved the ability to cut through the noise and clamor of unscientific measures, even as we risk making some noise and clamor of our own." This observation suggests that the radical question the AAPOR panel needs to address is whether the noise and clamor of "scientific" polls is any better than that of the unscientific ones.
1 Kathleen A. Frankovic, Presidential Address "Noise and Clamor: The Unintended Consequences of Success," Public Opinion Quarterly, Vol. 57, No. 3 (Autumn, 1993), pp. 441-447.
Which candidate is winning "the popular vote" in the race for the Democratic nomination? Back in April, I devoted twocolumns to the fuzziness inherent in summing up primary and caucus vote counts but, admittedly, struggled to reduce the problem to a simple thesis. This week in The New Yorker, Hendrik Hertzberg does in just under 300 words, what I struggled to do with many more (link added):
In a nominating process, especially this one, the “popular vote” is an elusive phenomenon. RealClearPolitics.com, an independent Web site whose numbers political reporters and operatives tend to trust, maintains six separate tallies. At the moment, Obama leads in four of them. With or without participants in the caucus states of Iowa, Nevada, Maine, and Washington (i.e., states where voters’ preferences were expressed by gathering in corners and the like, and whose numbers can be estimated but are not pinpointed), and with the totals for both Florida (whose primary was unsanctioned by the Democratic Party, with the consent of all the candidates, and where no one campaigned) and Michigan (also unsanctioned, and where Obama’s name was not even on the ballot), Clinton’s claim that more people have “voted” for her is factual. But her claim to be “ahead” depends entirely on a tally for the Michigan primary that is distinctly North Korean: Clinton, 328,309; Obama, 0. However, if the bulk of the 238,168 Michiganders who voted “uncommitted” are assumed to have been Obama supporters—a reasonable assumption—then Obama leads by every possible reckoning. And if only Florida is included, then Obama leads whether or not those four caucuses are counted.
Next week, after the three remaining primaries—Clinton is expected to sweep the largest of them, Puerto Rico’s—the likelihood is that each candidate will be able to point to “metrics” showing that he or she is the people’s choice. Obama will almost certainly have the better case, especially in view of opinion polls showing that his national lead among Democrats has been growing, but the reality is that the two have been almost equally strong. Obama will remain the leader in the delegate count, owing largely to a more astute strategy, and he will be the nominee.
Read the full column for Hertzerg's thoughts on Clinton's claim's regarding the "popular vote" and the "loftier lessons" of it all (via: Greg Sargent/TPM).
"A new Gallup Panel survey, conducted May 19-21, finds 61% of Democrats saying they are confident their party will win the election, including 35% who are "very confident." Meanwhile, only 39% of Republicans are confident, with only 13% saying they are very confident.
Notably, Democrats who prefer Obama for the nomination are much more confident in the party's chances of winning the November presidential election than are Clinton supporters, 70% to 49%. Also, as would be expected, a majority of Clinton supporters believe Clinton would give the party the best chance of winning in November; only 22% of Clinton supporters believe Obama gives the Democrats the best chance of victory."