March 29, 2009 - April 4, 2009


US: Bush, Cheney Favs (USAToday-3/27-29)

USA Today / Gallup
3/27-29/09; 1,007 adults, 3% margin of error
Mode: Live Telephone Interviews


Favorable / Unfavorable
Bush: 35 / 63 (Jan 09: 40 / 59)
Cheney: 30 / 63 (July 07: 30 / 62)

"Bush and Cheney have only begun to emerge publicly after departing Washington, D.C., in January. Thus far, Bush has promised to stay out of the political fray while Cheney has stepped squarely back into it. If this distinction holds over time, it could conceivably contribute to a divergence in how Americans perceive the two -- with Bush's personal image among Democrats softening while views toward Cheney become even more polarized. Of course, other factors will come into play as Bush and Cheney establish their post-presidential roles in public affairs, and as the Bush presidency continues to be assessed. At this early juncture, however, public attitudes about the two former leaders have not yet changed."


Profiling Obama's Polling Operation

Ben Smith has a must-read piece into today's Politico that provides new details on President Obama's polling operation. While recent reports have included passing references to Obama's pollsters, Smith nails down who is doing what:

Data from pollsters Joel Benenson and Paul Harstad has become increasingly important to shaping the White House’s message as the crucial battle over the president’s budget intensifies.

“The pace [of polling] is picking up,” said one source familiar with the data.

In addition, David Binder, a San Francisco-based focus group expert, also has been traveling the country taking the national temperature on issues like energy and health care, others close to the White House said.


A political aide, Larry Grisolano, confirmed the outlines of the White House polling operation, which is paid for through the Democratic National Committee.

“Harstad and Benenson poll for the DNC, which shares data with some folks in the admin[istration], as has been the practice in past administrations,” he said in an email.

Smith's piece describes the Obama polling operation so far as falling somewhere between the practices of the Clinton, whose pollster Mark Penn polled once a week, and the Bush, who cut back to just six national polls a year and depended more on analysis of "the growing pile of publicly available data" Also, where Bill Clinton "studied Penn’s polls with 'hypnotic intensity,'" Obama "leaves political guru David Axelrod to sift through the results."

The article also fleshes out the profile of Joel Benenson, the "first among equals" among Obama's team of low profile pollsters during the fall campaign:

“It became clear over the course of the campaign that there needed to be one person who knew the polling inside and out who’d be able to brief Barack and be able to be a part of debate prep, and that was Joel,” said another former campaign adviser.

A bearded, combative former newspaper reporter, Benenson and Axelrod, also an ex-journalist, have a warm rapport-- one former aide described them as a “vaudeville act” together. And Benenson has an unusual resume: Before becoming a reporter for the New York Daily News, he was a beer distributor. Politico's Roger Simon described him, amid the chaotic 1977 New York blackout, seated in front of his shop with a 12-gauge shotgun. He jumped from the Daily News to New York Governor Mario Cuomo’s 1994 reelection campaign, then went to work for Penn on Clinton’s 1996 race.

The full article has much more -- it's worth reading in full.

SD: 2010 Senate (DailyKos-3/30-4/1)

DailyKos.com (D) / Research 2000
3/30 - 4/1/09; 600 likely voters, 4% margin of error
Mode: Live Telephone Interviews

South Dakota

Favorable / Unfavorable
Sen. John Thune (R): 57 / 32
Tom Daschle (D): 50 / 43
Rep. Stephanie Herseth Sandlin (D): 53 / 31
Pres. Barack Obama (D): 47 / 45

2010 Senate - General Election
Thune 53, Daschle 40
Thune 51, Herseth Sandlin 39


US: National Survey (FOX-3/31-4/1)

FOX News / Opinion Dynamics
3/31-4/1/09; 900 registered voters, 3% margin of error
Mode: Live Telephone Interviews


Obama Job Approval
58% Approve, 32% Disapprove (chart)
Dems: 90 / 5 (chart)
inds: 52 / 35 (chart)
Reps: 23 / 66 (chart)

Congressional Job Approval
38% Approve, 52% Disapprove (chart)

Favorable / Unfavorable
Obama: 62 / 32 (chart)
Limbaugh: 31 / 52
Biden: 51 / 35
Palin: 45 / 45

When do you think the current economic recession will be over?

    5% This year
    25% By the end of next year
    32% In two to three years
    11% In three to four years
    20% More than four years

(story, results)

Robert Groves Nominated to Head Census

Topics: AAPOR , Census , Robert Groves , Sampling

In the survey world, this is very big news (from AP via The Page):

President Barack Obama is tapping Robert M. Groves, a University of Michigan professor who has pushed the use of statistical sampling, to be the next census director.

A Commerce Department official who demanded anonymity said the White House will make the announcement later Thursday.

Groves is an expert in survey methodology and statistics who served as an associate director of the Census Bureau from 1990 to 1992. He and others recommended that the 1990 census be statistically adjusted to make up for an undercount, only to be overruled by then Republican Commerce Secretary Robert Mosbacher, who called it political tampering.

It is something of an understatement to describe Robert Groves as "an expert in survey methodology." He is one of our nations' most respected survey methodologists and arguably the leading authority on the subject of non-response in surveys. He has served as the president of the American Association for Public Opinion Research (AAPOR), and won many of its awards including the career award for exceptionally distinguished achievement. Interests disclosed: I had the good fortune to study under Groves in classes I took at the University of Maryland's Joint Program in Survey Methodology (JPSM), a program that Groves helped found.

As the first three three paragraphs of the AP make clear, some see nothing but "political tampering" in any reference to "statistical sampling" regarding the census. For those tempted to label Groves as the pawn of partisans in the White House or the Democratic party, I have a warning: The notion of Bob Groves yielding to partisanship is laughable. As in rolling on the floor laughing out loud laughable. Groves is well known and universally respected among survey researchers and Census Bureau professionals alike. He is an ideal choice for this appointment.

I conducted the interview with Groves below, on the topic of non-response, at last year's AAPOR conference. The Bob Groves in this interview is the scientist and professional his students and peers know well:

Update:  AP has updated their story with initial reactions to the appointment (via @AAPOR).  This didn't take long:

House Republicans quickly expressed dismay Thursday over the selection of Groves, saying Obama's choice raised serious questions about an "ulterior political agenda."

"The fight to protect the accuracy and independence of the 2010 census has just begun," said Rep. Patrick McHenry of North Carolina, the top Republican on a House subcommittee overseeing the census. "President Obama has made clear that he intends to employ the political manipulation of census data for partisan gain."

Also, the Washington Post has more background on Groves and the sampling controversy, including this:

Groves served as the bureau's associate director from 1990 to 1992 and currently is director of the university's Survey Research Center. He has researched why people participate in statistical surveys, worked to develop surveys with lower non-response errors and studied how data is collected for surveys.

A congressional aide familiar with Census matters said Groves has "bulletproof scientific credentials" and is "really highly regarded by his peers as a low-key, determined guy who's been really focused on reducing error in survey research for his whole career."

Update 2Time's Amy Sullivan, who kindly linked to this post, reviewed the census sampling controversy when it came up in connection with Judd Gregg's withdrawl as Commerce Secretary nominee. 

Who Knew?

Topics: Barack Obama , Behavioral Scientists

We've blogged some about the behind the scenes work done by Democratic campaigns in 2008, particularly the creation of the Analyst Institute and its applications of the experimental work of Yale's Alan Gerber and Donald Green on the effectiveness of campaign techniques. But this report from Time's Michael Grunwald (via Mike Allen via Ben Smith) adds a whole new level of intrigue:

SCOOP/THE BIG IDEA - Michael Grunwald of Time, "How Obama Is Using The Science of Change: It's more than a campaign slogan. Inside the White House's plan to employ behavioral economics to promote its agenda--and fundamentally alter the way Americans live: Two weeks before election Day, Barack Obama's campaign was mobilizing millions of supporters; it was a bit late to start rewriting get-out-the-vote (GOTV) scripts. 'BUT, BUT, BUT,' field director Mike Moffo wrote to Obama's GOTV operatives nationwide, 'What if I told you a world-famous team of genius scientists, psychologists and economists wrote down the best techniques for GOTV scripting?!?! Would you be interested in at least taking a look? Of course you would!!' Moffo then passed along guidelines and a sample script from the Consortium of Behavioral Scientists, a secret advisory group of 29 of the nation's leading behaviorists. The key guideline was a simple message: 'A Record Turnout Is Expected.' That's because studies by psychologist Robert Cialdini and other group members had found that the most powerful motivator for hotel guests to reuse towels, national-park visitors to stay on marked trails and citizens to vote is the suggestion that everyone is doing it. 'People want to do what they think others will do,' says Cialdini, author of the best seller 'Influence.' 'The Obama campaign really got that.'

"The existence of this behavioral dream team--which also included best-selling authors Dan Ariely of MIT (Predictably Irrational) and Richard Thaler and Cass Sunstein of the University of Chicago (Nudge) as well as Nobel laureate Daniel Kahneman of Princeton--has never been publicly disclosed, even though its members gave Obama white papers on messaging, fundraising and rumor control as well as voter mobilization. All their proposals--among them the famous online fundraising lotteries that gave small donors a chance to win face time with Obama--came with footnotes to peer-reviewed academic research. ... President Obama is still relying on behavioral science. But now his Administration is using it to try to transform the country. Because when you know what makes people tick, it's a lot easier to help them change.

The research in question sounds like less experiments directly geared toward politics (the sort of thing Gerber and Green do) than advice offered from a group of economists and behavioral scientists grounded in the work they have done in other areas. Still, very interesting and the first I've heard of it.

Thoughts anyone?

Update:  One who knows tells me this project was about more than offering advice.  The "dream team" helped design experiments conducted by the Obama campaign that ultimately guided various tactics, including the most effective media used to reach younger voters.  

Update 2: Grunwald's article is now posted online.

CT: 2010 Senate (Quinnipiac-3/26-31)

Quinnipiac University
3/26-31/09; 1,181 registered voters, 2.9% margin of error
Mode: Live Telephone Interviews


Job Approval / Disapproval
Pres. Barack Obama (D): 71 / 22
Gov. Jodi Rell (R): 72 / 22 (chart)
Sen. Joe Lieberman (i): 46 / 47 (chart)
Sen. Chris Dodd (D): 33 / 58 (chart)

Favorable / Unfavorable
Sen. Chris Dodd (D): 30 / 58
Rob Simmons (R): 39 / 12
Sam Cligiuri (R): 8 / 3
Tom Foley (R): 12 / 6

2010 Senate - General Election
Simmons 50%, Dodd 34%
Caligiuri 41%, Dodd 37%
Foley 43%, Dodd 35%

View all data for the 2010 Connecticut Senate Race here.


US: Obama Approval (USATOday-3/27-29)

USA Today / Gallup
3/27-29/09; 1,007 adults, 3% margin of error
Mode: Live Telephone Interviews


Obama Job Approval
64% Approve, 30% Disapprove (chart)
Economy: 56% / 39% (chart)
Foreign Affairs: 61% / 28%


April Fools "Outliers"*

Topics: Outliers Feature

*Guarantee: Only one link to a lame April Fools gag. It shouldn't be hard to spot:

David Hill praises the AAPOR NH/primary polling report.

Eric Kleefeld asks Scott Rasmussen to explain a poll question about a threat that doesn't exist; Steve Benen has more.

Greg Sargent finds something nice Republicans will say about Obama.

Gary Langer reminds that Americans were ready to throw Wagoner under the bus.

Jennifer Agiesta explores Michelle Obama's formidable favorables.

Chris Cillizza parses recent polls on the role of government.

Matt Gottleib reviews the Hotline-Diageo results on rebounding optimism.

Nate Silver assesses how presidential approval ratings predict off-year election results.

Democrats hint at a "model" of absentee voting patterns in NY-20.

Andrew Gelman calculates the odds of a tied U.S. House race.

Mark Mellman likes the Republican leadership but (re: NY-20) forgets Al Gore's "little known third category."

We ask: is Domenico Montanaro the new Nate Silver?

Lori Weigel reveals some true polling baloney.

Lee Sigelman shares results from the government's massive survey asking the federal workforce to rate their own agencies.

Sarah Palin's PAC denies polling in Iowa and New Hampshire; Geoffrey Dunn finds the real sponsor.

The Boston Public Health Commission conducts an "informal poll" of Boston teens on dating violence, Rihanna and Chris Brown (via Boston Globe via Patrick Sauer).

The New York Times sweeps the Malofiej infographic awards (via @alexlundry).

Pollster.com contributor Kristen Soltis debuts the The Right Idea, a video podcast from the Winston Group.

US: National Survey (Quinnipiac-3/24-30)

Quinnipiac University
3/24-20/09; 2,326 registered voters, 2% margin of error
Mode: Live Telephone Interviews


Obama Job Approval
58% Approve, 25% Disapprove (chart)
Dems: 90 / 4 (chart)
inds: 56 / 32 (chart)
Reps: 25 / 60 (chart)
Economy: 55 / 37 (chart)

Favorable / Unfavorable
Obama: 59 / 26 (chart)

State of the Country
30% Satisfied, 68% Dissatisfied (chart)
Economy: 5% Excellent/Good, 95% Not so good/Poor (chart)
Economy: 19% Getting Better, 32% Getting Worse (chart)


NY-20 Vote: 50-50 with Absentee Out

Ny20Votes.pngWith 100% of precincts reporting the NY-20 special election has come down to Scott Murphy (D) 77,344 to James Tedisco (R) 77,279, a 65 vote margin. Call the Minnesota Senate Case lawyers! TPM reports some 5907 absentee votes remain outstanding so that will surely be interesting to count.

The district is made up of 10 counties or parts of counties.  The chart above shows how the vote in those counties have shifted from 2002 to 2009. The Republican seat shifted a bit Dem in 2004 and then was swept over to the Democratic column in 2006 with 53.1% of the vote. In 2008 that margin (aided by incumbency) expanded to 62.1%.  Without an incumbent it is no surprise we saw a closer election tonight, but would it be more like the 53% of 2006 or fall short of that, perhaps even back to a Republican majority? As it happens, back to 50-50.

Tedisco was helped by a strong performance in Saratoga county, which largely overlaps his state Assembly district and is also the largest turnout county in the district, about 1/3 of the district. That was a county that supported Gillibrand in her win in 2006 and landslide in 2008, in both cases at about the same rate as the district-wide vote. Such was not the case tonight. Saratoga went for Tedisco over Murphy by 30,247 to 25,837. It was almost enough.

 It is revealing to compare the 2009 and 2006 election results, since 2006 represents a close, 53.1%, Dem win. Murphy had just over 3% to give. 
Murphy lost just over 6% from Gillibrand's 2006 total in Saratoga county, Tedisco's best gain.

Other counties shifted towards the Republican candidate by 3 points, more or less, with Renssalear moving the most, almost 5 points pro Rep.  Dem gains were more limited and in smaller counties, Washington, Warren, Essex and Delaware. Net effect: Murphy lost all he had to give, save 65 votes. 

How did the polls do? The last Siena poll had the race 47-43 for Murphy (inside the margin of error.) Our trend estimate put it at 47.0-42.4. Both over estimated Murphy's margin, and had nothing to say about the 10% undecided. The simple polling average, however, had Tedisco up 45.6 to 37.6, a larger overstatement of Tedisco's margin. 

If the Minnesota Senate case is any indication, NY-20 may still be waiting for a winner on the 4th of July. Who got the absentee vote out, and who did the best job of filling out the ballot right? Stay tuned. 

Wednesday Update:

Turnout in Tuesday's special election was down but still pretty healthy.
With just over 150,000 votes case, that was down 100k from the close 2006 midterm, a 40% decline. But given typically low turnout in special elections held in the spring, not a bad effort by voters. Note that the results of the past four election don't vary in any important way with turnout. But the key question now is those 5-10 thousand absentee ballots, or 3.3-6.6% of ballots cast on election day. 

US: National Survey (DemCorps-3/25-29)

Democracy Corps (D) /
Greenberg Quinlan Rosner Research (D)
3/25-29/09; 1,000 2008 voters, 3% margin of error
Mode: Live Telephone Interviews


Obama Job Approval
58% Approve, 34% Disapprove (chart)

Favorable / Unfavorable
Obama: 61 / 27 (chart)
Geithner: 32 / 21

State of the Country
40% Right Direction, 50% Wrong Track (chart)

2010 House Ballot *
48% Democratic Candidate, 42% Republican Candidate

(SPLIT A) As you may have heard, President Obama recently released his plan for the
federal budget. From what you have heard about Obama's budget plan, do you favor or oppose

    52% Favor
    40% Oppose

(SPLIT A) If your Representative in Congress voted in favor of President Obama's budget
plan, would it make you more or less likely to support your Representative for reelection it two
years, or does it make no difference?

    30% More likely
    30% Less likely

* Correction: 'Where the incumbent is known to be running, that candidate's name was inserted, otherwise the generic "the Democratic candidate" and "the Republican candidate" were used. In the landline sample, the incumbent names were inserted preceded by party identification. Generic "the Democratic candidate" or "the Republican candidate" inserted for the opposition. For the cell phone sample and open seat districts, both candidates were given as generic.'


US: National Survey (Hotline-3/26-29)

Diageo / Hotline
3/26-29/09; 800 registered voters, 3.5% margin of error
Mode: Live Telephone Interviews


Obama Job Approval
63% Approve, 31% Disapprove (chart)
Dems: 95% / 4% (chart)
inds: 63% / 30% (chart)
Reps: 28% / 64% (chart)

State of the Country
39% Right Direction, 49% Wrong Track (chart)

Favorable / Unfavorable
Obama: 64 / 30 (chart)
Geithner: 37 / 30
Pelosi: 32 / 42
Reid: 14 / 26
Reid: 18 / 31

Has What You Have Seen, Heard Or Read About Bonuses For Employees And Executives Of Financial Services Companies Made You More/Less Likely To Support Gov't Programs To Spend Money To Help Companies In Other Industries?

    11% More likely
    63% Less likely
    15% No difference
    5% Depends on the industry


VA: 2009 Gov (PPP-3/27-29)

Public Policy Polling (D)
3/27-29/09; 740 likely Democratic primary voters, 3.6% margin of error
Mode: IVR


'09 Governor-Democratic Primary

    Brian Moran 22%
    Terry McAuliffe 18%
    Creigh Deeds 15%

All historical data here.


NH: 2010 Senate (ARG-3/27-30)

American Research Group
3/27-30/09; 535 registered voters, 4.2% margin of error
Mode: Live Telephone Interviews

New Hampshire
2010 Senate: Hodes 42%, Sununu 36%

All 2010 New Hampshire Senate data here.


US: National Survey (ABC-Post-3/26-29)

ABC News / Washington Post
3/26-29/09; 1,000 adults, 3% margin of error
Mode: Live Telephone Interviews


Obama Job Approval
66% Approve, 29% Disapprove (chart)
Economy: 60% Approve, 38% Disapprove (chart)

State of the Country
42% Right Direction, 57% Wrong Track (chart)
Economy: 27% Getting Better, 36% Worse, 36% Staying the Same (chart)

How much blame do you think [ITEM] deserve(s) for the country's economic situation - a great deal, a good amount, only some or hardly any?

    Banks: 80% Great or Good Amount / 19% Hardly or Only Some
    Bush: 70% / 29%
    Business: 80% / 19%
    Consumers: 72% / 28%
    Obama: 26% / 72%

(ABC story, results; Post story, results)

Trends and Averages in NY-20


The New York 20th congressional district special election is upon us tomorrow. Many commentators, on both sides, have considered it as a referendum of sorts, on Obama, or Michael Steele, or on Congress (either side) or on the stimulus or the budget or the war on terror. Take your pick.

In my more parochial world of polling analysis I tend to see it as a test of how we think about polling aggregation. Should we focus on estimating the current positions of the candidates based on trends or should we use simple averages of recent polls? RealClearPolitics has always preferred the simple averages, though so far as I can find they haven't posted a NY-20 average. I've always preferred trend estimates because they capture the all important dynamics of the race. The polling in NY-20 this time presents the difference in these two aggregation approaches rather vividly.

The chart shows the five polls we have for the race along with the trends and the simple averages (as diamonds on the election day line to the right.) This is a race in which the trends are both very strong and the polls surprisingly consistent with those trends. The better known political veteran, Jim Tedisco, started out at 50% support to novice Scott Murphy's 29%. That was February 3-4 polling. Since then Tedisco has slipped a little to 43% while Murphy has surged to 47% in the Siena poll done March 25-26. That's a lot of movement for any campaign. This last poll shows Murphy leading for the first time. (The Tedisco campaign claims their internal polls continue to show him in the lead. A Democratic source claims their recent polls show a 2 point Murphy lead. See that part of the story here.)

What I'm interested in is the different conclusions we reach based on simple averages versus trend estimates, and when they may differ from each other.

When a race is stable, with no trend either way, then the trend estimate and the simple average will agree with each other. A "flat" trend is just the average. But when we have sharp trends, as in this race, the two can differ a lot. The simple average of the five available NY-20 polls has Tedisco at 45.6 and Murphy at 37.6. In contrast, the current trend estimate reverses this with Murphy at 47.0 and Tedisco at 42.4. Which should we believe?

What makes the trend persuasive in this case is how consistent the polls have been in showing Murphy's gains and Tedisco's (more moderate) declines. While there are only five polls, the results for both candidates are remarkably close to the trend lines. And the trends aren't affected by dropping any single poll. The NRCC poll done by Public Opinion Strategies in early February is just as close to the trend lines as the DCCC poll done by Benenson in late February- they differ due to the trend, not due to one being out of line with the rest of the data. Likewise the three Seina polls have shown the same close match to the trend line and a steady gain for Murphy and slow decline for Tedisco. It is this consistency across polls that makes the trend compelling.

Imagine if we mixed up the order of the polls. Ignoring order we have Tedisco at 45, 50, 43, 46 and 44. An average of 45.6 +/- about 3. Likewise ignoring order we have Murphy at 41, 29, 47, 34 and 37. An average of 37.6 +/- about 9. If this random order were really what we were seeing, we'd be justified in using the simple averages, and would want to comment that while Tedisco's results are within sampling error, Murphy's are considerably more variable than we'd expect. That would be a story of noisy polls, and the best we could do is the simple average (with due note of the noise level.)

But these really aren't noisy polls. If we calculate the deviation of polls from trends, we get a surprisingly small range: only 3.4 points for Tedisco and 0.8 points for Murphy. That's another way of saying the points are all really close to the trend lines, especially in Murphy's case. That's very different from the conclusion we'd reach based on the hypothetical unordered sequence of the previous paragraph.

When we have sharp trends like these, the simple average will lag behind the current trend because the average is ignoring the order of polls, while the trend uses that order as a central element for understanding where the race stands. The trend estimate is always an estimate of where the race was when the last poll was taken, while the simple average is an estimate across all polls regardless of date. In the case here, it is clear we get very different conclusions between the two methods of aggregating polls.

With so few polls, the linear fit is more reliable than the local regression trend I normally use. For less than about 15 polls the local trend can be affected quite a bit by an outlier or two. But in this case I stretched the point and show the local fits in the chart as well. It is clear that the local trend follows the linear trend quite closely, with a small exception of the Tedisco result for the Benenson/DCCC poll. I'd make nothing of this because there isn't enough data for the local fit to be reliable on it's own. The fact that it isn't far from the linear just confirms the obvious-- the polls tend to follow a straight line trend quite well.

So this leaves us with two aggregations that predict different results tomorrow night. The trend estimates have the race tighter, and have Murphy ahead. The averages reverse that order with a wider margin for Tedisco. It is certainly a stretch to say that Murphy is a lock. There are just five polls, even if the trend is strong. And my colleague Mark Blumenthal posted here last week on the difficulties of polling in special elections. I tend to side with experienced candidates over novices, so I'd give the benefit of the doubt to Tedisco on that score. But the trends are quite consistent and point to at least a close finish and a modest advantage for Murphy. We'll see tomorrow night.

(You can also check the NY-20 race in our usual interactive chart here.)

What Happened in NH? AAPOR's Answer

Topics: AAPOR , New Hampshire

Most political junkies remember two things about last year's New Hampshire primary. First, Hillary Clinton's surprising three point win and the fact that the pollsters were the "biggest losers" as the final round off of pre-election polls had shown Barack Obama surging ahead. A dozen different surveys showed Obama leading by a range of 3 to 13 points, and by roughly six percentage points on our final trend estimate. Fewer remember that polling errors were even bigger in subsequent states and fewer still will recall that the American Association for Public Opinion Research (AAPOR) announced formation of an ad-hoc committee to study and report on the problems of the New Hampshire and other primary polls.

Well today, more than fourteen months after the 2008 New Hampshire primary, the AAPOR Ad Hoc committee has released its full report. While those hoping for an obvious smoking gun will be disappointed, the report represents a massive collection of information that does shed new light on what happened in New Hampshire. The evidence is spotty and frequently hedged -- "definitive tests" were "impossible" -- but AAPOR's investigators identify four factors as contributing to polls having "mistakenly predicted an Obama victory." From the AAPOR committee press release:

  • Given the compressed caucus and primary calendar, polls conducted before the New Hampshire primary may have ended too early to capture late shifts in the electorate's preferences there.
  • Most commercial polling firms conducted interviews on the first or second call, but respondents who required more effort to contact were more likely to support Senator Clinton. Instead of continuing to call their initial samples to reach these hard‐to‐contact people, pollsters typically added new households to the sample, skewing the results toward the opinions of those who were easy to reach on the phone, and who more typically supported Senator Obama.
  • Non‐response patterns, identified by comparing characteristics of the pre‐election samples with the exit poll samples, suggest that some groups who supported Senator Clinton--such as union members and those with less education--were under‐ represented in pre‐election polls, possibly because they were more difficult to reach.
  • Variations in likely voter models could explain some of the estimation problems in individual polls. Application of the Gallup likely larger error than was present in the unadjusted data. The influx of first-time voters may have had adverse effects on likely voter models.

In other words, what happened in New Hampshire wasn't one thing, it was a likely lot of small things, all introducing errors in the same direction. Various methodological challenges or shortcomings that might ordinarily produce offsetting variation in polls instead combined to throw them all off in the same direction. Polling's "perfect storm" did not materialize this past fall, but that label seems more apt for the New Hampshire polling debacle.

The report also produces evidence that rules out a number of prominent theories, among them the so-called "Bradley Effect." The authors claim they saw "no evidence that white respondents over-represented their support for Obama," and thus, no evidence of "latent racism" benefiting Clinton. Fair enough, but they do report evidence of a "social desirability effect" that led respondents to report "significantly greater" support for Obama "when when the interviewer is black than when he or she is white" (although Obama still led by smaller margins among when interviewers were white -- see pp 55-59 of the pdf report).

As should be obvious, this very quick and cursory review just scratches the surface of the information in the 123 page report. There is a story here about the sheer breadth of the information provided. For example, today's release also includes immediate availability through the Roper Archives of full respondent level data provided by CBS News, Gallup/USA Today, Opinion Dymamics/Fox News, the Public Policy Institute of California (PPIC), SurveyUSA, University of New Hampshire/CNN/WMUR for polls conducted in New Hampshire, South Carolina, California and Wisconsin.  [Update: I'm told that a small glitch in the documentation is holding up release of some or all of the Roper data until, hopefully, later today].

But aside from the admirable disclosure by the organizations listed above, there is also a story here about an outrageous lack of disclosure and foot-dragging, including three organizations that "never responded" to AAPOR's requests for information over the last fourteen months: Strategic Vision (for polls conducted in New Hampshire and Wisconsin), Clemson University and Ebony/Jet (for polls conducted in South Carolina).

Stay tuned. I will have more to say later today and in the days that follow on this new report. Meanwhile, please share your thoughts on the report in the comments below.

For further reading, see my first review of the theories for the New Hampshire polling flap, our bibliography of reaction around the web and the rest of our coverage from 2008.

Update: ABC's Gary Langer shares his first impressions including one thought I negelected to include:  "The volunteer AAPOR committee members who produced [the report], led by Prof. Michael Traugott of the University of Michigan, deserve our great thanks."

Interests disclosed:  As a member of AAPOR's Executive Commmittee from May 2006 through May 2008, I voted to create the Ad Hoc committee.  I did not serve on the committee but our Pollster.com colleague Charles Franklin did participate.

Risk and Political Preferences

Anyone who has ever watched Deal or No Deal has noticed that some people are far more willing to take risks than others. Not only does a person's tolerance for risk affect their decisions about whether to open another suitcase on a game show, but it also influences countless daily decisions like what to eat or whether to drive over the speed limit. But what might a person's tolerance for risk have to do with their political views?

This past Fall, I was part of a team that had a module on the 2008 Cooperative Congressional Election Study. To shed some light on the role that risk might have on political preferences, we borrowed a set of questions developed by economists to gauge an individual's tolerance for risk. Essentially, the questions ask individuals the circumstances under which they would be willing to take a new (and equally good) job. The chart below shows the distribution of responses.


Not surprisingly, the results suggest that the American public is quite risk averse. Most Americans were not willing to take a new job even if the potential for increasing their income was greater than the potential income loss. In fact, over half of the respondents would not take a job even if it offered them an even chance of either doubling their income or cutting it by just 20%. These findings are similar to those of other studies that have looked at how risk tolerant (or intolerant) the public is. But what does this have to do with politics? Take a look at the charts below which show the partisan/presidential vote breakdown among each of the four levels of risk tolerance.



How a person feels about risk is related to that person's choice of party as well as their vote choice. In particular, as one becomes less tolerant of risk, they become more likely to affiliate with the Republican Party (and less likely to be a Democrat). The relationship was even stronger when it came to vote choice. The most risk tolerant respondents preferred Obama by more than a 2-to-1 margin. The most risk averse respondents went for McCain by a margin of 6%.This relationship persisted even when I controlled for other factors that tend to influence vote choice.

When you think about it, there are lots of reasons that an individual's tolerance for risk would influence their political views. It makes sense that the most risk averse Americans were less likely than others to get behind the candidate who was viewed as relatively inexperienced. (Risk averse Americans are probably more likely to vote for incumbents as well.) It also seems logical that those who are more averse to risk would be more likely to affiliate with the party that tends to be more conservative.

There are several recent academic papers on this topic (including my own). Yet, despite the growing evidence showing that risk matters for how people think about politics, pollsters rarely include questions that allow us to capture respondents' feelings about risk. Is it time that they started to do so?