National

Favorable / Unfavorable
Hillary Clinton: 56 / 37 (chart)
Nancy Pelosi: 27 / 49
Sarah Palin: 37 / 53 (chart)
Joe Biden: 45 / 41
Harry Reid: 20 / 40

Obama Job Approval
50% Approve, 44% Disapprove (chart)
Dems: 79 / 15 (chart)
Reps: 13 / 84 (chart)
Inds: 44 / 47 (chart)
Economy: 46 / 47 (chart)
Health Care: 46 / 46 (chart)

Congressional Job Approval
16% Approve, 58% Disapprove (chart)

2010 House: General Election
46% Democrat, 37% Republican (chart)

State of the Country
35% Right Direction, 52% Wrong Track (chart)

In February, Congress passed President Obama's $787 billion economic stimulus bill. Do
you think the stimulus bill is working?
27% Yes, 48% No

Overall, given what you know about them, do you support or oppose the proposed changes
to the health care system being developed by Congress and the Obama Administration?
51% Favor, 49% Oppose (chart)

Do you favor or oppose having a "public option" which would allow individuals to purchase
health insurance coverage from the government?
45% Favor, 31% Oppose

Do you favor increasing the number of troops in Afghanistan?
41% Increase, 42% Decrease, 17% Keep the same

National

Favorable / Unfavorable
Barack Obama: 55 / 38 (chart)
Nancy Pelosi: 38 / 53
Harry Reid: 33 / 56
Mitch McConnell: 16 / 66
John Boehner: 15 / 63
Democratic Party: 42 / 50
Republican Party: 23 / 66

State of the Country
41% Right Direction, 55% Wrong Track (chart)

National

In your view, is Barack Obama doing enough to cooperate with the Republicans in Congress, or
not?
49% Yes, 49% No

In your view, are the Republicans in Congress doing enough to cooperate with the Barack Obama,
or not?
31% Yes, 67% No

From everything you have heard or read so far, do you favor or oppose Barack Obama's plan to reform health care?
45% Favor, 53% Oppose (chart)

As you may know, several health care bills have been passed by committees in the U.S. House and Senate and they can be brought before Congress for debate and a final vote at any time. Which of the following do you think Congress should do:
26% Continue working on those bills this fall and make relatively minor changes before passing final legislation
33% Continue working on those bills this fall but pass final legislation only if major changes are made
24% Start work on entirely new bills that would not be ready until some time next year
15% Stop working on any bills that would change the country's health care system

Now thinking specifically about the health insurance plans available to most Americans, would you favor or oppose creating a public health insurance option administered by the federal government that would compete with plans offered by private health insurance companies?
55% Favor, 44% Oppose

New Jersey. Our final trend estimate based on all pre-election polls was dead even, with each major party candidate receiving 42.0% of the vote and independent Chris Christie 10.1%. Christie had a one-point lead on the RealClearPolitics average of the last five non-partisan polls (+1.0%), roughly the same margin as using our more "sensitive" trend line (+1.1%).   

The unofficial count, as of this writing, has Christie leading by 4.3% (though as noted yesterday, all of these unofficial results are likely to change slightly as provisional and absentee ballots are counted). So the average polling error in New Jersey was between 3.3% and 4.3% depending on the average. Nate Silver did a compilation of comparable New Jersey polling errors (compared to final averages) on 9 previous elections that ranged from a low of 0.5 to a high of 4.8. So the error yesterday, while higher than average, fell well within recent experience.

At the same time, nearly everyone has noticed that the average of the final polls from three organizations using an automated methodology (sometimes refered to as "interactive voice response" or IVR) had Christie ahead by four percentage points (46% to 42%) -- roughly the same as his unofficial margin -- while the last three live-interviewer telephone polls had Corzine leading by an average of one point (41% to 40%)

As I wrote on Monday night, what makes that gap between automated and live-interviewer polling interesting is that it was not some random fluke on the last few polls, but persisted throughout the campaign to a degree that we did not see in Virginia this year or in most states during the 2008 presidential election. My conclusion was that the consistency in the estimate of Corzine's vote on so many recent polls suggested a looming "incumbent effect," that voters had largely made up their mind on Corzine but that a small but critically important number were still weighing whether to support Christie or Daggett. So, the theory goes, the IVR polls did better by removing the live interviewer and simulating a secret ballot, thus pushing voters harder to make a choice and more accurately recording their true intentions over the phone.

And what happened to Daggett? Our final trend estimate had him at 10%, but he received only 5.8% of the vote. Although it had been rising until mid-October, Daggett's support ultimately followed the traditional pattern. Many voters that had been intrigued by his candidacy ultimately concluded that their votes would be wasted and opted to support either Christie or Corzine. The Fairleigh Dickinson Unversity poll provided a hint of where Daggett's support was heading in an experiment conducted on their last survey: They found that Daggett received just 6% -- the same number he won on election day -- when they only named Corzine and Christie as candidates but accepted Daggett as a volunteered choice. When they offered a three-way-choice that included Daggett, his support jumped to 14%.

Virginia. Republican Bob McDonnell's victory in Virginia was never in doubt during the final weeks of the campaign, so political junkies were less obsessed with the polling numbers, but the polling errors in Virginia were, on average, about the same as in New Jersey. Our final trend estimate had McDonnell ahead by 13.4% (54.7% to 41.0%). The unofficial tally has McDonnell leading by 17.4% (58.7% to 41.3%) so the error, as of this writing, averages 3.7 points on the margin.

In Virginia, the gap between the results of automated and live interviewer polls was not nearly as big or as consistent as in New Jersey. The average of the final automated polls in Virginia conducted by PPP, SurveyUSA and Rasmussen had McDonnell at 56% compared to 54% on the final polls in the last week conducted by five organizations using live interviewers, while both sets of poll gave Democrat Creigh Deeds an average of 41% of the vote. However, the final automated polls by SurveyUSA and PPP along with the live interviewer survey by Virginia Commonwealth University are closest to the final margin (as of this writing).

New York City. Our final trend estimate had Mayor Michael Bloomberg leading Democratic challenger William Thompson by a 14-point margin, (53.1% to 39.0%), but Bloomberg won by less than five (50.6% to 46.0%) so the polling error is large (9 points on the margin) -- roughly the same as the infamous New Hampshire polling debacle).

What happened? Marist pollster Lee Miringoff describes it as a "text book case of pre-election poll analysis:"

It is not unusual in contests between a well-known incumbent (Bloomberg) and a relatively unknown challenger (Thompson) that the incumbent ends up getting pretty much the same number he was attracting in pre-election polls. Undecided voters tend to find the challenger or not vote at all, having already rejected the incumbent.

He refers, of course, to the "incumbent rule," a subject I speculated about at length in 2004, only to see it generally not apply that year, in close races in 2006 or 2008. That said, it does appear to have returned in New Jersey and New York City on Tuesday.

But that apparent reemergence raises an important question: If the rule is no longer a "rule," but rather a phenomenon that occurs only occasionally, how do we know to expect it? Miringoff wrote yesterday that Marist's polls "showed the trend that Democratic voters were 'coming home' to Thompson." That result would have been a helpful warning sign. Problem is, I can't find any reference to it in Marist's final poll release. Instead, I find this prediction: "If today were Election Day," they wrote on Wednesday without qualification, "Mayor Michael Bloomberg would handily win a third term."

If anyone deserves to say "I told you so" in New York, it is Thompson pollster Geoff Garin, who released a survey last week showing Thompson gaining (he said), trailing by only 8 points (38% to 46%) and by only 3 points (41% to 44%) among those who said they were certain to vote. The release prompted Bloomberg spokesman Howard Wolfson to retort that it "gives new meaning to the term margin of error." Not exactly. (And yes, we managed to miss this poll and omit it from our chart -- apologies to Garin and our readers for that oversight).

I asked Garin for his thoughts and he agrees that "undecideds split against incumbent" in the New York race and that such a split was knowable in advance, but argues:

[I]t is stupid to think they would split 100 to nothing. There was a high undecided in NYC because voters were cross pressured -- they did not want to reward Bloomberg for his bad behavior on term limits, but they didn't know enough about Thompson to know whether he would be up to the job.

Garin also thinks their sample made a difference:

I think the main reason we did better and the public polls were off is that we worked off the voter file, and were persnickity about who we took into what was very likely to be a low turnout election. Even among whites, the smaller the turnout scenario the better for Thompson. I am sure the public polls let in too many people.

Maine Question 1. Polling on the gay-marriage referendum was far more limited -- just seven public polls released over the course of the campaign -- and the complicated ballot language and the error prone nature of prior referenda poll warned us to expect the unexpected. Yet while the differences between the final polls were relatively small, it is worth noting that the automated survey from PPP was the only one that showed more support for the anti-gay marriage position than opposition. Our final trend estimate showed the No side (pro gay marriage) with a two point lead (49.4% to 47.1%) but Question 1 won by nearly six (52.8% to 47.2%).

While this one experience is far from a conclusive test, there are at least theoretical reasons to think that automated surveys have an advantage in measuring true preferences on issues like gay marriage, where the presence of a live interviewer might introduce some "social discomfort" that would make the respondent reluctant to reveal their true preference.

* * *

So were automated IVR polls the big winners on Tuesday, as Mickey Kaus, Taegan Goddard and PPP's Tom Jensen argue? If what you care about most is predicting the winners, it is clear that the automated surveys provided a more accurate gauge of the outcome, especially in New Jersey where the closer simulation of the secret ballot probably gave us a heads up of an imminent "incumbent rule" effect favoring Christie. SurveyUSA also deserves credit for coming closer than most pollsters to the final margin in New Jersey, Virginia and New York City.

But that said, consider that we count on polls to do much more than predict the outcome. In addition to the points raised by Brian Schaffner here yesterday, consider two things:

First, as a live-interviewer media pollster pointed out to me yesterday, there were some inconsistencies with subgroups, particularly by race. As the table below shows, despite relatively small sample sizes, the three automated surveys showed Republicans Christie and McDonnell winning a greater percentage of the African American vote than the final live-interviewer surveys and the exit polls (though there were a few inconsistencies; namely Rasmussen in New Jersey and Marist in New York City).

If you believe the exit poll result, then the automated surveys provided a generally misleading sense of whether the Republican candidates were about to make bigger inroads than they did among African-American voters (consider also commenter RussTC3's observation about big differences between job approval ratings as measured by PPP and the exit polls -- as Mike Mokrzycki reminds us we do polls for reasons other than predicting the outcome).

Second, there is one last contest we need to review....

New York 23. Although three last minute polls on the special election in New York's 23rd Congressional District conducted after Republican Dede Scozzafava withdrew from the race last Saturday showed Conservative Doug Hoffman leading Democrat Bill Owens by margins of between 5 and 17 points, Owens prevailed by 4 points (49.0% to 45.9%). Whatever shortcomings we might identify in the polling, the far bigger error was the interpretation applied by pundits, most notably me, who foolishly assumed that the trend in Hoffman's direction was unstoppable and that normal assumptions about last minute developments would apply. In retrospect, it is obvious that there was nothing normal about the last 72 hours of this particular campaign.

Moreover, we should have paid closer attention to the evidence of growing voter uncertainty in the final Siena Research Institute poll. Their final survey, conducted on Sunday night, showed Hoffman with modest but not quite statistically significant lead (41% to 36%) but also a doubling of the undecided (from 9% to 18%) in just a few days. So their poll showed that voter uncertainty was surging at a time when it is usually nonexistent. To his great credit, Siena pollster Steven Greenberg also argued that Owens might still gain from the Scozzafava endorsement on Sunday since "most voters are not political junkies" and had not yet heard the news" (an argument I boldly dismissed since few undecided voters had a favorable impression of Scozzafava -- apologies to Greenberg for that).

But while we might plausibly reconcile the results of the Siena poll with the outcome, the PPP survey is another story. While their estimate of Owens' support (34%) was within a few points of the other polls, PPP had Hoffman receiving five percentage points more support (51%) than he ultimately received (45.9%). A late shift among the undecided voters cannot explain the difference.

I am planning to look more closely at this example, but the important point for now is that while the automated polls turned in a strong performance in New Jersey, Virginia and Maine, the PPP poll in NY-23 was highly misleading.

The larger lesson is this: Automated polls have been maligned, unfairly in my view, as inherently "unreliable." Yet when it comes to predicting election outcomes they continue to prove, NY-23 aside, at least as reliable as surveys done by conventional means. In New Jersey this week, they were more accurate in predicting the winner. At the same time, however, it would be wrong to jump to the opposite conclusion and place inherently greater trust in all automated surveys,  especially when used for purposes other than predicting election outcomes.

All polls have their limitations. Rather than trying to divide them into two categories, "reliable" and "crap," we might do better to try to understand their limitations and interpret the results we see accordingly.

National

Most Closely Followed Story
29% Reports about swine flu and the vaccine
22% Debate over health care reform
17% Reports about the condition of the U.S. economy
10% The U.S. military effort in Afghanistan
8% The Major League Baseball World Series
2% News about political instability in Pakistan

Are you hearing mostly good news about the economy these days, mostly bad news about the economy or a mix of both good and bad news?
5% Mostly good news, 31% Mostly bad news, 62% A mix of good and bad news

California

2010 Governor: Republican Primary
37% Whitman, 15% Campbell, 6% Poizner (chart)

2010 Governor: Democratic Primary
46% Brown, 19% Newsom (chart)

National

Favorable / Unfavorable
Barbara Boxer: 33 / 37 (chart)
Carly Fiorina: 11 / 24
Chuck DeVore: 8 / 15
Al Ramirez: 10 / 12

Maryland

Job Approval / Disapproval
Gov. O'Malley: 48 / 40
Sen. Cardin: 46 / 26
Sen. Mikulski: 57 / 28
Pres. Obama: 60 / 33

Maryland will have an election for Governor next year. At this point, would you like to see Martin O'Malley re-elected or would you like to see somebody new get elected?
39% Re-elect, 48% Someone new

2010 Governor
O'Malley 47%, Ehrlich 40%

Maryland will also elect a U.S. Senator next year. At this point, would you like to see Senator Barbara Mikulski re-elected or would you like to see somebody new get elected?
53% Re-elect, 36% Someone new

National

I'm Going to read you a list of possible Republican candidates in the 2012 presidential election. Please tell me whether you would, or would not, seriously consider supporting each for president

Among Republicans:
Mike Huckabee: 71% Yes, 25% No
Mitt Romney: 65% Yes, 31% No
Sarah Palin: 65% Yes, 33% No
Newt Gingrich: 60% Yes, 35% No
Tim Pawlenty: 32% Yes, 48% No
Haley Barbour: 26% Yes, 52% No

Among all adults:
Mike Huckabee: 40% Yes, 51% No
Mitt Romney: 39% Yes, 51% No
Sarah Palin: 33% Yes, 63% No
Newt Gingrich: 29% Yes, 63% No
Tim Pawlenty: 18% Yes, 60% No
Haley Barbour: 15% Yes, 65% No

Please tell me whether you think each of the following people is qualified or is not qualified to be president
Mike Huckabee: 50% Yes. 36% No
Mitt Romney: 49% Yes, 39% No
Sarah Palin: 31% Yes, 62% No
Newt Gingrich: 44% Yes, 46% No
Tim Pawlenty: 25% Yes. 51% No
Haley Barbour: 18% Yes, 57% No

National

State of the Country
38% Right Direction, 57% Wrong Track (chart)

Obama Job Approval / Disapproval
53% Approve, 43% Disapprove (chart)
Dems: 82 / 15 (chart)
Reps: 21 / 78 (chart)
Inds: 45 / 41 (chart)

Congressional Job Approval
29% Approve, 68% Disapprove (chart)

Has the U.S. economy turned the corner on the current crisis, is the worst yet to come, or have things stabilized but not yet begun to improve?
7% Turned the corner, 32% Worst yet to come, 59% Stabilized but yet to improve

2010 House: Generic Ballot
48% Democrat, 41% Republican (chart)

As of right now, do you favor or oppose the healthcare reform proposals presently being discussed?
39% Favor, 49% Oppose (chart)

Creation of a public entity to directly compete with existing health insurance companies.
51% Favor, 43% Oppose

Legislation to permit the creation of insurance co-operatives NOT run by the government.
57% Favor, 35% Oppose

Party ID
34% Democrat, 22% Republican, 44% independent (chart)

Kentucky

2010 Senate: Democratic Primary
Mongiardo 40%, Conway 37%

First, there is no way to control for other reasons that these polls might have generated different results, including different approaches to screening for likely voters and how undecideds are dealt with. With regard to the latter issue, it is important to note that the pollsters using live interviewing in New Jersey were showing more than twice the percentage of undecideds as those using IVR.

This leads to a second important point (related to the first): comparing these pollsters based on the final result presupposes that each pollster that has been entered into this fictitious competition was actually trying to get the final result correct in the first place. If that was the goal, then it seems as though each polling firm would have allocated all of their undecided respondents into one camp or another.

Third, one of the reasons for concerns with IVR polling is that citizens with only a cell phone cannot be reached by these pollsters and these citizens now comprise at least one-fifth of the population. Yet, while the cell-only problem may generally be an issue for IVR technology (and for live interview pollsters who aren't calling cell phones), it is less of a problem for polling on elections, and particularly in low turnout elections. This is because the types of people that do not have landlines are less likely to be voters (and particularly less likely to be voting in low turnout elections). Ultimately, an off-year low turnout election may actually be less of a challenge for IVR-based polls because the non-coverage bias should be smaller for these contests. Where these polls may run into greater challenges is when they attempt to make inferences about the American public rather than registered (or likely) voters.

Also, the following analysts and organizations have posted in-depth analyses of the exit poll results:

• ABC's Gary Langer on Virginia and New Jersey.
• CBS's Jennifer De Pinto on New Jersey.
• CBS's Fred Backus on Virginia.
• Associated Press on Virginia and New Jersey.
  
• Politico's Erika Lovely on Virginia and New Jersey.
• Monmouth University's Patrick Murray on New Jersey.
• Edison Research's Joe Lenski on New York City.
  

[I'm confident this list is missing similar analysis from other organizations, so please add a comment or email us if you see articles worth linking to].

If I had $100 for every time I've been asked that question by a reporter on the Wednesday morning after an election, I could retire early. And after five years of blogging on this beat, it's a question I'm determined to refuse to answer today.

Why?

First, all the votes are not yet counted (including 7% of the precincts in NY-23), and the counts that are available do not yet include the absentee and provisional ballots that will be added later and are not reflected in those percentage-of-precincts-reporting statistics you see on all the media vote counts morning. Take a look at this snap judgement from November 5, 2008. It declared a "big winner" among prognosticators on the assumption that Barack Obama won by 6.1 percentage points (52.4 to 46.3), but when all the ballots were counted the margin was 7.2 (52.9% to 45.7%). So that particular snap judgement picked the wrong "big winner."

Second, the whole notion of crowing a "big winner" based on a handful of polls in a handful of states is foolish. The final polls yesterday had random sampling error of at least +/- 3 percentage points. If a poll produces a forecast outside its margin of error, that's important. But if several polls capture the actual result within their standard error, chance alone is as likely as anything else to determine which one "nails it" and which miss by a point or two.

Third, there are sometimes other problems with making too much of "hitting a bullet with a bullet" on the final poll, when the polls leading up to it provide different results.

Yes, there are several good stories about what went right and what went wrong with yesterday's polling, including some important lessons about the value of automated polling. Some pollsters certainly did better yesterday than others. And I'm hoping to have something written and posted on that subject later today, provided that I don't get bogged down by the calls and emails from reporters wanting me to tell them, "who was the most accurate pollster yesterday?

National

Do you think the country is -- or is not -- more deeply divided this year on the major issues facing the country than it has been in the past several years?
68% More deeply, 29% Not more deeply

Regardless of which presidential candidate you preferred, do you think the Obama administration will or will not be able to do each of the following?

Increase respect for the United States abroad:
60% Yes, will, 38% No, will not

Improve the health care system:
46% Yes, 52% No

Reduce unemployment:
51% Yes, 46% No

Control federal spending:
31% Yes, 67% No

Keep the U.S. safe from terrorism:
57% Yes, 40% No

Bring U.S. troops home from Iraq in a way that is not harmful to the U.S.:
56% Yes, 41% No

Bring U.S. troops home from Afghanistan in a way that is not harmful to the U.S.:
46% Yes, 50% No

Heal political divisions in this country:
28% Yes, 69% No

National

2010 House: Generic Ballot
50% Democratic Candidate, 44% Republican candidate (chart)

Would you say you will definitely vote for the _____, or is it possible you
would change your mind between now and the election?
27% Definitely Democratic, 23% Democratic could change,
22% Republican could change, 22% Definitely Republican

Would you be more likely to vote for a candidate for Congress who supports President Obama, or
more likely to vote for a candidate who opposes President Obama?
54% Supports Obama 41% Opposes Obama

Kentucky

2010 Senate: Republican Primary
35% Rand Paul, 32% Trey Grayson, 3% Roger Thoney, 2% Bill Johnson, 1% Brian Fouglas Oerther

2010 Senate: Democratic Primary
39% Dan Mongiardo, 28% Jack Conway, 5% Lillie Miller-Johnson, 2% Darlene Fitzgerald Price,
1% Maurice Marion Sweeney

2010 Governor: General Election
43% Grayson, 39% Conway
48% Grayson, 38% Mongiardo
44% Conway, 39% Paul
43% Mongiardo, 43% Paul

Republicans Are Poised for Gains in Key Elections
Outcomes in New York, New Jersey and Virginia Are Unlikely to Forecast Much About National Races in 2010, History Shows

Republicans appear positioned for strong results in three hard-fought elections Tuesday. But isolated, off-year contests aren't always reliable indicators of what will happen in the wider federal and state races held in even-numbered years.

Wall Street Journal, 11/4/09:

Republicans Win in Key States

A Republican sweep in Virginia and New Jersey on Tuesday shifted the political terrain against President Barack Obama only a year after his historic election.

PS For the record, the WSJ was right the first time. Despite what the press will tell you, a handful of off-year elections don't tell us much about the "political terrain" facing Obama and the Democrats. As Matthew Yglesias points out, we have these things called "polls" that we can use to measure people's political beliefs and opinions. Perhaps we should consider using those instead.

Update 11/4 11:41 AM: Dave notes in comments on my blog that the first story includes a similar passage about the election potentially revealing "much tougher political terrain," which I missed:

A Republican sweep in Tuesday's key contests would at minimum show that Democrats face much tougher political terrain than they did a year ago.

I'm not sure what the passage means (the metaphor of "political terrain" is not well-defined) but it seems to contradict the lede of the story, which states that off-year elections are not reliable indicators. The point remains that the ledes are in tension (if not in direct contradiction).

It's also worth noting note the contradiction between the election "show[ing]... political terrain" (11/3) and the results actually "shift[ing] the political terrain" (11/4). Maybe it's time to retire the metaphor, which lets reporters vaguely suggest that things have changed without specifying how.

Update 11/4 8:49 PM -- Eric Boehlert at Media Matters has a virtually identical item on the AP's election coverage:

The AP on Tuesday:

To be sure, it's easy to overanalyze the results of such a small number of elections in a few places. The results will only offer hints about the national political landscape and clues to the public's attitudes. And the races certainly won't predict what will happen in the 2010 midterm elections.

The AP on Wednesday:

To be sure, each race was as much about local issues as about firing warning shots at the politically powerful. But taken together, the results of the 2009 off-year elections could imperil Obama's ambitious legislative agenda and point to a challenging environment in midterm elections next year.

(Cross-posted to brendan-nyhan.com)]]> http://www.pollster.com/blogs/wsj_vs_wsj_on_2009_elections.php http://www.pollster.com/blogs/wsj_vs_wsj_on_2009_elections.php 2009 Wed, 04 Nov 2009 09:24:27 -0500 charles@politicalarithmetik.com (Charles Franklin) by Charles Franklin

I'm headed to bed but here are a couple of snapshots of election night.

First, above, New Jersey then and now. Whatever else you say about the race, Corzine lost support across all regions of the state and by relatively constant amounts. This "uniform swing" shows that he didn't just lose in Rep areas, or Dem areas, or urban centers. The decline in Corzine support was very widespread and quite even. An across the board loss.

In NY23, Hoffman generally outperformed McCain's vote in 2008, but not by enough to take the race. These are based on 87% of precincts reporting, so not quite final data.