New Hampshire 2010 Senate
Judd Gregg (R-i) 54%, Carol Shea-Porter (D) 35%
Gregg 47%, Paul Hodes (D) 40%

That's how ABC News polling director Gary Langer begins a blog post explaining that polls from ABC News predicted today's report of a 2.2 percent decline in holiday sales by the International Council of Shopping Centers. He points to results reported in November signaling "a Dismal Retail Season" and showed 51% saying "they'll spend less this year than last on holiday gifts, matching the sharpest consumer retreat in polls dating back 23 years." The results from their December survey were worse. He concludes with this appropriate point:

As with politics, there's been far greater detail in our economic polls, especially in our mid-December survey's extensive look at the roots and directions of the public's economic anxiety. But as good data help us understand the contours of public opinion, so they anticipate the results of those attitudes. It's why the vast bulk of survey research isn't carried out by news organizations seeking to report public views, but by corporations seeking to understand how such attitudes will impact their bottom line.

But I link mostly because of the "Told Ya" headline. Langer does concede that the phrase is not "terribly polite," but his use of it gives me the chance to blog this unforgettable Barney Frank quote from Jeffrey Toobin's profile in this week's New Yorker:

"There are three lies politicians tell," [Frank] told the real-estate group. "The first is 'We ran against each other but are still good friends.' That's never true. The second is 'I like campaigning.' Anyone who tells you they like campaigning is either a liar or a sociopath. Then, there's 'I hate to say I told you so.' " He went on, "Everybody likes to say 'I told you so.' I have found personally that it is one of the few pleasures that improves with age. I can say 'I told you so' without taking a pill before, during, or after I do it."

New York State
2010 Senate: Caroline Kennedy (D) 51, Peter King (R) 33%

As soon as I returned to the keyboard after publishing the item -- which emphasizes details provided by Obama pollster Cornell Belcher on how the campaign dealt with "racial aversion" -- my RSS reader produced links to a new Carl Bialik item on the role race played in Obama's victory. It summarized another article by two political scientists, Stephen Ansolabehere and Charles Stewart, who used exit poll data to argue that “Obama won because of race — because of his particular appeal among black voters, because of the changing political allegiances of Hispanics, and because he did not provoke a backlash among white voters.”

Crucial to their argument is that Obama barely gained among white voters compared to Sen. John Kerry in 2004; Obama won 43% of white votes, compared to 41% for Kerry. That slight gain didn’t tilt the election to Obama; instead it took blacks’ and Latinos’ rising share of the electorate, coupled with Obama’s big win among both groups — far bigger than Kerry’s. (Obama won 95% of black votes and 67% of Latino votes, compared to 88% and 53%, respectively, for Kerry.)

“Had the racial composition of the electorate stayed the same in 2008 as it was in 2004, and had whites remained as supportive of Republicans as they were in 2004, Obama would still have won the popular vote, albeit by a much smaller margin,” Ansolabehere and Stewart wrote. “But, had Blacks and Hispanics voted Democratic in 2008 at the rates they had in 2004 while whites cast 43 percent of their vote for Obama, McCain would have won.”

A few hours later, Andrew Gelman chimed in with more details from his analysis of county-level voting patterns:

You can also slice up the vote swing geographically, by counties in different regions of the country, and you find that Obama did close to uniformly better than Kerry nearly everwhere, except for Republican-leaning poor counties in the South (where Obama pretty much stayed even with Kerry). The geographic patterns are striking (see graph at the end of this post).

Race matters, yes, but we're still seeing a national swing.

He added:

I think Ansolabehere and Snyder are right on the money when they write, "the results of the 2008 election challenge much of what has been conventionally thought about race and politics in America. Barack Obama has accomplished an astonishing political move [by] disproportionately energizing nonwhite voters and converting erstwhile Republican supporters within the minority community without alienating white voters."

My summary: as Carl said, the election outcome is multidimensional. Because Steve and Charles were writing a short article, they very properly focused on a single feature of the election--race. I'd say that the #1 feature of the election was a bad economy that produced a national swing toward the Democrats in general and Obama and particular. But once you want to break this down by demographics, I agree that ethnicity is the biggest factor.

Next, John Sides linked to all of the above and took issue with my post on two points. First, he interprets my post as an argument that political scientists forecast an Obama landslide on the basis of political fundamentals. "Neither the fundamentals nor the existence of racial prejudice," he writes, "should have led a sensible analyst to predict a landslide. Most political scientists certainly didn’t. I railed against the perception that this race should have been a landslide here."

The use of the term "landslide" was Ambinder's. I have to admit, I'm not sure what would constitute a "landslide" in the context of last year's election, as defined by Belcher, the unnamed "Obama advisors" he differed with or anyone else. Was Obama's 7.2% margin in the popular vote or his 365 to 173 margin in the electoral college a "landslide?" I'm not sure, although I read the Ambinder passage as implying skepticism from Belcher "in the fall" that Obama's margin would be as wide as it ultimately proved to be.

Second, Sides hears me arguing, perhaps inadvertently, that “political scientists will think they’re right no matter what happens.” Well, no, I didn't mean to imply that all political scientists tend to declare themselves right regardless of the outcome (or even that some do), only that had the outcome been different, another set of scholars somewhere would have been ready to declare (rightly), "I told you so."

Most electoral outcomes are "multi-dimensional." Rarely, if ever, are they about just one thing. And I will grant that the machinations of the campaign -- rallies, paid advertising, field organizing, etc. -- tend to be less consequential in presidential general elections, where voters get massive, direct exposure to the candidates through televised debates and a year's worth of news coverage, than in almost all other types of elections. However, I think the tendency within political science to use presidential general election forecasting models to dismiss the notion of "campaign effects" is overdone. That point aside (or perhaps on that point as well), Sides, Gelman and I mostly agree.

Finally, in the midst of writing this post, I stumbled on yet more from Obama pollster Cornell Belcher via Marc Ambinder. The latter posts a 14-page analysis (PDF) by the former produced for the Democratic National Committee based on data collected in two post election surveys. The memo includes data on "the surge among new voters of color" and much more.

Alaska 2010 Senate
Republican Primary: Murkowski (R-i) 57, Palin (R) 33

(via Hotline - subscription required)

Traditionally, the Hotline has only published live-telephone interview surveys while excluding interactive voice response (IVR) polls, despite the increased media coverage of many of these so-called "robo-polls." In our constant effort to remain tuned to industry developments, and to determine if such distinctions are fair and valid, the Hotline will begin running selected numbers from IVR polls during the upcoming cycle. Specifically, head-to-head matchups, favorability ratings and approval ratings from IVR outfits will appear on an interim basis in the Hotline's Latest Edition through the '10 midterms. This data -- from firms such as InsiderAdvantage, Public Policy Polling, Rasmussen Reports and SurveyUSA -- will be published alongside live-telephone data, but will be clearly labeled as IVR results.

For those who are unfamiliar, The Hotline has been a DC institution for more than 20 years, serving up a daily political news summary chock full of polling data since the days when the preferred mode of delivery was the fax machine. They have long refused to publish surveys that used an automated methodology rather than live interviewers, so in our small world, their decision to publish IVR results, even if only on an "interim" basis, is important and, in my view at least, a welcome step.

The short version is that Obama gained sufficient credibility and support among African-American voters to allow his campaign to focus later in the campaign on wooing uncertain white voters, many of whom demonstrated what Belcher describes as "racial aversion." The eventual lopsided margin among black voters, Ambinder writes,

also highlights what Obama did not have to do: he did not have to pander to black leaders; he did not have to target specific messages at the black community with the attendant risk of exacerbating economic tension between blacks and whites. He did not have to bring up race. And that was key, because Belcher's polling confirmed that culturally anxious whites were willing to vote for a black candidate so long as they did not meditate on the candidate's blackness. Obama was able to credential himself as an African American without engaging in overt racial politics. Or, rather, the black community credentialed Obama without his resorting to racial politicking, something that white Democratic candidates had to do.

The article -- well worth reading in full -- has more details on how Belcher measured "racial aversion" and on the conclusions he drew from the data.

Ambinder also reports on Belcher's candor, back in September, about how race might limit Obama's support:

In the fall, when some Obama advisers began predicting a landslide, Belcher would have none of it. "No one with any real post-civil-rights understanding of our national political contours could with a straight face predicate a Democratic national landslide," he told me in September.

It's worth contrasting that statement with the post-election assessments of many political scientists. They found Obama's ultimate margin "not surprising" since it roughly matched what statistical models based mostly on "fundamental factors" (such as perceptions of the economy and the Bush administration) had predicted (for more details, see comments by Larry Bartels in the Brookings post-election roundtable or the concise summary, with ample links, by John Sides).

But Belcher's words of caution remind me of a comment from one of John Sides' readers in reaction to Sides argument that "the fundamentals" mattered more than the campaigns:

[T]he fact that Obama--as a black man--was able to pull within the margin of a usual victory speaks to the ability of his campaign skills. Again, coming from a sociological perspective, race is so incredibly salient in so many aspects of our lives as Americans, it is astounding that so many Americans were willing to put those sentiments aside and vote for a black man. I suspect that this is what you might mean by "It may be that the campaign helped move voters in line with the outcome that the fundamentals predict" -- but I think that understates how amazing Obama's accomplishment to be the first African American President really is.

See the same link for Sides' reply.

I can't help but thinking that if the election had turned out differently, we might have heard a chorus of "I told you so's" from a different set of political scientists reminding us of the lessons of 30 or 40 years of academic opinion research on how racial attitudes shape political preferences. It didn't happen that way, and Ambinder's piece helps explain why.

National

As you may know, the governor of New York will need to appoint someone to replace Hillary Clinton in the U.S. senate once she becomes secretary of state in January. Caroline Kennedy, the daughter of former president John F. Kennedy, has been mentioned as a possible replacement. Would you like to see Caroline Kennedy appointed to this seat or would you rather see someone else get the appointment?

45% Would like to see Kennedy appointed
36% Would rather see someone else appointed

Which of the following would you like to see the state of Illinois do to fill the open senate seat...?

16% Allow Roland Burris to serve until 2010 when the next election is scheduled
23% Keep the seat open until the situation with Blagojevich is resolved and allow him or a new governor to appoint a senator to serve until the 2010 election
52% Hold a special election as soon as possible to fill the seat

How do you think the senate should handle the situation when Roland Burris arrives to fill the open Illinois senate seat?

27% Allow Burris to fill the seat
51% Block Burris from filling the seat

(source: Kennedy, Burris)

In May 2002, I sat down to draft a questionnaire for a new client, a wealthy businessman who was then considering a challenge to an incumbent U.S. Senator two years hence. It was just six years ago, back in the days before my MysteryPollster blog, back when conducting such surveys for Democratic candidates was my full-time occupation. And, as it turned out, this client would be my last "major" race (a designation that, in my mind, falls somewhere between a contest with an uncertain outcome and one with some national significance).

My client's eventual campaign struck some as a microcosm of conventional (and thus flawed) politics. One of the consultants for one of our opponents called our campaign "an exercise in the technology of politics...They do polls and see what people want to hear and then they put ads on the air and run them again and again. There's nothing new about what they're doing." There was truth in this critique, and the ultimate demise of this campaign played a not insignificant role in my evolution from consultant to blogger.

But it was something that happened at the very beginning of that campaign that sticks in my memory now. On that day in May, I started with my usual process when drafting a benchmark "message testing" survey. I took all of my notes from initial discussions with the client, suggestions from other consultants and information gleaned from news clippings and the Internet and synthesized it into a single document listing all of the "messages" -- candidate profiles and arguments about them pro and con -- we hoped to test on the benchmark survey.

I found my "notes for questionnaire" for that project on my hard drive yesterday, still there after moving with me through at least three different computers over the last six years. My first cut consisted of roughly 700 words. About half involved the incumbent Republican, about a third concerned our client. I had also typed a two sentence profile of the potential primary opponent, a former Democratic Senator that had been turned out of office a few years before.

Finally, the document also included the names -- just the names --of three potential candidates that most everyone considered extreme long-shots. The paucity of detail on these three spoke to their status as very long shots in this particular Senate race. Anyone looking over my notes right now, however, would immediately notice one now familiar name among the also-rans:

"Barack Obama."

The survey we fielded a few weeks later showed that just 4% of likely general election voters had a favorable impression of Obama, while 5% rated him unfavorably. Nearly four out of five (79%) had never heard of Obama. He did a little better among likely Democratic primary voters -- 11% favorable, 6% unfavorable -- but still ran far behind, winning just 6% of the vote, in a five-candidate primary matchup that also featured former Senator Carol Moseley Braun. Obama did a little better (rising to 12%) when we omitted Braun, but Obama still ran far behind state Comptroller Dan Hynes (with 34%).

Of course, my client, Chicago businessman Blair Hull, started with even less recognition and support (2% of the vote, to be precise), but his candidacy and its ultimate demise was another story altogether. The consultant who dismissed our campaign as "nothing new" was a truly forward looking guy named David Axelrod. But I digress.

It is still hard to believe that a State Senator who seemed like the longest of long shots for the U.S. Senate so recently is about to be sworn in as the 44th President of the United States. I look back at my notes from six years ago, ponder all that has happened since and just shake my head in wonder.

This story teaches many lessons, of course, but the most relevant to readers of this site is that no poll or statistical model could have predicted Obama's ascent. Yes, we could see from the "internals" of that first survey that Obama had the potential to be a formidable contender in the 2004 Senate primary, especially after both Braun and incumbent Peter Fitzgerald announced that they would not run. But no survey or model in 2002 could have predicted Obama's 52.8% majority in the seven-candidate 2004 Senate primary or the nearly 70% of the vote he won in the general election, much less all that followed in bid for the White House in 2008. I'm certainly a believer in opinion surveys and statistical models, but they have their limits.

The Obama story from 2004 provides no direct corollary to the ongoing arguments about the "electability" of potential Senate candidates in Illinois, New York or elsewhere, but I do see a warning against leaping to conclusions about a candidate's prospects based only on early favorable ratings or horse-race numbers. "Electability" is an appropriate topic whenever a party selects its nominee (or appoints someone to serve out the term of a departing legislator), but quantifying electability through polling, or through predictive models derived from it, is a shaky enterprise at best. Obama's rise -- in the face of early data that led many to question his electoral potential in both 2004 and 2008 -- is a testament to such challenges (see also this pertinent example from September 2007).

This story also says something bigger about the potential for "change" within the American electoral system. Typically the day-to-day business of politics is mundane and static. We fight the same fights over and over, and little seems to change. Politics seems to be mostly about compromise, mostly "the art of the do-able." Yet every once in awhile, usually in the context of a presidential election, some out-of-the-blue candidacy reshapes our perception of what is possible. It shows us that sometimes, if we're lucky, we get to see history being made in the midst of mere electoral politics.

I look back to my own experience in the 2004 Senate election and realize that there was a silver lining in an unsatisfying campaign that amounted to, admittedly, little more than "an exercise in the technology of politics:" It offered me the opportunity to witness (and, yes, literally chart) one of the most meteoric success stories in American political history from the very beginning. And from this perch at Pollster.com, I got the chance to keep that ringside seat, charting and writing about the most exciting campaign of my lifetime.

So I want to thank Pollster.com's readers for coming along with us for the ride, and especially those who have stayed now that things have calmed own a bit. We hope you will stick around and let your friends know as we continue to track public opinion in 2009. You will be seeing a transition in Pollster.com over the next month or so, as we move the 2008 data off the front page and begin to feature charts and data that track the performance of the new administration and the concerns of American voters. But as we do, let's not lose sight of an idea we have tried to stress here from our first post: Using survey data well requires that we understand its limitations.

[Typos fixed and one badly constructed sentence revised].

New York State 2010 Senate

Tested: Attorney Gen. Andrew Cuomo (D), Caroline Kennedy (D), Rep. Peter King (R)

Cuomo 48, King 29
Kennedy 46, King 44

(source)

New York State

If the choices were Andrew Cuomo and Caroline Kennedy, who would you prefer Governor Paterson appoint to replace Hillary Clinton in the US Senate?

58% Cuomo
27% Kennedy

How has your opinion of Caroline Kennedy changed since she started publicly campaigning for appointment to Hillary Clinton's Senate seat?

23% More Favorable
44% Less Favorable

(source)

The Pew Internet & American Life Project releases a survey on voter engagement after the election (via Amy Sullivan).

Jennifer Agiesta finds new optimism among Democrats.

Brenden Nyhan sees no sign of the Obama honeymoon ending.

Gary Langer digs into how veterans voted in 2008.

Tom Jensen shares a tongue in cheek anecdote on response rates.

Chris Bowers considers public opinion among the Palestinians (PS: Get well soon).

One more:  John Sides shares an analysis of voter turnout among Virgos.

National

"Because of these ups and downs, Bush's 49% approval average for his presidency will rank him in the middle of the pack (7th of 11) of post-World War II presidents. His average to-date of 49.4% is similar to Richard Nixon's 49.1% but slightly better than Harry Truman's and Jimmy Carter's historical lows below 46%."

(source)

It will be a new year full of new polls and new challenges, and we are looking to bringing you new charts and analysis to follow it all.

Until then, from all of us here at Pollster.com, we wish you a Merry Christmas, a Happy Hanukkah or joy in whatever way you celebrate the holiday season.  And if I don't get the chance to say it online, a happy New Year too.

According to the most recent ABC News poll, "Americans' Christmas gift wish list has a decidedly more prosaic cast this recession year."

56% of Americans say Gov. Rod Blagojevich has been the "naughtiest" political figure this year, according to a recent CNN / ORC survey; 25% say Gov. Eliot Spitzer, 19% say John Edwards.

A recent Marist poll says that 19% of Americans choose It's a Wonderful Life as their favorite Christmas movie; less than 1% choose Gov. Schwarzenegger's tour de force.

Mark Murray reports on another debriefing by McCain pollster Bill McInturff.

NPR reviews public opinion on single payer national health insurance.

Jennifer Agiesta looks behind the Post/ABC job loss numbers.

Jon Cohen finds support for the popular vote rather than the electoral college.

The Pew Research Center shows the Internet overtaking newspapers as a news source.

Curtis Gans (PDF) releases the CSAE turnout report (via Hotline On Call)

Swing State Project is crowdsourcing results from the presidential race by congressional district (via Jensen).

John Sides graphs corruption by state -- which state is #1?

David Hill sees little chance for the GOP to leverage the Blagojevich scandal.

Mark Mellman plays the economic anxiety Grinch.

Survey Practice looks at how interviewers view race matching, a British analysis of the American Elections, audio recording in in-person surveys and more.

Aleks Jakulin points to a Slashdot review of a "manga" comic book* about statistics, Andrew Gellman is underwhelmed. *See AySz88's comment.

Now that the '08 tally is official, we note that for the second election in a row, the IBD/TIPP Poll not only came closest to the final margin, but was right on the money -- tantamount to hitting a bullet with a bullet.

My thoughts of these snap judgements of poll "accuracy" over the last two months have had a common theme: We are placing far too much emphasis on "the last poll." Nothing supports that argument as much as the claims in this IBD release. And the story is worth telling in detail, so pull up a chair and let me begin at the beginning.

IBD/TIPP released results of daily tracking from October 13 through the night before the election, Monday, November 4. Their first release was based on interviews conducted over seven days (October 6-12). Their field period narrowed to five days as of the October 16 release and to four days on their second-to-last release on November 2.

As the chart shows below, the IBD/TIPP poll showed a persistent "house effect" favoring McCain. Twenty-two (22) of 23 releases before 11/4 produced an Obama margin that was below our trend estimate. It produced an Obama margin that averaged roughly four percentage points over the course of October, compared to six to eight points for our trend estimate over the same period. A roughly 2-3 point difference on the margin is not large in absolute terms, but it does put the margin reported by the IBD polls at the lower end of the range of those reported by other national polls during October 2008. Other polls produced leads for Obama on the opposite end of that range, but the average margin of roughly seven points during most of October matched the final result almost perfectly.

Without undecideds allocated, the final IBD/TIPP polls showed Obama leading by 5.1 points (47.5 to 42.4%), a result only slightly better for Obama than their average result over the previous four weeks. However, their final projection on Monday afternoon allocated to Obama two-thirds of remaining 6% that were still undecided. That allocation produced an projection that was remarkably close to what the poll averages showed earlier that day. The final IBD/TIPP projection had Obama ahead by 7.2 percentage points (51.5% to 44.3%), exactly the same as our trend estimate as of Monday morning (Obama +7.2 points; 51.5% to 44.3%) and just one tenth of a percent different from the RealClearPolitics average that day (Obama +7.3 points; 51.6% to 44.3%).

Obama's final margin -- as of this writing and based on final results in each state -- turns out to be urned out to be 7.25 percentage points (52.9% to 45.7%), within a whisper of the final IBD/TIPP projection and the result that all other polls, collectively, had been showing for weeks (at the same time, the actual 1.2% support for candidates other than Obama and McCain fell far short of IBD/TIPP's projection of 4.1% -- something not accounted for in their boast of "top honors").

However, the "house effect" that made earlier IBD/TIPP results more favorable to McCain was not lost on those producing the IBD/TIPP analysis. Here is the first line of their first poll release:

In contrast to other polls, which show Obama leading McCain by 4 points (Reuters/C-SPAN/Zogby) to 11 (Newsweek), the IBD/TIPP Tracking Poll debuts today with Obama up just 2 points with 13% (including 25% of independents) undecided.

They came back to this theme repeatedly. On October 22:

Contrary to other polls, some of which show Obama ahead by double digits, the IBD/TIPP Poll shows a sudden tightening of Obama's lead to 3.7 from 6.0.

And again on October 29, the IBD/TIPP analysis pointed out that other polls were "migrating" (briefly, as it turned out) to their closer than average result:

The race tightened again to 3 points Wednesday, a margin IBD/TIPP has shown for six days and to which other polls appear to be migrating. For example, the Rasmussen and Gallup polls, each of which had Obama up 5 points two days ago, now have him at 3.

The relative closeness of the IBD/TIPP polls was also repeatedly noted by supporters of John McCain (all quotes via Nexis transcripts):

• Bill Bennett on CNN, October 22: "Wolf, I don't know if you saw them or James, AP poll and IBD poll which had these races much closer and in some of these states had McCain ahead."

• Hugh Hewitt on CNN, October 23: "I know they're not covered much in the media, but if you look at the Investor's Business Daily poll today, the poll, it's down to 1 point. Yesterday, it was three point seven."

• Sean Hannity on Fox, November 3: "Investors Business Daily, which was the most accurate poll in the last presidential election, now has it down to a two-point race."

• Mary Matalin on CNN, November 4: "[The McCain campaign is] looking at their polls, which are better than the public polls and do match like the IBD poll, all those that are out there but in the target states where they need to close the gap...this is [a] harsh [environment], as you're saying, but certainly not undoable."

Not surprisingly, many conservative bloggers relied on the IBD/TIPP polls to support the same conclusion, namely that the race was closer than other polls made it appear. Ironically, to strengthen that argument, many pointed to IBD/TIPP's boast of being the "most accurate" pollster in 2004, a message that appeared prominently alongside each day's results. "An analysis of Final Certified Results for the 2004 election," they wrote, "showed IBD's polling partner, TIPP, was the most accurate pollster of the campaign season."

TIPP's final projection in 2004, showing a 2.1 percentage point lead, did come closest among the final national polls to the ultimate 2.5 percent margin, although it is worth noting that IBD was one of the few to report results to one decimal place. Four other pollsters showed Bush leading by margins of 2 or 3 percentage points on their final 2004 poll. Since they rounded off their results to the nearest whole number, we cannot say for certain who would have been "closest," although reporting results out to one decimal is largely meaningless for individual polls given margins of error of 3 percentage points or more.

The same can be said this year: Seven pollsters other than IBD/TIPP reported final results or projections that rounded to Obama leads of 7 or 8 percent. Again, IBD/TIPP can claim to be "closest" to the final margin mostly because they chose to report results to one (largely meaningless) decimal place. Other pollsters may have been just as close. Moreover, remember random sampling error. How close a pollster comes to the illusion of "pinpoint accuracy" on any given survey (or even on any two surveys) is still largely a matter of chance.

But the issue raised here that many will wonder about is the allocation of two-thirds of the undecided voters to Obama, especially given the analysis I linked to yesterday by Mickey Blum about the so-called "convergence mystery." She noted that allocations that moved estimates closer to the poll averages helped explain the sharply decreased variance among the final polls that David Moore identified (see also my commentary). So what do we make of the fact that the IBD/TIPP allocation produced a result converged almost perfectly with the poll averages?

First, in their defense, IBD/TIPP is certainly not the only pollster to wait to allocate undecided voters until their final release. Gallup has followed that practice for decades, and five other national survey organizations did the same this year.

Second, Raghavan Mayur, president of TechnoMetrica, explained in a comment posted here yesterday that they used a "Hierarchical Heuristic algorithm" in both 2004 and 2008 that had been developed using their data for the decided. Such algorithms create mutually exclusive clusters of voters based on combinations of variables that, in this case, maximize the difference across those subgroups in terms of support for Obama or McCain. They divide up the decided electorate, creating some subgroups that are very likely to support Obama, some that are very likely to support McCain and others that fall somewhere in between based on what Mayur describes as "dominant demographics" that predict candidate support.

In response to my email query last week, Mayor provided more details on the workings of their algorithm. In this case they created groups of clusters based on combinations of seven variables (race, party identification, age, income, self-reported ideology, 2004 vote and religion). Once they had used those variables to create segmentation based on the decided, they developed allocation rules based on the demographic segments (and, presumably, on the level of support for Obama and McCain in each segment) that they used to allocate each undecided respondent.

So Mayur's explanation is essentially that their allocation was based not on a gut hunch or an arbitrary "rule" based on past experience but rather a statistical model based driven by individual level data. He also sticks doggedly to the initial IBD line that other pollsters had it wrong:

Other commentators questioned why we often had the Obama-McCain race tighter than other polls. The fact is, the race was tight right down to the last week, when undecided voters swung to Obama. The final 7.2-point spread was the widest we showed since our polling started Oct. 13.

Unfortunately, I find no evidence anywhere else to support the contention that two-thirds of undecided voters "broke" to the Democrat in either 2004 or 2008. Not in the exit polls. Not in comparisons between other polls and the final results. Only the IBD/TIPP poll achieved "awesome results" (as Mayur put it in an email), by assuming a 2-to-1 break to the Democrats in 2008 and nearly as much in 2004. More specifically:

• The average or trend estimate of the other national polls (Obama +7.6 on both our final trend estimate and the RCP final average) was less than four tenths of a percent from the actual result.   

• Of the seven pollsters that allocated undecided voters for their final projection, IBD/TIPP was the only one that allocated more to Obama than McCain. At least one of those -- the Pew Research Center -- used regression modeling of the decided to inform their allocation, which favored McCain slightly. Charles Franklin and I also did a similar application of a model developed among the decided in early October and found a near even split among the undecided.

• To justify an undecided percentage on his final poll (9%) that was higher than others (our trend estimate showed 5-6% undecided for much of October), Mayur reminds us that according to the exit polls, "the share of electorate deciding in the last weekend in 2004 was 9%, and in 2008 was 7%." True. However, those same exit polls show the same late deciders dividing evenly in 2008 (49% Obama, 48% McCain), and slightly in Kerry's favor (53% to 44%) in 2004.
  

There is also some confusion about the number of variables used to create the algorithm that allocated the undecideds. When the Wall Street Journal's Carl Bialik asked Mayur about it shortly after the election, Bialik reported that TIPP's allocation "method uses 10 variables, such as the respondent's party, gender and 2004 vote, to predict the 2008 vote." When I asked Mayor to specify the 10 variables, he listed only seven (race, party, age, income, ideology, 2004 vote and religion), explaining that "we also look at gender but do not use it in the model." However, the document he attached to the same email has one listing of classification rules that does not include the age variable, and a listing of computer code that does (Mayur asked that I treat the additional details contained within that document as "strictly 'confidential" so that I would not "give away our 'bread and butter' to other firms").

The point here is that the underlying model used to allocate undecided voters still leaves a lot of room for subjective judgements. How many variables are used to create the segmentation? Which subgroups from that segmentation are used to allocate undecided, and which are not? What is the criteria that determines whether a given subgroup gets allocated to Obama or McCain?   

Writing here yesterday, Mayur offered a "moral" of their allocation story:

1. Close to ten percent of the electorate makes decision in the final weekend (Saturday, Sunday, and Monday).

2. They don't break even. Democrats have a clear advantage among them, at least in the past 3 races.

Respectfully, those are the wrong lessons. Here is what we have learned:

• We see a conspicuous disparity between IBD/TIPP's final projection and the results released earlier in October.

• The accuracy of the final poll projections from IBD/TIPP are useful, at best, only in evaluating subsequent final horse race projections from that organization, not the results that come earlier.   

• The polling industry, and those of us that follow it, place far too much emphasis on "the last poll" in judging poll accuracy and reliability.   

Finally, let us be clear that we are in no position to discern an inappropriate motive on the part of TIPP or Investor's Business Daily. The more important issue here is what we make of the results they produced.

PS: I asked Raghavan Mayur for his comment on my argument that we put too much emphasis on final polls generally, and on the final IBD/TIPP polls specifically, in measuring survey accuracy. His full response appears after the jump.

National

In general, do you think there is too much, too little, or about the right amount of government regulation of the stock market and financial institutions?

22% Too much
59% Too little
18% Right amount

As you may know, a financial advisor named Bernard Madoff has been accused of running an illegal scheme in which many people lost millions of dollars. Do you think Madoff is an isolated case that does not reflect on other financial advisors and institutions, or do you think Madoff's behavior is common among financial advisors and institutions?

23% An isolated case
74% His behavior is common

New York State

As you may know, Senator Hillary Clinton has been nominated to be Secretary of State and Governor Paterson will pick her replacement in the Senate. Who do you think Governor Paterson should pick to replace Hillary Clinton in the United States Senate; Caroline Kennedy, Andrew Cuomo, Kirsten Gillibrand or someone else?

33% Kennedy
29% Cuomo
4% Gillibrand
24% Someone else

Do you think Caroline Kennedy is qualified to be a United States Senator or not?

40% Yes
41% No

Alaska 2010 Elections

Governor:
General Election: Palin (R-i) 55%, Knowles (D) 38%

Senate:
GOP Primary: Palin 55%, Murkowski (R-i) 31%
General Election: Palin 53%, Knowles 39%
General Election: Murkowski 49%, Knowles 41%

House:
GOP Primary: Young (R-i) 33%, Parnell (R) 27%, Harris (R) 11%
General Election: Young 49%, Berkowitz (D) 46%

National
Pres. Bush Job Approval: 28% Approve, 67% Disapprove

National

The federal government will provide some of the major U.S. auto companies with more than thirteen billion dollars in loans in order to prevent them from going into bankruptcy. In exchange for the loans, the government wants the auto companies to produce plans by the end of March that show how they would become viable businesses in the long run. Do you favor or oppose this decision?

63% Favor
37% oppose

If the major U.S. auto companies ask for more money next year, do you think the federal government should give them any additional assistance, or should the government let them go into bankruptcy?

28% Additional assitance
70% Let them go into bankruptcy

How would you rate Dick Cheney in comparison to other vice presidents in U.S. history? Would you say he is the best vice president the U.S. has ever had, very good, good, poor, very poor, or would you say he is the worst vice president the U.S. has ever had?

1% Best
4% Very good
30% Good
28% Poor
13% Very poor
23% Worst

National

As you may know, Illinois Governor Rod Blagojevich has been arrested on charges of corruption. Based on what you have heard or read, which of the following statements best describes your view of any recent contact between Blagojevich and top aides to Barack Obama:

12% Some Obama aides did something illegal
36% No Obama aides did anything illegal, but did something unethical
43% No Obama aides did anything seriously wrong

National

Apart from dealing with the economy, do you think Obama should or should
not...

... withdraw U.S. forces from Iraq within the next 16 months?

70% Should
26% Should not

... make major changes in the U.S. health care system?

77% Should
20% Should not

... implement policies to try to reduce global warming?

75% Should
20% Should not

... close the U.S. military prison at Guantanamo Bay in Cuba?

40% Should
44% Should not

... expand federal funding of embryonic stem-cell research?

52% Should
42% Should not

In their new issue, Survey Practice publishes responses from five survey researchers who deserve the title "expert:" Paul J. Lavrakas, Michael Traugott, Micheline Blum, Cliff Zukin and Don Dresser. David Moore follows up with a new article of his own that begins with this helpful summary:

The two major types of explanations offered by our experts for the convergence of polls right before the election are based on 1) changes in the pollsters’ methodology and 2) changes in the certainty of the vote choice.

The first explanation suggests that in the final weeks of the campaign, many pollsters adjust their likely voter models (mentioned by Lavrakas and Blum) or they increase their sample sizes (Dresser).

Lavrakas argues that the adjustment of the voter models, even when done explicitly to make their outcomes more consistent with other polls, should be seen as a positive action rather than as a “suspicious” activity. However viewed, such last minute changes could account for some of the convergence.

Dresser mentions the tendency of many pollsters to substantially increase their sample sizes for their final pre-election polls, to insure as small a margin of sampling error as they can reasonably afford. He specifically mentions Pew, Harris and ABC/Washington Post.

Blum suggests that the outlier polls during the campaign were either a) less likely to poll in the final three days, or b) more likely to allocate their undecided voters, thus bringing them closer to the mean.

Zukin speculates that the convergence has less to do with pollsters’ methods and more to do with the “phenomenon we are measuring.” Specifically, as voters become more certain about their choices, polls will tend to converge toward each other. That sentiment is also found in the explanations by Lavrakas and Blum.

Moore goes on to offer his own thoughts and asks why polls, "conducted at the same time, using virtually the same wording - [are] supposedly more accurate (reliable) when opinion is more crystallized?" For those intrigued by the "mystery," It is all worth reading in full.

Among all of these efforts to explain, I was most intrigued by Blum's analysis, which was the only one of the five to scrutinize the poll data directly. Among the national polls, at least, much of the variance in mid-October can be explained by a small handful of "outlier" results two-weeks out from the election:

Of the six organizations with outlier polls, three reported margins larger than 11 points, and all three consistently showed larger margins in their polls. In the final three days, however, one of these organizations allocated the undecided, giving more to McCain, one did not release a poll, and one had a margin of exactly11 points on its final poll. Of the three organizations releasing four polls with margins smaller than 5 points, one organization (with 2 “outlier” polls) allocated the undecided in the final three days, one did not release a poll in the final three days, and one “converged.” So, of the 6 organizations, 2 organizations (accounting for 3 “outlier” polls) allocated undecided in the final three days in the direction of the previously underestimated candidate, 2 did not release polls in last 3 days, and only 2 “converged.” If we remove the 7 “outlier” polls from the 29 released in the week of 10/21-27, the variance is reduced to only 2.9 points.

Her conclusion:

Basically then, both of the explanations examined, the allocation of the undecided by seven organizations in the final three days and the absence or favorable allocation of a few “outlier” organizations, appear to be major contributors to the “convergence” seen. Apportioning the undecided in the favorable direction and the absence of previous outliers virtually guarantees less variance and the appearance of “convergence. So, perhaps, rather than convergence, what we saw was that much of the earlier variance was due to a few outliers and that the final three days benefited from their absence or favorable apportionment of their undecided vote.

Blum's observation prompted me to take a closer look at the national polls that allocated the undecided. I find seven projections by six organizations (the two pollsters for the GWU/Battleground poll, Republican Ed Goeas and Democrat Celinda Lake, once again produced competing projections with different allocations). The process of allocating undecided should not, by itself, raise any suspicion. Of the seven allocations, three changed the initial margin not at all or within rounding difference of the averages, two moved the margin farther away from the poll averages as of Monday 11/4, and one moved it closer. However, that one move -- by the IBD/TIPP poll -- was important in helping produce the reduction in variance that Blum describes.

National

Which of the following would you say best describes the current economic situation in the United States?

60% Biggest crisis in lifetime
16% Crisis, but not the worst
18% Major problem
4% Not a major problem

How likely do you think it is that the United States will be in an economic depression within the next two years?

35% Very
39% Somewhat
21% Not too
4% Not at all

Colorado 2010 Senate
John Hickenlooper (D) 54%, Tom Tancredo (R) 37%
Hickenlooper 54%, Bill Owens (R) 40%
John Salazar (D) 53%, Tancredo 40%
Salazar 52%, Owens 43%

National

Caroline Kennedy, the daughter of President John F. Kennedy, is being considered as Hillary Clinton's replacement in the U.S. Senate now that Mrs. Clinton is becoming secretary of State. Is Caroline Kennedy qualified to serve in the Senate?

37% Yes
37% No

Do you have a very favorable, somewhat favorable, somewhat unfavorable or very unfavorable impression of Caroline Kennedy?

67% Very favorable/Somewhat favorable
23% Very unfavorable/Somewhat unfavorable

Hawaii
Sen. Daniel Inouye (D-i) 53, Linda Lingle (R) 42

Gawker responds with its usual snarky flair but makes a reasonable point, if you can get past the gratuitous nastiness: This week's column spends more time hawking Penn's book than supporting his arguments with data. They point out, for example, that Penn/Zalesne attach this footnote their first use of the term "Impressionable Elites:"

* "Impressionable Elites" is the term we used for educated, affluent people who focus more on personality than issues when it comes to evaluating political decisions. For more, please see pages 131 to 135 in "Microtrends."

There's a stranger footnote. The central contention of the column is that Bernard Madoff's bilking of a lot of very rich people "proves the point" that "elites have become more impressionable -- more removed from everyday problems, more trusting of what they hear, and more likely to adopt unthinking viewpoints based on brand or emotion."

The data that supports this claim?

[O]ur research** shows that the top 1% is heavily swayed by gut and impression, not numbers and facts. They vote more on the basis of personality in campaigns; buy products more on the basis of brands; and invest more on the basis of the tip than on sound logic.

And what research is that? Another footnote tells us:

** A PSB [Penn, Schoen & Berland] poll of 806 telephone interviews among likely 2008 presidential voters, including an oversample of 400 very likely Democratic presidential primary voters.

Two issues here. First, sample size? I will leave it to readers to decide exactly how many interviews such a sample will yield for the "top 1%." Presumably, non-voters fall disproportionately in the bottom 99%, so it may be that the wealthiest or best educated 1% of the population contributed slightly more than 1% of this sample. Still, I still have a hard time seeing how that top 1% subgroup could have yielded much more than n=15, unweighted.

Second, consider the reference to the "oversample" of likely "Democratic presidential primary voters." It certainly sounds like a survey done during or before the Democratic primaries on behalf of Mr. Penn's most well known recent client. It is not hard to imagine such a survey including tests of negative messages about Obama ("numbers and facts") that failed to persuade his very wealthy supporters (for an example of such a message test -- click here, scroll to pp. 11-12). Please tell me that the referenced evidence of elite impressionability depends on more than that. Please.

Update: Reader DS points out that the Microtrends chapter referenced in that first footnote includes results that give a good clue as to the evidence referenced in the second footnote. Specifically, on pp. 132-133, Penn and Zalesne write (the table is my reproduction of the one in the book):

This isn't just my gut. Let's look at the data.

A standard poll question I ask in campaigns is what people consider most important in voting for a candidate: (1) issues, (2) character, or (3) experience. I ask it because I know that all three are important in a leader, and that it can be tough to rank them.

According to a recent poll we did, a large plurality of voters -- 48 percent -- believe that a candidate's stand on the issues is most important, with character a distant second at 32 percent. That preference for issues holds steady whether or not voters have been to college, whether or not they are religious, and across race. Where it does not stay constant, however, is across income. Once voters reach the magic line of $100,000 per year, their priority shifts to character, by a significant margin. As the table below shows, people earning under $100,000 prioritize issues over character by a serious 51 to 30 percent. But once they reach $100,000, the switch, to character over issues, 45 to 37 percent.

That is a 29-point swing. A shift barely ever gets clearer in polling.

So if this is "research" referenced in the footnote, there's a good chance that the data used to make inferences about the "top 1%" was based on the 25% or so of voters that report an income over $100,000. Also, as DS puts it in his email, there isn't any more survey evidence on the book on this point: "no regression analysis, no experiment, just a simple, dumb question."

As it happens, the soon-to-be-extinct LA Times/Bloomberg poll asked a similar question in their September 2008 campaign survey. I have reproduced the results below. Notice two things: First, the Times/Bloomberg question showed 31% choosing "all" or "some" of the categories. The PSB had no such category and reported only 1% as "don't know." If allowing interviewers to take "all of the above" as an answer draws in nearly a third of the respondents, we have a pretty good clue that the underlying attitudes are soft.

Second and more important, the Times/Bloomberg poll showed a clear pattern by party and ideology: Democrats and Liberals were more likely to choose "positions on issues," while Republicans and Conservatives were more likely to choose "experience" and, to a lesser extent, "character." Here's a theory: Voters perceived McCain as strong and Obama as weak on "experience," so McCain supporters were more likely to say they consider experience important and Obama supporters less likely to choose that answer. That result is consistent with my experience that voters rarely answer such questions in a completely abstract way. They tend to answer with high profile candidates in mind, especially when the question comes in the context of a survey that has already asked about those candidates.

So before making too much of the pattern in Penn's cross-tab, I would want to see whether any other variables show similar correlations and whether controlling for other variables (through a regression analysis) would explain the pattern by income.

And of course, as DS points out in his email, there are many sources of political survey data that would allow a more complete test of this thesis. He says, for example, that he examined ANES data "to see if highly educated / high income respondents were more likely to be impressed by character qualities rather than issues," but found no connection.

First, yesterday, the Center for Disease Control and Prevention's National Center for Health Statistics released its latest biannual report on the prevalence of households without wireless or standard telephone service (via Bialik). The CDC monitors the cell-phone-only population because it conducts huge ongoing health "surveillance" surveys via telephone, and as such, ask questions about telephone usage on their ongoing, in-person National Health Interview Survey (NHIS). Their latest report, which covers the first six months of the year, shows that 16.1% of adults were reachable only by cell phone, while another 2.1% lacked telephone service of any kind.

As the chart above shows, the latest survey continues an ongoing, near linear upward growth in the cell-phone-only population. It is worth noting that the hint of a plateau in the trend seen between the second half of 2006 and the first half of 2007 was likely the result of a very slight change in question wording that took effect in 2007. Except for the brief near-pause, the trend has been steadily and consistently upward since 2005.

The second news is the latest report from the Pew Research Center on their efforts to survey voters via cell phone during 2008 (summary, full PDF). It is hard to overstate* the influence of the Pew Center's work on cell phones in the political polling industry. At the PAPOR conference in San Francisco last week, Professor Mike Traugott (a past president of AAPOR and chair of AAPOR's Special Committee on 2008 Primary Polling) noted that earlier this year, "the conventional wisdom in the Spring [among pollsters] was that we didn't have to worry about cell phone only people." At the end of the summer, however, "this conventional wisdom changed drastically" when the Pew Center released its report showing that the omission of cell phone only voters could understate Barack Obama's margin over McCain by two to three percentage points." By the fall, many national media surveys included supplemental cell phone samples in their surveys.

The new Pew report confirms that the patterns they saw, "that estimates based only on landline interviews were likely to have a pro-McCain tilt compared with estimates that included cell phone interviews," persisted through their final survey in late October. It was a difference of a point or two on the margins that "while statistically significant, was small in absolute terms -- smaller than the margin of sampling error in most polls." Of possibly greater importance going forward, they also found similar differences in party identification and self-reported ideology, and bigger differences in (not surprisingly) the use of Internet as a news source and social networking sites like Facebook or MySpace. See the report for the details.

Now that the conventional wisdom on cell-phone only households has shifted among national pollsters, from "it doesn't matter" to "we better account for the them," the Pew Report also points us to the next issue for the c.w. to chew over: Simply interviewing by cell phone (and screening for "cell only" respondents) may not be enough. Pollsters will also need to confront the more difficult issue of how to handle and weight the interviews among what some call "cell phone mostlys:"

Unlike most other polling organizations, Pew's election surveys involved a "full dual frame design," in which people reached by cell phone who also have a landline are interviewed, as well as cell-only respondents. In contrast, most pollsters who included cell phones in their election surveys screened their cell samples for cell-only respondents.

The difference between these two approaches can be seen as a tradeoff in methodological challenges. Supplementing a landline sample with cell-only respondents has the advantage of not "double covering" respondents who have both types of phones. This makes combining the samples more straightforward, but assumes that the landline sample is capable of accurately reaching all adults equally. If some adults have landline phones that they rarely or never answer because they favor their cell phones, they will be underrepresented in these surveys. Pew's approach of interviewing all adults in both the landline and cell phone samples ensures that every adult with a telephone is covered by the survey, but raises challenges in combining the data because some adults had a greater chance to participate if they have more than one telephone. Pew's methodology accounts for this double coverage by weighting respondents with both kinds of phones according to their probability of selection and the regularity with which they use each kind of telephone.

The report goes on to present data showing that the slightly different results produced by the two approaches. Duel users reached by cell phone were more likely to support Obama (53%) than those reached by landline phone (46%). The cell-phone-mostlys reached by cell also identified with the Democrats (54%) more often than those interviewed by landline phone (47%).

The report concludes with a section on the "practical considerations" of interviewing by cell phone. As they have found previously, once they offer a monetary incentive to potential cell-phone respondents ($10), the contact and cooperation rates are comparable to what they get on landline phones. So cell phone interviewing can be done. The downside is that cell-phone interviews cost Pew "nearly two-and-a-half times as much as landline interviews" for reasons they explain in detail.

As always, my summary does little justice to the full report. Go read it all.

*[Mangled use of "understate" corrected]

Colorado

With Senator Ken Salazar being nominated as Secretary of the Interior, Governor Bill Ritter will have the opportunity to appoint a new United States Senator. Some of the candidates being mentioned are Congresswoman Diana Degette, Denver Mayor John Hickenlooper, State Treasurer Cary Kennedy, Congressman Ed Perlmutter, former Denver Mayor Federico Pena, House Speaker Andrew Romanoff, and Congressman John Salazar. Which of these would be your preference for the Governor to appoint?

23% Hickenlooper
15% Salazar
12% Romanoff
11% Pena
8% Degette
6% Perlmutter
2% Kennedy

Historically, NCPP has focused their analysis on the national polls. Here are their main conclusions on the performance of the national polls in 2008:

In terms of Candidate Error, the average is less than one percentage point (0.9), whether the pollster choose to allocate undecided voters at the end of not. That is the same as the 0.9 percentage point error reported by NCPP for this analysis in 2004. It is slightly less than the 1.1 percentage point average in 2000. In 2008, estimated errors ranged from 0.1 to 2.4 percentage points.

Thus, despite widely discussed concerns such as the growing size of the cell-phone-only population (and this year the possibility of a repeat of Bradley/Wilder effect), there was no change in poll average error.

NCPP is a consortium of media pollsters, and as such concentrates on evaluating the performance of the polls (plural) rather than on rating or ranking individual pollsters or methodologies. So while the report has some useful data for making year-to-year, industry-wide comparisons, it will likely frustrate those trying to find "best" or "worst" pollster.

That said, any thorough effort to rank the pollsters, to separate "good" from "bad" on the basis of the accuracy of the last poll is bound to frustrate for reasons the NCPP report identifies in an easily overlooked, next-to-last paragraph:

No method of judging the error works perfectly. Other evaluations of poll performance based on other methods may produce different conclusions.

The NCPP report includes two methods of measuring the poll error that differ slightly from the eight first proposed in 1948 by the renowned Harvard statistician Frederick Mosteller in his chapter of the report of the Social Science Research Council on the polling failures that year (an still used by many who score pollster error). The NCPP measures also differ from the odds-ratio scoring proposed three years ago in the pages of Public Opinion Quarterly by Martin, Traugott and Kennedy. I have looked at state level pollster error using some of these methods, and can confirm that different methods can and do produce different rankings in 2008.

The reasons is that four factors can affect the size of the error scores, especially when aggregated for any given pollster, and these are not comparable across organizations:

1) The number of polls conducted - Generally, if we average errors across multiple polls, those who do more polls should show lower average errors by the logic of regression to the mean. Any one poll can produce a large error by chance, but as we average more and more surveys, the average errors should be generally lower (there is an exception

2) The number of interviews for any given poll - More interviews should mean less random error, and different pollsters use different sample sizes. The sample sizes for some individual pollsters can also vary widely from state to state. So if we aggregate errors across pollsters, some will do better simply because their sample sizes are bigger.

3) How the scoring handles or interprets the "undecided" category - In general elections, "undecided" is not a choice on the ballot, so any reported undecided is an error, in a sense. What complicates the analysis is that some pollsters allocate undecided voters on their final poll and some do not. Some error scores effectively ignore the undecided (either by allocating or by focusing on the margins separating the candidates), while some scores penalize pollsters that leave undecideds unallocated. This issue remains a matter of considerable, unresolved debate among pollsters.

4) The lag between the dates of interviewing and the election -- A longer delay between the field dates and election day creates a greater potential for error due to last minute shifts in voter preferences. Those that field late have an inherent advantage over those that conclude earlier, although the size of any such advantage in any given election is debatable and hard to evaluate. And ignoring all the polls that came before "the last poll" opens the possibility of a misleading measure, especially when polls do seem to converge around a common mean on the last round of polls (at least they did in 2008, see our posts here, here, here and here).

All of these are reasons why we have been cautious (so far) in producing a "best-to-worst" ranking of individual pollsters for Pollster.com. A few weeks ago, Mark Lindeman and I ranked pollsters based on their statewide surveys using 12 different scores and time frames (don't bother searching, as we have not yet posted these online).** Even when we narrowed the list to the 15 or so organizations that produced at least five "final" poll results in statewide contests, we found seven different pollsters ranking 1st or 2nd at least once, five ranking lowest or second lowest at least once and three that ended up in both categories (best and worst) at least once. And none of these rankings ranked the pollsters in a way that controlled for the number of polls conducted or the sample sizes used, ranking each pollster against the standard of how well it should have done.

The NCPP report takes a first stab at that sort of analysis by comparing what they call candidate estimate error to one half the of the margin of error. "A total of 53  of the state polls," the report tells us, "or 12.8 percent had results that fell outside of the sampling margin of error for that survey."*** Given that the margin of error is based on a 95% level of statistical confidence, if the surveys (and these comparisons) were perfect, we would expect only 5% of the results to fall outside the margin of error. Caveat: They arrive at this statistic by calculating the error on the margin predicted by the poll, dividing that number by two (to get an estimate on the error for each candidate) and comparing it to the reported margin of error for that poll.   

Do some pollsters do better than others when judged by that standard? I will try to assess that in my next post.

**I haven't posted those scores, mostly because the endless number of tables adds up to no obvious conclusion. I'm willing to post those tables, in all their glorious and confusing detail, if readers demand it. But I would much rather try to find ways to evaluate pollsters that attempt to control for the four factors listed above. As always, readers suggestions are welcome.

***When I wrote this post, the links on the NCPP web site pointed to earlier drafts of the tables and analysis that were not based on the final results in each state.  As a result, the original version of my entry quoted an earlier computation of the percentage of polls falling outside the sampling margin of error, which I have now corrected. 

Utah

As you may know, Congress is considering giving about 35 billion dollars in emergency loans to help or bail out Ford, General Motors, and Chrysler. From what you know, are you for or against this relief or bailout package by the United States government?

22% For
74% Against

National

Do you approve or disapprove of the job President-elect Barack Obama is
doing on the transition?

63% Approve
10% Disapprove

Do you approve or disapprove of Barack Obama's choice of Hillary Clinton
as Secretary of State?

65% Approve
27% Disapprove

New York

"When asked who Governor Paterson should pick to replace Clinton, 26 percent of voters - 30 percent of Democrats - said Cuomo, while 23 percent - 28 percent of Democrats - said Kennedy. Congresswoman Kirsten Gillibrand received seven percent support, Congressman Brian Higgins, six percent, and Nassau County Executive Tom Suozzi, five percent."

"While Cuomo has a slim lead over Kennedy for who the Governor should pick, 31 percent of voters think he will pick Kennedy, while only 16 percent think he will choose Cuomo,"

If David Paterson runs for Governor in 2010, would you vote to elect him or would you prefer someone else?

36% Elect
31% Prefer Someone Else

National

Job Approval / Disapproval
Bush: 29% / 67%
Reps in Congress: 25% / 69%
Dems in Congress: 37% / 55%

National

Do you approve or disapprove of the federal government's overall response to
the economic situation?

23% Approve
72% Disapprove

How much do you think Obama will be able to do to improve the economy - a
great deal, a good amount, only some or not much at all?

46% Great Deal/Good Amount
52% Some/Not Much

Overall, which party, the (Democrats) or the (Republicans), do you trust to
do a better job in coping with the main problems the nation faces over the next
few years?

56% Democrats
23% Republicans

Connecticut
Sen. Joe Lieberman (D) (I)*:
38% Approve, 54% Disapprove

Sen. Chris Dodd (D):
47% Approve, 41% Disapprove

Gov. Jodi Rell (R):
68% Approve, 20% Disapprove

How likely are you to vote to re-elect Chris Dodd for United States Senator in 2010? Do you think you will definitely vote for him, probably vote for him, probably not vote for him, or definitely not vote for him?

44% Definitely/Probably
47% Definitely/Probably Not

How likely are you to vote to re-elect Jodi Rell for Governor in 2010? Do you think you will definitely vote for her, probably vote for her, probably not vote for her, or definitely not vote for her?

54% Definitely/Probably
31% Probably/Definitely Not

Would you support or oppose amending the Connecticut State Constitution to ban same-sex marriage?

33% Support
61% Oppose

* apologies

National

On another topic, all in all, considering the costs to the United States versus the benefits to the United States, do you think the war with Iraq was worth fighting, or not? Do you feel that way strongly or somewhat?

34% Worth fighting
64% Not worth fighting

Do you think the United States is or is not making significant progress toward restoring civil order in Iraq?

56% Is making significant progress
40% Is not making significant progress

Thinking now about Afghanistan, all in all, considering the costs to the United States versus the benefits to the United States, do you think the war in Afghanistan was worth fighting, or not?

55% Worth fighting
39% Not worth fighting

Do you think (the United States must win the war in Afghanistan in order for the broader war on terrorism to be a success), or do you think (the war on terrorism can be a success without the United States winning the war in Afghanistan)?

51% US must win war in Afghanistan for success
40% Can be a success without US winning war in Afghanistan

National

On a different topic, as you may know, Illinois Governor Rod Blagojevich has been charged with conspiracy in trying to profit from naming Barack Obama's replacement in the U.S. senate.

(Asked of a half sample) Just your best guess, do you think any member of Barack Obama's staff has done anything illegal in connection with this matter, or not?

26% Yes, illegal
60% No, not

(Asked of a half sample) Just your best guess, do you think any member of Barack Obama's staff has done anything unethical in connection with this matter, or not?

29% Yes, unethical
60% No, not

Dilbert (above) does cell phones and surveys.

Mike McDonald posts his final 2008 turnout rates based on official or certified votes for president (131.2 million adults or 61.6% of eligible adults - via AP, The Page ).

Gary Langer sees Iraq as the "prime agent" in the Bush's historic unpopularity.

Mark Mellman ponders what happened in Indiana

David HIll urges Republicans in Texas, and elsewhere, to say "howdy" to non-Republicans.

John Harwood talks to campaign pollsters and ponders Obama's rising approval amidst bad news.

Chris Bowers offers six reasons why Obama's popularity is "remarkable."

PPP's Tom Jensen has developed remarkable powers.

Carl Bialik digs into the meaning the "probability of precipitation."

and (overlooked a few weeks back), Gary Andres find support for means testing Medicare (in a survey sponsored by the Coalition to Advance Healthcare Reform)

The panelists (including PPIC's Jennifer Paluch and the LA Times' Jill Darling) generally agreed that support for the "yes" vote was trending upward in the final week of the campaign and that the final shift in the yes direction could be explained by the conflicting views of roughly one-in-five of those supporting the "no" side as of mid-October. The final PPIC survey (fielded 10/12-19), for example showed the "no" vote leading, 52% to 44%, while also showing California likely voters divided evenly (47% favor, 49% oppose) on "allowing gay and lesbian couples to be legally married." As such, 19% of those who said they were voting no also said they were opposed to same-sex marriage. In PPIC's post-election survey, only 8 percent were opposed to same-sex marriage (see also Mark Baldassare's recent op-ed, "Why the Same Sex Marriage Ban Passed).

In the interview, Field's DiCamillo repeated an argument he made in a guest contribution on our site shortly after the election, suggesting a "Sunday before Election" church effect:

The Field Poll, completed one week before the election, had Catholics voting at about their registered voter population size (24% of the electorate) with voting preferences similar to those of the overall electorate, with 44% on the Yes side. However the network exit poll shows that they accounted for 30% of the CA electorate and had 64% of them voting Yes. Regular churchgoers showed a similar movement toward the Yes side. The pre-election Field Poll showed 72% of these voters voting Yes, while the exit poll showed that 84% of them voted Yes.

I asked both Baldassare and DiCamillo about the long term increase in support for gay marriage and how that might impact future campaigns to overturn Prop 8. Both were cautious about expecting too much from "generational" change, particularly over the next 3 to 4 years.

(See also the Pollster.com Prop 8 polling chart).

National

How likely is it that President-elect Obama or one of his top campaign aides was involved in the Blagojevich scandal?

23% Very likely
22% Somewhat likely
35% Not very likely
11% Not at all likely

Who is more typically more corrupt...politicians or CEO's of Major companies?

48% Politicians
25% CEO's of major companies

Texas Gubernatorial Republican Primary
Hutchison 55%, Perry (R-i) 31%