December 21, 2008 - December 27, 2008
Over these last two years, we are thankful to have had the opportunity to share the most exciting, compelling election of our lifetimes with you from this unique vantage point. The last year in particular has been long and sometimes grueling, so my family and I are looking forward to taking the next week or so off. Eric will be checking in from time to time, but I will be pretty much off the radar until the new year.
It will be a new year full of new polls and new challenges, and we are looking to bringing you new charts and analysis to follow it all.
Until then, from all of us here at Pollster.com, we wish you a Merry Christmas, a Happy Hanukkah or joy in whatever way you celebrate the holiday season. And if I don't get the chance to say it online, a happy New Year too.
The latest USA Today/Gallup survey says that while 80% of Americans identify with a Christian faith, 93% indicate that they celebrate Christmas.
According to the most recent ABC News poll, "Americans' Christmas gift wish list has a decidedly more prosaic cast this recession year."
56% of Americans say Gov. Rod Blagojevich has been the "naughtiest" political figure this year, according to a recent CNN / ORC survey; 25% say Gov. Eliot Spitzer, 19% say John Edwards.
A recent Marist poll says that 19% of Americans choose It's a Wonderful Life as their favorite Christmas movie; less than 1% choose Gov. Schwarzenegger's tour de force.
Gary Langer offers a requiem for a fallen comrade, the Los Angeles Times poll, dead at age 31.
Mark Murray reports on another debriefing by McCain pollster Bill McInturff.
NPR reviews public opinion on single payer national health insurance.
Jennifer Agiesta looks behind the Post/ABC job loss numbers.
Jon Cohen finds support for the popular vote rather than the electoral college.
The Pew Research Center shows the Internet overtaking newspapers as a news source.
Curtis Gans (PDF) releases the CSAE turnout report (via Hotline On Call)
Swing State Project is crowdsourcing results from the presidential race by congressional district (via Jensen).
John Sides graphs corruption by state -- which state is #1?
David Hill sees little chance for the GOP to leverage the Blagojevich scandal.
Mark Mellman plays the economic anxiety Grinch.
Survey Practice looks at how interviewers view race matching, a British analysis of the American Elections, audio recording in in-person surveys and more.
Aleks Jakulin points to a Slashdot review of a "manga"
comic book* about statistics, Andrew Gellman is underwhelmed. *See AySz88's comment.
In the eight weeks since the elections, I have seen a fair number of self-congratulatory press releases from pollsters boasting of their successes during 2008, but none had quite the audacity of the release put out last week by Investor's Business Daily about the polls they conducted along with the TechnoMetrica Institute of Policy & Politics. "IBD/TIPP Takes Top Honors Again," read the headline, continuing in bold print below:
Now that the '08 tally is official, we note that for the second election in a row, the IBD/TIPP Poll not only came closest to the final margin, but was right on the money -- tantamount to hitting a bullet with a bullet.
My thoughts of these snap judgements of poll "accuracy" over the last two months have had a common theme: We are placing far too much emphasis on "the last poll." Nothing supports that argument as much as the claims in this IBD release. And the story is worth telling in detail, so pull up a chair and let me begin at the beginning.
IBD/TIPP released results of daily tracking from October 13 through the night before the election, Monday, November 4. Their first release was based on interviews conducted over seven days (October 6-12). Their field period narrowed to five days as of the October 16 release and to four days on their second-to-last release on November 2.
As the chart shows below, the IBD/TIPP poll showed a persistent "house effect" favoring McCain. Twenty-two (22) of 23 releases before 11/4 produced an Obama margin that was below our trend estimate. It produced an Obama margin that averaged roughly four percentage points over the course of October, compared to six to eight points for our trend estimate over the same period. A roughly 2-3 point difference on the margin is not large in absolute terms, but it does put the margin reported by the IBD polls at the lower end of the range of those reported by other national polls during October 2008. Other polls produced leads for Obama on the opposite end of that range, but the average margin of roughly seven points during most of October matched the final result almost perfectly.
Without undecideds allocated, the final IBD/TIPP polls showed Obama leading by 5.1 points (47.5 to 42.4%), a result only slightly better for Obama than their average result over the previous four weeks. However, their final projection on Monday afternoon allocated to Obama two-thirds of remaining 6% that were still undecided. That allocation produced an projection that was remarkably close to what the poll averages showed earlier that day. The final IBD/TIPP projection had Obama ahead by 7.2 percentage points (51.5% to 44.3%), exactly the same as our trend estimate as of Monday morning (Obama +7.2 points; 51.5% to 44.3%) and just one tenth of a percent different from the RealClearPolitics average that day (Obama +7.3 points; 51.6% to 44.3%).
Obama's final margin -- as of this writing and based on final results in each state -- turns out to be urned out to be 7.25 percentage points (52.9% to 45.7%), within a whisper of the final IBD/TIPP projection and the result that all other polls, collectively, had been showing for weeks (at the same time, the actual 1.2% support for candidates other than Obama and McCain fell far short of IBD/TIPP's projection of 4.1% -- something not accounted for in their boast of "top honors").
However, the "house effect" that made earlier IBD/TIPP results more favorable to McCain was not lost on those producing the IBD/TIPP analysis. Here is the first line of their first poll release:
In contrast to other polls, which show Obama leading McCain by 4 points (Reuters/C-SPAN/Zogby) to 11 (Newsweek), the IBD/TIPP Tracking Poll debuts today with Obama up just 2 points with 13% (including 25% of independents) undecided.
They came back to this theme repeatedly. On October 22:
Contrary to other polls, some of which show Obama ahead by double digits, the IBD/TIPP Poll shows a sudden tightening of Obama's lead to 3.7 from 6.0.
And again on October 29, the IBD/TIPP analysis pointed out that other polls were "migrating" (briefly, as it turned out) to their closer than average result:
The race tightened again to 3 points Wednesday, a margin IBD/TIPP has shown for six days and to which other polls appear to be migrating. For example, the Rasmussen and Gallup polls, each of which had Obama up 5 points two days ago, now have him at 3.
The relative closeness of the IBD/TIPP polls was also repeatedly noted by supporters of John McCain (all quotes via Nexis transcripts):
- Bill Bennett on CNN, October 22: "Wolf, I don't know if you saw them or James, AP poll and IBD poll which had these races much closer and in some of these states had McCain ahead."
- Hugh Hewitt on CNN, October 23: "I know they're not covered much in the media, but if you look at the Investor's Business Daily poll today, the poll, it's down to 1 point. Yesterday, it was three point seven."
- Sean Hannity on Fox, November 3: "Investors Business Daily, which was the most accurate poll in the last presidential election, now has it down to a two-point race."
- Mary Matalin on CNN, November 4: "[The McCain campaign is] looking at their polls, which are better than the public polls and do match like the IBD poll, all those that are out there but in the target states where they need to close the gap...this is [a] harsh [environment], as you're saying, but certainly not undoable."
Not surprisingly, many conservative bloggers relied on the IBD/TIPP polls to support the same conclusion, namely that the race was closer than other polls made it appear. Ironically, to strengthen that argument, many pointed to IBD/TIPP's boast of being the "most accurate" pollster in 2004, a message that appeared prominently alongside each day's results. "An analysis of Final Certified Results for the 2004 election," they wrote, "showed IBD's polling partner, TIPP, was the most accurate pollster of the campaign season."
TIPP's final projection in 2004, showing a 2.1 percentage point lead, did come closest among the final national polls to the ultimate 2.5 percent margin, although it is worth noting that IBD was one of the few to report results to one decimal place. Four other pollsters showed Bush leading by margins of 2 or 3 percentage points on their final 2004 poll. Since they rounded off their results to the nearest whole number, we cannot say for certain who would have been "closest," although reporting results out to one decimal is largely meaningless for individual polls given margins of error of 3 percentage points or more.
The same can be said this year: Seven pollsters other than IBD/TIPP reported final results or projections that rounded to Obama leads of 7 or 8 percent. Again, IBD/TIPP can claim to be "closest" to the final margin mostly because they chose to report results to one (largely meaningless) decimal place. Other pollsters may have been just as close. Moreover, remember random sampling error. How close a pollster comes to the illusion of "pinpoint accuracy" on any given survey (or even on any two surveys) is still largely a matter of chance.
But the issue raised here that many will wonder about is the allocation of two-thirds of the undecided voters to Obama, especially given the analysis I linked to yesterday by Mickey Blum about the so-called "convergence mystery." She noted that allocations that moved estimates closer to the poll averages helped explain the sharply decreased variance among the final polls that David Moore identified (see also my commentary). So what do we make of the fact that the IBD/TIPP allocation produced a result converged almost perfectly with the poll averages?
First, in their defense, IBD/TIPP is certainly not the only pollster to wait to allocate undecided voters until their final release. Gallup has followed that practice for decades, and five other national survey organizations did the same this year.
Second, Raghavan Mayur, president of TechnoMetrica, explained in a comment posted here yesterday that they used a "Hierarchical Heuristic algorithm" in both 2004 and 2008 that had been developed using their data for the decided. Such algorithms create mutually exclusive clusters of voters based on combinations of variables that, in this case, maximize the difference across those subgroups in terms of support for Obama or McCain. They divide up the decided electorate, creating some subgroups that are very likely to support Obama, some that are very likely to support McCain and others that fall somewhere in between based on what Mayur describes as "dominant demographics" that predict candidate support.
In response to my email query last week, Mayor provided more details on the workings of their algorithm. In this case they created groups of clusters based on combinations of seven variables (race, party identification, age, income, self-reported ideology, 2004 vote and religion). Once they had used those variables to create segmentation based on the decided, they developed allocation rules based on the demographic segments (and, presumably, on the level of support for Obama and McCain in each segment) that they used to allocate each undecided respondent.
So Mayur's explanation is essentially that their allocation was based not on a gut hunch or an arbitrary "rule" based on past experience but rather a statistical model based driven by individual level data. He also sticks doggedly to the initial IBD line that other pollsters had it wrong:
Other commentators questioned why we often had the Obama-McCain race tighter than other polls. The fact is, the race was tight right down to the last week, when undecided voters swung to Obama. The final 7.2-point spread was the widest we showed since our polling started Oct. 13.
Unfortunately, I find no evidence anywhere else to support the contention that two-thirds of undecided voters "broke" to the Democrat in either 2004 or 2008. Not in the exit polls. Not in comparisons between other polls and the final results. Only the IBD/TIPP poll achieved "awesome results" (as Mayur put it in an email), by assuming a 2-to-1 break to the Democrats in 2008 and nearly as much in 2004. More specifically:
- The average or trend estimate of the other national polls (Obama +7.6 on both our final trend estimate and the RCP final average) was less than four tenths of a percent from the actual result.
- Of the seven pollsters that allocated undecided voters for their final projection, IBD/TIPP was the only one that allocated more to Obama than McCain. At least one of those -- the Pew Research Center -- used regression modeling of the decided to inform their allocation, which favored McCain slightly. Charles Franklin and I also did a similar application of a model developed among the decided in early October and found a near even split among the undecided.
- To justify an undecided percentage on his final poll (9%) that was higher than others (our trend estimate showed 5-6% undecided for much of October), Mayur reminds us that according to the exit polls, "the share of electorate deciding in the last weekend in 2004 was 9%, and in 2008 was 7%." True. However, those same exit polls show the same late deciders dividing evenly in 2008 (49% Obama, 48% McCain), and slightly in Kerry's favor (53% to 44%) in 2004.
There is also some confusion about the number of variables used to create the algorithm that allocated the undecideds. When the Wall Street Journal's Carl Bialik asked Mayur about it shortly after the election, Bialik reported that TIPP's allocation "method uses 10 variables, such as the respondent's party, gender and 2004 vote, to predict the 2008 vote." When I asked Mayor to specify the 10 variables, he listed only seven (race, party, age, income, ideology, 2004 vote and religion), explaining that "we also look at gender but do not use it in the model." However, the document he attached to the same email has one listing of classification rules that does not include the age variable, and a listing of computer code that does (Mayur asked that I treat the additional details contained within that document as "strictly 'confidential" so that I would not "give away our 'bread and butter' to other firms").
The point here is that the underlying model used to allocate undecided voters still leaves a lot of room for subjective judgements. How many variables are used to create the segmentation? Which subgroups from that segmentation are used to allocate undecided, and which are not? What is the criteria that determines whether a given subgroup gets allocated to Obama or McCain?
Writing here yesterday, Mayur offered a "moral" of their allocation story:
1. Close to ten percent of the electorate makes decision in the final weekend (Saturday, Sunday, and Monday).
2. They don't break even. Democrats have a clear advantage among them, at least in the past 3 races.
Respectfully, those are the wrong lessons. Here is what we have learned:
- We see a conspicuous disparity between IBD/TIPP's final projection and the results released earlier in October.
- The accuracy of the final poll projections from IBD/TIPP are useful, at best, only in evaluating subsequent final horse race projections from that organization, not the results that come earlier.
- The polling industry, and those of us that follow it, place far too much emphasis on "the last poll" in judging poll accuracy and reliability.
Finally, let us be clear that we are in no position to discern an inappropriate motive on the part of TIPP or Investor's Business Daily. The more important issue here is what we make of the results they produced.
PS: I asked Raghavan Mayur for his comment on my argument that we put too much emphasis on final polls generally, and on the final IBD/TIPP polls specifically, in measuring survey accuracy. His full response appears after the jump.
Continue reading ""Hitting a Bullet with a Bullet" - A Cautionary Tale"
CNN / ORC
12/19-21/08; 1,013 adults, 3% margin of error
Mode: Live Telephone Interviews
In general, do you think there is too much, too little, or about the right amount of government regulation of the stock market and financial institutions?
22% Too much
59% Too little
18% Right amount
As you may know, a financial advisor named Bernard Madoff has been accused of running an illegal scheme in which many people lost millions of dollars. Do you think Madoff is an isolated case that does not reflect on other financial advisors and institutions, or do you think Madoff's behavior is common among financial advisors and institutions?
23% An isolated case
74% His behavior is common
12/17-21/08; 834 registered voters, 3.4% margin of error
Mode Live Telephone Interviews
New York State
As you may know, Senator Hillary Clinton has been nominated to be Secretary of State and Governor Paterson will pick her replacement in the Senate. Who do you think Governor Paterson should pick to replace Hillary Clinton in the United States Senate; Caroline Kennedy, Andrew Cuomo, Kirsten Gillibrand or someone else?
24% Someone else
Do you think Caroline Kennedy is qualified to be a United States Senator or not?
DailyKos.com (D) / Research 2000
12/15-17/08; 600 likely voters, 4% margin of error
(400 likely Republican primary voters, 5% margin of error)
Mode: Live Telephone Interviews
Alaska 2010 Elections
General Election: Palin (R-i) 55%, Knowles (D) 38%
GOP Primary: Palin 55%, Murkowski (R-i) 31%
General Election: Palin 53%, Knowles 39%
General Election: Murkowski 49%, Knowles 41%
GOP Primary: Young (R-i) 33%, Parnell (R) 27%, Harris (R) 11%
General Election: Young 49%, Berkowitz (D) 46%
American Research Group
12/16-19/08; 1,100 adults, 3% margin of error
Mode: Live Telephone Interviews
Pres. Bush Job Approval: 28% Approve, 67% Disapprove
CNN / ORC
12/19-21/08; 1,013 adults, 3% margin of sampling error
Mode: Live Telephone Interview
The federal government will provide some of the major U.S. auto companies with more than thirteen billion dollars in loans in order to prevent them from going into bankruptcy. In exchange for the loans, the government wants the auto companies to produce plans by the end of March that show how they would become viable businesses in the long run. Do you favor or oppose this decision?
If the major U.S. auto companies ask for more money next year, do you think the federal government should give them any additional assistance, or should the government let them go into bankruptcy?
28% Additional assitance
70% Let them go into bankruptcy
How would you rate Dick Cheney in comparison to other vice presidents in U.S. history? Would you say he is the best vice president the U.S. has ever had, very good, good, poor, very poor, or would you say he is the worst vice president the U.S. has ever had?
4% Very good
13% Very poor
CNN / ORC
12/19-21/08; 1,013 adults, 3% margin of error
Mode: Live Telephone Interviews
As you may know, Illinois Governor Rod Blagojevich has been arrested on charges of corruption. Based on what you have heard or read, which of the following statements best describes your view of any recent contact between Blagojevich and top aides to Barack Obama:
12% Some Obama aides did something illegal
36% No Obama aides did anything illegal, but did something unethical
43% No Obama aides did anything seriously wrong
ABC News / Washington Post
12/11-14/08; 1,003 adults, 3% margin of error
Mode: Live Telephone Interviews
Apart from dealing with the economy, do you think Obama should or should
... withdraw U.S. forces from Iraq within the next 16 months?
26% Should not
... make major changes in the U.S. health care system?
20% Should not
... implement policies to try to reduce global warming?
20% Should not
... close the U.S. military prison at Guantanamo Bay in Cuba?
44% Should not
... expand federal funding of embryonic stem-cell research?
42% Should not
Survey Practice, AAPOR's online publication, has a new issue out that includes a follow-up to David Moore's provocative piece on what he dubbed the "convergence mystery." Moore observed that the national polls often produced "contradictory estimates and trends" -- and more specifically, a greater variance of results -- that significantly converged over the last few days of the campaign. I posted a follow-up here that showed a similar pattern in state level polling and posed a theory that might explain some of the convergence, and Moore responded with more data and a different theory.
In their new issue, Survey Practice publishes responses from five survey researchers who deserve the title "expert:" Paul J. Lavrakas, Michael Traugott, Micheline Blum, Cliff Zukin and Don Dresser. David Moore follows up with a new article of his own that begins with this helpful summary:
The two major types of explanations offered by our experts for the convergence of polls right before the election are based on 1) changes in the pollsters’ methodology and 2) changes in the certainty of the vote choice.
The first explanation suggests that in the final weeks of the campaign, many pollsters adjust their likely voter models (mentioned by Lavrakas and Blum) or they increase their sample sizes (Dresser).
Lavrakas argues that the adjustment of the voter models, even when done explicitly to make their outcomes more consistent with other polls, should be seen as a positive action rather than as a “suspicious” activity. However viewed, such last minute changes could account for some of the convergence.
Dresser mentions the tendency of many pollsters to substantially increase their sample sizes for their final pre-election polls, to insure as small a margin of sampling error as they can reasonably afford. He specifically mentions Pew, Harris and ABC/Washington Post.
Blum suggests that the outlier polls during the campaign were either a) less likely to poll in the final three days, or b) more likely to allocate their undecided voters, thus bringing them closer to the mean.
Zukin speculates that the convergence has less to do with pollsters’ methods and more to do with the “phenomenon we are measuring.” Specifically, as voters become more certain about their choices, polls will tend to converge toward each other. That sentiment is also found in the explanations by Lavrakas and Blum.
Moore goes on to offer his own thoughts and asks why polls, "conducted at the same time, using virtually the same wording - [are] supposedly more accurate (reliable) when opinion is more crystallized?" For those intrigued by the "mystery," It is all worth reading in full.
Among all of these efforts to explain, I was most intrigued by Blum's analysis, which was the only one of the five to scrutinize the poll data directly. Among the national polls, at least, much of the variance in mid-October can be explained by a small handful of "outlier" results two-weeks out from the election:
Of the six organizations with outlier polls, three reported margins larger than 11 points, and all three consistently showed larger margins in their polls. In the final three days, however, one of these organizations allocated the undecided, giving more to McCain, one did not release a poll, and one had a margin of exactly11 points on its final poll. Of the three organizations releasing four polls with margins smaller than 5 points, one organization (with 2 “outlier” polls) allocated the undecided in the final three days, one did not release a poll in the final three days, and one “converged.” So, of the 6 organizations, 2 organizations (accounting for 3 “outlier” polls) allocated undecided in the final three days in the direction of the previously underestimated candidate, 2 did not release polls in last 3 days, and only 2 “converged.” If we remove the 7 “outlier” polls from the 29 released in the week of 10/21-27, the variance is reduced to only 2.9 points.
Basically then, both of the explanations examined, the allocation of the undecided by seven organizations in the final three days and the absence or favorable allocation of a few “outlier” organizations, appear to be major contributors to the “convergence” seen. Apportioning the undecided in the favorable direction and the absence of previous outliers virtually guarantees less variance and the appearance of “convergence. So, perhaps, rather than convergence, what we saw was that much of the earlier variance was due to a few outliers and that the final three days benefited from their absence or favorable apportionment of their undecided vote.
Blum's observation prompted me to take a closer look at the national polls that allocated the undecided. I find seven projections by six organizations (the two pollsters for the GWU/Battleground poll, Republican Ed Goeas and Democrat Celinda Lake, once again produced competing projections with different allocations). The process of allocating undecided should not, by itself, raise any suspicion. Of the seven allocations, three changed the initial margin not at all or within rounding difference of the averages, two moved the margin farther away from the poll averages as of Monday 11/4, and one moved it closer. However, that one move -- by the IBD/TIPP poll -- was important in helping produce the reduction in variance that Blum describes.
USA Today / Gallup
12/12-14/08; 1,008 adults, 3% margin of error
Mode: Live Telephone Interviews
Which of the following would you say best describes the current economic situation in the United States?
60% Biggest crisis in lifetime
16% Crisis, but not the worst
18% Major problem
4% Not a major problem
How likely do you think it is that the United States will be in an economic depression within the next two years?
21% Not too
4% Not at all