Articles and Analysis


First Impressions: A Good Day for Averaging

Topics: 2006 , The 2006 Race

Despite exhaustion and sleep deprivation, we want to take a few minutes today to a very quick and very preliminary look at how the preelection polls did as compared to yesterday's results.  Since some precincts are still out and some absentee and provisional ballots are still being coutned, this quick looks is inherently preliminary and subject to change, but at the statewide level, the average of the last five polls in each races did reasonably well.  In every case that we have examined so far, the leader in the average of the preelection polls was the leader on election day. 

The following table includes only the most competitive Senate races that we tracked for the Slate Election Scorecard.  It shows the curernt unofficial result in each state as compared to our final last-5-poll average.  Since the preliminary results we gathered had been rounded to the nearest whole digit, we did the same with the final average.  Again, every leader in the polls ran ahead yesterday.


[Note:  For brevity's sake, the table above displays the results for Joe Lieberman in the Republican column, although Lieberman ran under the "Connecticut for Lieberman" party and has pledged to caucus with the Democrats in the Senate].

The list of the most competitive Gubernatorial races shows the same pattern.  While the averagse did not predict the winners perfectly, the leader in the prelection polls was the leader on election day in every case.



[Update:  The original version of the above table omitted the Minnesota Governor's race, which as several commenters noted, is the one state where the nominal leader in the averages was not the winner on election day.  My apologies for the omission -- more details in a comment below.

Averaging results is obviously an imperfect solution to pre-election poll variation.  The outcome in many races was off the "last-5-poll" average by as much or more than the Minnesota Governor's race:  The Pennsylvania Senate race, the Maryland races for Senate and Governor, and the races for Governor in Alaska and Michigan all featured results differing from the final average that were as large as Minnesota].

We hope to have a far more comprehensive analysis in a few days looking at more races and using vote return data that is closer to complete.  And these comparisons obviously make no effort to allocate undecided voters or use any of the more sophisticated measures of survey error.   But for now, the bottom line is that the last-five-poll averages gave a pretty good impression of the likely outcomes of each of these competitive races.




Thanks for all your hard work on this. The polls seem to predict the Senate results really well. Which brings me to my question. A few days ago you posted a neat graph of actual percent of elections won for each polling vote share based on the 2000 and 2002 Senate, governors and state level presidential polls. In that graphic, there is one extreme outlier, which is that in Senate races where the Dems were ahead by 13 points or so, they won only 1 out of 3 times (or maybe 3 out of 8 times). What is that race? Is it just one race, or several?




Well, I'm from Maryland, and I'm not convinced. It looks to me like the margin in both Maryland races was seriously underestimated.


MattF -- I noticed the same anomaly in Maryland but jumped to two different conclusions: 1) the race effect (see Tom Bradley, Harvey Gantt) is still in effect and black candidates underperform on Election Day compared with closing polls (a factor not borne out yesterday in Tenn Senate, Ohio Gov and Mass Gov) or 2) the Democrats had a helluva GOTV effort in Maryland (a factor that is borne out by Md Gov).

Congratulations to pollster.com. A magnificent effort this fall.


PS. should have added Pa Gov as another race with no apparent race effect.


Nadia Hassan:

I was wondering about polling certain sub-groups of the electorate. Is it difficult? For example, several times people brought up McCaskill struggling with black candidates, but she wound up winning them by 90%, and prognosticators suggested that she would need in the mid-80s. Some of the polling, even that showed her ahead had Talent above 15%.

Earlier, I also saw a Newsweek poll that showed Democrats up +2 with White Catholics, but Pew had a double digit advantage.



um- is there a reason that Minnesota isn't in the Governors table- other than it would be the only one which didn't follow the predictions? :)


The biggest difficulty is sample size and margin of errors. It seems like the poll that had Talent at 15% amongst blacks probably would have had a HUGE moe.



Kudos to everyone associated with this resource. Exceptional job.

Has anyone determined the total number of votes cast for Democratic Congressional candidates, and how this figure (%) translated into seats. I curious as to whether the recent UberGerrymandering has negatively impacted the Democratic Party's opportunity to gain seats.




Kudos to everyone associated with this resource. Exceptional job.

Has anyone determined the total number of votes cast for Democratic Congressional candidates, and how this figure (%) translated into seats. I curious as to whether the recent UberGerrymandering has negatively impacted the Democratic Party's opportunity to gain seats.




You have averaged the democratic and republican differentials to obtain an average. This is like putting one foot on the stove and the other on the refrigerator to be comfortable. You need to take the square of both numbers, add them, then take sq root to get a better assessment of accuracy.



Hey what happened to your polls in the New Hampshire congressional race involving Bradley and Shea Porter. She won and your polls have her behind on average by 12 points.


Jerome Armstrong:

Good stuff. But MN is where it didn't work for the Gov races.




If you run the numbers, you'll find doing the mean of the squares or the mean of the absolute-value doesn't make much of a difference, especially since there are so few negative results.


Mike Rappeport:

I redid your analysis for the senators, but using polls from the beginning of October instead of the last five polls (i.e. polls primarily in November a month later). Specifically I used the first five polls which started on October 1 or later from Real Clear Politics (except I omitted all Zogby interactive polls as obviously unreliable at any time). When I did this I found that there was essentially no difference in predictive ability between using the last five polls or using the beginning of October polls.

Chart shows
Average of last five polls (i.e. repeats your analysis)
Average of first five October polls
Actual result from your table
Noe showing comparison of results

State Last 5 First Oct 5 Actual
Ariz -7 -8 -9 Oct 1% closer
Conn -12 -12 -10 no difference
Mary 3 5 10 Oct 2% closer
Minn 17 19 20 Oct 2% closer
Mo 2 3 3 Oct 1% closer
Mont 3 6 1 Final 3% closer
NJ 6 5 8 Final 1% closer
Ohio 12 9 12 Final 3% closer
Penn 10 10 18 No difference
RI 6 6 6 No difference
Tenn -6 0 -3 No difference
Virg 1 -2 1 Final 3% closer

Average deviation of final 5 from actual =2.5%
Average deviation of first 5 October from actuaql = 2.83%
Final closer in four caases
First five in October closeer in four cases
N difference in four cases

I conclude from this that net effect of October campaigns, at least in these senate races, is very close to nil. I cannot tell whether that is because campaigns make no difference at all, or campaigns cancel each other out.


Rick Brady:

I think someone should cherry-pick three races (say, Maryland & Penn Senate and Mich. Gov), compute the 1-tailed probability of the deviation between the tried and true pollsters and the "official vote count", generate some insane odds ratio, and then write a paper (then a book) about how the Dems stole the election and the polls prove it.

Ok... I'm a cranky Republican trying to have a little fun...


Gary Kilbride:

I'm usually pretty good at figuring things out. Let me take a stab at Minnesota governor not being listed.

Last night CNN screwed up and left that race out of its convenient one-page lineup of the competitive gov races. It was frustrating all night, to the point I had to keep a separate windown open with that race alone. I emailed CNN several times but they might have been otherwise busy. Likewise, the Idaho gov race was omitted by CNN. That one had infrequent polling but was expected to be close, certainly tighter than it finished.

I'll take a flyer that Mark Blumenthal used CNN as the source for the margins and Minnesota was left out simply because CNN missed it. It wasn't hide-the-innacuracy.

The Maryland gov margin was surprising, one of the pleasant surprises of the night. If Ehrlich had been close and Steele trailed his poll consensus badly, I would have full confidence it was the mustache factor with Steele. Men with mustaches always lag their poll numbers by considerable margin. I've mentioned that on sites for years. In 2002 it was a four bagger, two senate nominees and two senate candidates, two Ds and two Rs and all losing and trailing the poll numbers. It was Kirk in Texas and Strickland in Colorado, plus the Republican gov nominees in Wyoming and Kansas. No examples in 2004, at least none I remember.

I was very relieved when Democrats nominated Cardin instead of Mfume, since Mfume had a mustache also.

I realize it sounds bizarre, but that goes back to Jimmy the Greek Snyder's theory, based on a poll, that women did not prefer men with mustaches and would not vote for them, even if subconsciously. He used it to make a huge wager on Truman over Dewey in '48. The late Paul Wellstone shaved off his facial hair in 2002 when the senate race against Norm Coleman became surprisingly competitive. I doubt Wellstone's sharp move was coincidence. Someone on MyDD posted the specifics regarding major statewide and national office holders with facial hair. It's extremely low, certainly below the national average.

Every Republican needs a mustache. I cheered Steele the entire campaign. Clever puppy commercials yet he didn't realize it was negated and then some, by a hairy handicap on his upper lip.

I did okay ($) on Cardin.

Of course, since Ehrlich in the same state similarly zig zagged in terms of polling and final result, perhaps there's another explanation. But not as fun and I'm not throwing away the mustache factor until it repeatedly fails.


Gary Kilbride:

Oh damn, I just reread the thread and saw the criteria, the races tracked in the Slate Election Scorecard. Maybe Minnesota was missing on that list since it popped up late in the cycle as competitive.

I liked my CNN theory better.

At least it allowed an excuse to throw in the mustache bit:)


In principle, when calculating the average one ought to weight the individual polls by the inverse squares of the quoted margin of errors. One ought also calculate the average's margin of error as the inverse square root of the sum of the inverse squares of the individual polls.

The first, I imagine, has little numerical impact on the results. The second, however, ought to give a good answer to the following question: are the uncertainties primarily statistical (i.e. w/in the margin of error), or primarily systematic?

I suspect you'd find the Pearson chi-squared for the whole set is absurdly large. (Don't forget the various factors of two that come from the convention of defining the MOE as 95%, and from correlated uncertainties on the difference.) A large value for chi-squared (and corresponding tiny probability) indicates that systematic effects (LV modeling, differential turnout, etc) dominate the statistical uncertainty.



About MN - I don't understand why it was ever off the radar. It was within about 5 points in the pools from June through the election, yet no one ever seemed to pay attention. Any thoughts as to why? It did end up the closest gov. contest of the night, after all.


Mark Blumenthal:

It's embarassing to admit it, but Gary's second theory was the right one. That and doing this post yesterday in very little time and on very little sleep.

Back in August when we put the Slate feature together, we took some educated guesses about which Governor's races would be most competitive. The embarrasing part is that Minnesota wasn't one of them. I kept a spreadsheet during the campaign that updated the average for each of the close races (plus a few more that the Slate editors cut at the last moment when the feature started running). Yesterday, I just grabbed the results for each state in my spreadsheet, cut a few (like Washington) where results were still incomplete and and posted the results.

I'll confess that everyone else, I paid far much more attention to the Senate races than the contests for governor. And while I knew that Minnesota had shifted to toss-up status, I never added it to the sheet.

So yes, although we did classify the state as a toss-up, the Minnesota Governor's race was the one excpetion where the nominal leader in the averages was not the leader on election day.

The irony is that about 20 minutes after posting this item, I got a call from Sharon Schmickle at the Minneapolis Star Tribune who was working on a story that appears today about this particular exception.

Incidentally, my final quote in that story was more about the ability of polls to predict a winner when the margin is a close as it was in Minnesota: 23,000 votes out of over 2 million. Polls can tell you the margin is close -- as they did in Minnesota -- but accurately forecasting the winner is tough.


Joel Rabinovitz:

About the Maryland polls: On October 29, the Washington Post published a poll showing the Dem candidates for Senate and Governor with 9 and 10 point leads, while the Baltimore Sun two days later showed the races with 1 point Dem leads. On November 2, the Post published an article explaining that the difference was due to the expected black turnout in the state: the Post estimated black turnout at 25% of all voters (above the historical 22%), based on the survey respondents� answers; the Sun�s poll estimated it at 19%. Looks like the Post got the final results about right. It would be interesting to see what the other Maryland polls (all showing narrow Dem leads) estimated black turnout at, and if that is why they were off.


Rick Matland:

After you've had some sleep, it would be interesting to know how the various polling firms did and especially how Zogby's internet polls did and how the automated calling polls did. Still it's safe to say that while the results look impressive, a move of just a few thousand votes in Montana and Virginia and suddenly the results are not as overwhelming. I think your previous post translates to "we can tell if someone's going to win if they've got a solid lead or we can tell if it's going to be close, but when it is really really close we need to rely more on luck than skill to get it right".


Does anyone know if the nationwide popular vote is posted online anywhere yet?



Thanks for the great site throughout the election.

And, to second a previous comment, what did happen to Minnesota Gov on your chart -- that one was a problem for everyone it seems.

On another note, my impressionistic memory is that for a decade or so polls in both New Jersey and Maryland are consistently skewed Republican, compared to actual results. A column on this -- is it true? if so, why? -- seems right down your alley.



Michael, I asked that very same question yesterday.

According to a WSJ story today, the Edison/Mitofsky exit poll indicates that 53% voted for the Democratic candidate. If we then surmise that 47% voted for the GOP, it results in a 6% point spread.

Here's the link to the WSJ story.

Vote Is a Blow to Republican Pursuit of Hispanics



J Gibson:

The Maryland results are curious-- but notice that in both cases, they hit the Republican number just about on the head. Is it possible the late deciders broke really heavily to the Dems?


Thank you for your work. With the closeness of your polls to the actual results it tells me that the Bush crime family stole the 2004 election. I have seen other polls in 2004 showing Kerry leading in various states with paper ballots and voting machines. The final results in the states with electric voting machines got shifted to steal it away from Kerry, especially in Ohio.

Well the Republicans couldn't steal these elections. We need civil servants registering voters and keeping track of those registrations. We also need vote by mail with paper ballots.

We then get rid of electronic voting machines and polling places.



In MD there was a case of intentional voter misinformation as a last-minute GOP campaign tactic, with "Sample Democratic Ballots" being drawn up and distributed to likely Democratic voters. The sample/suggested ballots were accurate with listing Dems correctly, with the exception of the key close, high level races.

It wasn't an isolated or small endeavor, and post-election the GOP has acknowledged the action as being deliberate and planned for some time. While I'd normally expect "dirty politics" to be distributed on a bell curve and just cancel itself out by and by, this struck me as a bit of a new and potentially disruptive low. Between that, and the reports of misleading robo-dialing as a tool of dampening enthusiasm to vote, I wonder if there might be a correlation between a predominance of one-sided voter misinformation being conducted in some areas and related/caused departure from predicted poll results.


Post a comment

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.