Paul Horwitz:

So what results does he get with his Kalman filter approach. Come on, don't leave us in suspense!

Posted on November 2, 2006 8:48 AM

 Nathaniel Lichtin:

The results he finds don't seem to match reality. Less than 1% chance of maryland going republican or 1.5% for New Jersey or only 14.5% for montana. These races are closer than the results he finds.

Posted on November 2, 2006 9:30 AM

 Benjamin Schak:

Yeah, the Kalman filter is a good method. I don't have time to do any election prediction this cycle, but it'd be my basic tool if I did.

I used a multivariate Kalman filter (the 51 variables being the 50 states and DC) to predict the 2004 election. That had the benefit of allowing measured swings in states that get polled to influence one's predictions of results in states that don't get polled.

I've never gotten around to writing up what I did, but here are a couple results I remember off the top of my head: 1) I discovered several months before the election that OH would be the state most likely by far to produce a situation like FL in 2000 (with WI as a distant runner-up). 2) I discovered several months before the election that Kerry was far more likely than Bush to win the electoral college while losing the popular vote, so I wasn't the least bit surprised when that almost happened. 3) I called every state except WI correctly. (And I blame WI on the preponderance of biased numbers from the Badger Poll.) 4) I correctly predicted both candidates' shares of the two-party popular vote with some insanely low margin of error. (I think it was something like 0.15%.)

One question for you about an issue that I struggled with: In its basic form, the Kalman filter requires you to assume some value for the additional daily variance that accrues as polls grow stale. How much variance do you add per day, and why? (I see someone above criticized your results as being unreasonable. This might be because your daily added variance is too low, particularly towards the end of the race.)

Posted on November 2, 2006 9:49 AM

 Jacob Eisenstein:

Nathaniel -- One very important caveat is the "percent chance of winning the election" numbers assume that the election is held TODAY. The method does not predict future events. If things break their way, the Republicans can come back and win in New Jersey. But as of today, something like nine of the last ten polls in New Jersey show Menendez ahead -- so if the election were held today, I'm very confident that he'd win.

Posted on November 2, 2006 10:30 AM

 Jacob Eisenstein:

Benjamin --

I think it's a really good point that variance in support increases towards the end of the race, since that's when people are paying attention to politics. I haven't figured out how to model this yet, but it's something I plan to think about after the election. What did you do?

Posted on November 2, 2006 10:37 AM

 Eric Applegate:

How well would this approach have modeled past elections? For instance, using poll data from 1-2 weeks before the election, what would the results have been for the Kalman filter, versus the actual results?

Posted on November 2, 2006 11:32 AM

 Bruce Caswell:

In response to the second poster, this is not a forecasting technique, it is a more precise way of describing the interpreting the results of the "snapshot" of the election taken by the polls. Hence, the "1% chance of the Republican winning in Maryland" means that there is only a 1% chance of the average of the polls being wrong at this time. It is a statistical statement on who is winning now, not who will win on election day. There are many things that could happen between now and election day that could change the final outcome.

Posted on November 2, 2006 12:53 PM

 Amit Lath:

If bias is indeed hardwired, say pollster A always favors Democrats, then your analysis should be able to measure and correct for it. You can compare his output to actual (previous) election results. Of course pollsters can change their methods after the reality check of an actual election, but hopefully this is a second order effect.

Posted on November 2, 2006 3:14 PM

 Alan:

I wrote Jacob offline, but I'll repeat my comments here and add an additional remark.

That is a nice piece of work and puts the social scientists to shame ;-) (I'm also an engineer, so if you are a social scientist, please don't take the comment seriously :-)

The number that's currently called "the chance of winning each seat" is really the chance of winning that day's unbiased poll. If you really wanted to use poll figures to estimate the probability of winning an election, I think you could/should add a "fudge factor" to account for the differences in what happens in a poll vs. what happens on election day. In other words, if you could poll every voter, your sampling error or so called "margin of error" would be zero, but you still would not know with 100% certainty what the outcome of an election would be even if it were held the same day. I guestimate that the difference between a well-done poll and actual election results in a senate race is something on the order of 2-4% ON TOP OF the sampling error. Therefore, I think you would get a better estimate of the chance of winning each seat if you added this "fudge factor", i.e., use 6-8% as the uncertainty of each poll, rather than 4% (the margin of error). What do the real social scientist think about this number? Is an additional 2-4% too much or too little?

Also, the little "ramps" in the graphs from one day to the next are a little strange. In reality the changes in the estimates reflect information impulses. The ramp comes from drawing a straight line from one day's estimate to the next, as if the time axis were continuous. In fact, its not continuous, its discrete (sampled) with a sample period of one day. The actual model and how it works would be more clear if you drew the center lines using only horizontal and vertical line segments, i.e., make each change a step up or a step down, not a ramp. Similarly, the 95% confidence intervals could be drawn using continuously widening cones with the wide ends cut "flat", i.e., drawn with vertical ends.

Alan

Posted on November 2, 2006 4:25 PM

 Alan:

P.S. You could in fact get an estimate of what might happen on election day if you ran the Kalman filter forward to Nov 7 without any new inputs. The cones would widen, reflecting greater uncertainty over what might happen in the future, but I believe the math behind the numbers would continue to be valid. This would work as long as the cones stay near the center of the graph, because of course, the cones can never extend below zero or above 100, so the linear math would break down if you tried to run the filter too far into the future.

Posted on November 2, 2006 4:34 PM

 Amit Lath:

Alan, shouldn't a well done estimate (poll) come closer than 2-4% of the actual measurement (election)? Rather than take this "fudge factor" as an additional error, one could figure out the mechanism and correct for it (and of course, there will be some uncertainty associated with this correction factor, but hopefully it will be less that the correction itself).

My thought on getting these correction factors was to look at individual polling firms and how far off they were in 2004.
Maybe that's not the best idea for estimating this factor, but I find it curious that polling firms that are known to be off don't pay a higher price in terms of error bars in the final fit.

Posted on November 2, 2006 4:58 PM

 Jacob Eisenstein:

Amit --

Just to be clear -- the model already corrects for pollster bias. It does this by counting pollster bias as a source of "error" in polling results, and correcting for it. A poll will be assigned a high bias if its results are consistently different from those of other polls.

Considering previous election results might be a more accurate way to do this, but as you say, there is a feedback phenomenon.

Posted on November 2, 2006 5:02 PM

 Alan:

Hello Amit, I'm assuming the factor is random, unknown or unmodelable, so you can't "correct" for it but you can account for it. One way to determine the factor might be to run the Kalman filter against past elections and determine what factor best accounts for the variance in the difference between the Kalman filter's best estimate and the actual election results. This would also be a good check of the Kalman filter: if its a good model, then the mean difference should approach zero.

Posted on November 2, 2006 6:10 PM

 Amit Lath:

Thanks Jacob. From your original post and your webpage I understand that bias is estimated by scatter (by the way, dangerous, no? the guy out on the tail could be the one guy who's actually managed to phone the factory workers on the owl shift...)

But never mind all that. However you estimate bias, do you correct the reported values, or do you take the bias and blow up the error bars so they have a bigger chisquared/lower weight in the overall fit?

I do think you are doing pretty much the right thing here. But one foolproof way to check you get bias/correction factors right is: apply the corrections and the variance gets smaller. But if you are actually using the distance from the mean to estimate the bias, then by definition the scatter is going to be lower after corrections. If you did have another way to estimate bias, you could see how well it worked by looking at what it did to the variance, no?

Of course, if EVERYONE is biased one way, the
this correcting by looking at the scatter fails overall.

Anyway, thanks for replying. I was course 8 many moons ago but 6.002 and 6.003 were some of the most fun courses I took.

Posted on November 2, 2006 6:28 PM

 Amit Lath:

Thanks Alan. It is a good idea, running Kalman on past elections. So I am sure Jacob must have done that (preprints?) One could then ask how much of the change in the numbers is due to a real time dependence and how much is scatter due to bias.

But assuming the correction is random is kind of a model. Then the mean of N polls will converge to the right point, modulo the time dependence that Jacob is accounting for with the Kalman.

But what if it is not random? Here is a toy model: the scatter is due to a pool of voters that all vote one way (say Republican) that various pollster sample with different efficiencies. This is a simple, one-parameter model, but no way to get at it by looking at the scatter. Looking at real elections you may have a shot.

Of course, to make things more complicated, I gather most if not all of the pollsters correct their data for numbers of R vs D, men vs. women, old vs young, and probably don't even add the systematic uncertainty from this correction to their numbers.

Posted on November 2, 2006 7:09 PM

 ekg:

In your attempt to model "momentum", do you explicitly account for the implied preferences of the undecideds who become "decideds" between one poll and its immediate follow-on? i.e. if prior undecideds seem to breaking for one candidate over another, is this explicitly considered? There are some polls out there (e.g. IL Governor, ID Governoe, AK Governor) with well over 10% undecided at this late stage.

Posted on November 3, 2006 10:09 AM

 Jacob Eisenstein:

Hi ekg. The model pretty much ignores undecideds -- it just considers who's ahead among decided voters, and how that lead compares to the variance. You could say that this is equivalent to assuming undecided voters will break 50/50. It would be interesting to look at all the ways in which election results deviate from polls; for example, that undecided voters usually break 2:1 for the challenger. Maybe in 2008.

Posted on November 3, 2006 1:13 PM

 Ian Valenzuela:

Wow, it seems as though each and every prediction was on target. Interesting to note, skeptical reactions to Maryland and New Jersey projections was misguided; the races there were much less close than conventional wisdom allowed; whereas tight races in MT, MO, TN, and VA were also predicted, as were the eventual winners (assuming Webb's lead remains). My advice is take these prediction results and market yourself to the media in the next election cycle.

Posted on November 8, 2006 4:54 PM

 Benjamin Schak:

Jacob--

I did something fairly ad hoc. I think I took whatever my assumed daily variance was (actually, in my case it was a variance/covariance matrix since I used a multivariate version of the KF) and just multiplied it by factors during the week before the election. Essentially, I ended up treating 10/28 as if it were two days, 10/29 as if it were three days, and so on. I think I boosted the assumed variance on some other days too, like the major parties' conventions.

Congratulations on your good results!

Posted on February 23, 2007 3:55 PM