Pollster.com

Articles and Analysis

 

Ford: Our Model vs. 538

Topics: modeling , Politics Home , UK elections

As noted yesterday, we are now following the PoliticsHome UK poll tracking and seat projection model developed by political scientists Robert Ford, Will Jennings, Mark Pickup and Chris Wlezien. I asked Ford if he could explain how their efforts differ from the model developed by Nate Silver and his colleagues at FiveThirtyEight.com. This is his response.
--Mark Blumenthal

How does our model differ from Nate Silver's recently unveiled model of UK elections? The very brief answer is that our model involves applying a modified version of "uniform swing" - the same change of vote in each seat, with some modifications - while Nate's involves proportional swing where the change in each seat relates to the balance of party power beforehand. Under Silver's model, we should see a greater swing against Labour where Labour start more strongly, and this effect should increase proportional with Labour's starting strength.

Empirically, there is little support for Nate Silver's conception of proportional swing, as shown in this recent paper by my colleague David Voas.

There is no evidence of larger swings in recent elections (including 1997) where parties start off more strongly. There is some evidence that swings are larger where the parties are competing more closely, but in our view Nate's model is a poor way to capture this dynamic.

We agree with Nate that there is plenty of evidence that a naive application of uniform swing is misleading, however we feel the best approach is to improve on uniform swing rather than abandon it entirely. Two major factors are seldom accounted for in popular applications of uniform swing. Firstly, uniform swing is generally applied deterministically, making no allowance for random variation in swing between seats. Secondly, it is applied too rigidly, making no allowance for systematic deviations identified in the data. We apply a probabilistic model, based upon a formula developed by John Curtice and David Firth for application in the 2005 General Election, where it was employed very successfully to project the result from exit polls. The model allows for a non-normal distribution in swing variations, and calculates a probability of each party winning each seat based on the vote shares expected (from opinion polls or exit polls). The seat totals are simply the sum of the probabilities.

This model also incorporates systematic differences in swing suggested by the polling data. We anticipate stronger Conservative performance in the marginal seats where they are competing directly with Labour by allowing an extra 2 points of swing to them in such seats. We also anticipate a different pattern of party performance in Scotland - which has its own government and a different party system - by incorporating the latest polling data estimates from Scotland, and adjusting the change in the rest of England and Wales to ensure the aggregate changes sums up the same. These adjustment are based on differentials which have shown up robustly in several recent polls of marginal constituencies and of Scotland

Nate also makes a variety of adjustments of this kind, but his changes are not as well grounded in empirical evidence from the polling data. Firstly, the transition matrix he applies to vote shares is based upon a weak evidence base - while pollsters provide details of respondents' recalled 2005 vote, the transition matrices calculated from this are subject to bias due to respondents' tendency to misremember their votes - in particular remembering voting for the winning party when they did not. This phenomenon is well established, and British pollsters attempt to correct for it in their weighting. However, any model which uses transitions in vote from polling data is likely to overestimate the extent of switching from the current governing party to opposition parties, because many people who say they voted for the governing party last time did not actually vote for them. We suspect this may contribute to Nate's high estimate of change from Labour to the opposition parties.

Secondly, the changes Nate makes for regional differentials in swing are based on polling data that is two years old and was collected in a very different political environment to the current one - the Conservatives were a long way ahead in the polls while the Lib Dems were far below their current tally. We considered incorporating regional swings based on this data, but rejected the change due to the age of the data. We incorporate changes for Scotland as we have a good evidence base from Scotland specific polling, which is regularly updated.

We do not attempt to model "tactical voting", or the effects of incumbent retirements because we simply do not have good quality, recent data on the pattern or level of such effects. Our own regression analysis of incumbent effects did not reveal robust effects of incumbent retirements in recent elections, so we are rather surprised to learn that Nate has uncovered some. Modelling effects such as these, where the statistical evidence is weak requires making strong assumptions. We prefer not to make such assumptions, sticking only to effects where the evidence base is very strong.

On top of our votes to seats projection, we also make efforts to develop a robust estimate of current public opinion. Nate freely admits that his public opinion figures are "educated guesses based on recent cross-tabular results". We employ a state space model to estimate current public opinion every few days, while controlling for systematic "house effect" differences between the pollsters and differences in the sample sizes they employ in their polls. The polling data inputted into our model is therefore based on a more systematic aggregation of available public opinion, although to be fair our current estimate of public opinion is quite close to Nate's.

To sum up, we believe our model has a stronger basis in existing analysis of UK voting patterns, and is based upon techniques that were employed successfully in 2005. Our approach is more sophisticated than other available UK resources, both in terms of its poll aggregation technique and in terms of its seat projection technique. We disagree with Nate's claim that uniform swing models are a low bar to clear - a model based upon a modified uniform swing approach, which employed the probabilistic techniques we use, got the Labour majority in 2005 exactly right based upon exit poll data and early seat declarations. This looks to us like rather a high bar to clear!

Of course, this election is perhaps the most difficult to predict since polling began in Britain, and it may be that uniform swing fails miserably, and that proportional swing of the form Nate proposes manifests strongly next Thursday. We prefer to navigate these uncharted waters with tried and tested methods as a guide, Nate suggests a radically new environment requires radically new methods. We will all know for sure in a week!

For those interested in learning more, the model used to forecast the 2005 election based upon exit poll data and early results is detailed here. Our seat projection techniques are based on those used in this model.

Further details of the model are also available on our PoliticsHome.com page.

 

Comments
Farleftandproud:

Living in Britain sounds really great right now. It would be really nice to live in a country where the most up and coming third party is to the left of the labor party!!! I would love that!

____________________

hobetoo:

Nate Silver's response and takedown of Ford can be found here: http://www.fivethirtyeight.com/2010/05/nerdfight-uk-election-model-methodology.html

____________________

Henry:

Pollster.com and Robert Ford just got pwned by Nate Silver.

I am a bit surprised that Pollster did not at least fact check Ford's post before putting it up.

____________________

Robert Ford:

Henry, I don't really view this as a competition, and I certainly don't think I have been "pwned" (whatever that is) by Nate Silver.

There are no factual errors in my post and a new post clarifying my reasoning and responding to Nate's comments will be on its way soon.

Regards

Rob Ford

____________________

Henry:

Ford, you share a name, and a mentality?

http://en.wikipedia.org/wiki/Robert_Ford_(outlaw)

The song version, see Elton John's "I feel like a bullet in the gun of Robert Ford":

http://www.lala.com/#song/432627056448266996
http://en.wikipedia.org/wiki/I_Feel_Like_a_Bullet_(In_the_Gun_of_Robert_Ford)

____________________

Henry:

Robert Ford-

Look it up, there is this cool new information exchange mechanism called the internet.

After reading both articles and the citations in yours, I do agree with Nate's analysis over yours, but I look forward to your response. I would also note that Nate's use of cross tabular data has proven very robust in the United States, and you should know that.

____________________

Henry:

Robert-

I have found your website at politicshome.com to be pretty clunky, in both Chrome and Explorer so I have not been able to go through all your work. That said, I have been able to go through your CV, it is impressive and you are certainly prolific (that said, who puts things they DECLINED on their CV?). Your CV in some ways reminds me of my own, where I was a molecular and quantitative geneticist attempting to comment on stats models for human gene mapping - suffice it to say I got pwned by the real stats folks. I would recommend you be a bit careful in this fight, your model is less complex and probably less robust as you simply do not bring the depth of math knowledge and modeling experience that Nate has. That doesn't mean you won't be closer to the real election results, partially because of your better gut instincts for the electorate in this case, but, that does not mean you have a better model.

____________________

Robert Ford:

Henry, it is up to each reader to make a judgement about which model is best given the current circumstances. I hope my more recent post clarifies the reasoning behind our model. I would add that we never intended the model, or the pollster.com post, as an attack on Nate Silver or as part of some sort of competition with him. My team and I welcome a more rigorous and nuanced debate of British public opinion, and Nate has certainly contributed to this.

I'm glad you found my cv impressive - that is generally the aim with cvs! Many academics put posts they have declined on their cvs, if these posts were offered at the end of an open competition, as this one was.

I would never claim to be a "real" stats person, but I don't think Nate would claim to be one either. I make very clear throughout my posts that the model is a collaboration involving others with far greater stats experience than me, and in turn draws upon research done by a string of very eminent British statisticians and political scientists, including David Butler, Clive Payne, Philip Brown, and David Firth. I stand on the shoulders of giants with this work, and I freely ancknowledge that.

With regards my modelling experience, I have plenty of it in the UK context, and my colleagues have plenty more in other contexts. I don't know how much experience Nate has modelling UK elections, but I am a big fan of his work on US elections. I am not particularly concerned about whether our model gets close than Nate's to the election results, as you say this doesn't really illuminate which model is best in any event. If Nate's model performs, I will be happy to acknowledge it and it will open an interesting debate about whether modelling of British elections needs to change.

____________________

Robert Ford:

I should add John Curtice to that list of colleagues and predecessors - his work in this area has been invaluable.

____________________

Henry:

Thanks Robert-

Maybe it is a UK thing, as we generally do not even put declined awards/positions on CVs here, no big deal, just looks weird to us over here. It might also be a "soft science" v "hard science" CV difference.

I think it is hard to critique, particularly as aggressively as you did, without leaving readers and the person being critiqued feeling as if you are competing, and in reality, you are and you know it. The whole idea that somehow human nature goes away in the ivory tower is laughable and we both know it.

____________________

Robert Ford:

Actually, it was an American who recommended I do it. Perhaps it is more of a social science thing.

I honestly don't regard this as a competition, and I don't really have anything to gain or lose by this. Whether you believe me is up to you.

I guess the initial wording seemed aggressive and I have apologised for that. I have little experience with blogging, and in my academic day job I have taken part in much stronger exchanges than this. I wrote the post after receiving an invive from Mark Blumenthal to do so. It was written in the same style as I write reviews of academic papers - focus on the key points differentiating our approaches, and explain what I view as the virtues of our approach.

____________________

I was wondering what the objective foundation is for the supposition of 2% extra swing in the marginals? Presumably it makes quite a difference to the prediction if the swing is to be only +0.5 or as much (say) as +4.
How confident can we be that the stated intentions mean 'This is the party I support' or 'This is the party for which I will vote for tactical reasons'?
The international science of polling must have extreme difficulies in the UK since we seem to be the only country where tactical voting exists to a serious degree. I will forgive the pollsters if they get it all wrong.

____________________

Modeling has an important role in most of the issues we do.
All we have to do is to follow the footprints given by models

____________________

Modeling has an important role in most of the issues we do.
All we have to do is to follow the footprints given by models

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR