Pollster.com

Articles and Analysis

 

Ford: Response to Nate Silver

Topics: modeling , Nate Silver , Politics Home , Robert Ford , UK elections

This morning, Nate Silver responded to yesterday's guest post by Robert Ford on the PoliticsHome UK poll tracking and seat projection model. In this entry, Ford responds on behalf of his team of political scientists that also includes Will Jennings, Mark Pickup and Chris Wlezien.

In a previous post at pollster.com I explained the model we have developed with politicshome. Nate Silver has since posted a lengthy critique of our approach, which I will respond to here on behalf of the team.

First I'd like to clarify a little the background to the post. We were asked by pollster.com to provide an explanation of the differences between the two approaches, and we did so. We provide our projections free of charge to politicshome.com as a way of contributing to the understanding of the current state of play, which in Britain as in the US is too often driven by a focus on individual polls, on spurious margin of error changes and on naive applications of uniform swing. Nate's model is also a valuable contribution to analysis of the British situation and as such we view it as a complement to our work, not as a competitor.

We agree with Nate that naive uniform swing performs poorly. We disagree that this implies abandoning the swing approach entirely, as the evidence from past elections suggests it is relatively straightforward to modify the swing model to improve its accuracy.

Modified swing models such as this have been employed with considerable success to the task of forecasting British election results from exit polls over the last 35 years. We attempt to build on this work, rather than start afresh. We do not consider this stubbornness or traditionalism but rather an appropriate approach given our aims: we wanted to provide a better tool for understanding and interpreting the polls, so we turned to research tools with a strong track record. We do not believe that this approach is necessarily superior to the approach Nate takes, and we agree that science is well served by putting alternative approaches to the problem out in the public domain in as much detail as possible.

We do, however, think there are two important problems with the approach Nate takes. The first is that it is necessarily more subjective than ours: the data needed to construct the matrices Nate uses simply do not exist, and the modeller therefore needs to construct them based on his own judgement. Nate quite rightly deals with this by adopting a scenario based approach to his forecast, so we can see how different assumptions lead to different outcomes. Again, I'd like to emphasise that we don't think this approach is wrong - it may well lead to a better forecast - it is just not our approach. We do feel that when a model involves subjective judgements like this, there should be a lot of clarity about how the decisions are made. Of course, model selection always requires some exercise of judgement, but we prefer an approach which requires fewer decisions, and therefore leaves less subjective judgement to justify. Nate's response to our post has certainly clarified his modelling process a great deal, although we still have a number of unanswered questions. For example: how do his team decide what to put into each cells of their matrix? What might lead him to change their entries? How are the matrices changed for subsets of the data- how does the vote split down in Scotland, for example? Where do the votes lost by retiring incumbents go? I'm sure good answers exist for all of these questions, I would just like to learn more about them.

The second problem is that the proportional swing methodology Nate proposes does not have a good intellectual basis. Proportional swing supposes that most voters have a roughly similar propensity to switch votes, so when a party starts with a high level of support it will lose more than when it has a low level of support. As was first pointed out by Iain Maclean in 1973, there is no reason to suppose that all voters have an equal propensity to switch in this way. Many voters may be committed to one party, and may never consider voting for another. If the propensity to switch votes is unrelated to the strategic situation in the seat - in other words, if "floating voters" are equally distributed across seats - then uniform swing is more likely than proportional swing. We would still, however, observe proportional swing if floating voters were disproportionately influenced by local factors, and if these local factors tended to drive them away from the locally dominant party.

Yet in reality, the distribution of British floating voters is fairly uniform across seats. David Voas' analysis of the 2005 British Election Study shows no difference between marginal and safe seats in the proportions of voters who are undecided, who are thinking of changing their votes, and who have changed their minds about whether to vote at all. As Voas notes, in such a situation we would expect a uniform swing if the influences driving voters' decisions are primarily national. Our view is that the influences in British elections generally are national - the television and print media markets operate at a national level, the parties are national operations, the operation of government is national. The most salient issues - unemployment, the recession, immigration - are national issues. The unique new factor in 2010 - the "Clegg Bounce" was the consequence of a debate aired on national television. There is therefore every reason to suppose that floating voters are being swayed by national factors. And as they are distributed evenly across seats, there is therefore also every reason to suppose that the change in vote will be distributed evenly. We may, of course, be wrong about this, as about every other aspect of our model. But we would welcome a clearer explanation from the fivethirtyeight.com team as to why they think proportional swing should operate in Britain, given the even distribution in floating voters and the dominance of national issues, national parties and a national media.

So our overarching justifications for using models based on uniform swing are that they have a long and strong track record, and a strong intellectual grounding. We apply similar criteria of strong empirical and intellectual grounding when making our adjustments. We adjust the swing in Scotland because Scotland has a uniquely distinct political culture, with a strong devolved Parliament where a different party currently governs, a distinct national media, and a different party system. The empirical evidence from repeated polling also confirms that the pattern of swing is very different there. We haven't made other regional adjustments because both the intellectual case and the evidence base are weaker. I should apologise to Nate for misunderstanding which regional data he used to make his adjustments. The data he is using is fine in terms of recency but the differences in it are not very large and some of the sample sizes for individual regions are quite small.

We adjust the swing in marginal constituencies because we know that the parties concentrate their spending and campaign resources in such seats, and we know from past research that such campaigning efforts make a difference. We also have a good evidence base from a series of recent polls of Labour held marginal constituencies, all of which have shown around a 2% swing bonus to the Conservatives. I'd like to spend a little time clarifying all the decisions here, as Nate has described this section of the model as its "weakest facet" and considers the choices to be "arbitrary". We apply the swing bonus in Labour held seats where the party holds 6 to 14 point majorities. There are two reasons for this choice- again they are intellectual and empirical. The intellectual aim here was to capture the subset of seats where the Conservatives would be concentrating their resources in order to win a majority. Seats requiring very small swings are almost certain to fall, and so are likely to receive less campaign resources, which is why we apply the 6 percent cut-off: the polling in 2010 has consistently shown a swing from Labour to the Conservatives of well more than 3 percent, implying nearly all of these seats should fall. The 14 point cut-off is chosen as if they capture seats above this point, the Conservatives are almost certain to have a majority. As achieving a majority is their primary goal, the seats needed to achieve it should receive the most resources. The empirical justification is that the 6 to 14 point range roughly equates to the range of seats that have been polled. The choices are therefore not arbitrary, although they must involve a degree of judgement. We are not wedded to them and will happily adopt more elegant solutions to modelling this issue - we would welcome suggestions on this front. However, the choices we currently make are well grounded both theoretically and empirically.

We do not apply such adjustments to seats involving the Liberal Democrats, again for both intellectual and empirical reasons. Intellectually, the Lib Dems have far fewer resources available, and until recently they were focussing most of these on defending seats from the Conservatives, not winning them from Labour. We do not see any strong theoretical reason to expect the recent pickup in Lib Dem fortunes to apply most strongly in Lab-Lib Dem seats. We also have very little polling data on this subject - there has been one poll suggesting the Lib Dems are doing better in Labour held seats than Tory held ones - but it has a relatively small sub-sample of each.

It is also by no means clear that applying a marginality adjustment to Lab-Lib Dem marginals would have a dramatic effect. Current polling suggests around a 7 point rise in Lib Dem support from 2005 and a nine point decline in Labour support. There are only 28 seats where the Lib Dems are close enough to win seats on the basis of such an 8 point swing from Labour to the Lib Dems. Allowing a 2 point bonus only brings about another ten seats into view. Applying the bonus to Conservative seats would have a somewhat larger effect, although it would be dampened because the Conservatives are also expected to improve their vote, reducing Lib Dem opportunities. However, we would not rule out strong Lib Dem performance in such seats entirely, and our approach allows us to model it effectively. We have not chosen to do so yet because we don't think there is enough evidence to do so but we could certainly explore what the effect will be on our estimates. If time permits, we will do so.

With regards incumbent effects, I concede that I was not clear about what I meant by robust effects and was too harsh in my assessment of Nate's modelling choice here, though to be fair he had not previously provided details of the source of his estimates. We decided not to add an incumbency adjustment for two reasons. Firstly the pattern of effects changes quite considerably between elections. A quick regression analysis of the 2001 election, identical to Nate's, shows a negative Labour incumbency effect twice as large as in 2005, and a Conservative effect which is about the same. The Lib Dem effect - which is the largest in Nate's model and the most consequential - is the least robust. A 3 point negative retirement effect becomes a one point positive effect. In 2005, the Lib Dems did much worse when the incumbent MP retired. In 2001, they did slightly better. Secondly, there are strong reasons to expect the incumbent effect to operate very differently in this election. Parliament was convulsed by a massive expenses scandal in the summer of 2009, with many incumbent MPs abusing their privileges to buy property and luxury goods at the tax payer's expense. This is widely expected to have a significant impact on many races, and has significantly altered both the pattern of retirements and the value of incumbency. Voters may choose to punish the worst offenders, or reject all incumbents as tainted. We simply do not know. To make strong assumptions about how incumbency works in 2010 derived solely from how they worked in 2005 seems imprudent to us given the circumstances.

Nate's conclusion makes three arguments: that uniform swing models make strong, unfounded assumptions, that uniform swing models have failed badly in some elections and that uniform swing models are inelegant. I think each is a little unfair. Uniform swing's assumptions are strong, but there has been a lengthy and fruitful academic debate in Britain about their foundations, and there is more intuition and empirical evidence to support them than to support the assumption of proportional swing. Nate is right that a basic, naive uniform swing model performs poorly in elections like 1997, but this is an argument for improvement, not abandonment. In fact, a modified probabilistic uniform swing model, with data based differential swing adjustments was employed in 1997 to model the result based on the exit poll, and performed very well. Finally, Nate accuses the uniform swing model of inelegance. I disagree - it is true that the model can predict negative votes, but this is simple to correct for. A negative vote simply suggests the party's achieved vote will be very low. I don't see what is so inelegant about that, and I find much that is elegant in a model that can condense vote changes into a small number of coefficients which can easily be derived from commonly available polling data.

My colleagues and I agree with Nate that models should be constantly analysed, tested and improved. We have attempted to build a model that incorporates thirty years of such analysis, testing and improvement in the realm of BBC exit poll forecasting. Our view is that the technology developed in this context can provide a valuable resource for understanding how current polling will translate into results, and this was the motivation for making a set of projections based upon it available via the politicshome.com website. It is possible that the developments of the past few weeks have rendered this technology obsolete and require a radically new approach. We remain unconvinced, but in the end it is the British voters will provide the final verdict on the debate, at least for this election cycle.

 

Comments
Henry:

Thanks for the update, and thanks for not starting it with an attack this time (e.g. "there is little support for X, see Y" and then X does not support Y).

I remain unconvinced that Brit voter truly acts like a sheep and reacts the same way across all socio-economic classes and locations - but it could be true, you all have to watch the BBC.

It may be true that Nate's model will not work well this election, but based on your posts I think it is because the pollsters and the demographic data in your electorate seems to be extremely poor for a developed country - if the Lib-Dems get some good micro-targeting they could run the place by the next election.

____________________

Robert Ford:

For interested readers, I'm attaching some links here to some of the academic papers I refer to in the above. Unfortunately, most of them are "gated" so you will have to access them through an institution with a subscription to them.

Maclean, Iain (1973) : The Problem of Proportional Swing, Political Studies

http://www3.interscience.wiley.com/journal/119674797/abstract

Brown, P and Payne, C (1975) "Election Night Forecasting", Journal of the Royal Statistical Society

http://www.jstor.org/pss/2345211

Brown, P, Firth, D and Payne, C (1999) “Forecasting on British Election Night 1997”;
Journal of the Royal Statistical Society

http://www.jstor.org/pss/2680579

Curtice, J and Firth, D (2007) "Exit Polling in a Cold Climate, Journal of the Royal Statistical Society

This is available free of charge at the Royal Statistical Society website, and summarises many of the developments in the previous papers:

http://www.rss.org.uk/pdf/CurticeFirthOct2007.pdf

____________________

Robert Ford:

I'm posting here some of the links to the academic literature on this subject. Unfortunately, many of them are gated so can only be accessed with a subscription.

Iain Maclean (1973) "The Problem of Proportional Swing", Political Studies

http://www3.interscience.wiley.com/journal/119674797/abstract

Philip Brown and Clive Payne (1975) "Election Night Forecasting", Journal of the Royal Statistical Society

http://www.jstor.org/pss/2345211

Philip Brown, David Firth and Clive Payne (1999) "Forecasting on British Election Night 1997", Journal of the Royal Statistical Society

http://www.jstor.org/pss/2680579

John Curtice and Clive Payne (2007) "Exit Polling in a Cold Climate", Journal of the Royal Statistical Society

This last one is freely available at the RSS website:

http://www.rss.org.uk/pdf/CurticeFirthOct2007.pdf

____________________

"given... the dominance of national issues, national parties and a national media".

Then why did voting preferences became so regionalized in the first place? I assume because clase, race, traditions, identities, party organization, etc., are also regionalized. And aren´t the reactions to national events mediated by these factors? A "national issue" may generate different (even opposite) reactions in different areas. (e.g., a government policy increasing welfare benefits would increase its popularity in certain areas and reduce it in others). The uniform swing may be useful, but I´m not convinced that the salience of national issues is a very solid reason.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR