June 9, 2008
On Polls and Astrophysics
Last Friday's New York Times included an op-ed column by Neil deGrasse Tyson, an author and astrophysicist who used current polling state-by-state polls to conclude that "if the general election were held today, Barack Obama would lose to John McCain, while Mr. McCain would lose to Mrs. Clinton."
His assertion depended on "a new method of analysis on the statistics of polls" developed by two astrophysicists, J. Richard Gott III and Wes Colley, "that has been accepted for publication in the journal Mathematical and Computer Modeling."
Here is the gist of their method:
[I]n swing states, the median result of all the polls conducted in the weeks prior to an election is an especially effective predictor of which candidate will win that election — even in states where the polls consistently fall within the margin of error.
So the idea is this: Ignore sampling error and award states to the candidates if they are "ahead" (even by a single percentage point) in a greater number of polls. That method may work out roughly the same as an average in a state with a large number of polls. Tyson does not say how many polls the Gott-Colley method used in each state in 2004, but in that year the Kerry-Bush results remained reasonably stable over time (so counting up all polls over a 2-3 month period may have worked as well as watching the final week). Moreover, you could have skipped the "rocket science" in 2004 and just examined the final RealClearPolitics averages. The candidate leading on their final averages won every state except Wisconsin.
However, apply that counting method to the "past six weeks of polls," as Tyson did, and you run into a problem: Very few polls, so the counting method starts to break down. Thus, for the crucial 20 electoral votes available in the state of Ohio, Tyson tells us:
In Ohio, for example, Mr. McCain beats Mr. Obama two polls to one. But Mrs. Clinton beats Mr. McCain two polls to nothing. So Ohio, which Mr. Kerry did not win in 2004, would go into Mrs. Clinton’s column, giving her an additional 20 electoral votes.
Shazam! That's not a "two to one" margin, that's three polls, total: One (from SurveyUSA) showing Obama leading by 9, one from Quinnipiac University showing McCain leading by 4, and the tie-breaker from Rasmussen Reports showing McCain with a single percentage point advantage. Never mind that if you average the three (as RealClearPolitics does), they give Obama with a one-point advantage. Or if you draw a regression line through all the available polls -- as we do -- you get a the same one-point Obama edge. Altogether, these data suggest a very close race in Ohio, as of mid-May, not a clear leader. But never mind all that. Tyson applies the Gott-Colley procedure and gives the state to McCain.
The bigger problem with the Tyson column is is the phrase, "if the general election were held today." It isn't. The notion that polls are just a "snapshot" of opinion may be a hackneyed cliche, but it is nonetheless true. The current batch of horse-race polls tell us about voter preferences over the last six weeks. The next six weeks may be different.
Andrew Gelman reproduced a chart from a paper he co-authored 15 years ago to remind us all that polls conducted five or six months before an presidential election are often poor predictors of the final outcome (h/t Monkey Cage):
By Mark Blumenthal on June 9, 2008 11:37 AM | Permalink
Comments
I've always enjoyed Dr. Tyson's commentary on astrophysics (a field in which I'm not even a qualified (but interested) amateur.) I must say, however, that he should stick to his field of expertise. And I'm afraid the same might be said of Mssrs. Gott and Colley.
I missed the original article in the Times, but I suspect Dr. Tyson has already found it to be an embarasssment. If not, he soon will.
And by the way, Mr. McDonald, the universe is shaped like MY head, not Mr. Blumenthal's. In fact, it exists IN my head.
Don't forget, guys, that Mark Blumenthal is a long-time Democrat. He may disagree with this method just because of his partisan favoritism.
At the moment, RCP's polling average has the Electoral College at 272-266 for Obama. This is not factoring in his post-unity bounce. electoral-vote.com, which was accurate in 2004, has it at Obama 287, McCain 227, Ties 27. Poblano's fivethirtyeight.com, which had a good track record in the primaries, has it at 272-266 for Obama.
So, I don't know where this astrophysicist is dreaming up his conclusions. Probably the most reliable predictor I've ever heard of is the direction of the second-quarter unemployment rate in an election year, i.e., the difference between June and March.
In every presidential election since 1948 with the exception of 1956, when the UE rate went up in Q2, the incumbent party lost the White House. When the UE rate went down, the incumbent part won the White House. When it stayed the same, the elections were very close, with the incumbent party losing. (Note: Two of those cases, 1960 and 2000, were probably stolen by the out-of-power party.)
Here is the data series for UE for every month since January 1948. May 2008 isn't included; the UE rate rose to 5.5%, which leaves it 0.4% above March. Watch me be wrong, but from what I've read the chances are that UE will take another leap in June. If so, then McCain (and, more importantly, his donors) ought to cash in their chips right now.
A further note on 1956. It was a blip upward that year, quickly reversed. In other election years, the UE rate, even when changing slightly in Q2, was usually part of a longer-lived trend.
One interesting year in 1992, when the UE rate turned around too late to do George Bush Sr. any good. The research shows that changes of direction after Q2 generally are ignored by voters, who don't have enough time to feel them in their pocketbooks.
This year, I think the official UE stats have masked the degree of distress being felt in the population. There are all kinds of reasons for that, relating to statistical anomalies that have been observed elsewhere. But even the official numbers (especially May UE) show an unmistakable trend in 2008.
Add in skyrocketing gas prices and plummeting house prices, and it's hard to imagine the Republicans having anything but a rotten year, even if the election were held tomorrow.
From two astrophysicists, I'd expect more. I take that back, from two engineers or statisticians, I'd expect more. The theory guys sometimes have no idea what's going on w.r.t. to actuality...
In partial or provisional defense of Gott and Colley, Tyson isn't applying their method as he describes it. Using "the median result of all the polls conducted in the weeks prior to an election" might actually work, at least sort of. (One could squeeze a bit more statistical sophistication into the method without breaking a sweat, and for all I know, Gott and Colley actually did.) Using the median result of a few polls conducted months prior to the election... I guess that's the Tyson method.
Something occurred to me: Maybe everyone has it wrong. Maybe they are astrologers and not astrophysicists.
Dear kingsbridge77:
"Don't forget, guys, that Mark Blumenthal is a long-time Democrat. He may disagree with this method just because of his partisan favoritism."
I guess it takes one to call one. The article presents a reasoned argument that anyone can evaluate. I know presenting a reasoned argument can be hard for Republicans, but you should really try it. Maybe you will reach a valid conclusion some day and get past lazy sliming as a method of argument and commentary.
Just a guess, but I suspect that the Mathematical and Computer Modeling journal did not solicit peer review from someone who specializes in public opinion polling or election forecasting. Worse, the NY Times published this without fact checking. It's not the first time, nor will it be the last, time that the media allows pseudo-science into the mainstream discourse from people who are not trained in the subject area. I suggest that we band together and publish an article in the Public Opinion Quarterly that argues the universe is shaped like Mark Blumenthal's head.
Posted on June 9, 2008 2:37 PM