Articles and Analysis


Housekeeping: Our Classifications

Topics: ASU/Cronkite , Barack Obama , John McCain , Pollster , Pollster.com , Trend lines , Zogby

As of this morning we have made some small changes to the algorithm that classifies leaders in each state on the maps we display for the Presidential, Senate and Gubernatorial races. As quite a few readers noticed earlier in the week, there were some odd inconsistencies in the way the margins separating the candidates translated into "toss-up" or "lean" status. So we have changed the classification slightly to make the process more consistent and intuitive.

The bottom line is that the new criteria shifts three states into the yellow toss-up designation, Virginia, North Carolina and Arizona (more about that last one below). Also, two states that had been designated strong Obama (Michigan and Iowa) are now lean, while Georgia moves from strong to lean McCain.

The inconsistency between states was partly technical (having to do with the way we calculated the margin between the candidates) and partly a function of calculating a confidence interval around the trend line. That process meant that states with more polls in their chart were producing much narrower "margins of error" than polls with few.

So we have changed to using a confidence interval based on the average sample size for the available polls in each state (essentially the same approach we used during 2006). That means that the margin necessary to be classified a leader will be slightly smaller in states where pollsters tend to conduct more interviews (such as Pennsylvania, Connecticut and Texas). But overall, things will be more consistent.

This choice may tend to overstate the uncertainty about a leader in some cases, especially where we have a huge number of polls (such as the National trend). Given all of real-world variability (and volatility) built into opinion surveys, particularly the difficulty of approximating the "likely" electorate at this stage in the contest, we believe it is better to err on the side of less certainty rather than more.

We are also using a totally subjective standard which will on occasion produces some strange results. We believe the "smoothed" loess regression trend lines that appear in the charts provide the best visualization of the trends that underlie available data. It also has the very helpful feature of essentially ignoring "outlier" values in most instances. One poll in ten, no matter how out of line with the trend, rarely budges the trend line.

But there is one circumstance that can produce squirrelly results: When a new "outlier" poll appears in a state with relative few polls, especially if several weeks or months have passed since the last poll.

That, unfortunately, is exactly what happened in Arizona, not once but twice in late June. First, the ASU/Cronkite released a survey based on 175 interviews showing 34% of registered voters as undecided and producing much lower than usual percentages for both McCain and Obama. Next, the always shaky Zogby Internet Panel survey showed a three point Obama lead (and included Barr at 7%). The combination has narrowed the McCain's trend-line margin to just 3.5 percent, just enough to edge below the "toss-up" line.

If we throw out the Zogby poll, the trend lines would now show McCain leading by 10 percentage points. If we left the Zogby poll in, but threw the ASU/Cronkite poll out, McCain would lead by more than 6 points. Either margin would easily classify Arizona as lean McCain. The next poll out in Arizona will most likely have the same effect.

And as long as we are on the subject of housekeeping, please note that by popular demand we added back a "blog roll" of links to all 50 state charts and all of the charts with available data in race. That collection of links now appears on our front page and everywhere else on Pollster.com. So if you are not a fan of the maps, or just prefer to jump from chart to chart with one click, the links are back in place.



Mark, thanks so much for the explanation. In the end, of course, there always has to be some allowance for judgment. I would just throw out any poll with an N=175, because of the high risk that it's done in an amateurish way as well as the high sampling error.

You also mention that the appearance of a new poll in a state with few recent polls can strongly change the apparent vote preference.

The paucity of previous polls, as well as their (lack of) recency, provides one rationale for Nate Silver's approach on 538.com. He provides a baseline in his demographic-historical "538 estimate" that in essence dampens the effects of poll results on the projection of the winner; this dampening effect is greater in states with a paucity of polls. He also uses polls from "similar states" and the "national" levels to make up for the paucity of polls in a given state.

One can argue with such an approach -- especially as we move close to election day -- but at least it uses relevant "other information" to dampen the movement in response to the latest poll. I'm not saying it's preferable to Pollster's approach, but it does address the problem you mention in a systematic way.

One question that perhaps you could address in a future article. This concerns the length of the field period of polls. Most state and national polls are in the field for several days. A seeming disadvantage of this is that it's harder to assess the impact of events that may have occurred on a given day. But an advantage is that the results aren't strongly affected by just one day's news.

I am concerned that single-day polls for the horse races, especially in states with a paucity of polls, create a lot more volatility in the apparent trends in opinion than is meaningful. When state polls that occur episodically are treated something like "overnight polls," don't we have more reason to question how to weight them in our summaries of trends?




I realize it makes little difference in the greater scheme of things, but the inclusion of Zogby's "interactive" survey results simply undermines my confidence (very slightly) in the site. I'm perfectly willing to accept the partyID weighting of Rasmussen though I think it's illegitimate. And there are undoubtedly a number of methodological shortcuts in other polls that I don't even know about. In other words, I don't think I'm a purist about this stuff.

Having said that, I think inclusion of Zogby's internet-based polling simply crosses the line to inclusion of virtually any straw poll no matter how ridiculous. And the fact that it is defended as somehow legitimate because of the large "sample" size and undisclosed weighting scheme only makes it look even more like a Literary Digest rerun.

As far as I can tell, Zogby's "interactive" polling is nothing other than a marketing ploy designed to sell cheap surveys for market research where it MIGHT be more appropriate. I don't think this site should be engaged in promoting it.


In terms of the National numbers, my biggest concern is the extraordinary weighting that Gallup and Rasmussen get for their tracking polls. One of the reasons for looking at an average or trend line poll is to balance out "house effects." With Gallup or Rasmussen getting a new entry every three days, don't their polls begin to overwhelm the others. This year, for instance, they have been showing a consistently closer rate than the other "houses" except FOX.



You raise some great points. I'll say more on this later, but one of the issues is whether we are trying to predict election outcomes or to characterize the polling. At Pollster, we are more about the latter. To the extent polls predict elections, then we accomplish the former as well. And of course everyone wants to know the state of play so we have a map and an electoral vote estimate! But I think our effort is more about where the polls stand today and how they have trended over time. Our classification of races is based only on the polling for that reason. At some point we might create a true forecasting model. The 538 model is a good example of the kinds of things you would put in such a model. But it by definition is going beyond what the polls show.

Bottom line is we are after slightly different things, and that leads us to concentrate more exclusively on the polls here.

I'll also take a look at the field periods. There are some extremely long ones, often from universities where they are as much about teaching students about polling as they are about measuring opinion. As an academic, I approve of teaching students, but worry about quality. For professional polling, though, most field periods are 2-3 days, so I don't think there is a huge amount of variation. Of course some IVR polls are single day. So I agree this is worth looking at, but don't expect to see much variability.


Also excellent point and one we struggle with. Our decision rule has been to err by being completely inclusive. The political world is filled with cherry pickers who like polls that agree with them and dismiss everyone else. We decided two years ago not to go that route. But the issue of variation in poll quality (and not just Zogby internet polls) is a real one and we haven't addressed it as well as we should. I've posted from time to time on the impact of individual polls on our trend estimate, and the results are usually pretty encouraging-- no one poll has a huge impact. But the exceptions are when a poll is at the end of a series, especially after a break from other polling, or when the series is very thin and one pollster accounts for the bulk of polls. We run that risk especially in the state polling where a handful of firms account for lots of the polls.

I'm not interested in promoting anyone's commercial interests here, but I'm also anxious not to dismiss polls based solely on their methodology. I have some reservations about IVR polls, for example, and university polls done by undergrads and phone polls with poor disclosure and anything that looks like an outlier.

So I hope to be able to give you a better, and empirically based, way of thinking about survey quality in the near future. One of the important issues in the survey profession is the fear that low quality cheap polls drive out high quality but expensive research. I share that fear. But I also fear expensive polls that can't demonstrate their added value but which limit how much data we can observe.

Thank you both for the stimulating comments, and stay tuned for some more data-based answers.




You snuck in there while I was composing my comment! Didn't deliberately ignore yours.

We've done a bit of looking at this issue of the trackers overwhelming the other data. There are some differences but the influence on our overall trend is not huge. What we need to do is give you a "Pollster Compare" post like we do for presidential approval so you can easily see what the differences are and what impact they have on the trend estimates.





Because different pollsters "push" the respondents for an answer to differing degrees, may or may not include "leaners", and may or may not include third party candidates, it is problematic to average or "trend" the raw numbers in a state. Its sort of like averaging apples, oranges and pears. IHO, a better metric to determine who is ahead and how much is to average or "trend" the normalized difference:

nd = (Obama - McCain) / (Obama + McCain)

I think this quantity would be less sensitive to differences in how the polls are conducted.

Such an approach might for example give you less problems with a situation like Arizona, but even without that specific case, I think it is overall more accurate and has a more sound theoretic basis.



The biggest problem with one day polls IMO is the risk of a biased sample, i.e., the folks who are home and answering their phone at that hour on a Monday night might be representative of the electorate as a whole, even after making demographic adjustments.




Thanks for the comments. You folks do a marvelous job aggregating results and I don't envy your task of trying to separate wheat from chaff. And as you note, there are a number of methodological shortcuts that impact results. Trying to be completely pure would leave you with little to report.

Keep up the good work. I'm sure I'm not the only one who appreciates it greatly.


Post a comment

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.