November 27, 2006

House District Polls vs. Results - Part I

Continuing with our post-election review of how the polls performed, I want to turn to the polls conducted in individual House races. As I discussed last week, the final results among likely voters for the national generic vote varied with each other beyond sampling error, and the overall average of those results overestimated the support for Democrats. When we look at the polls we tracked on Pollster.com in individual districts, the story is much different. As we should expect, the overall average of polls in individual districts compares remarkably well to the overall average of the actual results.**

To review: On Pollster.com we tried to track and report every poll we could find within individual districts. In the end, we logged over 400 polls in 87 districts. I have obtained unofficial, but mostly complete results in all 87 districts. As should be obvious, these more competitive districts do not have the problem on unreported results in uncontested districts. And while final certified results may change the district level results by a percentage point here or there, any such changes are unlikely to make much difference in the average results across many districts.

The table below compares the overall average poll result to the overall actual result when averaged across all districts for which polls were available. The first line of the table (a) shows the average of the last-five (or fewer) polls still posted on our House summary page in the 87 districts in which at least one poll was available. The next three lines show comparisons for three more averages, (b) only polls released after October 1, (c) only polls released after October 15 and (d) only the final poll for each pollster released after October 15. The next four rows (e through f) show each of these averages but include only independent, non-partisan polls.

Of course, as we start putting restrictions on the types of poll used to calculate averages, the number of districts with available polls declines (something I discussed in reviewing the poll data in October). As such, it was necessary to calculate a different vote count average for each method of averaging. Not surprisingly, the Democratic margin tended to increase (in both the polls and the results) as the number of districts declines. Democrats did better in the most competitive districts. The less competitive districts were more likely to be held by relatively safer Republican incumbents.

Of course, comparing the raw poll results to the actual vote count presents the perennial problem of what to do about undecided voters. So I created the following table, which shows the Democratic percentage of the two-party vote assuming an equal split of the undecided/other vote. A proportional division (D/D+R) increased the error very slightly across all categories but the differences were so slight that I omitted them from the table. .

Contrary to my own expectation, all of the poll averages of the Democratic share of the two-party vote come remarkably close to the overall average result regardless of the averaging method used. The differences between the methods are negligible, although somewhat surprisingly, the most predictive average was based on the last five (or fewer) polls regardless of date, including many polls conducted in September or earlier.

Although the differences are small, throwing out the partisan polls made the overall averages slightly more accurate in terms of predicting the result, although it also meant losing available data for a handful of districts in each case.

Why did the district level polls perform, on average, so much better than the "generic" vote on the national surveys? It is all about what pollsters call "measurement error," something that occurs when the question does not measure the thing we hope to measure.  Polls conducted at the district level ask respondents to choose between the names of the actual candidates. The "generic" national vote asks about generic party labels and assumes the respondents know the names of candidates.  Present the choice as it appears on the ballot, and the poll gets more accurate.   

Of course, the tables above just show how well the House race poll data worked on average. How well did these polls perform in individual districts? And how did some of the more prolific House race pollsters compare to each other in terms of poll accuracy? I will take those questions up in subsequent posts.

**PS: And I have to note that Pollster reader Mark Lindeman beat me to this observation in a comment over the weekend.

-- Mark Blumenthal

November 27, 2006 in The 2006 Race