Pollster.com

Articles and Analysis

 

Partisan Composition of Samples in 2006 Generic Congressional Ballot Surveys: Greater Discolsure, Less Controversy

Topics: 2006 , Exit Polls , The 2006 Race

Today's Guest Pollster Corner Contribution comes from Alan Reifman of Texas Tech University, who takes a closer look at this fall's pre-election polls.

In the months leading up to the 2000 and 2004 general elections, presidential election polls showed considerable variation -- both across different pollsters and within the same pollster at different times -- in the percentages of self-identified Democrats, Republicans, and Independents comprising the samples. Sample composition itself probably would not concern many people, but when these sampling variations seemed to affect the polls' candidate vs. candidate "horse race" numbers, people got agitated.

Discrepancies in polls' partisan compositions almost inevitably raise the issue of whether survey samples should be weighted (i.e., post-stratified) to match party ID figures from sources such as previous elections' exit polls, much like polls are weighted to match gender and other demographic parameters from the U.S. Census. Underlying the question of whether pollsters should weight by party ID lies another question: How fixed and enduring are voters' identifications with a party? Again, experts differ. Zogby was the first major pollster to weight on party ID, with Rasmussen following suit later. Most, if not all, of the remaining pollsters do not weight by party.

I track polls' partisan compositions at my sample weighting website. I am neither a pollster nor a political scientist, but I am a social scientist who teaches research methods and statistics, and I've spent much time studying and collecting data on party identification. I also took a graduate statistics class many years ago from Pollster.com contributor Charles Franklin.

If I had to summarize developments on the sample composition/weighting front for 2006 (where the main point of interest was the Generic Congressional Ballot), I would identify two trends:

1. Thanks to the efforts of the Mystery Pollster himself and others who raised the issue over the past few years, full "topline" documents (also known as polls' "internals"), which included party ID numbers, were freely accessible via the web for most of the national pollsters during this past election season.

2. The margin between the percentages of self-identified Democrats and Republicans (D minus R) comprising most national polls over the final two months of the campaign season was pretty stable. As a consequence, questioning of polls' partisan breakdowns was relatively rare this year.

On my website, I used Rasmussen's party ID readings as my benchmark for comparison, due to the large numbers of interviews involved (500 daily interviews, aggregated over the 90 days preceding the start of each new month). Most of the time, Rasmussen had the D-R margin at roughly 4.5 percent. As shown in the major chart on my website, when multiple independent polls that were in the field during roughly the same time frame (and which released the necessary party ID numbers) were available, I averaged their partisan percentages. Four polls (not including any from Rasmussen) taken from October 18-22 inclusive showed averages of D 34.4 and R 29.8, well in line with Rasmussen's margin. (The average of five polls from an earlier period, October 5-8 inclusive, had a wider Democratic margin: D 36.9, R 29.6.)

In the chart, I also provided brief verbal commentary on how each poll's partisan breakdown matched up with Rasmussen's. As can be seen, polls' D-R margins were sometimes described as "about right," with instances of "D edge understated" and "D edge overstated" almost perfectly balancing out over the final two weeks.

In the end, the New York Times exit poll (N = 13,251) showed the national electorate for U.S. House races to consist of 39% Democrats and 36% Republicans. This 3-point difference is slightly smaller than would have been anticipated from some of the late polls, but only slightly. It should also be noted that, even with its huge sample size, the Times exit poll still is a sample survey and thus carries a small margin of error (about +/- 1).

One final note: As animated as I get by party ID percentages, I must acknowledge that they are not the whole story. For example, among the final batch of polls, FOX, Pew, and Time all had Democratic respondents outnumbering their Republican counterparts by either 3 or 4 percent. Yet these polls differed widely in their Generic Ballot readings, with FOX and Time having Democrats up 13-15 percent (with FOX's sample explicitly described as consisting of "likely" voters), whereas Pew had them up only 8 (among registered voters) or 4 (among likely voters). Other traditional issues of survey methodology -- such as question wording and order effects -- thus have to be examined for their possible role in these polls' varying D-R margins on the Generic Ballot.

 

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR