Pollster.com

Articles and Analysis

 

Partisan variation in 2007 polling

Topics: Measurement

1PIDBoxPlots.png

The Newsweek poll released this weekend was quickly criticized for under-representing Republican partisans. For the adult sample, Newsweek had 22% Republicans and 35% Democrats, while for registered voters their numbers were 24% and 36% for Reps and Dems. Independents were 36% of adults and 37% of registered, with 7% of adults and 3% of registered refusing or unable to answer the question.

How out of line are these results? The graph above shows the distribution of party id for all polls taken since January 1, 2007. The groups are for three populations (Adults, Registered Voters and Likely Voters) and for whether the party ID question is "leaned", that is does it include those who say they "lean" to one party or the other, after first saying they are independent or non-partisan. In the legend for the graph A.N is adults, not leaned, RV.L is Registered Voters, Leaned, and so on.

It is clear that the distribution of Republicans, Democrats and independents depends quite a bit on the population and the question wording. For the adult, not leaned, sample that Newsweek used for presidential approval, the 22% Republican they found is at the low end of samples in 2007, where the median is 25%. Among registered voters, not leaned, the median in 2007 is 29%, compared to Newsweek's 24%. For Democrats, the Newsweek sample at 35% for adults compared to a median of 35%, while among registered voters their 36% compares to a median of 35%.

So the Newsweek results are a bit low (-3%) for adult Republicans, and a bit lower (-5%) among registered voters, but quite close to the median for Democrats. While the adult Republican number is the lowest of the year in comparable samples and questions, it is only 3 points from the median. That is indicative of the relatively low spread of estimates of Republicans in the adult population this year, where half of all polls fall between 24% and 28% and only 3 of 24 polls fall above 30%. Among registered voters with unleaned questions, half of all polls fall between 27% and 32%, with four polls above 32% and six polls below 27%. Newsweek's registered voter sample is the second lowest reading for Republicans of this group.

The relationship between percent Republican and percent Democrat is not terribly strong within categories of population and question wording, as shown below.

2PIDRepByDemByPopByQ.png

If anything, there is a positive correlation of the two partisan groups, ranging from .44 to .81 for adults and registered voters. (The four cases of likely voters have little variation and produce negative correlations, and I discount this.) The positive correlation is a good indication that what is at work with partisanship is that some questions or survey organizations produce more or fewer independents, driving down or up both Reps and Dems simultaneously.

The variation we see above is a reminder that party identification is a central concept, but subject to quite a lot of variation in measurement depending on population and question wording (and probably "house effects" due to survey organization, a topic ignored in this post.)

Given the sample size of each partisan group, the range of variation we see across polls is not surprising, and that raises an important point. One of the first things we ask about a poll is what its partisan split is. That is an important diagnostic to have available, and Newsweek deserves to be praised for openly disclosing this bit of data. All pollsters SHOULD release this information but a number do not. The current criticism of Newsweek is exactly the disincentive these organizations have for such release. If your party id is even a bit off, it becomes an immediate target for criticism and partisan cries of "fraud". Such is an uninformed view. Normal sampling variation will produce a range of party id estimates, and the data above show that in general this variation is well behaved, doing about what we'd expect given the sample sizes involved. But it does provide a tempting target for anyone unhappy with a particular poll.

This is not to say we should not scrutinize pollsters based on their data. That is entirely what we do here, of course! And I think I was the first to point out that the latest Newsweek poll appears to be an outlier on presidential approval. But that conclusion was based on a great deal of evidence from over 1400 polls and a consistent methodology I apply to every new poll regardless of source and regardless of results.

Finally, it is easy to overstate the impact of the distribution of party id on other survey estimates, such as presidential approval or horserace results. While party id is a powerful individual level predictor of those things, a discrepancy of 3-5 percent in partisanship, as in this case, does not translate directly into a similar discrepancy in every question with partisan overtones. Partisans are not completely homogeneous in their views, so the impact of variation in party id is mitigated by the variation within partisan category in approval or candidate preference. If we want to analyze a poll's results, it is better to focus directly on the question of interest, rather than on another variable like party id which may influence but not entirely determine the result we care about.

Cross-posted at Political Arithmetik.

 

Comments
Stephen Clermont:

Also, PSRA and Newsweek do not weight by Party ID. They weight every national poll the same way, to the Current Population Study and factor in sex, age, education, race and ethnicity, and region. Republican identifiers might be low in this survey, but that is how the sample came out. Nothing has been done differently here than has been done in more than 100 previous surveys done by PSRA for Newsweek. Also, the Pew Research Center has recently shown how Republican party ID has fallen over the last five years with the GOP-Dem split going from 43-43 to 50 Dem-38 GOP.

While this poll might be an outlier, it does show a broad trend identified in other research conducted by Princeton Survey Research.

____________________

Michael:

What was the male/female ratio on this poll? Usually that is a good indicator of bias. I bet this was at least 55-45 female, which is why they didn't publish it.

____________________

Stephen is right about the long term trend in partisanship. This is a bit of a challenge to those who want to say what the "right" pid balance is at any given moment. I wrote a lengthy piece on this last spring (before others noticed it) which you can read at

http://politicalarithmetik.blogspot.com/2006/03/partisanship-moves.html

So PID is a moving target, though not one that fluctuates wildly.

Michael-- Most surveys are weighted to balance male/female population ratios, as well as age, race and education (and sometimes more), as Stephen noted above. The Newsweek/PSRA sample was weighted in this way. That means the sex ratio doesn't vary much from poll to poll in the weighted data. Only if you believe that the women who answer the poll are systematically different from the women who don't (and ditto for men) is this a big problem.

I do agree that ALL pollsters should release their complete demographic profile for each poll, both weighted and unweighted. Those are important data and don't involve any proprietary information. The reluctance to do so is that with partisan critics looking for every flaw to use to undermine the credibility of results they don't like, the pollster is opening themselves up to spurious attack by providing better, more open, information. I feel their pain, but still think all demographics should be disclosed.

Charles

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR