Pollster.com

Articles and Analysis

 

Abramowitz: A Note on the Rasmussen Effect

Topics: Automated polls , House Effects , IVR Polls , job approval , Measurement , Rasmussen

Alan I. Abramowitz is the Alben W. Barkley Professor of Political Science at Emory University in Atlanta, Georgia. He is also a frequent contributer to Larry Sabato's Crystal Ball.

In his recent post, Mark Blumenthal provides an excellent discussion of some of the possible explanations for the differences between the results of Rasmussen polls and the results of other national polls regarding President Obama's approval rating. What needs to be emphasized, however, is that regardless of the explanation for these differences, whether they stem from Rasmussen's use of a likely voter sample, their use of four response options instead of the usual two, or their IVR methodology, the frequency of their polling on this question means that Rasmussen's results have a very disproportionate impact on the overall polling average on the presidential approval question. As of this writing (December 4th), the overall average for net presidential approval (approval - disapproval) on pollster.com is +0.7%. The average without Rasmussen is +7.1%. No other polling organization has nearly this large an impact on the overall average.

A similar impact is seen on the generic ballot question reflecting, again, both the divergence between Rasmussen's results and those of other polls and the frequency of Rasmussen's polling on this question. The overall average Democratic lead on pollster.com is 0.7%. However, with Rasmussen removed that lead jumps to 6.7%. Again, no other polling organization has this large an impact on the overall average.

According to Rasmussen, Republicans currently enjoy a 7 point lead on the generic ballot question among likely voters. Democracy Corps, the only other polling organization currently using a likely voter sample, gives Democrats a 2 point lead on this question. To underscore the significance of this difference, an analysis of the relationship between popular vote share and seat share in the House of Representatives indicates that a 7 point Republican margin of victory in the national popular vote next November would result in a GOP pickup of 62 seats in the House, giving them a majority of 239 to 196 over the Democrats in the new Congress. This would represent an even more dramatic shift in power than the 1994 midterm election that brought Republicans back to power in Congress. In contrast, a 2 point Democratic margin in the national popular vote would be expected to produce a GOP pickup of only 24 seats, leaving Democrats with a comfortable 234 to 201 seat majority.

One of the biggest problems in trying to compare Rasmussen's results with those of most other polls is that Rasmussen is almost alone in using a likely voter sample to measure both presidential approval and the generic ballot. Moreover, Rasmussen has been less than totally open about their method of identifying likely voters at this early stage of the 2010 campaign, making any evaluation of their results even more difficult. However, there is one question on which a more direct comparison of Rasmussen's results with those of other national polls is possible--party identification. Although the way Rasmussen asks the party identification question is somewhat different, reflecting its IVR methodology, Rasmussen's party identification results, like almost all other national polls, are based on a sample of adult citizens. Despite this fact, in recent months Rasmussen's results have diverged rather dramatically from those of most other national polls by showing a substantially smaller Democratic advantage in party identification. For example, for the month of November, Rasmussen reported a Democratic advantage of only 3 percentage points compared with an average for all other national polls of almost 11 percentage points.

Rasmussen's party identification results have only a small impact on the overall average on this question because they only report party identification once a month. However,
Rasmussen's disproportionately Republican adult sample does raise questions about many of their other results, including those using likely voter samples, because the likely voters are a subsample of the initial adult sample. If Rasmussen is starting off with a disproportionately Republican sample of adult citizens, then their likely voter sample is almost certain to also include a disproportionate share of Republican identifiers. Of course, there is no way of knowing for certain whether Rasmussen's results are more or less accurate than those of other polling organizations. All we can say with some confidence is that their results are different and that this difference is not just attributable to their use of a likely voter sample.

 

Comments
Joe Kristovich:

While I understand the concerns about the different methodology, how can Rasmussen's overall accuracy be explained? I know that the "accuracy ratings" can be somewhat subjective, but Rasmussen is consistantly at the top of the list.

____________________

jack:

Correct. Rasmussen nailed the Presidential elections with a 1% margin on the margin. That's impression. Gallup, for example, predicted an 11% victory by Obama.

____________________

StatyPolly:

I have another theory for Rasmussen's variation from others.

As Mark's original piece pointed out, the smaller variation in the approval column may be caused by to the likely voters vs. adults samples. Makes perfect sense.

The larger variance in the "disapprove" is trickier. Here is what I have:

Most other polls - two prompts approve/disapprove - Pass/Fail

Rusmussen - four prompts - strongly approve, somewhat approve, somewhat disapprove, or strongly disapprove - Grades A,B,C,D

Additionally, Rasmussen has far fewer Not Sures/Don't Knows than most.

If I am asked whether I approve or not, I may think the grade is a "C". C is a passing grade, so it's approve. Or it maybe a don't know/not sure. So while other polls end up splitting the C's between approves and DK's, Rasmussen puts them into disapproves. That's how he ends up with very few Don't Knows/Not Sures and more Disapproves.

I am in a rush, can't lay it out more fluently right now, but it makes a lot of sense if you think about it.

____________________

jack:

The worst question is asked by Harris, in my opinion. Since when is "fair" more negative than positive? People who think Obama have done a "fair" job might simply think he's done an average job. Here's the definition of "fair" in the American Heritage Dictionary: "Moderately good; acceptable or satisfactory: Example: gave only a fair performance of the play; in fair health." http://education.yahoo.com/reference/dictionary/entry/fair

____________________

D:

Rasmussen is consistently at the top of the list.

Depends on the list. Since Rasmussen persists in linking to this list, I'll continue to revise it to, among other things, reflect how things stood after all the votes had been counted:

CNN (10/30-11/1), Democracy Corps (D) (10/30-11/2), Economist/YouGov (10/25-27), FOX (11/1-2) and Ipsos/McClatchy (10/30-11/1) tied for FIRST with a 7% margin per pollster.com.

ARG (10/25-27), Harris Interactive (10/20-27), IBD/TIPP (11/1-3) and NBC/WSJ (11/1-2) tied for SIXTH with an 8% margin per pollster.com.

AP/Yahoo/KN (10/17-27), Pew (10/29-11/1), Rasmussen (11/1-3) and YouGov/Polimetrix (10/18-11/1) tied for TENTH with a 6% margin per pollster.com.

ABC/Post (10/30-11/2), CBS (10/31-11/2) and Marist College (11/3) tied for FOURTEENTH with a 9% margin per pollster.com.

Diageo/Hotline (10/31-11/2), DailyKos.com (D)/Research 2000 and GWU (Lake/Tarrance) (11/2-3) tied for SEVENTEENTH with a 5% margin per pollster.com.

CBS/Times (10/25-29), Gallup (10/31-11/2) and Reuters/ C-SPAN/ Zogby (10/31-11/3) tied for TWENTIETH with an 11% margin per pollster.com.

Newsweek (10/22-23) was TWENTY-THIRD with a 12% margin per pollster.com.


I also did some state polling comparisons:

Nine* pollsters polled in nine or more states in the ten days prior to the election. More than one of the nine polled in 38 states. Determination of who was "closest" to the true result in those 38 states is based on polling/victory margin followed (where there were ties) by candidate percentages. Ties remained in Georgia, Missouri, North Carolina and South Carolina so the "closest" total is 43.

Closest is bolded and off by more than 5% is italicized.

The base: YouGov/Polimetrix (50 states & DC, 11 closest, 21 off by over 5%, 13 lone pollster)

Alabama, Alaska, Arizona, Arkansas, California, Colorado, Delaware, Florida, Georgia, Illinois, Indiana, Iowa, Kansas, Kentucky, Maine, Massachusetts, Michigan, Minnesota, Mississippi, Missouri, Montana, Nevada, New Hampshire, New Jersey, New Mexico, North Carolina, North Dakota, Ohio, Oregon, Pennsylvania, South Carolina, South Dakota, Utah, Vermont, Washington, West Virginia and Wisconsin (lone pollster Connecticut, District of Columbia, Hawaii, Idaho, Louisiana, Maryland, Nebraska, New York, Oklahoma, Rhode Island, Tennessee, Texas and Wyoming)

ARG (14 states, 2 closest, 4 off by over 5%)

Alaska, Arizona, Arkansas, Colorado, Florida, Indiana, Missouri, Montana, Nevada, New Hampshire, North Carolina, Pennsylvania, Virginia and West Virginia

CNN/Time (10 states, 4 closest, 2 off by over 5%)

Arizona, Colorado, Florida, Georgia, Missouri, Nevada, North Carolina, Ohio, Pennsylvania and Virginia

Mason-Dixon (15 states, 0 closest, 4 off by over 5%)

Arizona, Colorado, Florida, Georgia, Illinois, Indiana, Iowa, Kentucky, Minnesota, Missouri, Montana, Nevada, New Hampshire North Carolina, Ohio, Pennsylvania, Utah and Virginia

PPP (15 states, 7 closest, 2 off by over 5%)

Colorado, Georgia, Indiana, Michigan, Minnesota, Missouri, Montana, Nevada, New Mexico, North Carolina, Ohio, Oregon, Pennsylvania, Virginia and West Virginia

Rasmussen (27 states, 5 closest, 6 off by over 5%)

Alaska, Arizona, Arkansas, California, Colorado, Florida, Georgia, Illinois, Indiana, Iowa, Kentucky, Maine, Michigan, Minnesota, Mississippi, Missouri, Montana, Nevada, New Hampshire, New Jersey, New Mexico, North Carolina, Ohio, Oregon, Pennsylvania, South Dakota and Virginia

Research 2000 (18 states, 4 closest, 4 off by over 5%)

Alaska, Arizona, Georgia, Illinois, Indiana, Iowa, Kentucky, Minnesota, Mississippi, Montana, New Jersey, North Carolina, North Dakota, Oregon, South Dakota, Vermont, West Virginia and Wisconsin

Survey USA (23 states, 7 closest, 7 off by over 5%)

Alabama, California, Delaware, Florida, Georgia, Indiana, Iowa, Kansas, Kentucky, Massachusetts, Minnesota, Missouri, New Hampshire, New Jersey, New Mexico, North Carolina, Ohio, Oregon, South Carolina, Pennsylvania, Virginia, Washington and Wisconsin

Zogby (9 states, 3 closest, 1 off by over 5%)

Florida, Indiana, Missouri, Nevada, North Carolina, Ohio, Pennsylvania, Virginia and West Virginia

So best (in order by percentage) as "closest" where more than one pollster polled: 1. PPP, 2. CNN/Time, 3. Zogby, 4. Survey USA, 5. YouGov/Polimetrix, 6. Research 2000, 7. Rasmussen, 8. ARG and 9. Mason/Dixon.
So best (in order by percentage) on being off by less than 5%: 1. Zogby, 2. PPP, 3. CNN/Time, 4. Rasmussen & Research 2000, 6. Mason/Dixon, 7. ARG, 8. Survey USA and 9. YouGov/Polimetrix.

* Strategic Vision was originally included in this comparison.

____________________

Alan Abramowitz:

Thanks for these excellent comments. Very impressive data analysis here. But I think the most interesting aspect of this is that Rasmussen has not been an outlier when it comes to polling on candidate preference in elections. He's been pretty close to the average most of the time. The interesting question to me is why, then, he's an outlier when it comes to other questions such as presidential approval and now party identification where there is no easy way of verifying accuracy as there is with election polling.

____________________

jack:

Here's what a cynic would think, Alan: Rasmussen is simply trying to sway public opinion. He wants to keep Republicans (he's a right-winger) happy so they won't lose interest; He also wants Drudge to link to his website. This means traffic and business. Then, he will adjust his crappy numbers near the end, because he does not want to ruin his own reputation. He will be in line with the average of all posters, as usual.

Not saying I think that way. I'm just saying. :)

____________________

Mark Sanford:

Why did you not mention Gallup when discussing the party identification and generic ballot questions? Gallup showed Republicans ahead in the a generic ballot and showed a narrowing of the party ID number to the point it was almost non-existent. While I understand Gallup only uses a registered voter model for those questions, this only bolsters Rasmussen's results, as likely voter polls generally favor Republicans.

I think one thing most people here would probably agree on though is that polls which arrive at 13-14% (I have even seen 25% for a Newsweek "poll")differences in Party ID are completely bogus. You have to be one heck of a partisan to believe that only 20%, or less, of the country identifies itself as Republican, a number which Larry Sabato recently stated on Twitter, in response to numbers from a Washington Post poll, was ridiculous.

And it always fascinates me when people make accusations of bias against Rasmussen and then cite polls put forth by news organizations that hardly have a spotless record in that regard. I hate to keep mentioning it, but the Washington Post (as well as CBS which actually had him at 58%) poll had Obama's job approval at 56% at the same time six different polls showed him at 50% or below, with five of those showing him below 50%. CNN now has him at 48%, Gallup at 49%, PPP at 48%, and a few others I have forgetten have him below 50%. How exactly is Rasmussen the outlier in regards to presidential approval?


When Rasmussen ceases being one of the most accurate pollsters around, maybe he will start being dismissed and will then go the way of Zogby, a man pretty much no one listens to anymore. Until then, criticizing only him, particularly by comparison to an explicitly partisan polling company (Democracy Corps) smacks of partisanship.

____________________

Alan Abramowitz:

Note: I'm comparing Rasmussen with the overall average, Gallup included. D-Corps has an excellent reputation and generally shows no clear house effect.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR