Pollster.com

Articles and Analysis

 

Newsweek Poll 'Cooked?' Please

Topics: Generic House Vote , Newsweek , Party Weighting , Todd Eberly , Weighting

Is the latest Newsweek poll "fishy?" As we reported yesterday, their latest sample of registered voters split evenly on the question of whether they plan to vote for a Democrat or a Republican for Congress this year. Over the weekend, Todd Eberly, an assistant professor of political science at St. Mary's College of Maryland, argued that the poll seemed "fishy" and "cooked." Jim Geraghty gave Eberly's post a plug on National Review Online and, as a result, commenters on my report on Pollster.com have been howling with outrage that we gave any credence to a "dishonestly weighted" poll.

As I noted yesterday, the Newsweek poll did produce a result on the more positive end of the bell curve for Democrats. Make no mistake: A simple average of recent polls (including Newsweek) shows a roughly five-point Republican advantage on the so-called generic House ballot -- a result that points to Republicans winning 50 or more seats and with it, control of the House. Moreover, the trend is moving in the Republican direction. So no one should interpret anything that follows as evidence that "all is well" for the Democrats.

2010-08-31-Blumenthal-PollsterGeneric.png

If Eberly had confined his criticism to Newsweek's headline and story, which focused only on the Newsweek poll and thus concluded that Democrats "may not be headed for a bloodbath," I would be sympathetic. But Eberly goes much farther and alleges that the data are "fishy," that "someone at Newsweek cooked the books and hoped we wouldn't notice."

On that score Eberly has his math -- and the facts -- flat wrong.

The crux of his argument -- the evidence that he oddly alleges the Newsweek pollsters hoped we wouldn't notice -- appears at the very top of the "complete poll results" document produced by Newsweek's polling firm, Princeton Survey Research Associates (PSRA; interests disclosed: PSRA CEO Evans Witt is a neighbor and friend). Because they provide results for the entire survey tabulated by party identification, PSRA also discloses the unweighted sample sizes for all the party subgroups, Democrats (280 registered voters), Republicans (284) and independents (247) as well as the total of all registered voters interviewed (856).

Eberly finds that partisan mix inconsistent with the results that Newsweek reports for the generic ballot. "[I]t is mathematically impossible," he writes "for Democrats and Republicans to be tied at 45%" given that party breakdown.

Well of course it is. The party breakdown is unweighted. PSRA also discloses, on the same front page of their questionnaire, that their data are "weighted so that sample demographics match Census Current Population Survey parameters for gender, age, education, race, region, and population density."

Now in fairness, PSRA's report does not explicitly say that the subgroup sample sizes are unweighted -- an omission which often leads to this sort of confusion -- but they do provide weighted results for party identification at the end of their report. Among registered voters the weighted result is 32% Republican, 35% Democrat, 29% independent and the rest volunteering that they have no party (5%), are a member of another party (1%) or are unsure (3%).

"Now it's possible," Eberly concludes, "that after weighting for gender, age, education, race, region, and population density the partisan ID of the sample would change."

Yes. It's also likely. Any national pollster will tell you that weighting a sample of adults to match census statistics will typically make the sample a few points more Democratic. The four-point shift seen on this survey is slightly bigger than usual, but that's the way random variation works.

Newsweek does not weight it polls by by party. They weight their adult samples demographically and then, in this case, report on the results among registered voters. Most national media pollsters use the same procedure. A simple Google search on "weighting by party ID" will quickly yield a full explanation and more.

But Eberly is having none of that. His smoking gun? When he enters Newsweek's results-by-party ballot into a spreadsheet, and plugs in the "reported [party] breakdown" (36% Democrat, 32% Republican), he can't reproduce the 45% to 45% tie on the they report on the generic House ballot. By his calculations, "the Republicans still lead by 47.4% to 42.6% -- [so] the poll is pure nonsense."

Professor Eberly? Did you notice that the the unweighted sample sizes of Democrats (280), Republicans (284) and independents (247) add to just 811, not the 856 registered voters that Newsweek reported?* Did you wonder why? Did it occur to you that your tabulations *omitted results for 45 interviews conducted among the registered voters whose answers were "other," no party or unsure and that the omission might explain why your calculations don't match what Newsweek reported?

Apparently not.

Now there's nothing unusual about Newsweek's omission. Few public pollsters report results for subgroups of less than 100 interviews, and for good reason. The margin of error on 45 interviews would yield a margin of error of at least +/- 15%. But I asked PSRA to make an exception in this case and they kindly disclosed that the 45 other/none/unsure respondents support the Democratic candidate in their District rather than the Republican by a 40% to 29% margin. Put those numbers into a spreadsheet along with the rest of the result-by-party, apply the weighted party composition reported at the end of the questionnaire (36% Democrat, 32% Republican, 27% independent, 5% other/none/unsure), and I get a result on the generic ballot of 45.8% Democrat, 44.6% Republican. The slight difference from the 45% to 45% reported by Newsweek is likely due to the rounded numbers we plugged into the spreadsheet.

2010-08-31-Blumenthal-NewsweekByParty.png

Eberly calls on Newsweek "to release fully the effects of it's weighting." I have no idea what he means, but readers should know that Newsweek discloses more about its weighted and unweighted party identification results than most pollsters. Can you point to any Rasmussen poll of registered or likely voters, for example, that discloses either its unweighted or weighted party identification breakdown?

Now again, the results of this Newsweek's poll are arguably on the optimistic end of the bell curve for Democrats, but given the reported +/-4% margin of error, the 45%-to-45% result does not differ significantly from our 45.6% Republican to 41.1% Democrat trend estimate (as of this writing) based on all available public polls.

The charge that Newsweek and PSRA intentionally "cooked the books and hoped we wouldn't notice" is nonsense. Eberly owes them a retraction and an apology.

**Hat tip to Pollster reader John who did notice the discrepancy.

[Cross-posted to the Huffington Post]

 

Comments
CompCon:

Rasmussen leans to the right and you write dozens of articles slamming them and people leaving comments on this site have actually called for a criminal investigation into Rasmussen.

Newsweek cooks the books and you fall all over yourself to defend them because they twisted their results to the left - which you and your owner, the huffington post like.

It doesn't take a genius to see what is going on here.

Everytime there is a poll that is really horrible for democrats - an obvious outlier trying to make things look good for democrats somehow mysteriously appears out of thin air.

And when it is a worthless liberal rag like Newsweek that is putting these polls out - it makes people defending them look like either complete idiots or partisan hacks.

____________________

AmazingThings:

In light of the recent scandals involving Strategic Vision and Research 2000, the complete lack of credibility by certain firms such as Rasmussen, and the wildly divergent results that we're seeing between Newsweek and Gallup (as the most recent example), I think we've reached a tipping point where political polling can no longer be trusted as the industry is currently composed.

Because public polling can influence public opinion and ultimately election results, and because of the lack of transparency by many firms, I think it is important that the government step in to regulate and license pollsters. This is the only way that they will gain the public trust, and to avoid the hint of bias or worse, outright fraud.

____________________

CompCon:

I hope that was a joke, AmazingThings. You aren't seriously talking about the government licensing and controlling the press, are you? As much as I was disgusted with daily kos making up poll results - there is no way in hell I would advocate the government stepping in and taking over polling. That's insane.

____________________

lat:

CompCon,

This is one of the few times I would ever agree with you, but you are correct. As a strong Civil Libertarian as far as I am concerned if Newsweek wanted to say the dems were ahead by 50 points then they can do that and the same would go for Ras on the gop side. Nobody is forced to read Newsweek or Ras Polls and it's up to us (we the people) to decide what we do and do not believe. We don't need the government meddling in that arena.

____________________

John:

@CompCon

Firstly there haven't been 'dozens' of articles slamming rasmussen. There have been a few questioning if rasmussen (and usually more generally IVR polls) deviations from more normal methodology are problematic, but in all the articles I have read, the author has been careful to point out that the results from these polls have been pretty good. Given this is a polling site, discussion on new methodology does seem reasonable. There have also been a number of articles on why there have been at time divergence between rasmussen and other polls, but these were not 'slamming' rasmussen but just pointing out where the differences might be.

Secondly, do you not see the difference between saying the poll is, quite far out there compared to the average, or it has a large house effect etc, than actually saying without any proof the poll is 'fishy', 'fraudulent' or 'someone at Newsweek cooked the books and hoped we wouldn't notice'?

____________________

AlanSnipes:

@compcon:
All over your posts reflect the whining and victim syndrome of everyone on the right. You decry health care reform as wasteful spending, as opposed to the current health care system's increasingly wasteful spending that hurts american business and the budget?
In a democracy, we are supposed to be self-governing, so all of your whining about government makes no sense to those who believe in democracy (obviously, you don't).
You are chortling with glee over every poll that has Republicans winning this fall, yet you refuse to acknowledge the wins of the Democrats in 2006 and 2008, as if when your side loses it doesn't count, that you somehow speak for the majority in America despite these results.
You actually speak for and do the bidding of a couple of right wing billionaires (the Koch brothers)who are bankrolling the right's organizations and act like they speak for average americans when you or they don't.
When there is a poll you don't like, it was biased. Of course, you are not biased, you are perfectly objective.
You never offer any solutions to our problems other that tax cuts for the rich and "cut the deficit" with no plan on how to do so, never putting forth the spending cuts and tax increases that would have to take place to reduce the deficit. In short, you are an empty shell.

____________________

hobetoo:

@Mark: Excellent dissection of Eberly's rancid diatribe.

____________________

Matt Sheldon:

Why is the weighting dishonest?

Let me count the ways.

First, they weight an RV sample to match the General Population Survey from the Census. Why is this wrong?

...because the population of RVs is not the sane as the general population.

This should be obvious to an expert polling site, but apparently was lost on the author.

Secondly, Newsweek provides no indication of what the weighted sample looks like.

Mark seems to harp on methodological transparency, except when the poll results fit the desired outcome of the HuffPo audience.

The bigger issue with the original post is that Mark suggested the difference was simply that of two random bouncing balls around the true mean. Wrong.

One if these is a random sample. The other us a weighted sample. Gallup bounced towards the GOP, and Newsweek bounced straight at the overall mean.

...then they weighted it. Then it bounced to the Democrats.

One is a random fluctuation, while the other is guided by inappropriate weights.

____________________

John:

"First, they weight an RV sample to match the General Population Survey from the Census."

Not quite sure what you mean by that, the poll was an adult sample which was weighted. Respondents were asked if they were certain if they were RVs. The ballot question was made up of those people. I would imagine that is much the same as Gallup.

"Secondly, Newsweek provides no indication of what the weighted sample looks like."

Do you mean by party-id?, It's the last question.

"One if these is a random sample. The other us a weighted sample. Gallup bounced towards the GOP, and Newsweek bounced straight at the overall mean."

They are weighted by almost the same categories (Gallup seems to also weigh by cell-phone usage), from the same source.

____________________

Matt Sheldon:

John, back to basics.

The Gallup poll is unweighted. You keep claiming otherwise and it is making you look uninformed. Anyone who follows polling knows this. That is why it is so bouncy.

The Newsweek poll uses inappropriate weights.

Their topline 45-45 tie is if RVs, not adults.

YET, they weighted it as if it were a survey of adults. The n=856, not 1,067 for the generic ballot.

Their numbers show an incredibly high number of registered voters among all adults (84%).

Using general population weights to derive an RV result is inappropriate because the population of RVs is different.

...older, whiter, more married, more kids. More GOP.

Newsweek fails to provide demographic banners for their RV weighted sample for a reason...because it is the wrong profile.

That is why this is dishonest.

They used a fairly transparent trick to get an RV sample to poll like an adult sample.

Mark missed it, and so did you.

____________________

Matt Sheldon:

I got the wording confused on my previous post. Gallup is weighted by a different set of factors.

I was referring to the unweighted Newsweek results. I reread and it is backwards. The unweighted result has the GOP ahead.

Here is how Gallup does it.


Samples are weighted by gender, age, race, Hispanic ethnicity, education, region, adults in the household, cell phone-only status, cell phone-mostly status, and phone lines. Demographic weighting targets are based on the March 2009 Current Population Survey figures for the aged 18 and older non-institutionalized population living in U.S. telephone households. All reported margins of sampling error include the computed design effects for weighting and sample design.

Newsweek does not provide their weighted demos, so we have no idea what the sample is.

____________________

John:

@Matt

The newsweek poll demographics comes from the "Census Current Population Survey" same as Gallup.

It is an all adult survey so why wouldn't they weigh it to all adults?

They didn't poll just RVs, they polled all adults and then asked a screening question to get the sub-sample of RVs.

The ballot question includes only those who are 'absolutely certain they are registered to vote at their current address' which is 76% of the poll.

____________________

AmazingThings:

CompCon and lat: You are both engaging in typical right-wing knee-jerk reactionism. I never suggested that the government "control" or "take over" pollsters or the press. I am suggesting common sense regulation and a licensing requirement. The government oversees regulation and licensing of numerous facets of our lives, from aviation to pharmaceuticals to the radio waves (that's the press, too, right?), without "controlling" or "taking over". It's time the polling industry have some basic oversight, the same way most other industries have. Otherwise they have no credibility.

____________________

CompCon:

@AT: "Otherwise they have no credibility. "

So what? Why is it important for pollsters to have credibility? It's not up to government to license media content. They don't currently have control of the radio waves to the extent that they decide how credible anyone on the radio is. If they ever do that - heaven help us all.

Polls published for public consumption are entertainment - nothing more, nothing less. I can't believe that liberalism has gotten so out of control that people want opinion polls to be licensed and regulated.

____________________

CompCon:

@John: "Firstly there haven't been 'dozens' of articles slamming rasmussen."

Yes, there have. I spent about 5 minutes googling pollster.com and I found articles criticizing Rasmussen on the following dates:

10/09/08
12/1/09
06/03/10
01/06/10
06/13/10
07/07/08
04/14/10
11/17/09
02/23/10
12/04/09
07/21/08

That's 11 in the first few pages of google results. That would lead me to believe that there are easily dozens here at this site and hundreds more at the huffington post - the owner of this site.

____________________

John:

@CompCon

Glancing over the articles you refer to (it will take me slightly longer than 5 mins to find and fully read 11 articles), most of them don't actually criticise (never mind 'slam') rasmussen at all, and just try and explain the difference between it and other polls, or what impact the volume of rasmussen polls might have on the pollster's averages. There was a couple of articles by guest pollsters which were slightly more critical (or perhaps suggestive would be a better description) but hardly enough to claim "dozens of articles slamming rasmussen."

I am not really sure what huffington post has to do it, it only recently bought over Pollster.com from YouGov, but Mark, as far as I know, keeps full editorial independence.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR