Pollster.com

Articles and Analysis

Job Approval, Outliers and Perspective

Charles Franklin | March 24, 2009

An absurd amount of attention has been given today to a new Zogby internet poll. In a very well managed PR campaign, Zogby leaked the results last night before releasing them a bit after noon today, while also debuting a Zogby presence on Twitter. A little Googling will turn up many hits citing the not-yet-released-at-the-time results. Nate Silver at fivethirtyeight.com helped the poll get attention by attacking it before it was even released. The right is delighted-- "Obama approval drops below 50%!!" The left (with Silver's help) denounces the poll. Twitter is abuzz on both sides. And media, both old and new, couldn't contain their chatter over the results. (My favorite, the Boston Herald's Joe Dwinell's lede "The honeymoon is over, a national poll will signal today as President Obama's job approval stumbles to about 50 percent over the lack of improvement with the crippled economy." The fourth paragraph notes that "Some polls show Obama coasting with a 65 percent job approval, but not in Zogby's tally.")

It takes five seconds to put the new poll in perspective. Take that long to look at the chart above. The Zogby poll stands out pretty clearly in the chart, no?

To make a lot of the Zogby poll is to deliberately ignore the context in which it appears. Yes, the left is correct that it is an outlier. A huge one. The right looks desperate or ignorant by embracing this result as meaningful.

But rather than suppress the poll or ban it from polling analysis, let's just put it in the chart and let the data speak! No stats needed to get the point.

Poll results are far more meaningful when we look at them with a bit of perspective. The chart also makes clear how the Gallup and Rasmussen daily tracking polls differ from one another as well. Gallup's daily tracker consistently results in higher approval ratings than do non-daily national polls. Rasmussen consistently results in lower approval ratings than do non-daily national polls. But put them all together and we get a trend estimate based on all the polls that matches the trend for conventional non-daily polls quite well. If we pick only Gallup or only Rasmussen, we bias our understanding of the state of public opinion. Not by a huge amount, but by a persistent two-or-three points up or down compared to the overall trend or to the non-daily polls. This is no surprise. House effects, the tendency of polling organizations to produce modest differences from one another, are well known and much written about. By putting all the data in the chart for everyone to see we let these house effects stand out in an obvious way.

If someone still wants to cherry pick a result to suit their partisan preferences, then fine. That's called politics, duh! I'm not against it. But let's not confuse willful distortion for partisan purposes with analysis of what the data actually show.

The Zogby internet polls have a considerable history that I won't repeat here. The point is not to condemn them but to put them in perspective. That done, I don't think much more comment in required.

By Charles Franklin | March 24, 2009 3:44 PM | Permalink | Comments (6) | TrackBacks (0)

Comments

RS:

So Gallup's a little high, Ras is a little low... Zogby is just erratic! :-)
So... will we see a Zogby op-ed in the WSJ about how Obama's crashing-and-burning?!

Posted on March 24, 2009 9:25 PM

____________________

hobetoo:

No, Zogby isn't "just erratic." It's consistently way low.

Posted on March 25, 2009 9:40 AM

____________________

OGLiberal:

A few things here...

First, as noted by the chart above, Zogby has consistently been an outlier.

Second, Zogby has 49% saying Obama is doing an "excellent or good" job and 50% saying he is doing a "fair or poor" job. Where I'm from, fair isn't bad. It isn't necessarily good and it's certainly not excellent, but it's not bad. How many of that 50% responded with "fair"?

Third, Obama's "favorability" rating in Zogby's poll only dropped 1pt, from 56 to 55. (Zogby's favorable ratings for the president are also an outlier, btw.)

Fourth - The right direction/wrong direction numbers continue to improve. In early January, Zogby had it at 14/70. On January 26th, it was 36/45. In early March it was 40/48. The poll he put out yesterday had it at 45/47. Doesn't this (steadily improving) number itself reflect well on Obama?

Zogby has one goal in mind with these weak online polls - bookings on FoxNews and a headline on Drudge. Poor Zogby - I don't even think Drudge paid attention to this one.

Posted on March 25, 2009 10:52 AM

____________________

hobetoo:

I think Zogby didn't get traction for two reasons: (a) he was truly an outlier, and (b) only the RW media paid much attention. It could also be that Silver's early debunking -- and reminder of Zogby's incompetence -- helped in dampening interest in the Zogby "story."

In any case there was a lot more news interest in other issues -- including Obama's planned presser -- than this poll by a somewhat discredited pollster.

Posted on March 25, 2009 11:29 AM

____________________

William:

I think we should not be reacting to a voluntary sample survey, regardless of who conducted it. This information is likely to be skewed due to participation bias.

Posted on March 25, 2009 12:27 PM

____________________

Bigmike:

You get no arguments from me. Zogby is obviously getting different numbers.

Where was this same analysis last fall when CBS and the Daily Kos were in the tank for Obama? I must have missed that graph.

Overall, I actually do like this site. There is lots of good info in an easy to understand format. But, like Zogby, take it with a grain of salt.

Posted on March 27, 2009 10:15 PM

____________________