Pollster.com

Articles and Analysis

 

Pollster Variation in Iowa

Topics: Iowa

1IADems1226.png

(LARGE graphs-- you probably should click once or twice to see them at full resolution in order to see the details.)

A new American Research Group (ARG) poll of Iowa has caused quite a debate in the comments at Pollster.com. Today Mark Blumenthal takes a close look at polling the "Dark Side of the Moon" in Iowa during the holiday season and the unknowns involved, including the unknowns of the ARG poll. So while the pollsters are busy trying to get one more Iowa poll in this week, let's look at the track records of the pollsters in Iowa.

The chart above shows the polling of the Democratic nomination race in Iowa since January. In 2007, twenty four different polling organizations conducted Democratic caucus polls. Of these, only 7 have conducted three polls or more. That means that for most pollsters, we have no way to separate random error from systematic "house" effects. To do that, we need multiple polls, and to do it reliably we need a number of polls from each organization. But we can at least look at the seven organizations with three or more polls, and see how they compare to the trend estimates. The trend, of course, reflects the best estimate across all pollsters, but until we see the actual vote (itself a very slippery concept in Iowa) we can't say if the trend was better than individual polls or not. Still, this reveals when pollsters seem to follow the trend and when they systematically seem to miss it.

The clear result of the comparison above is that ARG generally showed a much better performance for Clinton than the trend in the first half of the year. From January through June, ARG usually had Clinton some 6-10 points above the trend estimate, with one exception in which ARG agreed almost exactly with the Clinton trend. During this period, ARG was the most discrepant of all pollsters from the Clinton trend.

In the second half of the year, ARG's polling has generally been much closer to the trend estimates, usually less than 4 points away from the Clinton trend. During this time ARG has mixed in with other polling pretty consistently.

That is, until the most recent December 20-23 poll, which again shows a large Clinton deviation of about +6 points from the trend. (I'm using the standard blue trend line. The upturn in the red "sensitive" estimate is interesting, but it is also sensitive to the ARG poll, an example of why you might not entirely trust the red estimator.) This is the first large ARG deviation recently, and quite a change from ARG's previous poll of Dec 16-19 which was close to trend.

In an inversion of the Clinton effect, ARG consistently underestimated Obama's support, compared to the trend, in the first half of the year, but their polling fell much closer to the trend during the second half. At least until the latest poll which has Obama some 10 points below the standard trend. (Again, note the downturn of the sensitive estimator, which would be turning down even without ARG, but which is turning down somewhat more because of ARG.)

ARG's polling for Edwards has been more variable. Early 2007 polling fluctuated quite a bit, sometimes below and sometimes above trend. More recent ARG results for Edwards has generally been a few points below trend, with the latest result about as much below trend as has been "normal" for ARG recently.

ARG's results this year have been more heterogeneous than some oversimplifications claim. There was a substantial overestimate of Clinton early in the year, along with an underestimate of Obama's support. That was important during this period because there were relatively few other polls available for Iowa in this period. But in the second half of the year, the ARG results for Clinton and Obama have been much closer to trend estimates, though still with a small average advantage for Clinton and small disadvantage for Obama.

In light of this, the ARG poll for December 20-23 does look out of line with their own previous polling, certainly their polling of the last 4-5 months.

Let's look at their Republican results for Iowa.

2IAReps1226.png

On the Republican side, ARG's results have been especially favorable to John McCain, and to a lesser but still substantial degree to Giuliani. Unlike the Democratic results, these effects have not diminished much in the second half of the year. McCain has often been as much as 10 points above trend, with only one poll below trend and one more right on the trend. No other pollster has been so consistently far from the trend for McCain.

For Giulinai, the results are less far from trend, but still quite consistently above trend. Only 2 of the last 10 ARG polls have Giuliani below trend.

Romney has fared close to trend in the ARG surveys, though on average a bit below trend. The discrepancies for Romney are much less dramatic than for McCain or Giuliani.

The last two ARG polls show shifts of -3 and +4 points for McCain and Romney respectively, and a single point difference for Giuliani. (And a -5 and +6 for Huckabee and Paul.) For the Dems the shifts were +5, -6 and +2 for Clinton, Obama and Edwards respecively.

As Mark Blumenthal notes, the reasons for these discrepancies are largely matters of speculation. But the consistency of the ARG house effects are pretty clear in these data. The ARG results currently stand on the same side as their long term house effects: above trend for Clinton, Giuliani and McCain, and below trend for Obama, Edwards and Romney. Compared to other pollsters, these house effects for ARG appear to be the largest of any polling firm in Iowa.

Cross-posted at Political Arithmetik.

 

Comments
Paul:

ARG is even more off the average in NH and SC - same trend - a much bigger Hillary percentage than other pollsters.

____________________

Raj:

Something is wrong with either the Republican national polling data posted or the graph. The graph shows Huckabee at 22.8% and Guiliani at 21.8%, but there is no way that the data posted below support this - whether you consider the last three, four, five etc. polls for Huckabee, he doesn't average 22.8%, and in fact he's only over 22% in a single poll (I think the ninth newest poll now). Do you maybe have Giulini and Huckabee switched (i.e. Huckabee should be 21.8% and Giuliani should be 22.8%)?

____________________

Raj,

A bit off topic, but let me answer this.

Our trend estimate is just that, an estimate of the trends and where the race stands as of the latest data available. It is NOT a simple average of recent polling but a "local regression" estimate of support as of the most recent poll. So if you are trying to get our trend estimates from just averaging the recent polls, you won't succeed.

Here is a way to think about this: suppose the last 5 polls in a race are 25, 27, 29, 31 and 33. Which is a better estimate of where the race stands today? 29 (the mean) or 33 (the local trend)? Since support has risen by 2 points in each successive poll, our estimator will say the trend is currently 33%, not the 29% the polls averaged over the past 2 or 3 weeks during which the last 5 polls were taken.

Of course real data are more noisy than my example, so we have to fit the trend in a more complicated way than the example, but the logic is the same. Our trend estimates are local regression predictions, not simple averaging. If the data have been flat for a while, the trend and the mean will be quite close to each other. But if the polls are moving consistently either up or down, the trend estimate will be a better estimate of opinion as of today while the simple average will be an estimate of where the race was some 3 polls ago (for a 5 poll average-- longer ago as more polls are included in the average.)

And that's why we estimate the trends the way we do.

Charles

____________________

Rick:

I suspect this is not a mistake, but just a feature of using a local linear (LOESS) fit. Essentially a line is being fit to the last few points of the data, and since this line slopes upward for Huckabee, if you extend this line out to the next time unit, it is possible for the current estimate for Huckabee to exceed all of the values in the dataset. This is one problem with smoothing using LOESS, rather than kernel smoothing. In my opinion, kernel smoothing would be preferable.

____________________

twaneesha:

I was elected yesterday, Obama said. I have never set foot in the U.S. Senate. Ive never worked in Washington. And the notion that somehow Im immediately going to start running for higher office just doesnt make sense. So look, I can unequivocally say I will not be running for national office in four years, and my entire focus is making sure that Im the best possible senator on behalf of the people of Illinois. He further elaborated: Look, Im a state senator who hasnt even been sworn in yet. My understanding is that I will be ranked 99th in seniority. Im going to be spending the first several months of my career in the U.S. Senate looking for the washroom and trying to figure out how the phones work. So, in four years, Obama went from figuring out how to use the telephones and finding bathrooms, to becoming a foreign policy expert, while missing 1/3 of his votes and failing to hold a single hearing as the CHAIR of Senate Foreign Relations committee. Right?

____________________

frankly0:

Would it just kill people to wait a few more days and see if the latest ARG poll is supported or contradicted by further polls? Why try to manufacture meaning out of next to nothing?

____________________

wxdavid:

Come on use folks you brains. Does anyone really think mcCain is going to get 17% in is VERY strong 3rd place Iowa showing when he has NOT been in the state in 2 months?

as I said back on Monday... its does not make a difference If ARG showed Hillary way up in Iowa or William shatner way up.

ANY poll when compared to the group that is THAT much out of kilter is crap.

For example No other POLL has Ron Paul getting 10% in Iowa.

____________________

Paul Velleman:

Because there is a large enough house effect to make a difference, why don't you adjust for house effects before finding your overall trend? That kind of blocking is appropriate statistically and could give a better overall trend. It could also allow you to make your blue line trend more responsive without making it more jittery.


____________________

Paul-- only 7 pollsters have 3 or more polls in Iowa, so for the remaining 17 pollsters there isn't enough data to estimate a house effect either reliably (n=2) or at all (n=1). I could pool across states perhaps, to increase this some, but there would still be lots of pollsters with only 1 poll. (I left a comment to this effect earlier today on the sensitivity comparison post where you mentioned this.)

I agree in principle but for the primary polling the data are awfully thin for most pollsters.

Rick-- raises a good point that endpoints can be tricky with local regression, especially with a final outlier or a late change in direction. I'm less convinced that kernel smoothing would be better in the cases that are problematic for local fits, but will see if I can run some comparisons for you.


Charles

____________________

VirginiaDem:

Charles -- do you care that I cut & pasted some of your analysis here?

http://www.dailykos.com/story/2007/12/26/223317/00/437/426705

____________________

VirginiaDem:

Charles -- do you care that I cut & pasted some of your analysis here?

http://www.dailykos.com/story/2007/12/26/223317/00/437/426705

____________________

VirginiaDem-- that's fine. Thanks for asking,

Charles

____________________

Anthony:

many people have been claiming push polling or polls favoring the candidates who seem to be in the lead. It doesn't fit with the strategy of downplaying/lowering expectations. For instance, Hillary wouldn't want a poll showing her 15 points up because it could energize voters loyal to other candidates and keep some of her supporters at home if they believed she would win anyway. The best outcome would be to be down a few points in the polls and then to pull off a win!

____________________

frankly0:

So let's see.

ARG's poll showed Clinton above trend the first half of the year, and pretty much on trend for the second half. Now suddenly she's going up again, and that shows ARG is reverting to being above trend, and due to house effects? What happened to the house effects in the second half of the year?

This is science, for Christ's sake? Is it too hard to say you don't really know anything useful about whether this due to house effects or a real shift? Isn't the true scientific statement often that there isn't enough evidence yet to come to a conclusion?

____________________

Rick:

Just in case you're interested in doing kernel regression (also sometimes called Nadaraya-Watson estimation), there's a nice function in R called "ksmooth" that makes this very easy, and functions bw.nrd() and bw.nrd0() for choosing a default bandwidth.

Hope you're having a great holiday, and keep up the good work! Love the addition of the "more sensitive" curve!

Rick

____________________

Suzanne:

Yup, "twaneesha," Obama is one smart professor. And he actually understands the rule of law. I can't wait to have such a quick study in the White House, someone with well thought out convictions, who understands himself.

I think Hillary will make a fine White House adviser on women's affairs, if she really cares about anything.

____________________

low-tech cyclist:

Just looking at ARG in Iowa doesn't suffice, IMHO. Their polls have a history of seeing things no one else does, going back at least as far as Lieberman-Lamont in the CT general election last year. ARG did a couple of polls showing them essentially tied (and getting a lot of people's hopes up), when everyone else was showing Lieberman 10-12 points ahead.

This was the general election, remember, and Lamont in fact lost by 10. That's when I learned to be wary of ARG, and it's served me well since.

Showing Hillary and Obama tied in NH at the end of July, when everyone else had Hillary ahead by 15-20%? It's just ARG. Showing Edwards way up in SC (30% in late May, 22% in late June, 18% in late July, 24% in late August, then abruptly down to 7% in late September before bumping him back up to 18% in late October) when nobody else had him nearly that high, and only one other (small-sample) poll had him quite that low? It's just ARG.

Trust me, that's just the start of it. Their polls have an unusual history of seeing things nobody else sees. Maybe they're right and everyone else is wrong. But in CT last November, everyone else was right.

____________________

Rick:

Fascinating stuff, low-tech cyclist! What else can you tell us about the historical accuracy of the different pollsters? Which usually seems to be the most accurate? How did the different pollsters do in the primaries 4 years ago?

____________________

Joe:

I figured the different ARG results came from applying a more refined/revised "likely voter" screen. It seems like, a week out from actual caucusing, polls would get more specific in trying to identify actual voters, no?

____________________

MissLaura:

ARG was also, I seem to recall, the outfit that had Bush winning NH over McCain by a fair bit in 2000 - just before McCain beat Bush.

I don't trust them as far as I can throw them.

____________________

Ron Paul is going to get at least third place, with probably 15% to 20%, or more, of the vote. These polls are all rigged and we know it.

____________________

Richard :

Charles, what do you think of the L.A.Times/Bloomberg Iowa poll released on Dec. 28? It shows Clinton at 31% and Obama at 22%? Does this mean we now have TWO deviant polling firms acting in Iowa. or could it mean that the ARG trend is gaining substantiation?

____________________

Richard :

Charles, what do you think of the L.A.Times/Bloomberg Iowa poll endidng Dec. 26? It shows Clinton at 29% and Obama at 26%. The poll also shows that among likely voters in the Iowa caucuses Clinton has 31%, Edwards 25%, and Obama with 22%.

____________________

Rick:

But note that the LA Times/Bloomberg poll only had a sample size of 391.

____________________

Rick:

But note that the LA Times/Bloomberg poll only had a sample size of 391. So the margin of error is a bit higher there than in the two more recent polls.

I'm still interested in the historical performance of the various pollsters. Anyone have anything on this? Are there any sites that show what the polls predicted for Iowa in 2004?

____________________

Rick:

But note that the LA Times/Bloomberg poll only had a sample size of 391. So the margin of error is a bit higher there than in the two more recent polls.

I'm still interested in the historical performance of the various pollsters. Anyone have anything on this? Are there any sites that show what the polls predicted for Iowa in 2004?

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR