Pollster.com

Articles and Analysis

 

On Iowa and ARG

Topics: Iowa

The release of new surveys yesterday by the American Research Group (ARG) in Iowa, New Hampshire and South Carolina has generated quite a bit of buzz as well as questions about ARG of the sort I alluded to yesterday in my post on the screens pollsters use to select likely primary voters and caucus attendees. That post is the first in a series that will look at the methodological differences among the various pollsters active in the early primary states. For now, however, let me take up two issues specific to Iowa and ARG's surveys there.

The first question comes from reader AL:

I think your averaging is wrong on the Iowa race for the democrats. I think it should be Edwards at 25.7 and Hillary at 25.4. You have the averaging mixed up. You should correct it.

I am not sure how AL arrived at those numbers, but the current Pollster.com estimates in Iowa (Clinton 25.7%, Edwards 25.4%) are not "mixed up." The confusion may arise from the fact that our estimates are regression based estimators rather than true averages. Professor Franklin explains the difference in detail here.

07-30-IATopzDems_sml.png

The second issue concerns the Iowa surveys by ARG. As commenter jsamuel put it, "ARG seems to almost always under poll Edwards." While I lack Professor Franklin's flair for graphics and regression trend lines, some simple averages show that jsamuel is right. Sen. Clinton does consistently better ARG's surveys than those from other pollsters, while former Sen. Edwards does consistently worse:

07-30%20arg%20iowa.png

We logged in six new Iowa surveys during June and July. The two from ARG show Clinton ahead of Edwards by an average six points (31% to 25%), while the four from other pollsters give Edwards an average lead of five points (27% to 22%).

Surveys from February, March, April and May show essentially the same pattern. Clinton leads by two points (30% to 28%) in four surveys conducted by ARG, while Edwards leads by an average six points (27% to 21%) in 13 surveys by other pollsters.

Unfortunately, this is the sort of scenario in which averaging (or simple regression based estimation) can be potentially misleading. One pollster (ARG) is getting consistently different results and contributing a large number of polls to our overall estimate. So if ARG is both different and wrong, their polls are throwing off our estimates.

We know ARG's results in Iowa are different. Why? And what should we make of that difference? For some clues, stay tuned to my series on primary screens.

 

Comments
Andrew:

The reader's question was related to Clinton & Edwards. This is probably why the fact that ARG disfavors Obama even more than it does Edwards was left out of the analysis. In non-ARG polls, Obama is in close proximity to Hillary, while in ARG polls, Hillary holds a huge lead. This is true for both the June-July and the Feb-May periods.

____________________

DTM:

Following along with Andrew, it seems to me that more than anything else, ARG is different because it tends to poll Clinton higher (by about 9 points in both samples) and Obama lower (by about 4 points in both samples). By my standards Edwards is actually getting about the same numbers, but his relative position is changing because of the changes in Clinton's and Obama's numbers.

____________________

Good points, both.

What I wonder is if a polling firm like ARG, who is brand new to primary polling, is getting results that are inconsistently favoring one candidate and throwing off polling averages, which in turn influence all the news coverage and affect public opinion in that direction, doesn't there come a point where its responsible in the name of objectivity to pull that poll from the averaging? This is particularly concerning when one considers the establishment candidates in both parties were consistently favored by ARG's numbers; not only did ARG's numbers consistently favor Clinton over other Democrats, but until McCain's meltdown became impossible to hide at the end of Q2, ARG was favoring him over other Republicans also. I've been trying to call attention to this for months now; the first time I noted the ARG discrepancy with other primary polls was in a comment here back in February or March.

____________________

Brian:

I have a question in relation to your discussion on ARG "getting consistently different results" in regards to Clinton and Edwards. Is it likely that there is a similar explanation for Giuliani's lead over Romney in ARG's most recent Iowa/NH polls as well?

____________________

Mark Blumenthal:

@Andrew - Yes, I focused on the Clinton-Edwards difference because of the reader's question (and because it was late and I wanted to keep this brief), but your point is well taken.

@Shadow - fwiw, ARG is not new to primary polling, but I think your larger question is a very good one and one we continue to struggle with. Our goal has always been two-fold: Too create a comprehensive archive that tracks all trial heat results, good, bad and ugly and then provide readers the tools and commentary to try to figure out the difference.

What we are hard at work trying to create (in ways you cannot yet see on the site) is a system that provides multiple averages and charts that highlight "house effects" like the one above. We did a little of that in 2006 (see the "Pollster compare" format like this one in Senate and Governor races, and hope to implement more of the same for this cycle very soon.

@Brian - If ARG is doing something consistently different in its methodology than all other polls, then yes, that difference may well affect the Republican results too.

____________________

Mark - I am glad to hear this topic is getting due consideration, thank you. I was worried that there was no limit to how far off an outlying poll can consistently be before it loses credibility in an accurate polling average, which would theoretically mean that a single poll could consistently come out with a thirty point lead for a candidate, resulting that candidate getting an endless cycle of glowing media coverage due to the spike in the polling average skewed soley by that outlier.

____________________

Andrew:

Although ARG isn't my favorite, we can't measure a pollster's efficiency just by comparing it to the average of the rest. We don't know that the rest is right or wrong. ARG accurately predicted the winner of 45 of 50 states in '04. Let's wait until election day in those states to reach a conclusion.

____________________

Jim D:

To be sure, predicting the state winners in a presidential general election contest is not hard (except for the half-dozen or so swing states in the end). Anyone with a high school education can get at least half right, and anyone who pays attention could have probably guessed 45 out of 50 right, with or without polling.

____________________

I'll do a full post on this when I can, but a couple of quick points.

Reliably estimating house effects requires a substantial number of polls by a wide selection of pollsters. This year's state primary polls within each state are still pretty limited for that estimator. We'll do it as soon as the data are strong enough to support it.

On the effect of a single discrepant poll is an important issue. The regression estimator we use here is generally LESS affected by this than are moving averages of 5 or so polls. By definition, each included poll contributes 1/5 to the moving average while it is included in the average. The regression estimator we use here takes more data into account but discounts the effect of discrepant values. A poll that is far away from the rest of the polls will be given less influence on the trend estimate.

I will do a series on these issues later this month.

Charles

____________________

Anonymous:

The July ARG poll shows Edwards down by eight points, and Bill Richardson up by eight points (from five to thirteen) Have any other polls showed Richardson with this kind of movement? Has he been on tv in Iowa in the last month or something? Very strange that Edwards would lose eight points in a month, and even stranger that Richardson of all people would be the one gaining those points. I think maybe ARG got a bad Iowa sample this month, it happens.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR