Pollster.com

Articles and Analysis

 

Tracking Poll House Effects

Topics: House Effects

dailytrackers.png
There are six daily tracking polls currently reporting data, up from just two (Gallup and Rasmussen) during most of the year. How are they doing?

Compared to our Pollster.com trend estimate based on all public polls, not too bad. The trend based on trackers only is close to that for the all polls trend, with an average difference of just 0.35 percentage points, a very slight under-estimate of the Obama minus McCain margin. Recently the difference has been negligible, with most of this difference coming early in the year.

A bit of visual inspection shows the GW Battleground poll seemed more out of line until they shifted their party weighting plan after a few days. Likewise Hotline had a couple day "hiccup" but has returned to trend.

What about house effects? The range is moderate, from +4.3 points on the margin for Daily Kos, to -4.2 points for Zogby, though the latter has only just started polling so the confidence interval is wide.

TrackerHFX.png

 

Comments
RS:

I suppose this is a way of looking for the house effect in different pollsters' results. But I am not entirely happy.

For instance, the house effect is defined with respect to the average. But the average is defined by these polls! And over the past week, Gallup showed results similar to the dkos/R2K poll. So I wish there was a better way to isolate the house effect. A chicken-and-egg problem, seems it to be...

I usually look at the national trend after knocking off the dkos/R2K poll - that usually brings down the Obama advantage by 1%. But I could probably get that back by knocking off Reuters/Zogby - yes, almost. Oh well.

____________________

Observer:

What a curious piece. Surely the only time anybody can make a judgement as to "How they are doing" is after all the votes in the General Election are counted?

Interesting that Battleground were "out of line" until they shifted their party weighting. Being "out of line" is not the same as being wrong.

If all these polls operate on similar assumptions they are not confirming one another. They might all be right, but they might all be wrong. We have had one example of that in the recent past: The Dem New Hampshire Primary.

Personally I think they are probably about right for a current snapshot, which is all they claim to give. But we won't know for sure until after election day.

____________________

brambster:

Rasmussen made a sudden move mid-September to try to pull themselves in line with the other polls. This was evident on both the state and national level.

It seems that pollsters are generally rated by their last poll rather than earlier ones, so a pollster can in fact pull themselves in-line with the norm at the last moment and pretend to have been right all along.

As far as Rasmussen goes, I don't buy it at all. I have as much faith in Rasmussen's accuracy as Nate's Silver's accuracy. They're both biased, and they both tweak their models mid-stream in order to correct a result that is clearly out of sync with the rest. The difference with Rasmussen is that they are pretending to be a respectable pollster when in fact they are nothing but a propaganda machine for Republicans, much the same as the other pollsters that you place a (R) next to.

It bothers me a great deal that Rasmussen is given any sort of respect by the polling community. This is exactly like other news organizations giving Fox News respect. It's unwarranted, and it's detrimental to the entire field to prop up such an aberration.

____________________

Karl Smith:

If you are concerned about "House Effect" then this is the correct method.

Now a pollster may have a large house effect because everyone else is biased. But, if you take as a starting point a wisdom of the crowds mentality then one would be suspect of polls that are out of line.

Note that it is not enough for a pollster to be better, there has to be bias for the house effect to be a good thing. This is because the average of the polls has a lower standard error than any of the polls standing alone.

____________________

ReidH:

The house effect here seems to simply represent the number of dems vs repubs in their samples. Kos has it 35 dem 26 repub - and find that ratio is pretty much what they get every night. Gallup, I read, had a ten point dem advantage - don't know if that's forced by weighting or just how it pops up. Rasmussen uses the average from their independent polling on party ID. They were using the average of prior three months, but now use average of past six weeks. It's been running about 39Dem to 33 Repub. I saw Zogby a couple days ago at least had only a two point edge for dems - I assume he forced that result since he more than others uses his psychic powers for such things. I know Battleground initially wasn't weighting for party but now is - don't know the weight. But all the figures above are is - a measure of the relative party ID, I think.

____________________

judas_priest:

Rasmussen hasa earned the respect. It was among the most accurate firms in 2004. Scott Rasmussen is clearly a conservative Republican, but this shows up not in his results but in the topics he selects and the articles he links to.

The issue with RR is not the polling but the likely voter model, which he does not disclose (most pollsters do not). I suspect that what he is using is slightly skewed toward McCain, but that is a question of analysis rather than one of polling accuracy or bias.

It is improper to compare his work to that of Nate Silver at 538. Rasumussen is a polling firm, and stops there. Nate does no polling and is attempting to build a model that will predict the outcome of the election. This is his first yar at it and as he learns more he makes adjustments tothe model. These adjustmetns are fully disclosed, as is his entire model. At least one poster there successfully reproduced Nate's model from the informationhe posted. Hi smodel may indeed turn out to be wrong, but if so it's a result of faulty analysis, not bias.

If you don't like his model or his adjustments, do your own. Until then, unless you can point our specific problems with Nate's model, be a little cautious with your condemnations. Both are respected by people who know the field.

BTW, if Rasmussen is a biased Republican pollster, and Nate Silver a biased Dem analyst, why does Rasmussen link to Nate's site?

____________________

DTM:

I actually think the most useful part of this analysis is showing that the trend for each poll (minus the early Battleground poll) more or less follows the overall trend, which is useful to know.

For the reasons people have mentioned, measuring the house effect is not really all that useful when it comes to national trackers (more useful is measuring a house effect for state polls, which allows one to correct for a shifting mix of polls over time with respect to sporadically polled states).

____________________

brambster:

In the 2006 midterm elections for all of the senate and governors races that Rasmussen polled, they gave Republicans a 2.04 point advantage in the difference. 2/3 of the polled races were slanted towards the Republican. Just a month ago, Charles Franklin found that Rasmussen had a house effect of about 3 points pro-Republican.

On the day that Biden was picked as Obama's VP choice, Rasmussen placed an attack ad on their site for McCain in their "Breaking Poll" section just under the Biden polling. This wasn't a paid placement, it was a link to a video served from their site. There are countless other examples of their bias, but this was the most glaring.

Rasmussen is a source of propaganda disguised as a pollster. Their results should always be footnoted with a statement as to their leanings, and polling sites should be careful to not let Rasmussen's enormous volume cloud the results from other pollsters.

____________________

GrammarFreak:

This is an interesting article, particularly because it notes the effect of a house altering their polling method. Because every pollster wants to be viewed as "legitimate" they will make changes to shoot closer to the center (even the (D) and (R) pollsters need to seem like they are within reason). It's essentially an a cappella choir with good relative pitch all adjusting to keep harmony, but they may be merrily shifting away from the true pitch (the true %s) in an effort to keep up the chorus. Pollsters with an ulterior motive can even attempt to ensure that the chorus doesn't move in a particular direction by always staying just to one side of the average.

In that sense, it is almost useless to group a series of polls from one house, as the method may be different from one set to the next. If a poll doesn't follow some set of constant ground rules, it hardly deserves to be put on a chart like this under one name. It would be particularly useful to be able to run data through the pollsters' "old" models to see what direction the new models have moved them in. Of course, no one's method may be transparent enough for that type of analysis. It would be nice to be able to run the raw data through a constant method (even if that method allowed for changes in party ID using constant criteria) to see how the chorus of polls moves with respect to it over time.

____________________


My current thinking on this is that the tracking polls may be accurate regionally and not nationally. I would bet that McCain's numbers will look more like Zogby's in Ohio and Nevada and that they will look more like Research 2000's national tracking polls in New Mexico, Iowa and Colorado. I think there is a chance though that the much discussed "Bradley Effect" will play out differently in Virginia and North Carolina than it will in Ohio.

Racial politics in the South may be more polarized up front than up north- so I would be more concerned about under-performance by Obama in Wisconsin, Indiana, and PA than I would be about it in Virginia- where because of its history I think people will speak straightforwardly about whether they will vote for an African-American for president or not. Regional differential in the Bradley effect has not been as well studied as it needs to be. I also think that the Bradley effect may play differently at the Presidential level than at the state level because voting for president is more of a matter of identity than voting for a senator or governor.It will also depend upon how much misinformation is circulating out there unde rthe radar, and how many strange things happen like those 300 ballots that misprinted Obama's name (http://reframeit.com/comments/BEd6-qqkkaG)
That happened in New York, but I expect more weird turns shennaniganship to happen.

____________________

jme:

A couple of somewhat technical comments:

1.) In the interests of educating lay-people about the sources of error/variation in polling, perhaps it would be helpful to not implicitly equate "house effect" with (statistical) bias, which is what is being quantified here. Of course, you have included CIs for the bias term, but it would be helpful I think to explicitly compare pollsters with respect to their variances as well. That way, one can emphasize that a "house effect" is not really a measure of partisanship, rather a look into consistent differences in the (statistical) bias and variance of each pollster.

Obviously, this will likely just reflect sample size differences between pollsters, but one could control for that by dividing the variance of the residuals by the average sample size of that pollster, perhaps? That might give an interesting look into the variance introduced by a particular pollster, independent of sample size. (Actually, now that I think about it, maybe doing the comparison using standard deviations would be more intuitive, so that the scales are appropriate; so you'd divide by the square root of the average sample size?)

2.) This is just me being picky, but Charles hasn't explicitly mentioned this in any of his house effects posts (I think): I hope that the residuals for each pollster are being computed based on a trend line fitted w/out that pollster, right? It wouldn't make much of a difference for, say, Quinnipiac, but the daily trackers constitute such a large portion of the national polling now, I would think this would be necessary.

____________________

RWCOLE:

Informative study, thanks for doing it. I tend to discount Rasmussen because of his methodology and his bias, but your numbers show that his results are at least mainstream. Thanks.

____________________

jme:

Silly me...dividing by the average sample size! What a silly idea, that just makes things worse!

I think I'll have to get a piece of paper to work out what I meant exactly...

____________________

UpstateProgressive:

Two polls I find interesting to compare are the Research2000 (R2K) and Rasmussen (Ras). Both are solid pollsters, but Ras is widely considered to tilt GOP and R2K (being commissioned by DailyKos) might be assumed to have a Dem tilt, though Kos denies any such effect.

Anyway, what's interesting about the house effect is that it's so consistent. R2K has been coming in about 52-41 or 51-40 (with minor variants) for Obama for the last several days. Meanwhile Ras has been coming in at 50-44 or 51-45 for Obama day after day. This has been so consistent as become downright expected. Ras's Obama number is going to be one or two points below R2K, but always at or above 50. OTOH, R2K will have McCain in the low 40s and Ras will have McCain in the mid 40s but never above 45 (a number McCain has rarely cracked in any poll).

I'm not sure who is right, but there's something going on with the way these pollsters construct their polls that is leading to these very consistent results. I think that this is what Charles Franklin means by a house effect and comparatively, at least, R2K can be said to have a slight Dem lean and Ras a slight GOP lean, which is leading to almost the same Obama result (slight R2K edge), while giving McCain about a 4 point boost in the Ras poll. The net result is that both consistently see an Obama lead but Ras pegs it at about 6 points and R2K at about 11 points, with the pollster.com composite tracking right down the middle between them.

My guess is that it has mostly to do with their LV model and their party affiiiation balance, We'll only know who was right on election day when we find out who has the better turnout model.

____________________

Loyal:

I guess I like late night rambles and speculation here at Pollster.com.

Doesn't a lot of this come down to definition. If house effect is deviation from the mean or median, then measuring and describing it is pretty simple and it has nothing to do with accuracy. It is how far off the pack do you fall.

Alternately one could look at absolute deviation over standard error as another approach to looking at variance from the norm.

Or one could look at the mean coefficient of variation of the deviations from each house, which is I think what I would do. In other words I'd take the mean absolute deviation and divide it by the std deviation of the population of those means. That would give me a sense of where the deviation falls and how consistent it is. What do folks think of coefficient of variation of the population of deviations for each house as a "better" statistical descriptor of house effect?

Although I hesitate to mention it (accept over a glass of coffee or beer as Charles has previously suggested and I would gladly take him up on were I to visit Madison or if he were to come to NY (and Madison Ave?)) there might be lessons form meta analytic techniques that could be informative here too.

If we want to look at acuracy, obviously we have to use the last poll and the election to see how each deviates from reality, rather than the pack.

Thanks for a provocative analysis.

Loyal


____________________

jme:

A lot of interesting comments in this thread...here are a few more contributions:

The concern about not knowing the "truth" to compare each pollster's results to is an accurate observation, but I think slightly misguided. I'm kind of steeped in these sorts of analyses, so when I look at Charles' plot of house effects, I read it as a measure of the _relative_ (statistical) bias of the different pollsters. He could have compared each pollster to a "true" difference of, say, -40, and the _relative_ differences would have been the same. So comparing them to the mean really is harmless because any standard would be arbitrary.

If it _is_ harmful it's in leading people to assume that we have solid evidence that DailyKos runs around +4 from the truth, when really all we can say is that DailyKos runs around +4 compared to Rasmussen. That's still useful information, but quite different.

That said, the average of _all_ polls, as represented by the trend line, is in a real sense our _best_ guess at the truth, so it's a reasonable choice of baseline for comparison. And in practice, I'd be willing to bet (assuming sufficient polling density) it's a remarkably good estimate of the truth. (The central limit theorem works, folks! It ain't just hocus pocus!)

And this is all partly why I asked above whether Charles is using some sort of "leave one pollster out" cross validation scheme when doing this, to avoid RS's concern about circularity (overfitting, in statistical jargon).

There might be a more sophisticated way at getting at the idea of an absolute measurement of (statistical) bias rather than a relative one, using bootstrapping, but I'd have to think about it...

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR