Pollster.com

Articles and Analysis

 

National Trackers: Narrowing? Not Much

Topics: Daily Trackers

Here are the daily tracking results from today. All changes are 1-2 percentage points on the margin, which is well within the usual range of sampling noise. If we use the "expanded" likely voter model for Gallup (our usual practice), we have four tracking moving slightly in McCain's direction, two in Obama's direction and two with unchanged margins. If you prefer to watch the "traditional" Gallup likely voter model, the count is 4 to 3. Once again, this looks mostly like random variation.

081029-b trackers


You see a hint of a "narrowing" when comparing today's results to those reported a week ago, and even then the difference is slight.  Consider what our national trend looks like when filtered to include just the eight national tracking surveys.  This more apples-to-apples trend shows the slight narrowing that amounts mostly to a slight rise in the McCain percentage.
 

The Eight Daily Trackers:


The trend does look a little more abrupt with the other national polls included, but that may be an artifact of the "nose" of the trend line being influenced more on some days than others by the daily trackers (which include Zogby, IBD/TIPP, Battleground and Rasmussen, the organizations showing consistently closer than average national margins).


All National Polls:

 

Comments
DTM:

It will definitely be interesting to see what happens when we get a fresh batch of non-trackers.

____________________

vercordio:

That "Eight Daily Trackers" one is fascinating. Every day the numbers appear to tighten dramatically, my blood pressure goes up a bit. But looking at those lines, most of those days had zero effect in the grand scheme of the past couple weeks.

It does appear that, at the least, Obama's support has peaked and is leveling out. Normally that could be concerning, but probably not five days from the election...

____________________

THX1138:

My blood pressure just jumped too....

____________________

Why would you include a Gallup Registered Voter poll at this point? It causes me to greatly question the analysis.

Of course anyone can say this one is 'in the bag for Obama' or its '95%' certain Obama will win, but any objective observer sees this as a 3-6 (avg of about 5) point margin for Obama which is well within the margin for error for all polls.

I do not think the Obama people will be taking the weekend off even in 'safe' states like Iowa, Wisconsin, New Mexico, and Pennsylvania. It is still very possible for McCain to take any of those states based on latest state polls and their margins for error.

____________________

Allen:

When plotting regression lines, Pollster.com discards 2 out of 3 (or 4 out of 5) data points in the national trackers. The reason Pollster.com does this is so that the daily trackers are not overweighted when computing regression lines that also include non-tracking polls.

However, if you really want to get a good look at the movement in just the daily trackers, it helps not to discard any data points. Unfortunately, this cannot be done using Pollster.com.

Also, I believe it is misleading to report that the movement is within the margin of error when a poll includes weighting. Basically, a weighted poll exhibits smaller sampling error than an unweighted poll. The margins of error reported by the pollsters are generally incorrect. If you watch the polls over time, they in fact exhibit less variation than they should if they had the reported sampling error. In essence, a change in a weighted poll can be significant even if it is less than the reported margin of error.

I believe there has been a distinct narrowing of the national tracking polls over the last three days. This narrowing has not yet shown up in the state polls. That might mean the narrowing has only occurred non-battleground states that are not polled every day, and therefore it will have no affect on the outcome of the election. It is difficult to tell at this point, but it should become more apparent in the next few days.

If the narrowing does result in a shift in the election, it will probably be most apparent in the daily snapshot projections at http://election-projection.net . At the moment, these projections show no change in the race, but they are worth keeping an eye on.

____________________

DTM:

@darrow

Keep in mind that the concept of a MOE applies to an individual poll, not so much to the aggregation of many polls.

____________________

DTM:

@Allen

Pollster's policy with respect to tracking polls doesn't discard any data--every single daily sample taken by the tracking polls is included in the regression. What Pollster does not do, however, is count each daily sample multiple times, which is what would happen if Pollster included each report of the average of the last N daily samples.

____________________

Allen:

@DTM

I understand exactly how it works. And I stand by my statement: if you want to track the movement in a daily tracker, it is helpful to include every daily data point, not just every third or every fifth. This is especially the case if you are trying to track short term movement. If you don't believe me, go back and try to track short term movement over the last two months, using every daily point, and then using every third. You will find that it is easier when you include every day.

____________________

@ DTM

Respectfully disagree. Avg of all polls would include avg of all Margins for error. That would be 2.75% MOE. Think of it this way, all of the pollsters may have called all of the same voters (obviously they didnt, but used to prove a statistical point.) We have no way of knowing whom they have called, therefore we have to assume that the average of all polls also has the average of all inaccuracies.

____________________

DTM:

@Allen

I think the seeds of our disagreement lies in your phrase "if you want to track the movement in a daily tracker . . . ." I am actually not particularly interested in the task of tracking the tracker itself. Rather, I am interested in tracking voter preference, and polls are just a tool I use to do that.

So, it is certainly true that Pollster's charts would do a better job of tracking the trackers' reports by including all of the trackers' reports as if they were independent samples. But doing that wouldn't add anything in terms of data about voter preference, and indeed would just overweight certain samples. Again, I wouldn't want Pollster to do that because I see including tracking polls in these charts as a means to tracking voter preference, not as a means to tracking the tracking polls themselves.

All that said, I would be fine with breaking out each daily sample and treating it as a little mini-poll. Unfortunately, most tracking pollsters do not make that possible.

____________________

Gary Kilbride:

No narrowing today. I was glad of that. My ideal scenario is Obama winning by about 4-5 points, which is comfy enough not to sweat the national outcome but also allows the states I bet on McCain to fall my way, with the possible exception of North Carolina.

The simple method I always use is net points in national polls. Today was basically a wash. But if you look at the past week, I don't know how the drift toward McCain is denied. Among those 10 national polls, 7 of the 10 have a red move since 10/22, with 1 even, and a net of 16, or 1.6 points average.

____________________

Also remember that a poll should theoretically NEVER exceed its margin for error, given properly assumed circumstances. A Margin of Error should include ALL possible outcomes based on assumed set of circumstances.

This is why turnout assumptions are so important, i.e. they go to the assumed circumstances and not the MOE. Polls that are wrong usually assume the circumstances improperly. Crosstabs are umportant to establish the real MOE. For example, a simultaneous poll that judges 50% of independents will vote for Obama with a margin of Error of 2% is fundamentally flawed if Obama gets 53% regardless of the number of independents that turn out.

Watch this on exit polls. They are really a crock and need to be completely re-thought and analyzed.

____________________

DTM:

@darrow

But of course the pollsters aren't calling the same sample--the odds against that are astronomical. So at a minimum you should add up the sample sizes and recalculate the MOE.

But I actually don't think you should apply the concept of a MOE to poll averages at all. That is because different polls use different methods, so they really aren't just a bunch of subsets of a larger sample for which it would be possible to calculate a MOE. In other words, the concept of sampling error doesn't apply to a poll average, because it isn't really a cohesive sample.

____________________

DTM:

@darrow

You further say: "Also remember that a poll should theoretically NEVER exceed its margin for error, given properly assumed circumstances. A Margin of Error should include ALL possible outcomes based on assumed set of circumstances."

This is incorrect. The margin of error is typically set at 95%, meaning the real value is supposed to be within the MOE +/- the reported value 95% of the time. What that means is that about 1 out of 20 polls will end up outside the MOE from the real value simply due to sampling error.

And yes, that makes the term "margin of error" a bit misleading, since you can (and will, with enough polls) have polls outside this margin just due to sampling error.

____________________

Andareed:

@darrow: You will typically see MOE worded as "accurate to +/- 3%, 19 out of 20 times", meaning that 1 out of 20 times (5%) it will be outside the MOE. So you expect that poll to be outside of MOE 5% of the time.

____________________

@ DTM

That's the reason I added 'theoretically' and when we stray to far from statistical theory, then we get to the point where the polls become irrelevant for their ultimate predictive value.

Margin Of Error in statistical theory is an absolute concept. Qualifiers such as '19 out of 20' time within the MOE are added to keep pollsters in business. Frankly, if these guys had more integrity, they would just add the '5% outside the MOE' to their actual MOE. But that would mean they'd have to be honest about the variability within their 'accurate' polling methods.

What if Obama wins by 15? Does that mean we excuse ALL of the other polls because of their '19 out of 20' out language or do we commend and start adopting the methods used by Pew Research who predict this result

You say sampling error shouldn't apply to poll averages so . . . . .why should anyone average polls at all? Of course you can attempt to draw conclusions but you also have to attempt to apply error variances. For purposes of this leisure site, the best way to do that is to average the error. Remember error percentages in a pollster's poll apply equally to each person polled based on the polling METHOD. So we could have 1000 polls that each have a 2.5% MOE and the aggregate of all will also have a 2.5% MOE.

____________________

Allen:

@DTM:

The national poll margins are of little interest to me. At the end of the day, the only thing that matters is the individual states, and especially the battleground states that have the potential to swing the election. The states are not polled every day though, so it can be useful to watch the national tracking polls at a leading indicator of how the individual state polls might move. In that respect, only the movement in the national tracking polls is interesting, not the absolute margins, and that is what I use them for.

____________________

Theoretically, you could poll all of the actual voters on Nov 4 that have a method with a 2% margin for error (in this case the error would be in the questions rather than the sample size because your sample would be the actual universe)and your poll would be correct as long as it was within 4% of the ultimate margin. If you are outside 4%, throw away the method and get another job.

The good thing about math is that it isn't right 95% of the time. Properly applied, it's always right. I will concede that many pollsters haven't figured this out yet.

____________________

Allen:

@darrow:

So we could have 1000 polls that each have a 2.5% MOE and the aggregate of all will also have a 2.5% MOE.

Could you explain that statement? If I took 1000 polls, each with 100 respondents and averaged them together, I would have 100,000 total respondents. The sampling error with 100,000 respondents is much lower than the sampling error with 100 respondents. What leads you to say that the MOE would nonetheless be the same?

____________________

Allen:

@darrow:

AFAIK, the MOE reported in a poll only reflects the theoretical sampling error; it has nothing to do with the potential error in "the method" or "the questions". It is however a good idea to estimate those yourself and take them into account if you are trying to make a projection based on the polls.

____________________

IdahoMike:

Isn't the margin of error for a particular poll just a way of saying that if these exact same questions were asked in the exact same way of ALL people in the population, you would come up with a result within the given +/- 95% of the time?

The MOE is GUARANTEED, and DOES assume a set of circumstances - namely, the circumstances attendant to that very poll. Trying to take into account the fact that you aren't asking questiosn in a way that recreates the voting booth, and that you aren't asking people once they arrive at the voting booth is where the ART of polling comes into play. Who will show up, and how do the answers the sample gave translate into how the population will vote? Correcting for those variables is what's tough.

For instance, someone calls my wife and says "Hi, I'm conducting a poll for Fox News / Rasmussen, and . . . " CLICK! My wife won't talk to anyone assocaited with Fox News. But if they said they are conducting a poll for DailyKos, I can guarantee she participates.

It's all about re-creating the voting booth among the population that will actually vote.

____________________

darrow:

In your example, lets say each poll of 100 is conducted in the exact same manner and we know that each poll has a MOE of 1.0%. And remember. the REALITY is the REALITY (easy enough to say, obviously harder to predict.)

My point is that in EVERY poll, the pollster could get a result of 51/49, but in every poll the real result could be 49/51. And this would be an accurate poll!

Put differently, a larger sample size does not necessarily lead to a lower margin for error. In fact, it can actually increase the margin in some cases.

____________________

All this being beaten sufficiently to death, this race is close. I completely agree with the point made above that the State polls are at variance with the national polls and that it is far more likely that the state polls move towards the national than the other way around.

____________________

DTM:

I think some of the confusion here is from overlooking that the MOE is a measure of only one particular source of polling error, namely sampling error. So while it is indeed true a given poll could be inaccurate for some other reason, such as a methodological problem, that possibility isn't reflected in the MOE. Conversely, the MOE is a simple function of population size and sample size, and it does indeed go down as the sample size increases. Again, that doesn't address other forms of error, just sampling error.

On a slightly different point, the full distribution of sampling error effectively covers the whole range of possible results. So, for example, you could take a sample of 1000 likely voters and get 500 for Obama and 500 for McCain, but there is still a (very small) chance the actual population is either 99-1 for Obama or 99-1 for McCain. Accordingly, you have to make a choice about where you cut off the probability distribution (again, the standard cutoff is 95%), because otherwise it wouldn't be a useful number to provide.

____________________

DTM:

@Allen

I basically agree with you about the limited use of national polls, but I don't see how it follows that you would want Pollster's national chart to track the trackers as opposed to tracking voter preference. That is because presumably the connection between national polls and state polls is through voter preference, so that is what we should be trying to track if we are attempting to fill in the gaps of sporadic state polling with more frequent national polling.

Note, by the way, that Pollster's current methodology for trackers is never out of date. So you really do have all the national data available in the charts at any given time, and hence those charts are as current a measure of trends in voter preference as possible.

____________________

Allen:

@darrow

I understand what you are saying, but an actual poll includes errors due to both methodology and sampling. Assuming normally distributed errors, the properly weighted average of two or more polls always has a smaller MOE than any of the individual polls.

____________________

Bill:

@darrow

As a matter of statistical theory, the margin of error of the average of independent random polls is necessarily smaller than the margins of error of the individual polls.

However, there is a problem with averaging political polls because they are not exactly random polls: even if they select a random sample (which some of them do not do), they all (at least all that I know about) weight responses in the sample to reflect an assumed demographic. For instance, if a poll obtains responses from 1000 voters but happens to get 550 registered Republicans, they decrease the weight of those 550 responses to reflect the fact that less than 55% of the electorate is registered Republicans. Because different polls use somewhat different demographic models (for example: Zogby uses equal Repubican-Democrat weighting, which reflects 2004 Presidential voting patterns, while many other polls use a weighting that refects current party registration) averaging various polls is not the same as averaging independent random polls.

However, it should still be the case that the margin of error of the average of a large collection of polls should be much less than the margins of error of the individual polls.

As has already been pointed out, however, asking people whom they intend to vote for is not the same thing as counting their actual votes.

____________________

I agree. My argument tends toward pure math and not necessarily 'polling'. But let's not pretend that polling IS math. Frankly, if you are a pollster and use terms like MOE, you should absolutely be called on it when you fail. But your point about sample size is simply not accurate. More of a bad sample with bad questions can yield simply more of a bad result.

In a different context, the financial meltdown was caused by badly framed mortgage performance equations incorporated into statistical variance models which caused greater problems when their MOE's were given more credibility because of larger sample size.

____________________

Allen:

@DTM:

I'm not sure how else I can explain it. The best way to track the states is to use the state polls. However, the states are not polled every day, and therefore the state results tend to lag actual changes in public opinion. The national tracking polls can be used as a leading indicator of how the state polls might change. When the margin in the national tracker goes up or down, the margin in the state polls MAY move similarly. When used for this purpose, the absolute value of the national margin is meaningless--only the movement is relevant.

The methodology used by Pollster is intended to give you an accurate picture of each candidate's absolute standing, but it is not the best method to use if you want to monitor short term movements.

____________________

darrow:

As a matter of math, and therefore statistics, you are wrong. My example above clearly demonstrates that the appropriate MOE for 1,000 polls of 100 people each that each have an MOE of 1% is, in fact, 1%. It is not .5% or something less. A valid result for such a poll could be 49,000 to 51,000 when the poll registers 51,000 to 49,000. It is not somehow invalid because of some arbitrary LESSER MOE you would apply.

In fact, this example is straightforward enough. Nice big ropund numbers. Tell me, my friend, what should the MOE be for the larger sample? In mathematical terms please, not 'poll' math.

____________________

Allen:

@darrow:

If you are asking me to explain it further, I will pass. But your math is not correct because you are not accounting for the fact that every poll includes errors due to both the sampling and due to methodology (which could broadly refer to any other source of error). You are considering only the errors due to methodology, and further, it appears you are assuming that they are always correlated, i.e., all polls/pollsters are making the exact same errors. This is also incorrect.

____________________

DTM:

@Allen

We seem to agree that the connection between changes in national polls and changes in state polls is "changes in public opinion". We also seem to agree that the absolute number in national polls doesn't matter much, but the trends in the national polls can help us supplement sporadic state polls.

But where we seem to disagree is how to best measure the relevant sorts of trends in the national polls. Again, I concede if you want to track the reports of the tracking polls, you should include all those reports in your aggregation. But if you instead want to track actual "changes in public opinion", then I think you are overweighting the samples in tracking polls for no purpose by including all the tracking poll averages, as opposed to just the current fresh tracking poll samples and all nonoverlapping samples in the past.

In fact, I don't actually see how overweighting the tracking polls will help you pick up any national trends in voter preference faster. Again, it would be a bit different if you could include all the tracking poll daily samples as mini-polls (although I actually think that would buy you very little as far as picking up trends faster). But if anything you are actually overweighting slightly stale data by including all the tracking poll averages, since the latest daily is included only once, but the prior daily is included twice, and the dailies before that three times (and some more if it is longer than a three-day tracker).

And the bottom line is that if you follow your proposed approach, you automatically triple (or more) the weight of tracking polls in the chart. I think in the long run, that overweighting without justification is likely to introduce more errors than it could pausibly help you in picking up trends in actual voter preference faster.

____________________

Allen:

@DTM:

1. If you want to monitor short terms movements, you should keep the mix of polls the same, i.e., use only the daily trackers.

2. If you include only daily trackers then there is no overweighting. Overweighting is only an issue if you mix trackers and non-trackers.

3. The best method for detecting short term movements is to line-plot each tracker (i.e., connect the dots for each day) then somehow "aggregate" or synthesize a conclusion from totality of the individual lines. Drawing a single regression line through a graph containing all of the trackers plotted together (i.e., the Pollster graphs) is not the best method. But even if you were to draw a regression line like Pollster, including all of the days is helpful because gives you a better indication of when the changes occur (i.e., a sample point every day gives you better temporal location that a sample point every three days).

____________________

cloud9ine:


@darrow,

You have a ver confused idea of MoE versus methodology induced errors. MoE is simply and only sample size errors. The probability that the 100 people I radomly pick out of a 1000000 might not represent the distribution of the 1000000.

When you say things like, if they did it properly, they could avoid the 95% level, you are wrong. Say 10000 people out of 1000000 don't like pie, and the rest do. If you pick 100 people randomly, it is possible that you have no one who hates pie in your group. It is also possible that you have more than one person who hates pie in your group, all the way up to all 100 in your sample being pie-haters. What you do is put up two numbers that say, that this result is within this margin of error around the true value this percentage of the time.

It is part of reliability engineering and other statistical theory. For example, you test a lightbulb to say, we have 95% confidence that a lightbulb will survive 10 years of operation. To say this, we test a sample for a number of cycles to arrive at this result.

____________________

cloud9ine:

sorry i meant, you evaluate to say a lightbulb has 80% reliability at 95% confidence level which gives you a sample size.

____________________

franzneumann:

@darrow

You said: "Margin Of Error in statistical theory is an absolute concept. Qualifiers such as '19 out of 20' time within the MOE are added to keep pollsters in business."

I could not see anyone else correct this, so I must point out for you that when pollsters use the term "19 times out of 20" it's merely another way of stating a 95% confidence intreval. This is not the same thing as MOE. The MOE is the variability or radius you can expect around the confidence interval. You could state it this way, 95% of the time you'll have this same result +/- 3.1 percentage points (for a 1,000 case random sample of the general population).

This is not pollster mumbo jumbo designed to obfuscate unsuspecting buyers, but statistical terminology rooted wholly in scientific probabililty sampling.

As a further note to this discussion, a confidence intreval cannot be calculated for anything but a random probability sample. Since most of the internet polls are not based on a random probability sample (I believe Knowledge Networks is the only one), it is incorrect to apply any confidence intreval to them at all.

I would therefore argue that they should not be included in any of these averages.

Finally, MOE is applied to independent probability samples. I cannot be aggregated in some cumulative fashion as if the total n becomes one independent sample - it doesn't, it's a collection of x independent samples and therefore MOE cannot be calculated.

____________________

Several points.

First, my 'example' was wrong. It demonstrated a 2% MOE rather than a 1%. I am surprised that none of you caught this simple mathematical error.

I am glad I have caused all of you to question my logic. Now please do the same for the polls as they apply to the actual results on Tuesday. Someone (I am not going back to find out who) actually said their can be no appropriate MOE applied to the average of all polls. As if the average is true because its an average. Absurd. Of course there is an MOE and of course the best one to be applied is also an average of all MOEs. Remember an average of all polls necessarily reduces the number of possible results when an MOE is applied.

I'll end with this. If I flip a coin 100 times, and it ends up 70 heads and 30 tails, the % chance that I get a heads on the next flip is 50%, not 70. I am not sure how this applies (actually I am and lets discuss it on a differential mathematics site) , but think about it.

____________________

15isTheBest:

All the repubes hopefuls saying that an Obama lead of 5-6 points is 'well within the margin of error' should consider that that margin could also work the other way: Obama could win this with 52 - 39, easily.

Also, there's no reason to suspect that Indies will massively (more than 66 points) break to mccain.
The only surprise might be a MASSIVE bradley effect, in that hardcore repubes simply haven't talked to pollsters, at ALL.

Barring violence (and even then, president Biden or Clinton would win all 50 states plus 60 senate seats) this is over.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR