Pollster.com

Articles and Analysis

 

(Very Late) Status Update

Topics: Status Update

Well, obviously, we've been a little preoccupied with cleaning up some issues with the new charts today, so the "lunchtime" update is a bit late today. Since I did the update at the crack of dawn yesterday, this one incorporates 34 polls logged since then.

As has been the pattern for the last week or so, the new surveys are a mix of ongoing weekly tracking programs, updates from surveys conducted before the conventions, and first polls in each state for a given organization.  So it is hard to see clear patterns with apples-to-apples comparisons, but of the five surveys tracking since earlier in September, all five showed movement to Obama.  On one example of such movement, four new surveys in Michigan which helped move that state back to the "lean Obama" classification.

The bad news for Obama -- at least in terms of our map -- is that we had two new polls showing movement to McCain in three states, Montana, West Virginia and Maine, where polls have been rare this year. As explained earlier in the week, our trend lines in these states are considerably more sensitive to individual polls. Partly for that reason, we had both states classified as toss-ups. Both moved into the McCain column on our map. Montana moved all the way to strong Republican.  So the

But when we look at movement at the state level aside from Montana and West Virginia, the larger pattern is mostly in Obama's direction in the statewide polls.

2008-09-26 trends.png

The update of Charles Franklin's mashup chart of all of the state level polls (which he ran first thing this morning) continues to show a trend favoring Obama. Not obvious from this chart is that the slope of the "all states" trendline steepened since yesterday, with the Obama margin now stands at 4.1, up from 1.8 on Thursday.

2008-09-26_BluesRedsSwingAndTU.png

Starting today, we're adding a "national forces" chart that compares the trend in the Obama-minus-McCain margin from the national polls to the mashup of all state polls. The national poll margin had tightened slightly as of yesterday.

2008-09-26NationalForces.png

However, as as partly noted by TPM's Greg Sargent, four of the five 3-day tracking surveys showed some movement in Barack Obama's direction as compared to yesterday (and today's trackers are not yet included in Franklin's chart from this morning -- Sargent left out the GWU/Battleground survey which had a one point move to McCain).

This raises a side point I have been wondering about since the number of daily trackers expanded to five (and counting).  Typically, the movement on any one the daily tracking surveys is too small to be considered statistically meaningful. And when two move one way and three move the other, we can usually assume that we are mostly seeing statistical noise in the variation.

But if public opinion is unchanged, the movement of the Obama-McCain margin in one direction or the other should be as random as a flip of a coin. And it is hard to flip a coin five times and have it come up heads four out of five flips. I had been tempted at one point to plug some numbers into my favorite binomial probability calculator and figure out the odds of consistent movement by 4 of 5 tracks or by all 5 Of course, the day-to-day movement is not strictly comparable to the flip of a coin, since "no change" has no coin equivalent.

So I toss this question to our more remarkably statistically astute readers: Is it possible to calculate the odds of 4 of 5 tracking surveys moving, even slightly, in the direction of one candidate? What do we do about the odds of no-change? And would this be a one or two-tailed test?

 

Comments
Mark Lindeman:

Umm, well, I ginned up a little Gallup simulation with 900 responses per day and a rounded 3-day moving average, assuming that Obama and McCain each have 48% support. It worked out that both Obama and McCain were "unchanged" about 30% of the time. I imagine that different assumptions could shift that percentage substantially. For now, let me just assume that in each poll, the chances are: 35% Obama improves on the margin, 35% McCain improves, and 30% no (measureable) change.

Under that assumption for all five polls (where both candidates are stuck at 48% true support), the chance that one particular candidate will be up in four or five out of five polls would be a bit over 5%. The chance that either candidate will be up in four or five polls would be a bit over 10%.

That's as much as I can fit on the back of my envelope tonight. Subject to correction and extension....

____________________

serpounce:

Is there some reason why the daily tracking polls stopped getting posted?

____________________

My 'very late' take on the snap post-debate polls of who won the debate:

CBS, among undecideds: 40% Obama, 38% draw, 22% McCain.

CNN (reported on-air) 51% Obama, 38% McCain

I thought it was a draw. There was a lot of substance without a lot of zingers. McCain managed to hammer away on a few talking points, but Obama gave as good as he got. In the big picture, without a knockout or game changer for McCain, this is ultimately a win for Obama. I do not think the structure of the race will change much from where it has been for the past couple days. If anything, now that the conversation has changed again, perhaps there should be some stabilization of the recent McCain slide. I don't expect a 'surge' for McCain, though, from this debate.

@Mark L: I like the simulation approach. A good way to go in answering the question, though I would think one should be able to write out the joint probability...

____________________

jme:

Mark B. wrote: "So I toss this question to our more remarkably statistically astute readers: Is it possible to calculate the odds of 4 of 5 tracking surveys moving, even slightly, in the direction of one candidate? What do we do about the odds of no-change? And would this be a one or two-tailed test?"

Eesh. I agree (just off the top of my head) with Mark L.'s simulation approach. Calculating these probabilities exactly seems just too cumbersome given that in reality, each tracking poll should have a different variance (different sample sizes) and the probability of movement in a particular direction won't simply depend on the "true" support levels and the new round of interviews - the 3 day old interviews that are dropped from the moving average are crucial too.

So yeah, that seems way more complicated to me than a simple binomial (or multinomial) probability. But simulation seems promising...maybe...

____________________

jme:

Uh oh...just saw on 538 that we now have a _statewide_ tracking poll just for PA. Is this a curveball, or will pollster just treat it the same way the national trackers are treated?

____________________

Michael McDonald:

For those who miss the daily tracker update:

DailyKos/R2K
Obama 49%
McCain 43%

Diageo/Hotline
Obama 48%
McCain 43%

Gallup
Obama 49%
McCain 44%

Rasmussen
Obama 50%
McCain 44%

Sorry, no GWU Battleground over the weekend and no embedded hyperlinks.

Btw, just three days ago, Gallup was reporting that McCain had been doing better and had the race even, so Obama has had two very solid nights of Gallup polling. Thus, even if McCain (somehow) receives a debate bump, it will likely not show up in tomorrow's Gallup tracker.

____________________

Mark Lindeman:

Yeah, the more I thought about this problem, the less sure I was what joint probability I was writing (although if one has the probabilities for each tracker, the joint probability across trackers is straightforward).

FWIW I just ran 10,000 rounds of four scenarios (so of course these results are subject to sampling error).

n=900/day, 48/48/4: 29.3% repeats
n=600/day, 48/48/4: 22.5% repeats
n=900/day, 46/46/8: 26.7% repeats
n=600/day, 46/46/8: 20.6% repeats

Unbalanced percentages probably reduce these probabilities somewhat.

As the probability of a repeat decreases for any tracker, the probability that 4 out of 5 trackers will move in the same direction by chance increases.

____________________

jme:

@Mark L.

Just for my own understanding, you're simulating the 900 (or whatever) daily responses directly, right, assuming some true probability that a person supports McCain or Obama? And then doing that 3 times to get a single tracking poll estimate and so on...?

____________________

The test should be two-tailed unless you have some sound reason for supposing the polls will go in a particular direction. If choose(n,k) refers to the binomial coefficient for the number of ways to choose k items from among n, the number of ways five polls can split 5-0 or 4-1 is

choose(5,0)+choose(5,1)+choose(5,4)+choose(5,5)=1+5+5+1=12.

Since there are 2^5=32 permutations, there is a 38% chance that such movement would happen by chance. So this is no good. Requiring 5-0 is better but not great; the probability of such an occurrence is 6%.

To extract the most information, you really need to use the actual size of the changes. Simply compute the size of the swing in each poll, then calculate the mean and standard error of the mean (SEM). For N polls, SEM is standard deviation divided by sqrt(N). The size of the swing divided by SEM is the "z score" for whether a significant swing occurred. It's outside the 95% confidence band if z>2, more or less. More precisely, consult a table of significance values for the t-test.

An even better way that resists outliers is what I have done at the Princeton Election Consortium, which is to use the median. There is a way to get an effective SEM this way as well. In that case I found that McCain had a 9-10 point convention bounce (which of course has dissipated by now).

____________________

jgfellow:

Mark,

You've gotten some good comments on probabilities. Here are my two cents:

1) Monte Carlo Simulation is a good bet. I'd second (fifth?) that recommendation.

2) Simulate individual respondents, not polls. The polls have a (virtually) continuous distribution, but the respondents have a truly trinomial distribution (Obama/McCain/Other).

3) I think that someone else said this, but to be clear: a trinomial simulation will understate the probability of large swings, because of the weights that pollsters apply. You need to account for this in your simulation, but I believe that one can do this relatively easily.

4) I think that you ought to clarify what you really care about: Do you want to know the probability that 3, 4 or 5 polls break in the same direction? That can be done. Alternatively, do you want to test the probability that today's polls represent a significant difference from yesterday's? That can be done also, but it's a different simulation / test.

I love your website. I happen to be a predictive modeling actuary by trade. I composed a quick excel workbook which actualizes the simulation I describe above. I would be *delighted* to share it with you!

Please post to this thread if you have any interest...

Cheers!

____________________

In regard to trinomials and so on, I do not recommend going in this direction. Although the case I presented is a simplification (to the binomial case) it does gives an approximation of what you might expect. Using only the sign of the change discards information that can be used to help you get what you want.

It is common to rely on numerical (Monte Carlo) simulation. This has the risk of being a crutch. It substitutes brute computing force for an understanding of probability and statistics. It's worth taking a moment to design one's statistic with a better conceptual grounding.

In this case, you asked for a nonparametric test (i.e. number of increases/no change/decreases). But you are better off with a parametric test (i.e. a mean or median).

____________________

Sam Wang:

Postscript: To argue the other side, I do think numerical simulation is an excellent tool for building intuitions about how a statistic will behave. That's worth doing.

____________________

keytype123:

There is a possibility of a 269 tie based on the electoral-vote map (if Virginia goes red).What would it take for one of Maine's EV's to go to McCain i.e how close in the popular vote would he need to be in Maine or does he have to win a district and if so is this a possibilty-a rural district for example?

____________________

Loyal:

Reading this late at night after one of those weeks, so please forgive me if I'm not thinking this through completely, or I'm a bit longwinded.

Simulation makes sense as one approach -- it's not necssary for a reasonable estimate however. Still definitions are needed (this is moving towards Sam Wang's point). Is any movement (ie one more voter) significant enough to count as a bump? Probably not. But with a known variation within each poll, one could define a constant z-score of the swing to define as meaningful and then to translate in each poll to a percentage change (since we are not looking for statistical significance in any one poll, I disagree with Sam Wang here, the zscore used is completely arbitrary so long as you the same one in each poll; -- think about it as if you were doing a power calculation for a research project: N is defined in terms of the sufficency to detect an x SD increase y % of the time that it is realkly there and you get to define x and y for your own project).

Stay with me for a second here. We know that the +/- MOE is approximately twice the SD. So how about we consider a move of 0.5 sd to be arbitrarily meaningful for this discussion. Calculate that for each poll as MOE/4.

Although one could do simulations to give you probabilities of the following scenarios: move >0.5 sd to McCain, Move >0.5 sd to Obama or neither (starting with mean proportions for each of them as the actual proportions in the previous days poll, running this with specific numbers for each poll's n and previous findings). But this information is actually redundant with what you have, ie the sd which allowed you to calcualte the zscore move of each poll to define its sign (-1 , 0, or +1) - so the simulation becomes unnecessary.

Instead of a simulation, my suggestion for the next stage would be that you model these three possible outcomes: movement in either direction or and no meaningful movement (by our definition of 0.5 sd -- which is of course arbitrary as long as you are consistent across the polls).

So now you have 243 permutations (3^5), of which 2 (0.8 %) allow for all 5 to move meaningfully in either direction, 10 (4.1%) allow for 4 to move in one direction with the other not increasing, and another 10 with 4 in one direction and the other moving the opposite way.

So now you could say that the probability of 4 polls increasing by a meaningful amount and the fifth poll holding steady is 4.9% by chance alone. If you don't care if the other moves in the opposite direction. then you bump it to about 9% by chance.

FWIW there are 20 combinations (10 in each direction) whereby the movement is one way in 3 polls with no movement in the other direction. That means that if three polls move in one direction and none in the other, there is about a 13.3% chance of that happening by chance alone.

One key here, of course, is that we are distinguishing no movement from movement in the opposite direction.

As I said it's late and I may regret this in the morning, but as for tonight, this is my story and I'm sticking with it.

What's left for you to do, Mark B is to define meaningful movement and tell us how many move, or how small you would have to define movement ot make it 4 up, etc. A sensitivity analysis of different definitions might be interesting.

Happy figuring.

Loyal


____________________

Considering the sampling statistics is probably unnecessary because of the additional contribution from systematic variability among pollster methods. In other words, getting into MOE/sampling SDs is either gilding the lily or not gilding it enough. Besides, once one moves in that direction, more than ever the right tool is a parametric statistic.

Loyal's calculation regarding the number of permutations implicitly assumes that increase, no change (rounded to %), and decrease have equal probability, which is only true for a very large sample. In a sample of 3000 I believe the probability of no change in percentage is about 20%, and therefore 40% for a change in either direction. In this case the probability of a 5-0 or 4-1 split, assuming no change and "wrong direction" change to be equally meaningful, is the sum of
choose(5,0)*(0.4)^5
choose(5,1)*(0.4)^4*0.6
choose(5,4)*0.4*(0.6)^4
choose(5,5)*(0.6)^5
which comes out to 42%.

____________________

For the truly obsessed, to get into the weeds a bit:

Note 1: What does "no change" mean?

When you are looking at tracking polls, what is the probability of "no change"? It is approximately the probability of a change of less than 0.5% in either direction. For N respondents, the sample SD is 100/sqrt(N) %. Thus "no change" is the probability of moving less than +/-sqrt(N)/200 sigma. For N=1000-3000 this is 13-22%.

Note 2: How does one do a median-based parametric test?

Compute the following:

1) For P polls, calculate M, the median of all P swings. For example, if P=5 and the swings are [+3 +3 +2 0 -1] where the sign indicates one candidate, then M=+2%.

2) Calculate MAD, the median absolute deviation from M. In this example that would be median([1 1 0 1 2])=1%.

3) ESD, the effective standard deviation, is MAD/0.6745. In this example, 1.48%.

4) SEM, the standard error of the mean, is ESD/sqrt(P). In this example, 0.66%.

5) z score = M/SEM. In this example z=2.0/0.66=3.0. The critical value for P=5 is z=2.57 (see this table). So the median change, 2%, is outside a 95% confidence band.

In the example given, the changes were three in one direction, one no change, and one in the other direction. Our intuition looking at the data is that a change occurred. Yet looking at just the sign of the change is insufficient.

A similar example, [+2 +2 +1 0 -1], would give a z score of 1.45, not significant.

____________________

EarBucket:

Quick question--on your Virginia chart, the standard sensitivity shows a slight McCain lead, but change it to either high or low sensitivity and Obama takes the lead. What's up with that?

____________________

Loyal:

Not sure re Sam Wang's objections.

In my model, I defined no change as plus or minus 0.5 standard deviations. In that case you have approximately a 38% chance of no change and a 31% chance of change in either direction. Had I suggested using 0.43 sd, there would be virutally equal probabilities of no change and change in either direction.

My use of the permutations was a conservative estimate.

Let's calculate probabilities using 0.5 sd:

For the arbitrarily named first poll, assuming no real change, there is a 62% probability of change in either direction. From there on for a (5,0) distribution you have a 31% probability of random movement in the same direction for each subequent poll. Prob random movement of 5 polls in one direction = 0.62*0.31*0.31*0.31*0.31=0.0057 or just under 0.6%.

Make it exactly four with no movement in other direction and you have Prob= 0.62*0.31*0.31*0.31*0.38=0.007

Make it at least four in one direction with no movement in the other and you have 0.62*0.31*0.31*0.31*0.69=0.0127 or approximately 1.3% which is of course, the sum of the above two probabilities.

Now if we want to make this a 95% probability test, the definition of no change can shrink to 0.1 sd, which implies that there is a 46% chance of change in either direction and only an 8% chance of no change in any individual poll.

Still your probabilities are 0.92 (change in either direction in poll 1)*0.46*0.46*0.46(same direction change in three polls)*0.54(not change in other direction) and that gives you a conjoint probability of 0.0484, or just under 5%.

The key is that the compounding across polls makes monotonic movement unlikely by chance alone.

Am I missing something here?

Thanks, Loyal

____________________

My thought would be that you need to first assume some kind of (normal?) distribution for the stochastic movement of the polls; the mean is 0 and the s.d. depends on the sample sizes and so will differ for each of the polls. Because of rounding, the "no change" outcome is really an oucome of less than a 0.5 shift in either direction (assuming the previous day is exactly at the rounded number--a superficially false assumption, but it will work on average). Thus the odds of an observed movement in either direction is simply the % of the distribution to the left/right of the -0.5 / +0.5 boundary. I don't have a normal distribution table in front of me, but with the sample sizes of each poll, it should be trivial to calculate the odds of particular outcomes.

____________________

Loyal:

Forgive my ignorance here. Why are folks suggesting within +/- 0.5 points as the range of no interest? To me, the definition of menaingful change has to be tied to the distribution of the poll. Is this a polling thing, since I come from a different tradition of numeracy?

Loyal

____________________

@Loyal:

I just mean it in terms on what we see as the published results; we don't see the actual figure, and because of rounding we can't distinguish between no movement and a movement of

Though of course, since it's continuous (or quasi-continuous, encompassing 200 million potential voters), the probability of absolutely no movement is 0. So really, there's no such thing as "no movement"--just no perceptible movement. Because we're rounding to integers, the effective threshold will be, on average, +/- 0.5 points.

____________________

Something got cut off... end of first paragraph should be "a movement of less than half a point."

____________________

Loyal:

@Andrew
Thank you. I appreciate your response. To me, the issue is movement in terms of the distribution. I am accustomed to thinking of "no meaningful difference" instead of "no difference, " or even "no significant difference".

Of ocurse, the less detail that is published, the less certain we can be of teh relative movement, so the absolute observed difference is an importnat marker.
Loyal

____________________

Mark Lindeman:

Wheeeeeee! Wow, that got pretty intense once I stopped reading.

I agree that counting the number of trackers that move in the same direction is not the best way of testing for a change in underlying preferences. I thought it was an interesting question in its own right.

It seems to me that Sam Wang's note 1 above doesn't account for the moving-average aspect of the tracker, but maybe I'm too distracted to see it.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR