Pollster.com

Articles and Analysis

 

More on the "Convergence Mystery"

Topics: Convergence Mystery , David Moore

My colleague David Moore has posted here and on AAPOR's Survey Practice about the intriguing convergence of the national polls at the very end of this year's fall campaign (I made a similar observation in assessing the final round of national polls last month). I wondered whether the same phenomenon occurred at the state level. What follows is some preliminary data and my own answer to Moore's question about why this happens.

First, let's review Moore's data. His Survey Practice posting includes the following table, which presents both the average Obama lead and the variance in that lead on national surveys broken out across six different time periods on October.

081205-Moore_Table

As Moore points out, "Obama's average lead each week varies only slightly over the whole month of October," yet the "substantial" variability of the polls drops off significantly in the final week. "Why," Moore asks, "do different polls show such variability over the month of October, and then suddenly converge in the last week of the campaign?" He continues:

Of course, it's true that opinions "crystallize" in the final weeks, but why should that make polls so relatively unreliable during the campaign? Shouldn't polls conducted at the same time produce the same results, even if many people are still mulling over their decisions? Shouldn't different polls find the same proportion of indecisive people?

Moore adds that the editors of Survey Practice will be asking media pollsters for their explanation and analysis of the "convergence phenomenon," reactions that they plan to publish.

I wondered whether we would see the same pattern in statewide surveys. So I replicated Moore's calculation within the twelve battleground states in which we logged 20 or more polls during October and November (Colorado, Florida, Georgia, Minnesota, Missouri, Nevada, New Hampshire, North Carolina, Ohio, Pennsylvania, Virginia and Wisconsin). Then it calculated the mean and median variance across all 12 states, along with the average Obama margin over McCain in each.   

As the table below shows, the same general pattern occurs: Even though Obama's average lead remains roughly constant throughout October, the variance in that lead is even larger at the state level before a marked convergence over the last week of the campaign [I have also copied a table showing the values for each state individually, including counts of polls in each, after the jump].

081205_StateLevelTable

So what's the explanation for this convergence phenomenon? I too look forward to reading the response of the media pollsters in Survey Practice, but I have my own theory. I know that many will wonder, as reader DTM does, "if there was some deliberate convergence." A deliberate "thumb on the scale" may have happened in some cases. But I want to suggest a more benign explanation.

Just about every political pollster that I know cares deeply about the accuracy of that last poll before the election. It is, as the cliche goes, the pollster's "final exam" every two years. Our clients use that final poll as a test of our accuracy, and our credibility and business are at stake every election.

So now put yourself in the position of a pollster who is producing results in any given race that are out of line with other public polls. Trust me, everyone notices, and your paying clients will ask you why your survey is so different from the rest. Every pollster in that position will check everything they can about their own survey and methods -- the sample, the interviewing procedures, the weighting, the screen questions and so on. Keep in mind that even the most by-the-book media pollsters leave themselves a fair amount of room for subjective judgment calls about their likely voter screen questions, models and weighting procedures.

Now consider this basic observation: Pollsters that look hard enough for something amiss will likely find and "correct" it. Those that don't -- those that feel comfortable that their estimates are in line with other pollsters -- will not look as hard for "problems" and, thus, are less likely to find anything in need of "fixing." Take that underlying tendency and apply it to the actions of the many different pollsters over the final weeks in October and I think you have an explanation for why results tend to converge around the mean.

This is just a theory, of course, but whatever the explanation, it should leave us questioning the value of trying to sort out "good" pollsters from "bad" using accuracy measurements based only on "the last poll."

The following table shows the results for each state individually:


081205_AllStateData.png

 

Comments
William Ockham:

I think you and David Moore are asking the wrong question. Instead of asking why did the polls converge, you should ask why was there so much variability in October. All available evidence suggests that by October 1, the race had totally stabilized. Your own figures show that the average Obama lead for October polls was 7.3%. His actual win was (using the almost final numbers on the Wikipedia site) by 7.25%.

I believe that the stability of the race was subjectively obvious on October 1. Indeed, I made an electoral college prediction for every state on that day (proof available on request). The only state I missed was Missouri (and who knows what a recount there might haved done). I even called NE-CD2 for Obama. Now, maybe I got lucky, but the only states that were close on October 1 were Missouri, Indiana, and North Carolina (some folks thought North Dakota was close, but that was clearly poor polling).

So, if we re-phrase your question, to be something like this:

The 2008 Presidential race was remarkably stable for the last month, so why was there a huge variance in the polls?

Now, we can get a better answer. I would posit that pollsters are rewarded for being right on the last poll, so they make sure those as accurate as possible, but leading up to those last polls, they are rewarded for 'making news'. This leads to an unconscious thumb on the scales towards showing 'movement' whether or not any exists.

That leads to an obvious conclusion on how to evaluate pollster performance in this year's Presidential election. Assume that 7.25% is the correct value for Obama's lead for any poll done in October or November. For any particular poll, it is valid if 7.25% falls within it's margin of error. If not, it had a problem larger than sampling error.

____________________

cinnamonape:

Hmmm! If it is that they a "scrupulous" on their last poll because the "buyers" care about that one...why would anyone subscribe except for the final poll?

Basically it suggests that there are certain pollsters who are sloppy, using bad evaluation and weighting statistics, crappy methodology, poor staff, or whatever...and the subscribers don't care? Or perhaps the subscribers WANT TO hear the results their polls are providing?

Certain buyers, with certain ideological nits to pick, subscribe to certain polls...and then can use those results...to fundraisers, donors, volunteers, party heads, etc.

And why would these polls CONSISTENTLY make these errors in a particular direction, and THEN converge? How many jump from +4 to the trend one week, and then -3 another. That would show a "non-selectivity" to their errors. But clearly their errors are either recursive, or intentional.

And they can get away with this "give the customer what they want" approach - at least until the final poll. Then there is the actual, hard reality that catches up. No one can test the validity of a poll on October 4th, after all. But one on November 1st had better be close.

____________________

cinnamonape:

"Indeed, I made an electoral college prediction for every state on that day (proof available on request). The only state I missed was Missouri (and who knows what a recount there might haved done). I even called NE-CD2 for Obama."

I recall someone having a contest on accurate EV and Congressional seats. I actually got "informed" lucky as well...guessing 365. I can't recall what I guessed for the Senate seats but I think I said 58...and then there was Congress, and I guessed a gain of about 20 seats for Dems. So I'll likely be pretty close.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR