Pollster.com

Articles and Analysis

 

The Accuracy of Likely Voter Models


Several recent posts have addressed whether the likely voter (LV) models are more accurate than the results based on registered voters (RV). My sense is that this is not a question that can be answered in general, but rather has to be considered separately for each polling organization.

 

Let's look, for example, at Brian Schaffner's recent post, where he compared the RV vs. LV results for several pollsters in the 2004 election. He found that "On average, the RV samples for these eight polls predicted a .875 Bush advantage while the LV samples predicted a 2.25 advantage for Bush, remarkably close to the actual result." He concludes that "it does appear as though likely voters did a better job of predicting the result in 2004 than registered voters."

 

This quick look at the data is hardly conclusive, of course, which Brian acknowledges. He had only eight polling organizations in his analysis, and despite the average results, four of the eight polls showed no advantage to using the LV model, while the other four did. As he suggests, it would be important to look at other years, but also other types of elections and other polling organizations.

 

Even then, however, an overall conclusion would not be especially helpful. Each polling organization's LV model is so different from another's that each organization has to look at its own success rate over the years to determine whether the LV model is helpful. In 1996, Gallup's senior editor, Lydia Saad, showed that in some presidential elections, Gallup's LV and RV results showed little differences, but when they did differ significantly, the LV results were more accurate in predicting the election results. That was at the national level. In the New Hampshire primary Gallup poll results over time, however, it's the RV results that typically were marginally closer to the final election results.

 

Still, at the national level, I would always bet on Gallup's LV results being a better estimate than the RV results, an example that Brian found for 2004. Gallup's final RV results showed Kerry up by two percentage points, while the LV results showed Bush winning by two points. Pew, which uses the Gallup LV model, also showed a four-point swing, from a one-point Kerry victory to a 3-point Bush victory.

 

Other polling organizations, of course, could find different results. And trying to average the results across polling organizations, to determine whether "in principle" LV models should be used, I would argue, is not helpful. That question has to be addressed by each polling organization based on its experience with its own LV model.

 

 

Comments
Independent:

Shouldn't we be talking about Registered Voter samples and Likely Voter models? The former, whatever the uncertainty built into them, are the result of of a well understood procedure, while the latter is often the end product of a proprietary processing unknown to all but the particular pollster providing the results. LV models might very well be more accurate that RV samples, but the cavalier comingling of the two sets of numbers arrived at using essentially different sets of procedures seems hardly justifiable,

____________________

Anderson:

Is it also the case that LV models are less reliable this far out from an election? Does it make a difference whether we're talking about a Gallup LV result on November 1 or August 20?

____________________

SystematicError:

"Independent" has a good point about LV models.

You could make several models of LVs. You could start with a model of the parent population being two independent sets (say 18-22 yr olds, and older) and assume any given LV filter has unique efficiencies for passing members of these two sets. Each LV filter also has its own unique fake-rates for falsely passing non-likely voters from these two sets.

Then you could do pseudoexperiments to see how harder vs. softer LV screens distort the answer. If there were only one parent population, then you might want a hard screen, sacrificing efficiency to get high purity. But with two (or more) populations, maybe not.
Depends on how different the voting preferences of the two sets are.

Then one could compare to the various LV screens (sort them into a harder-to-softer spectrum somehow) and see if things move as predicted by the pseudoexperiments.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR