Pollster.com: Mellman on What Poll Responses Mean

November 15, 2007

We have been talking quite a bit lately about the difficulty pollsters have in identifying "likely caucus goers" in Iowa. Pollsters face similar challenges in any similarly low turnout primary (a phenomenon that is more common than you think) as well as in general elections (though not to the same degree). The underlying problem for all pre-election polls is the same: When you ask people if they are "likely" to vote in a future election, they tend to be far too optimistic in their predictions. That is especially true if the pollster makes it easy to say yes, by including "somewhat likely" as an answer category.

For different reasons, the candidate that voters prefer "if the election were held today," is often different from the candidate they actually choose on Election Day.

In a must-read column in The Hill, Democratic pollster Mark Mellman helps us understand why:

For 25 years, I've counseled clients and colleagues to consider psychological research demonstrating that people are very poor reporters of their own decision-making processes. It reveals that we have little reason to believe much of what people tell us directly in focus groups and polls about why they do what they do.

Though it is heresy for a pollster to say it, the evidence also suggests people are only mediocre predictors of their own behavior. Responses to horserace questions a year out may be a special case of faulty prediction.

Mellman also cites a paper by political scientists Gary King and Andrew Gelman titled, "Why Are American Presidential Election Campaign Polls So Variable When Voters Are So Predictable?" Over at the Horse Race Blog, Jay Cost unpacks the findings of the King-Gellman paper which focus on polling in general elections. He offers this quick synopsis:

Gelman and King offer what they call the "Enlightened Preference" Model. They assert that:
(1) Voters do not have full information throughout the campaign about the "fundamental variables" that ultimately drive vote choices.
(2) Voters do use all available information to make their decisions.
(3) Voters do not rationally account for uncertainty during the course of the campaign.

This explains how polls can vary so wildly, and yet final results can be so predictable. Voters base their election decisions on basic variables. Thus, their vote choices are quite predictable. But it is only at the end of the campaign that they have fully grasped the values of the variables. Additionally, they do not factor this lack of knowledge into their thought processes. And so, when pollsters dial them up - they rely on the data they have available, but give answers that are less certain than they realize.

Both Mellman and Cost go into more detail. Both posts are well worth the click.

-- Mark Blumenthal

November 15, 2007 in Measurement Issues, The 2008 Race