Pollster.com

August 20, 2008

The "Loopy" Zogby Polls

By David Moore

All pollsters, it seems, eventually find themselves with what Andy Kohut once referred to as "loopy" results. His comment was about the Gallup polls in the 2000 election, though in September 2004, Pew experienced such results itself, and of course several polls this campaign season have produced inexplicable or "wrong" numbers, as indicated by the subsequent primary election vote counts.

This time, it's Zogby's turn to confuse the masses. His latest Reuters/Zogby poll, based on a sample of 1,089 "likely voters" drawn from listed telephone numbers, conducted Aug. 14-16, 2008, shows McCain over Obama by 46% to 41%.

Two days earlier, Zogby reported substantially different results. His online poll (of self-selected people who want to be part of his Internet polling sample) of 3,339 "likely voters," conducted Aug. 12-14, showed Obama with a three-point lead, 43% to 40%.

By Zogby's own calculation of the margins of error of each poll, the difference between the two polls in McCain's support (46% in the later telephone poll vs. 40% in the earlier online poll) is statistically significant. The difference in Obama's support (41% vs. 43% respectively) would not be statistically significant. Still, the 8-point difference in the margin of McCain's lead would be significant - a McCain 5-point lead vs. an Obama 3-point lead in the earlier poll.

If we believe both polls, the period of Aug. 13-14 must have been a real bummer for Obama and an electoral high for McCain. Whatever it was that caused millions of voters to "change" their minds and gravitate toward the Republican candidate in the two-day period, however, escaped my notice. Perhaps others have been more observant.

Of course, there are reasons to discount both polls. Zogby has long been known for refusing to use sound methods in designing his samples. The use of only listed telephone numbers, and the self-selected samples of voters in his online surveys, are the two most salient problems. Still, his last pre-election polls often come close to the actual election results, and many news media outlets regularly publish his results.

Regardless of how loopy are Zogby's results, or his sampling methods, his polls contribute to what Kathy Frankovic, in her AAPOR presidential address in 1993,[i] referred to as the "noise and clamor" of the polls. Thus, they're worth noting, if only in disbelief.

By David Moore on August 20, 2008 9:58 AM | Permalink | Comments (8) | TrackBacks (0)

August 19, 2008

The Accuracy of Likely Voter Models

By David Moore

Several recent posts have addressed whether the likely voter (LV) models are more accurate than the results based on registered voters (RV). My sense is that this is not a question that can be answered in general, but rather has to be considered separately for each polling organization.

Let's look, for example, at Brian Schaffner's recent post, where he compared the RV vs. LV results for several pollsters in the 2004 election. He found that "On average, the RV samples for these eight polls predicted a .875 Bush advantage while the LV samples predicted a 2.25 advantage for Bush, remarkably close to the actual result." He concludes that "it does appear as though likely voters did a better job of predicting the result in 2004 than registered voters."

This quick look at the data is hardly conclusive, of course, which Brian acknowledges. He had only eight polling organizations in his analysis, and despite the average results, four of the eight polls showed no advantage to using the LV model, while the other four did. As he suggests, it would be important to look at other years, but also other types of elections and other polling organizations.

Even then, however, an overall conclusion would not be especially helpful. Each polling organization's LV model is so different from another's that each organization has to look at its own success rate over the years to determine whether the LV model is helpful. In 1996, Gallup's senior editor, Lydia Saad, showed that in some presidential elections, Gallup's LV and RV results showed little differences, but when they did differ significantly, the LV results were more accurate in predicting the election results. That was at the national level. In the New Hampshire primary Gallup poll results over time, however, it's the RV results that typically were marginally closer to the final election results.

Still, at the national level, I would always bet on Gallup's LV results being a better estimate than the RV results, an example that Brian found for 2004. Gallup's final RV results showed Kerry up by two percentage points, while the LV results showed Bush winning by two points. Pew, which uses the Gallup LV model, also showed a four-point swing, from a one-point Kerry victory to a 3-point Bush victory.

Other polling organizations, of course, could find different results. And trying to average the results across polling organizations, to determine whether "in principle" LV models should be used, I would argue, is not helpful. That question has to be addressed by each polling organization based on its experience with its own LV model.

By David Moore on August 19, 2008 11:10 AM | Permalink | Comments (3) | TrackBacks (0)

August 5, 2008

Why Should Pollsters "Cringe" at the Undecided Vote? (Panagakis-Moore, cont'd.)

By David Moore

Nick Panagakis' response to my column on a different approach to measuring vote choice reflects, I believe, the current conventional wisdom, that a forced choice vote choice question is the best predictor of how voters will cast their ballots. This approach, Nick argues, "historically comes close to the actual outcome." Not only that, he "cringes" when he sees pollsters hedge their bets on a poll, by saying "candidate A is up by 9 points - but 30% could change their minds." He says that reporting such numbers "devalues polls."

But what is the "truth" of the matter? Are we not interested in accurately portraying what the electorate is thinking "today"? If so, how can we say, as CNN does, that 100 percent of voters have made up their minds with more than three months before the election? Or as Gallup has been telling us for the past two months, that an average of 95 percent of voters have already made up their minds? Or even, as most other pollsters say, that over 90 percent have made a choice?

Pollsters get away with producing such dubious numbers, I think, because most pundits take a schizophrenic approach to the polls. At one level, they treat the results as though they are the Holy Grail. At the next moment, they dismiss the numbers as being irrelevant at this time of the campaign season, saying that we need to wait until after the conventions before people begin paying attention to the election. Dan Rather's recent column encapsulates this sentiment, headlined as "Summer polls in the presidential campaign are pure folly."

If we are concerned about devaluing polls, we might want to think about giving an accurate portrayal of what the public is actually thinking (or not thinking) weeks and months before an election. The current vote choice question clearly does not reveal the extent of public indecision, and thus, I think, undermines the credibility of polls more generally.

I am not arguing that shortly before election day, in their last pre-election polls, pollsters should not press voters for their choices. I agree that in most elections, even the "undecided" voters have an inkling of whom they will support. Barring last minute media coverage that favors one candidate or the other, the faint-hearted leanings of these undecided voters usually turn out to be decent predictors of how they will act when they get in the voting booth. (Notable exceptions at the national level occurred in the 1948 and 1980 presidential elections, of course, not to mention the 2008 New Hampshire, South Carolina, and California primaries, among others).

Still, during the campaign leading up to the election, why should pollsters "cringe" at reporting that a large segment of the population remains undecided? In fact, that's just what CBS News has done, commendably in my view, when it headlined its latest poll results as "Poll: Obama Leads, But Race Fluid." Nick, it seems, would not favor such a headline, nor apparently would most other media pollsters - at least as indicated by their own reports.  

There may be better ways to get at voter indecision, other than asking first, if people have made up their minds. Andy Smith of the UNH Survey Center said he will be experimenting this election season with other approaches, which could include the names of the candidates, as well as asking voters who they expect to vote for in November (not "today"), with the tag line, or haven't they made up their minds yet? A follow-up question could probe their leanings, but at least up front, the question would explicitly allow for the undecided voter to indicate such a sentiment.

By David Moore on August 5, 2008 10:50 AM | Permalink | Comments (4) | TrackBacks (0)

July 25, 2008

A Different Approach to Measuring Vote Choice (and Lack of Choice)

By David Moore

In the recent release of the Granite State Poll, Andy Smith (director of the UNH Survey Center) noted that Barack Obama led John McCain by three percentage points, 46 percent to 43 percent, with 3 percent favoring another candidate, and 8 percent undecided. In the next paragraph, however, he noted that "only 51 percent of likely voters say they have definitely decided who they will vote for, 21 percent are leaning toward a candidate, and 28 percent say they are still trying to decide."

The second sentence may seem incompatible with the first - 8 percent of voters undecided, at the same time there are 21 percent leaning and another 28 percent still trying to decide - but it's a compromise that allows Smith to use the standard vote choice question, while still measuring the extent of voter indecision.

The standard forced-choice (who would you vote for if the election were held "today") approach produces results showing that more than nine in ten voters have already made up their minds about whom to support for president. Such a finding defies credulity, as I argued in a previous post, because of many other indicators that suggest a substantial proportion of voters have not even begun to think about the election. This was a particularly problematic result during the early primary season, when anyone who had even a dollop of experience with elections knew that primary voters had not made up their minds weeks and months ahead of their respective elections, despite what the polls said.

During the New Hampshire primary season, CNN's Keating Holland and Smith experimented with a different approach for measuring voter preferences. I had suggested a dichotomous question up front, asking if voters had made up their minds (or not) who to vote for on primary election day, but Holland and Smith came up with an alternative three-part response - to ask up front if voters had definitely decided whom to support on primary election day, if they were leaning, or if they were still trying to decide. Following this question, regardless of the answer, all respondents were asked the standard vote choice question, whom they would vote for if he election were held today.

With this format, the pollsters were able to determine how committed voters were to a choice in January (primary election day), and also to measure their top-of-mind preference if the election were held "today." Asking the undecided question first did not appear to influence respondents' willingness to give a preference to the second question, thus allowing CNN and the UNH Survey Center to report numbers that were comparable to what other polling organizations were doing - but still being able to indicate the size of voter indecision.

In the final pre-election poll, the CNN/UNH Survey Center results were as close to the actual outcome on the Republican primary as any of the other polling organizations. On the Democratic side, the polling results were similar to the average of the other polling organizations, showing Obama over Clinton, when in fact Clinton won. But CNN and the Survey Center were able to announce up front that with three days to go, 21 percent of the Democratic voters said they were still trying to make up their minds - suggesting the potential for movement.

Because the experiment appeared to provide additional insight into the state of voters' minds, Smith has continued to ask the undecided question up front in the general election polling. That's what gave him the results noted at the beginning of this post.

There are several ways to report the results. In accordance with standard practice, Smith focuses on the "today" results. Alternatively, he could focus on the results that treat the "still trying to decide" as though, in fact, they are undecided. Both results are shown in the table below:

A more detailed table of filtered results would show the following:

By the way, it's clear that McCain does better than Obama among people who say they have not yet decided whom to support, which is why the margin is 6 points in the filtered version and just 3 points in the standard version. Table 3 shows the crosstabs:

If the above results are typical of national polls, then one reason that McCain may be competitive with Obama, despite the underlying factors that suggest a Democratic election year, is that voters who haven't yet made up their minds are more likely to have heard McCain's name. When pressed by pollsters who they would support "if the election were held today," they mention the more familiar name. That doesn't mean, however, that come election day, they will actually vote for McCain.

So, which is the more accurate representation of the results - the one showing just 8 percent undecided, or the one showing 29 percent undecided?

My own preference would be to report the results as shown in Table 2, or in Table 1 in the "filtered" column. Those results are not comparable to the way most polling organizations present their figures, but I think they give a more accurate picture of the state of the electorate's collective preferences than the standard approach. After all, it's difficult to argue that at this time in the campaign season, 95 percent of voters have already made up their minds.

However, the approach that Smith follows may be seen by the news media as more acceptable - initially focus on the standard vote choice results, but also follow up that presentation with figures showing how committed or undecided the electorate is, based on the undecided question that is asked first.

Comments?

Continue reading "A Different Approach to Measuring Vote Choice (and Lack of Choice)"

By David Moore on July 25, 2008 4:02 PM | Permalink | Comments (4) | TrackBacks (0)

July 17, 2008

Moore: "Swing Voters" Redux - CBS/NYT vs. Gallup

By David Moore

Today's Guest Pollster article comes from David W. Moore, a senior fellow with the Carsey Institute at the University of New Hampshire. He is a former vice president and senior editor with the Gallup Poll, where he worked for 13 years, and is the founder and former director of the UNH Survey Center. He manages the blogsite, Skeptical Pollster.com.

In a post last week, I suggested the size of the group that Gallup calls "swing voters" was probably a significant under-estimate of the actual proportion of the electorate that is up for grabs. A new CBS/New York Times poll seems to confirm my suspicions, reporting the equivalent swing voter group at one and a half times greater than what Gallup reported - 36 percent vs. 23 percent respectively.

The Gallup report defined "swing voters" as those who were "undecided" (6 percent), plus those who initially supported one of the two major candidates but then admitted they could change their minds before election day (17 percent). The same criteria, applied to the CBS/NYT poll, suggest a larger swing voter group because this poll has a larger undecided group than the Gallup poll (12 percent), and a larger number of voters who initially chose a candidate but said it was too early to say their minds were made up (24 percent).^*

There is almost a month between the two polls, Gallup's conducted June 15-19 and the CBS/NYT poll conducted July 7-14. So, the difference in the size estimates of the swing voter group could be a function of time. If so, that leads to the somewhat counterintuitive conclusion that this past month's campaigning has led to an increase in voter uncertainty, rather than the reverse, as more conventional frameworks might predict. I'm not a fan of this unconventional theory, though CBS reports that according to their poll, the undecided vote increased from 6 percent to 12 percent in the past month. Still, I suspect the differences between the two polls are mostly caused by house effects, but it would be hard to prove one way or the other.

In my post last week, I suggested the actual proportion of the electorate up for grabs is probably greater than the 40 percent figure, found by Gallup in September 1996 in the Robert Dole - Bill Clinton contest. The CBS/NYT poll lends credence to my suspicions, even though it also used the forced-choice format ("who would you vote for if the election were held today?") that Gallup used last month. In September 1996, Gallup first asked if voters had made up their minds, and then asked voters who they preferred. If the CBS/NYT poll had used that format this time, it almost certainly would have found a larger group of swing voters than the 36 percent they just reported.

In any case, CBS has rightfully emphasized in its headline the most important conclusion from these data: "Obama Leads But Race Looks Fluid." Too few pollsters, and too few news organizations, are looking at the fluidity of the electorate. Instead, like the New York Times article, they let stand the forced-choice horserace numbers as though such figures are solid estimates of voter intentions.

While I applaud CBS and Gallup for pointing to the fluidity of the race, I think it's worthwhile saying again that the uncertainty in the race is not because many voters may change their minds before election day, but rather because many voters have not yet made up their minds. The notion that 90 percent or more of voters have already come to a conclusion as to whom they will support (even a conclusion they can change) is highly misleading - an artifact of poor question wording that pollsters should have long since modified.

^* The CBS/NYT poll shows that 28 percent of those who initially made a choice then admitted their minds were not made up. The table shows that 86 percent made a choice (45 percent for Obama, 39 percent for McCain, 2 percent for other). The two percentages multiplied by each other give 24 percent. The latter figure added to the 12 percent who originally said they were undecided produces a 36 percent total for "swing voters."

By David Moore on July 17, 2008 10:27 AM | Permalink | Comments (5) | TrackBacks (0)

July 11, 2008

Moore: USA Today's Cluster Analysis of Voters - How Useful?

By David Moore

Today's Guest Pollster article comes from David W. Moore, a senior fellow with the Carsey Institute at the University of New Hampshire. He is a former vice president and senior editor with the Gallup Poll, where he worked for 13 years, and is the founder and former director of the UNH Survey Center. He manages the blogsite, Skeptical Pollster.com.

On Thursday, July 10, USA Today published an analysis of voter intentions that produced "six types of voters" who the paper claims "will decide the presidential election." The types included: true believers (30 percent of the electorate), up for grabs (18 percent), decided but dissatisfied (16 percent), fired up and favorable (14 percent), firmly decided (12 percent), and skeptical and downbeat (12 percent).^* As Mark Blumenthal indicated, this is a fascinating analysis, but how useful is it for understanding the election?

The six types of voters were produced using cluster analysis. This statistical technique is similar to factor analysis, except that it classifies respondents into distinctive groups, while factor analysis classifies various opinions into distinctive groups. Without going into the details of how the technique works, I think it's sufficient to note that the analyst has a great deal of control over the types of groups produced by cluster analysis. The analyst chooses the variables that are used to classify respondents, and also determines how many groups the cluster analysis produces. The fact that the analyst chose six clusters, instead of any other number between two and ten, was purely a subjective decision.

What is most surprising about the analysis is that it is issue free. The stereotypical complaint by political observers about the news media is that reporters focus on the horserace almost to the exclusion of any real substantive issues. This USA Today analysis fits that criticism to a T. I believe there is a widespread consensus among political observers these days that the war in Iraq (and national security more generally), the economy, and healthcare are among the most salient issues dividing the two major presidential candidates. Yet, there is nothing in the newspaper's analysis that groups voters according to their views on any of these major issues. Nor is there any mention of party identification, which often acts as a catch-all variable for a host of issues.

The variables chosen to classify respondents were 1) respondents' enthusiasm about the election, 2) whether respondents think the election would make a difference to them, 3) respondents' opinions (favorable or unfavorable) of each of the two major candidates, and 4) how certain respondents were to vote for the candidate of their choice. As these variable make clear, the classification scheme focuses almost exclusively on election turnout factors, with no mention of issues. Even the favorability ratings can be considered turnout variables in this context, because voters who are negative about both candidates are least likely to vote, while those who are positive about both candidates are mostly likely to vote. This is not to say that a mostly horserace-driven analysis, as this one is, doesn't provide some insights into the electorate. There are many different angles from which to analyze the electorate, and this is certainly a valid one. To me, it's just not as interesting as one that is more political in context.

Like most political junkies, I find intriguing almost any statistical analysis of polling data that goes beyond the simple marginals, and USA Today should be congratulated for making the effort. Still, I'd like to see a little more politics thrown into the mix - even if only to take these six types and describe their party identification, as well as their responses to other public policy questions. But mostly I would like to see a completely new cluster analysis that included policy attitudes as the defining variables for the groups. This is not to say that issues alone will determine the election. But I don't think we can get a good read on the electorate, and which types of voters will ultimately "decide the election," if we ignore issues altogether.

^* The percentages exceed 100 percent because of rounding error.

By David Moore on July 11, 2008 5:14 PM | Permalink | Comments (1) | TrackBacks (0)

July 9, 2008

Moore: Gallup's "Swing Voters" - A Major Underestimate?

By David Moore

Today's Guest Pollster article comes from David W. Moore, a senior fellow with the Carsey Institute at the University of New Hampshire. He is a former vice president and senior editor with the Gallup Poll, where he worked for 13 years, and is the founder and former director of the UNH Survey Center. He manages the blogsite, Skeptical Pollster.com.

In a recent post, Gallup's Jeff Jones reports that for the first time this election cycle, Gallup has measured the number of "swing voters" in the electorate. That's certainly a step in the right direction, but one might well wonder why it took so long for pollsters to admit that there is a substantial proportion of the public not committed to a candidate.

According to the post, Gallup finds that only 6 percent of "likely voters" are undecided as to which presidential candidate they will support. That number defies credulity. With five months to go in the campaign, neither major candidate the incumbent, no vice presidential candidates chosen, and no debates between the presumptive nominees, Gallup wants want us to believe that 94 percent of voters have already made up their minds? Yes, indeed! Not only that, CNN says 99 percent are decided. Time says 92 percent. Newsweek claims 87 percent. USA Today with Gallup says 97 percent. ABC/Washington Post - 96 percent. The NBC/Wall Street Journal poll says 90 percent. (For sources, see The Polling Report.)

With all these major media polls (not to mention numerous other polls not affiliated with the major news media organizations) in rough agreement that about nine in ten voters or more have made up their minds, any challenge to this conventional wisdom may seem futile. But here's something to consider. In a Sept. 3-5, 1996 Gallup poll, 40 percent of voters said they were undecided about whom they would support in the November presidential election between Robert Dole and Bill Clinton. How could such a large number be undecided in that poll - taken after the major party conventions and with just two months to go before the election, in which there was a popular incumbent candidate - and yet so few voters admit they are undecided in the current polls?

The answer, of course, lies in the way the voting question is asked. The standard vote choice question, which dates to 1935 when George Gallup first asked about presidential preferences, is deliberately designed to obfuscate the number of undecided voters. Gallup knew that the press wouldn't be interested in results that showed perhaps a majority of voters undecided months before an election, so he asked respondents which candidates they would vote for "today."¹ And for the past seven plus decades pollsters have blindly followed that same format. In the September 1996 poll, however, Gallup abandoned the standard vote choice question, and instead first asked voters if they had even made their decision as to which candidate they would support. In that context, 39 percent said they hadn't, and an additional one percent were unsure.

In the current Gallup report, mentioned at the beginning of the article, Gallup retains the forced-choice standard format, but follows up by asking respondents if they could change their minds before election day. Those who said they could - 9 percent who initially said they would vote for McCain if the election were held "today," and 8 percent who initially favored Obama - were added to the six percent who initially said they were undecided, producing a 23 percent group Gallup characterizes as "swing voters."

Thus, according to Gallup, about a quarter of the electorate is up for grabs. I'm skeptical about that number - I suspect the percentage is much higher, perhaps even greater than the 40 percent measured by Gallup late in the 1996 campaign. But at least it's a recognition that there is a substantial number of voters who are not yet committed to a candidate.

Still, I would argue that most of the swing voters are not people who "could" change their minds before election day, as Gallup asserts, but rather people who have not yet even decided whom to support. Gallup (and any other national poll), of course, could test that proposition. All they need to do is replicate the question that Gallup asked in its September 3-5, 1996 poll: Ask voters up front if they have made up their minds whom they will support in November.² My prediction - much more than a quarter of the electorate is up for grabs in 2008.

¹ Three times in 1935, Gallup asked if respondents would vote for Roosevelt "today." The first time he pitted Roosevelt against anyone was in January 1936, when he asked: "For which candidate would you vote today - Franklin Roosevelt or the Republican candidate?" See George H. Gallup, The Gallup Poll: Public Opinion 1935-1971, Volume One (New York: Random House, 1972), pp. 1-10.

² The exact wording is, "Have you made up your mind yet about who you will vote for in the presidential election this fall, or are you still deciding?"

By David Moore on July 9, 2008 1:43 PM | Permalink | Comments (4) | TrackBacks (0)

June 18, 2008

Moore: Hunter College's LGB Poll and Prevalence Rates

By David Moore

Today's Guest Pollster article comes from David W. Moore, a senior fellow with the Carsey Institute at the University of New Hampshire. He is a former vice president and senior editor with the Gallup Poll, where he worked for 13 years, and is the founder and former director of the UNH Survey Center. He manages the blogsite, Skeptical Pollster.com.

The new Hunter College poll of lesbian, gay, and bisexual (LGB) Americans provides important insights into the lives of this difficult-to-reach population. The poll is an excellent example of what polls can do best - reveal how people view their own experiences, thus providing history with important information on how people lived and thought at any given point of time. The forthcoming presentation¹ by the authors should provide additional information about the study.

One of the intriguing findings of the study is the percentage of people who identify as LGBs. The prevalence rate, 2.9 percent, is in line with other studies over the past couple of decades, which suggest that somewhere under five percent of Americans report they are homosexual. A decade ago, NORC's Tom Smith reported that "a series of recent national studies indicate that only about 2-3 percent of sexually active men and 1-2 percent of sexually active women are currently homosexual."² The Hunter College poll differs somewhat from these numbers, suggesting that the percentage of men and women identifying as LGBs is about equal - though half the women, and only a third of the men say they are bisexual.

It's important to recognize that these figures are lower-bounded estimates, and that the actual percentage of Americans who are LGBs is probably higher than what the polls can measure. While public acceptance of LGBs is higher now than it was, say, a couple of decades ago, there is still considerable public disapprobation of homosexual behavior. Such an environment cannot help but deter many LGBs from admitting their true sexual orientation.

In this context, it is noteworthy that the percentage of people willing to admit they are LGBs correlates with the political environment in which they live. The Hunter College poll shows that among people living in "strong Democratic states" (where John Kerry beat George W. Bush by five percentage points or more), the number of LGBs is about 3.6 percent; in swing states, it's about 3.2 percent; and in strong Republican states (where Bush won by five percentage points or more), it's about 2.0 percent.³ These differences appear to be statistically significant, though the authors could provide statistical tests to verify the observation.

If it is true that the percentages vary by political environment, there are at least two explanations. One is that LGBs move to states that are generally more accepting of homosexuals. The other is that LGBs are simply more willing to admit their sexual orientation when they live in a more favorable environment. One test would be to compare the figures by age by political environment - with the hypothesis that older LGBs might be more likely to move to friendly environments, while younger LGBs would not yet have had the time to do so. Thus, the correlation between political environment and willingness to admit that one is an LGB would be higher among older than younger people. If the rates are similar, it rules out the notion that the correlation is due to LGBs moving to a more friendly environment, and suggests instead that it is the environment itself that influences whether LGBs are willing to admit their sexual preferences.

Whatever the results, the poll itself deserves careful consideration of all of its findings. The methodology appears to be rigorous, while the findings provide innovative insights into the personal experiences and political orientation of LGBs.

² Tom W. Smith, "American Sexual Behavior: Trends, Socio-Demographic Differences, and Risk Behavior," GSS Topical Report No. 25, National Opinion Research Center, University of Chicago, updated December, 1998, p. 7.

³ These percentages are my recalculation of figures provided in the report in Table 2. The authors should be able to provide more precise calculations.

By David Moore on June 18, 2008 11:47 AM | Permalink | Comments (0) | TrackBacks (0)

June 6, 2008

Moore: The Frontrunner Myths

By David Moore

Today's Guest Pollster article comes from David W. Moore, a senior fellow with the Carsey Institute at the University of New Hampshire. He is a former vice president and senior editor with the Gallup Poll, where he worked for 13 years, and is the founder and former director of the UNH Survey Center. He manages the blogsite, Skeptical Pollster.com.

Eons ago, it seems, the press was touting Rudy Giuliani and Hillary Clinton as the dominant frontrunners in their respective party presidential contests. The press was wrong in doing this, of course, but the pollsters told them that was true, and journalists believed. Now ABC's Gary Langer has taken a "Look Back" at the 2008 primary season, and once again endorsed the myth of the two frontrunners:

"It was going to be short and simple: Hillary Clinton vs. Rudy Giuliani. Those were the long-ago and far-away days of initial preferences, when the two best-known candidates held commanding leads for their parties' presidential nominations. That it didn't end that way underscores an eternal truth of American politics: Campaigns matter."

I agree with Langer that campaigns matter, but disagree with his starting point. Indeed, that Giuliani was ever proclaimed the frontrunner is perhaps the most amazing myth of this whole campaign season.

The contest for delegates, as everyone knows, begins with voting in Iowa and continues from state to state, with election results in the early states inevitably affecting the results in later states. During the time that Giuliani enjoyed his so-called "commanding" frontrunner status (in the summer and fall of 2007), he was not the frontrunner in any of those early state contests - not in Iowa, not in New Hampshire, not in Michigan, not in Nevada, and not in South Carolina. He was the frontrunner in Florida, but if he didn't win any of the previous contests, it wasn't likely he would even be viable, much less the national frontrunner, by the time that primary was held.

This isn't just 20-20 hindsight.¹ Right from the beginning, critics challenged the media pollsters' use of "national Republicans" and "national Democrats" as indicative of what the voters were thinking. In fact, Langer acknowledged the problem back in July 2007, and it's worth citing his response:

"A colleague here sent me a nice pointed challenge to our latest election poll yesterday: National surveys by themselves are 'close to meaningless,' he said, because they measure national preferences in what'll really be a series of state caucuses and primaries.

"It's a fair complaint, and a serious one - because it cuts to the heart of just what our new survey, and its multifarious brethren, are all about. It's true, of course, that a poll of current preferences nationally does not tell us about current preferences in Iowa, New Hampshire or anywhere else. Without knowing who's thriving in Iowa and New Hampshire, it's hard to predict who survives to South Carolina, much less who wins where on Mega Tuesday and wakes up with the crown on Feb. 6....

"We ask the horse race question in our national polls for context - not to predict the winner of a made-up national primary."

Langer is absolutely right - national polls of the party faithful don't predict state winners, and without an idea of who they might be, there's no way to tell who the nominee might be. By this reasoning, no matter how well Giuliani might have been faring in the national polls, that said nothing about how he might do in the state contests and in his effort to win the presidential nomination. So, on what grounds was he the frontrunner?

It turns out, apparently, that all along ABC was using the national numbers of what Langer calls the "made-up primary" not just "for context," but in fact to predict the winner of the actual nomination process. That's the only way in which Giuliani could be called a frontrunner.

Of course, ABC was not alone. Every major media polling organization reported results, at one time or another, based on that "made-up national primary." And in the summer and fall of 2007, they all reported that Giuliani was the dominant frontrunner - while ignoring that he trailed in all of the early state contests.

Similarly, Hillary Clinton was hardly the "solid" favorite as virtually every major news organization claimed. It's true the polls showed her leading in the several primary states after Iowa, but in this latter state she was never dominant. She trailed John Edwards for the first seven months of 2007, until she moved into a modest lead in the late summer and fall. But there were many undecided voters, and if she lost in Iowa, who could predict how she might fare elsewhere? Howard Dean's experience four years earlier, when his leading status in New Hampshire evaporated in the two-day period following his loss in the Iowa Caucuses, should have been a cautionary note for pollsters.

The reality was that in the summer and fall of 2007, there was no Republican frontrunner, and the Democratic frontrunner had only a tenuous lead. That so many pundits and politicians and members of the general public still think otherwise, because that's what the pollsters told us, should be the biggest embarrassment of the polling industry since Dewey beat Truman in 1948.

By David Moore on June 6, 2008 3:18 PM | Permalink | Comments (17) | TrackBacks (0)

May 27, 2008

Moore: "Noise and Clamor" 2008

By David Moore

Today's Guest Pollster article comes from David W. Moore, a senior fellow with the Carsey Institute at the University of New Hampshire. He is a former vice president and senior editor with the Gallup Poll, where he worked for 13 years, and is the founder and former director of the UNH Survey Center. He manages the blogsite, Skeptical Pollster.com.

This month, the American Association for Public Opinion Research (AAPOR) gave its most coveted honor, the AAPOR Lifetime Achievement Award, to Kathleen A. Frankovic, director the CBS News polls and a former AAPOR president. In her acceptance speech, she referred to her presidential address of a decade and a half ago, when she leveled several incisive criticisms at the media polls - criticisms that deserve to be re-examined today.

Since joining CBS News over thirty years ago, Frankovic has amassed an impressive set of accomplishments, including being president of both AAPOR and its sister organization, the World Association for Public Opinion Research (WAPOR); a member of the Market Research Council; a trustee both of The Roper Center for Public Opinion Research and, separately, the National Council on Public Polls; and a former chair of the Research Industry Coalition. She is also the author of numerous published articles and book chapters on public opinion. If anyone could be considered a pillar of the polling industry establishment, she would be it.

Yet, her writings on polls have not been effusive encomiums to the presumed benefits they bring to society. While always attributing much importance to the role of media polls in American politics, she has also expressed concerns about them, nowhere more evident than in her 1993 AAPOR presidential address, "Noise and Clamor: The Unintended Consequences of Success."¹

Her theme is reflected in the title, as she raised questions about the increased frequency of polls and the lack of thought that goes into many poll questions - "Immediate response is more important than what the response is or what it really means. In other words, we may no longer have to think." She also worried about the decreased value and import of polls - "It's so easy to conduct polls now that it may actually cheapen the value of each one we do. Instead of meaning, we may just be getting noise - noise and clamor."

She noted that with the advent of scientific polls, we now have a "continuous ballot box," the dream of early democratic idealists. But is that good? Not always, apparently. In the several months prior to her presidential address, polls taken at two-week intervals had showed President Bill Clinton's approval rating bouncing all over the map, "from 58 percent to 53 percent to 59 percent to 53 percent to 57 percent to 49 percent to 57 percent to 45 percent." She lamented, "This is information, but how informative is it? It's almost like what Truman Capote once remarked about Jack Kerouac's novel, On The Road: 'That isn't writing-it's typing.' Continuing ballot boxes shouldn't bounce around so much." Indeed.

Today, the uninformative information provided by polls is even more acute, obvious to anyone who has followed the pollsters' fascination with the 2008 national Democratic electorate. It isn't the results at two-week intervals, but contemporaneous results that bounce all over the place these days. One has to look only at pollster.com from April 30 to May 4 to find five polls with three different results: Gallup by itself, reporting a dead heat (Obama up by two points); Gallup with USA Today and, separately, AP/Ipsos each reporting Clinton leading by 7 points; while CBS/NYT and Diageo/Hotline each reporting double digit leads (12 and 11 points respectively) for Obama. And this isn't the only time Gallup has contradicted itself this campaign season, or that different polling organizations have come up with contrasting results when interviewing in the same time period. (See comments by ABC's Gary Langer, Dec. 12, 2007 and Feb. 26, 2008; and Mark Blumenthal's "Dueling Gallups.")

Despite her criticisms, Frankovic proposed no remedies, nor special panels to investigate the problems, perhaps in recognition that they might entail a fundamental change in the way that polls are currently conducted. In the wake of the miscalls in the New Hampshire Democratic Primary, AAPOR's president, Nancy Mathieowetz, did in fact establish a special panel "to examine what occurred, provide a timely report of our findings, and promote future research on pre-election primary polls." No such panel has been established to examine all the subsequent conflicting polls, though the New Hampshire panel might want to consider broadening its scope. The "continuing ballot boxes" are not just bouncing around, they're running into each other going in opposite directions.

Frankovic concluded in her presidential address in 1993 that "We have achieved the ability to cut through the noise and clamor of unscientific measures, even as we risk making some noise and clamor of our own." This observation suggests that the radical question the AAPOR panel needs to address is whether the noise and clamor of "scientific" polls is any better than that of the unscientific ones.

By David Moore on May 27, 2008 2:00 PM | Permalink | Comments (0) | TrackBacks (0)