May 06, 2008

Response to Doug Usher

Humphrey Taylor has served as chairman of The Harris Poll, a service of Harris Interactive, since 1994.

Thanks to Doug Usher for his contribution to the debate on the panel-based online methodology for political polling. I am glad to see that he acknowledges the value and viability of this method for national polls. But I am puzzled as to why he thinks this method will not work in Congressional races (at least I think that is what he is saying). He writes of the "sour spot" based on the fact that political polls need to reach "a narrow population for which pollsters do not have well defined web contact information".

I assume he means by this that we cannot sample a geographic area because we do not know where people live. Of course we can, and we do this easily. We and others who have large panels, know the states in which people live, so that takes care of senate races.

What about congressional districts?

Some panels also have zip code information, and those that do not can screen for it. In so far as some zip codes straddle the boundary with another district we can screen for streets or even addresses if necessary. And of course this problem is the same ,or possibly worse, for RDD telephone samples ,as telephone exchanges may also straddle the boundaries between districts. Furthermore many people now take their telephone numbers with them when they move from one district to another.

My comment that was quoted by Doug Usher was taken out of context. I certainly believe that online political polling methods are "the wave of the future". My mention of cell phones was specifically in reference to telephone surveys of people aged 18 to 29. I am sorry if that was not clear.

--Guest Pollster on May 06, 2008 in Internet Polls | Permalink | Comments (0) | Trackback (0)

May 01, 2008

Predictive Accuracy of the 2008 Pennsylvania Primary Pollsters

Berwood Yost is Director of The Floyd Institute's Center for Opinion Research at Franklin & Marshall College. Kirk Miller is B.F. Fackenthal Professor of Biology and Senior Research Fellow at The Floyd Institute's Center for Opinion Research.

The 2008 Democratic presidential primary on April 22 put Pennsylvania in the national spotlight for a long six weeks. Members of the media followed the candidates into the Keystone State intending to learn more about its people and its politics. Not far behind the media came the pollsters--some media even brought their own pollsters. Pennsylvania voters were besieged by pollsters in unprecedented numbers. There were 39 publicly released surveys, which included more than 30,000 interviews with the state's voters, during only the last three weeks of the campaign. This is a tremendous increase in polling activity compared to the 26 polls released in the final three weeks of the 2004 presidential campaign in Pennsylvania or the 15 released during the final three weeks of the 2006 Senate campaign.

Taken together, the pollsters who pestered Pennsylvanians did an adequate job of predicting the final outcome: 36 of the 39 polls in April predicted a Clinton victory and the three outliers were all conducted by the same polling organization. We agree with Charles Franklin's assessment that the aggregate performance of the Pennsylvania pollsters was good. Figure 1 is a frequency distribution of the predictive accuracy of the 39 public polls released in Pennsylvania. It shows that there was a slight bias in the polling estimates toward Barack Obama (meaning the polls in Pennsylvania underestimated Hillary Clinton's margin of victory), but that this bias was small and, according to the exit polls, not surprising because late deciding voters moved in larger proportions toward Clinton.

Figure 1 Frequency Distribution of Predictive Accuracy

Some individual pollsters faired much better than others in the accuracy of their estimates. Figure 2 shows the predictive accuracy and corresponding confidence interval for each of the 39 polls conducted between April 1 and 22 in Pennsylvania, arranged by the number of days prior to the primary the survey was completed. Those pollsters who produced a biased estimate, meaning the confidence interval for their estimate did not overlap zero, are labeled in Figure 2. Three of the four polls conducted by Public Policy Polling (PPP) were biased and all were biased toward Obama. Two of the three polls conducted by American Research Group (ARG) were biased and one of SurveyUSA's three polls showed bias. One ARG poll showed that Clinton and Obama were tied; the other, seven days later, showed Senator Clinton ahead by 20 points. The SurveyUSA poll that missed also showed Senator Clinton ahead by 20 points. The measure of predictive accuracy we used shows that the pollsters' final estimates were mostly in line with the final election results.

Figure 2 Predictive Accuracy of Individual Polls by Date of Poll

The misses identified in Figure 2 are not related to sample size. Four of the surveys that missed had four of the eight largest samples; the other two that missed had sample sizes that were only slightly below the median size. There is a relationship in these analyses, as one would expect, between sample size and the widths of the confidence intervals, but there is no relationship between sample size, width of the confidence interval, and the likelihood that a survey was biased. We don't know what methodological choices matter most in producing unbiased polls without further examination of the methodological choices the pollsters make. Some might conclude that pollsters who use inter-active voice response (IVR) technology to collect data are more prone to bias because two of the three pollsters who produced biased estimates use IVR, but not all IVR pollsters produced biased results.

Another interesting question we tried to answer is whether the polls converged on the end result as election day approached. Depending on the method used, the answer is a qualified, "slightly." Figure 3 shows the predictive accuracy of each poll as a function of days before the Pennsylvania primary. The trend line fitted to the figure is produced by a LOWESS iterative locally weighted least squares regression. The red dots identify the six biased polls noted earlier. The curve indicates that the polls began to converge until about two weeks prior to the election, that they remained relatively constant for about a one-week period, and then began to converge again over the final days of the campaign. If the six biased polls are removed from the analysis, the convergence is not dramatically improved.

Figure 3 Predictive Accuracy of Individual Polls by Date of Poll with Fitted Regression Line

Measuring Predictive Accuracy

We used the measure of predictive accuracy developed by Martin, Traugott and Kennedy (2005) A Review and Proposal for a new Measure of Poll Accuracy. Public Opinion Quarterly, Volume 69 (3): 342 - 369. Their method compares the ratio of the estimated percent of voters voting for each candidate to the ratio of the final vote tally for each. The natural log of this odds ratio (ln odds) is used because of its favorable statistical properties and the ease of calculating confidence intervals for each estimate. The confidence interval for a poll that reasonably predicts the final outcome of the primary election will overlap zero. Senator Clinton's votes or projected votes were the numerators in all the ratios we calculated so negative values for ln odds represent an overestimate in favor of Senator Obama and positive values represent an overestimate in favor of Senator Clinton. According to this measure, a poll is biased if its confidence interval does not overlap zero. The polling results used in this analysis were taken from Pollster.com.

--Guest Pollster on May 01, 2008 in The 2008 Race | Permalink | Comments (1) | Trackback (0)

May 01, 2008

Disconnected: The Internet's Endless Shortcomings for Political Polling

Douglas Usher is the Senior Vice President of Widmeyer Communications and formerly Vice President at the Democratic polling firm, The Mellman Group.

"The survey research and marketing industries need to recognize that the Internet and cellphones, not landlines, are likely to be the wave of the future." So says Humphrey Taylor, chairman of the Harris Poll.

I met Humphrey Taylor once - in 1999. He pitched Harris Online services to the Democratic polling firm where I worked, and his team said that telephone surveys in politics would likely be replaced by web surveys after that election cycle.

Was he right about political polling? Hardly - in fact, he couldn't have been more wrong. Let me make this as clear as possible: no professional political pollster on either side of the aisle has ever used web-based surveys for quantitative research in their campaign practice.

And as any pollster.com reader understands - and all serious consumers of political polling know - you can count on one hand the number of public pollsters using online methodology for political polls. Even John Zogby, who claims that his firm has "since the mid-90's... utilized the Internet as a means of providing the public with instant access to the day's best public opinion research," has like most pollsters used telephone polling this cycle.

Internet polling is a growing industry. I use it all the time for my clients - indeed, it rules many aspects of consumer research. So, why the disconnect for politics?

Because quantitative political research for nearly all levels of American politics hits the "sour spot" of internet research.

Let me explain.

Internet-based research is perfectly suited for certain types of public opinion research:

• Qualitative research: in-depth, group level research designed to evaluate reactions to specific ideas, issues, and stimuli (like campaign ads) - research which provides rich feedback, but is not projectable on the population at large. The internet provides a (virtually) limitless pool of volunteers that will provide quick feedback about a candidate, product, print or television ad. It provides a reasonable - and often less expensive - alternative to focus groups, without the travel.

• Quantitative research among broad populations: For the broadest audiences - "adults" nationally, "likely voters" nationally, and "likely voters" in some states - internet research can provide a reasonable (and again, less expensive) alternative to telephone polling. The opt-in panels that internet research vendors build - if properly cleansed and refreshed on a regular basis - have been demonstrated to be reliable proxies to telephone research for point-in-time quantitative measurement.

• Quantitative research among narrow populations where e-mail contact is previously established: This type of research includes organization membership research, or a survey of loyal customers, or an internal corporate survey. One of the fallacies spread by those who sell public opinion research services on the internet is that because people are on the web, they are reachable on-line. But unless someone provides you (or an organization) with their e-mail address, it is nearly impossible to find them. However, for internal organization research, internet research conducted of a complete (or near-complete) population by e-mail has become an excellent alternative to phone surveys.

These three types of research describe most of the public opinion research for which clients pay money - hence, the internet has become a valuable research tool.

And qualitative public opinion research is well-suited for the internet (finally, the end of notoriously unreliable mall-intercepts!)

However, quantitative political public opinion research -- polling -- hits the internet's "sour spot" because it requires reaching a narrow population for which pollsters do not have well-defined web contact information.

How well do you think Harris Interactive's national panel maps on to likely voters in New York's 26th Congressional District? If you were polling Indiana's primary, would you feel comfortable that the list of e-mails that you bought from a vendor actually contained properly registered voters in the state with past primary vote history?

Some internet survey vendors claim that they have representative general election statewide panels. This may be true - but how many times can you go back to that panel before you exhaust it? Pollsters in competitive races will track data for 30 days or more - well beyond the capacity of internet vendors in even the largest state.

It's not because political pollsters are "old-fashioned" that they don't conduct web-based quantitative research - it's because there is no reliable way to reach their candidates' electorates online in a way that meets even a modest level of methodological rigor.

None of this is to discount concerns about telephone polling - ever-lower response rates, and caller-ID and cell-phone only households that makes reaching people on the phone more difficult than ever.

But, for political polling, internet-based research has not proven to be the panacea once (and continually) promised.

UPDATE: Humphrey Taylor responds.

--Guest Pollster on May 01, 2008 in Internet Polls | Permalink | Comments (1) | Trackback (0)

April 01, 2008

Hickman on the Gallup Daily

Yesterday, in a burst of blogger exuberance, I posted some charts emailed by my long ago employer Harrison Hickman, the Democratic pollster now associated with the firm Global Strategy Group who also conducted surveys earlier this year for John Edwards. I gave Hickman and his associate credit for the charts but then provided my own interpretation, comments I subsequently qualified. This morning, I did what I should have done in the first place, which is offer Hickman the opportunity to describe the charts in his own words. Harrison's summary follows below.
--Mark Blumenthal

The initial (and only) purpose of the line charts Ben Margolis and I sent Mark yesterday seems to have been obscured by our failure to provide explanation with the charts and some of the verbal vines their publication stimulated ("Day-of-Week Effect in Gallup Daily?"). Hopefully I can provide something of a corrective for the former.

1. We submitted the charts without explanation but with obvious doubts about their significance, statistical and otherwise. The subject line of my original e-mail to Mark was "spurious or what?"

2. The point of the exercise was to note that all the hoopla about Obama or Clinton being ahead or behind by more or less at a specific point was ignoring a persistent pattern in the data. (The time period covered was since the departure of the Sainted Senator Edwards.) The point was that the hoopla was misguided, not that the pattern itself is all-telling. We certainly never intended to suggest that particular changes could be associated with specific days or that there was any iron law of anything at work. Our message: if you don't like the results today, wait a couple of days. If you do, it might be wise to exercise some restraint. In that vein, Mark is correct in urging caution about reading too much into day-to-day changes. I would urge similar caution in the interpretation of two techniques under discussion here.

3. The rolling average technique was developed to introduce a cost-effective way to report opinion data more or less continually in critical points of a campaign, and there are a variety of different ways to calculate those averages. But it is important to note that the "smoothing" artifact is the reason the technique is useful, not the reason it is misleading. An on-going series of one-day polls would be more misleading for campaign professionals and poll consumers than rolling averages.

4. Perhaps the most important statistical point to understand about these types of polls is that a sample is not a sample until it is completed. Before its completion, a sample is not "random" even in the colloquial sense of the term. It is for this reason that no one should mistake the partial results of stand-alone samples as precise, no matter how extensively those partial data are weighted. This is particularly important to remember when confronted with early wave results of election day polls (exit polls).

5. One should be mindful of but not obsessed with any particular statistical test. Estimation error is the most reported but hardly the only type of error in opinion research. It is treated as more important than it is and than the other types of errors because (a) it has the veneer of precision because it is a number and (b) it easier to understand and better researched than other categories of errors. Here is a simple measure of the its importance: "sampling error" so-called is taught in the introductory course but other types of errors are saved for later in a student's learning. Here's another: If you read any questionnaire carefully and think seriously about the methods used to gather the data, you almost always will find sources for potentially greater "error" than estimation error in what is reported.

6. In fact, a legitimate argument can be made that estimation errors are not really an applicable statistic for most opinion polls we see. The underlying assumption of sampling error is that the sample in question is random, and random has a very precise statistical definition. For a host of reasons, the samples in most polls do not qualify as random in a strict sense and, in too many cases, even under the loosest standards. Harris or Gallup (forgive me for mot remembering which) used to report a table of mathematical estimation error ranges but also something called ranges generated "from observation." I do not recall that they ever explained the source of the observations but found the presentation refreshing as an implicit statement about the limitations of statistical error calculations.

7. Two final observations from reading comments. As consumers and practitioners, recognize that political arguments are still political arguments even when they are dressed up with statistical language. And, finally, do not assume that any pollster is part of a larger conspiracy against your preferred candidate until you have ruled out (a) incompetence and (b) the possibility that things are not as rosy as you want them to be.

Harrison Hickman
Global Strategy Group, LLC

P.S. Not to suggest that there is a day-(or period-)of-the-week effect in the Gallup data, but as of a few minutes ago, it seems that the stop-the-presses 10-point "lead" Obama enjoyed this weekend is now four points. An up-to-date version of our original charts is below, including a line based on calculation beginning the week after Super Tuesday.

--Guest Pollster on April 01, 2008 in Sampling Issues, The 2008 Race | Permalink | Comments (1) | Trackback (0)

March 28, 2008

Guest Pollster: Presidential Net Promoters

[Today's Guest Pollster contribution comes from Alex Lundry, research director at the Republican polling firm, TargetPoint Consulting.]

TargetPoint Consulting recently partnered with the Cook Political Report and RT Strategies, adding a new political research question called the Net Promoter Score (NPS) to their most recent national omnibus survey (March 6-9, N=802). This measure, adapted from the world of consumer research, attempts to measure voter enthusiasm and passion for a candidate. The results provide some new understanding to how the general election for President could shape up given either an Obama or Clinton candidacy.

First introduced by Frederick Reichheld in the Harvard Business Review and since popularized in his book, "The Ultimate Question," the NPS is used in the business world as a customer satisfaction metric, measuring a customer's likelihood to recommend a product, brand or company to someone else. This captures a number of difficult-to-quantify emotions, attitudes and preferences, by posing it as a recommendation. A recommendation is the ultimate endorsement, showing just how passionately you feel about a particular company or product. A recommendation means putting your own reputation on the line; an indication of loyalty, passion, and even the latent potential for word of mouth buzz.

The question is simple:

On a 0 to 10 scale, with 0 being "not at all likely" and 10 being "extremely likely," how likely is it that you would recommend voting for [INSERT CANDIDATE NAME] in the next election to a friend or colleague?

The NPS is calculated by subtracting the number of detractors (ratings of 0-6) from the number of promoters (ratings of 9 and 10). In the business world, +16 is the median score of more than 400 companies across 28 industries; CostCo has one of the highest known scores at +81. (See the NPS website for more details and similar statistics).

Studies have shown a direct and significant correlation between a business' score and company growth - specifically, a 7 point increase in overall NPS or a 2 point reduction in the percentage of detractors can each account for one percent of positive growth, thus indicating the potential electoral consequences of this measure once adapted to political polling.

To be fair, the NPS is not without it's own set of detractors and it's validity in the political world remains to be seen. For now we can only speculate about any correlation with electoral outcomes. Nonetheless, there is some promising historical data: TargetPoint began tracking the NPS on a generic congressional ballot in September of 2005 through August of 2006 and the results did seem to forebode the Republican fall from favor and the impending Democratic advances of that November. During that time the GOP NPS had a distinctively downward slope, falling from a high of +56 to a low of +11; meanwhile, the generic Democratic NPS trended upwards from a low of +32 to a high of +56.

But what about this year's election? In an Obama/McCain match-up McCain leads 45-43, but the NPS indicate some form of an "enthusiasm advantage" for Obama: among Obama voters, the NPS is +28 (53% promoters minus 25% detractors); while 48% of McCain voters are promoters and 31% detractors for a NPS of +17. Hence an Obama advantage of 11 points.

The Clinton/McCain ballot (McCain leads 47-45) again shows a Democrat enthusiasm advantage, though a slightly smaller one of 8 points (McCain: 44% promoter, 33% detractor, +11 NPS; Clinton: 48% promoter, 29% detractor, +19 NPS).

Though there is little surface difference between the candidates, deeper analysis indicates two critical demographic differences: enthusiasm among youth and Independent voters. The NPS among Independents voting for Obama (+30) is a stunning forty-nine points higher than the score among Independent McCain voters (-19). Interestingly, McCain actually wins Independents against both Clinton and Obama, but his Indy voters are much less enthusiastic than either Obama's or Clinton's. Clinton's NPS among her independent voters is also negative (-5), and a full 45 points short of Obama's. It appears that a Clinton candidacy would remove any passion or enthusiasm among Democrat-voting Independents.

Finally, we see nearly identical performance among 18-40 year olds. McCain's NPS among this age group is +7 and -5 against Obama and Clinton respectively; Clinton actually performs worse than McCain here, with a negative score of -13, while Obama dominates at +32. Again, we are left wondering what would happen to this youth enthusiasm should Clinton become the nominee.

Keep in mind that these are scores among people already voting for that particular candidate. While a vote is still all that matters on Election Day, a recommendation driven campaign can produce new votes faster, cheaper and in a more trustworthy and impactful way than traditional campaign appeals of advertising, direct mail and robo-calls.

--Guest Pollster on March 28, 2008 in The 2008 Race | Permalink | Comments (5) | Trackback (0)

March 14, 2008

Guest Pollster: The SurveyUSA 50 State Poll and the Electoral College

(Today's Guest Pollster's contribution comes from Professors Robert S. Erikson and Karl Sigman of Columbia University.)

In late February, SurveyUSA interviewed 600 registered voters in every state for a total of 30,000 interviews, ascertaining preferences in a McCain-Obama and a McCain-Clinton race. The focus was a new set of electoral maps of red and blue states based on who led each state in the survey. Based on who won each state in the SurveyUSA survey, Obama defeats McCain 280 to 258 while Clinton defeats McCain 276 to 262 in the Electoral College.

Of course SurveyUSA's mammoth undertaking at best presents a snapshot of the states at one point in time. And even if all the niceties of polling were perfectly met, the allocation of states as "red" or "blue" is problematic due to sampling error. Here, we take the analysis of the SurveyUSA 50 state polls one step further. Rather than assign states based on who leads in the state surveys, we assign states probabilistically to the Democratic or Republican candidate based on the SurveyUSA state polls. Then, based on these probabilistic estimates, we ask the question, given the SurveyUSA results, what are odds of an Obama or Clinton victory in the Electoral College?

To do this, we conducted one million simulations (in MATLAB) of the Obama-McCain contest and then one million more simulations of the Clinton-McCain matchup. In each case we assume that the state estimates were correct except for sampling error. Using sampling theory and the assumption of simple random sampling, we draw one million estimates of the vote for each state. In each case we draw from a normal distribution with the observed mean (percent Democratic vs. percent Republican) and the standard deviation determined by the number of respondents in the state reporting a preference (always slightly under 600).

What do our results show? First, we pooled the state polls to ascertain the national vote, weighing each state's percent in proportion to the size of its House delegation. We also assign the District of Columbia as a 436th district and assign each Democratic candidate 85 percent of the vote to McCain's 15 percent. With these assumptions, the national popular "vote" is tight as of late February. Obama wins 51.5 percent versus McCain's 48.5 percent. Clinton also wins by an even razor thin margin, 50.7 to 49.3. With 30,000 cases, both estimates are statistically significant. McCain would be in the actual popular vote lead less than one time in 20.

That being said, our simulations yield a 88% chance of Obama beating McCain (with 306 Electoral College votes on average versus 233 for McCain), and a 74% chance of Hillary beating McCain (with 285 Electoral College votes on average versus 253 for McCain). About one percent of our simulated outcomes were Electoral College ties. (We ignored within-state variation in Maine and Nebraska, which divide their electoral votes by district.)

On the one hand, we find the expected numbers of electoral votes (the average from the simulations) for Obama or Clinton to be slightly higher than SurveyUSA reports. On the other hand, there is sufficient variance in the outcomes, so that McCain wins a nontrivial portion of the simulations, even with Obama as the opponent. Our two million simulations remind us that the popular vote winner is not always the Electoral College winner, although probably due mainly to chance -- the lottery aspect of the Electoral College -- and not any identifiable partisan bias in the 2008 Electoral College.
________________________________________
We thank Linda Liu for her technical assistance.

--Guest Pollster on March 14, 2008 in Polls in the News | Permalink | Comments (40) | Trackback (0)

January 15, 2008

Norpoth: New Hampshire's Crystal Ball in 2008

(Today's Guest Pollster contribution comes Professor Helmut Norpoth of Stony Brook University).

New Hampshire voters may mystify pollsters and pundits, but they have acquired an uncanny sense of picking candidates that go on to the White House. Whatever accounts for Hillary Clinton's surprising showing in her party's primary in New Hampshire, that victory makes her the best bet for Democrats to win the general election in November; likewise, John McCain's victory in the Republican primary in New Hampshire makes him the best hope for the GOP to retain the White House in November. These predictions are derived from a forecast model I developed that uses primary performance as the sole short-term predictor of the vote in the general election (the "Primary Model"). I have applied the model, with slight modifications, in the last three presidential elections, in which it correctly predicted the winners of the popular vote several months before Election Day. (See my 2004 paper in PS: Political Science & Politics). A race between the two New Hampshire winners, so the forecast, would be a nail-biter, with Clinton edging McCain by a margin of just a single percentage point of the two-party vote.

The use of primary elections to predict the outcome of the vote in the general election has some compelling advantages. One, it puts the estimation of a forecast model on a firm footing by letting us use elections all the way back to 1912, when presidential primaries were inaugurated. Two, it makes it possible to include both incumbent and opposition candidates in the model; granted, the incumbent candidate's performance may prove more powerful, but the effect of the out-party's primary showing is not negligible. And finally, the use of primaries as a predictor permits an unconditional forecast of the November vote at a very early moment. No ifs and buts. If one is willing to go with the outcome of the New Hampshire Primary, one can do it right now. The only uncertainty that remains is which of the match-ups will result from the nomination process. Chances are we may not have wait until the national conventions.

To measure primary performance in a standard format that allows for comparison across elections with varying numbers of candidates, I use an equivalent of the two-party vote in general elections. A candidate's primary showing is expressed as his or her vote relative to that of the winner (or in case of the winner in relation to the second strongest candidate). For incumbent-party candidates, the measure is adjusted, depending on whether they are sitting presidents or not. Moreover, the New Hampshire Primary is used only since 1952, when the state switched to a presidential-preference type of primary; prior to 1952, the model relies on the vote in all primaries.

Even though primary performance is the key, giving the model its name, the Primary Model also enlists a cyclical pattern of the presidential vote: the tenure of a party in the White House typically lasts between two to three terms. A compelling explanation for that dynamic is the term limit in presidential elections. Except for FDR, American presidents have eschewed running for more than two terms; and have been barred from doing so since then. The rule guarantees that incumbent presidents are missing from those contests in some periodic fashion, as is the case in 2008. In many such instances the absence of a sitting president with a high degree of popularity may improve the chances of the opposition party of capturing the White House. Given his high approval rating, Bill Clinton's ineligibility in 2000 probably hurt the Democratic prospects that year, although the absence of a much less popular George W. Bush in 2008 may be a blessing for the GOP. In any event, elections without a sitting president in the race tend to favor the opposition party more than elections with an incumbent running for another term. The Primary Model handles this dynamic by way of an autoregressive process (the presidential vote in the two previous general elections). In addition, given the use elections as far back as 1912, the model applies an adjustment for pre-1932 long-term partisanship.

From 1912 to 2004, the out-of-sample forecasts of the Primary Model pick the winner of the popular vote in 23 of the 23 elections, with 1960 being the only exception (and yes, that record includes Gore's popular vote win in 2004). The prediction equation for the presidential vote in 2008 (expressed as the Democratic share of the major-party vote) is:

.361 (RPRIM - 55.6) (-1) + .124 (DPRIM - 47.1) +.368 (VOTE04) -.383 (VOTE00) + 50.7 = .361 (RPRIM - 55.6) (-1) + .124 (DPRIM - 47.1) + 49.4

where RPRIM and DPRIM represent the primary support of the Republican (incumbent party) and Democratic (opposition party) nominees for President, capped within a 30-70 percent range, and Vote04 and Vote00 the Democratic vote shares in 2004 (48.8%) and 2000 (50.3%). The measure for the Republican candidate is inverted (-1) because the Democratic vote is used as the dependent variable. The formula produces the following forecasts of match-ups between the leading contenders in either parties (the vote for each match-up being the Democratic percentage of the two-party vote):

The PRIMARY MODEL predicts that in a race of New Hampshire Primary winners, Democrat Hillary Clinton would narrowly defeat Republican John McCain in the November general election (50.5 to 49.5 percent of the two-party vote). The predicted margin of victory, however, is so small that the confidence attached to this forecast is less than 60 percent, given the size of the forecast standard error (2.5). In match-ups between the Republican primary winner and Democratic primary losers, McCain would end up in a virtual tie with Barack Obama (49.9 to 50.1 percent) while defeating John Edwards (52.1 to 47.9 percent) by a margin close to one unit of the forecast standard error (2.6). At the same time, in match-ups between the Democratic primary winner and Republican primary losers, Clinton would dispatch Mitt Romney, Mike Huckabee, and Rudolph Giuliani by margins way beyond that error range. Finally, in match-ups between primary losers, both Obama and Edwards would beat any of the Republicans, and quite handily so in most cases.

That is no sign of partisan bias. Rather, it has to do with the Model assigning more weight to the primary performance of incumbent-party candidates than to the performance of out-party candidates. Nominating a primary loser, or even a candidate with a lackluster primary showing, costs the incumbent party more dearly than it does the out-party. Candidates not listed in the forecast table would do no better than the weakest one in their respective parties.

--Guest Pollster on January 15, 2008 in New Hampshire 2008, The 2008 Race | Permalink | Comments (7) | Trackback (0)

January 13, 2008

Likely Voter Screens and the Clinton Surprise in New Hampshire

(Editor's note: Today's Guest Pollster contribution comes from Professors Robert S. Erikson of Columbia University and Christopher Wlezien of Temple University).

Does the world need one more explanation for the historic failure of the polls to predict Hillary Clinton's victory in the New Hampshire primary? We offer another possible account. Ours does not require unusual last-minute voter shifts in preference, voters lying to pollsters, or any disconnect between the campaign story line in the media and voter decision-making voters.

We suggest as the possible culprit the way pollsters' employ their likely voter screens. Pollsters may have been tricked not by voters shifting their candidate preferences but by a rapid shift in enthusiasm by Clinton supporters at the last minute. It may be that significant numbers of Clinton supporters were uninclined to vote at the time when the pollsters were doing their final interviews but then regained their interest just in time to vote. In short, the surge to Clinton could have been simply due to uncounted Clinton supporters who the pollsters dismissed as unlikely voters regaining their interest in voting.

According to most accounts, the late Clinton gains stemmed from sympathy for Hillary after her rough treatment in the media, Hillary's response to the questioning of her likeability in the final debate, and her tears on election eve. But how did this response come about? Was it due to truly undecided voters with their blank slates turning overwhelmingly to Hillary? Exit polls show no evidence of this. And it is unlikely that voters tuning in late would see the flow of the news moving in Hillary's direction. It is the idea that late-deciders could have done so that is so jarring to media watchers.

If late-deciders did not split for Hillary, maybe it was Obama supporters changing their minds? But it is even more implausible that voters who followed the campaign and settled on Obama as their choice would follow the late news and see a reason to vote for Hillary. Once people "make up their minds" in a campaign they rarely change and then only for seemingly good reasons. Did Obama supporters have reason to shift? Would the internal dialog of massive numbers of voters be: "I support Obama because he is such an exciting candidate...No wait, Hillary just shed a tear so I'll vote for her instead"?

Rather than voters deciding late for Hillary or shifting late to Hillary, we posit that her proportion of eligible voters in the New Hampshire primary was fairly steady in the final weeks. What changed was the enthusiasm of her supporters. It may be that Hillary supporters followed the news and became disillusioned by her decline in Iowa, her loss of momentum, and the general negative arc of her campaign. They were watching and they were responding to the media's storyline. Their response was not to shift to another candidate but to become dispirited. If interviewed by pollsters, their lessening enthusiasm placed them disproportionately in the "unlikely voter" column. Then, after the pollsters stopped calling, Hillary's supporters gained the enthusiasm necessary to motivate them to vote. This may be because Hillary showed her more human side late in the campaign or because it was her campaign was on the brink or for other less obvious reasons. The point is that the preferences of these voters were undercounted by pollsters. No unusual number of previously undecided voters or former Obama supporters is necessary to account for her late surge in the polls.**

Is our story true? We know that shifts in net enthusiasm from one candidate's supporters to the other's are more volatile than shifts in net preference. We also know that pollsters can be very sensitive to these shifts in enthusiasm when identifying likely voters. (See our paper from 2004 on "Likely Voters and the Assessment of Campaign Dynamics" in the Public Opinion Quarterly). Was it simply a very late shift in enthusiasm that caused the New Hampshire polls to go wrong?

Pollsters hold in their data banks the evidence that would tell if our conjecture is right or wrong. Our suspicion is that voter preferences among potential Democratic primary voters were more stable over the campaign's final weeks than generally realized. This shifting dynamic evident in the polls, we suggest, was exaggerated by daily shifts in enthusiasm that caused shifts in the composition of who gets counted as "likely voters." If likely voters first shifted against Hillary and then for, the shifting membership of the "unlikely voters" may have "surged" back and forth in the opposite way. It would be interesting to see if this was the case.

**Of course the pattern also could be explained by changes in enthusiasm among Obama supporters that mirrored what we have posited for Clinton supporter, flowing after the big victory in Iowa and then ebbing after the pollsters left the field.

--Guest Pollster on January 13, 2008 in New Hampshire 2008 | Permalink | Comments (11) | Trackback (0)

December 07, 2007

The Gender Gap In Turnout Likely To Widen

[Today's Guest Pollster's column comes from Margie Omero, President of Momentum Analysis, a Democratic polling firm based in Washington, DC.]

I posted last week about the "Single Anxious Female" moniker, and how coverage of this demographic group largely trivializes women. In fact, not only are women across marital status groups voting at a higher rate than men, this gender gap in turnout has existed for years, and is poised to widen further.

Census data here and here [2006 data found with this program] show that women have been turning out at higher rates than men in every Presidential election since 1980, and in every mid-term election since 1986. Not just raw numbers (there are more women than men, so even a lower turnout rate among women could still mean more women voters), but the percentage of adults who report voting. The graph below shows the difference between women and men's turnout rates (abbreviated as "women - men"). Note that the gap is more dramatic in Presidential years.

This pattern is not surprising. Not only have women been making societal gains in political influence, but women's educational attainment also increased dramatically during the same period. Again using census data, in the last mid-term election, for the first time more women than men had some college education (among adults 25 and older). This bodes well for a continued increase in women's turnout.

And despite the attention on young and/or single women not voting, it is younger women who comprise this gender gap in turnout. The table below shows 2004 turnout by age and gender. It is only among the oldest voters that men vote at a higher rate than women.

Naturally age and education are related. Younger women are more likely to have some college education than their male counterparts; the reverse is true among older men and women. But even within each education level (with one small exception), younger women are voting at a higher rate than younger men.

Looking at these numbers, I see a positive story not being told. The data suggest women's turnout will continue to increase, particularly in a Presidential year. And the gender gap in turnout is particularly large among younger voters, regardless of their level of education. But while women are becoming increasingly influential in elections, they are being told by the media their voting behavior is just another thing that requires improving. Surely there is a way to mobilize women and make their issues heard without hyperbole and finger-pointing. In a future post, I'll look at some of the assumptions made about why women aren't voting, and what the data really show.

--Guest Pollster on December 07, 2007 in The 2008 Race | Permalink | Comments (4) | Trackback (0)

November 27, 2007

Anxious about "Single Anxious Women"

[Today's Guest Pollster's column comes from Margie Omero, President of Momentum Analysis, a Democratic polling firm based in Washington, DC.]

It's almost official. Single women are poised to be the "Security Mom" or "Soccer Mom" of the 2008 election. They even have their own easy to remember moniker: the "Single Anxious Female." At first blush, it seems like a good thing for women. A woman top-tier candidate, a focus on women's issues and women's voters - it must be a good thing, right?

Actually, much of what you read about single women and voting is not borne out by the data. There is indeed a "Marriage Gap" among women. Married people vote at a higher rate than non-married people. But the marriage gap is actually larger among men. According to Census reports from the 2004 election, married men are as likely to vote (63% turnout) as married women (65%). But unmarried men (which includes single, divorced, separated, and widowed) are substantially less likely to vote (46%) than unmarried women (55%). The marriage gap is 10 points among women, and is nearly twice that (18 points) among men.

If you look specifically at single, never-married adults, this pattern holds. A majority of single women voted in 2004 (52%), compared to fewer single men (43%). This is even true with 18 to 24 year-olds (47% of single women in that group vote, compared to 40% of single men). The table below shows the turnout rate by gender and marital status. [Note: at the time of this post, the Census table contained an error, in that Row 88 (widowed men 18-24) should be blank, and all data currently in Rows 88-91 should be moved down one row. The error was corrected by email from the Census, but has not yet been updated on the site.]

You would never know about women's higher turnout by examining the press coverage. A CNN piece this month called single women, particularly younger single women, "notoriously difficult to get to the polling booth." An entire organization is devoted to closing the marriage gap among women . And women's advocates hypothesize about why single women don't turn out, making their own gender-based assumptions about women not recognizing their power.

More disturbingly, however, is what this focus on single women has wrought. Dubbed "Single Anxious Female," that cringe-inducing name has stuck, and has generated a sizable amount of press devoted to the caricature of the single woman. This group has become defined not by political views, but by their lack of gravitas. Several have called them the "Sex in the City voter". Feminist icon Naomi Wolf says they are more like Melanie Griffith in "Working Girl," as compared to Hillary's Sigourney Weaver. The CNN piece said this group is "more interested in showing off than in true political activism" and cited others who called single women "slutty" or "stupid." Advocacy has begotten dismissiveness.

Now, encouraging non-voters to vote is obviously important, and the point here is not to object to women-specific voting programs. And certainly campaigns should continue to reach out to women. But we need to change tactics. First, let's use the data correctly. Women vote at a higher rate than men. Unmarried women, however defined, vote at a higher rate than unmarried men. And this pattern holds across age groups. Second, it does not further the cause to allow women to be called anxious, show-offs, bubbly, stupid, or confused. These characterizations only perpetuate stereotypes about women, rather than work to improve our status.

--Guest Pollster on November 27, 2007 in The 2008 Race | Permalink | Comments (9) | Trackback (0)

November 19, 2007

GUEST POLLSTER: Alan Abramowitz on Iraq War Public Opinion

[On November 6th, Charles Franklin posted an analysis of recent trends in public opinion on the Iraq War. In response we've recieved a Guest Pollster's contribution from political scientist Alan Abramowitz, the Alben W. Barkley Professor of Political Science at Emory University in Atlanta, Georgia.]

The claim that there has been a significant shift in public opinion toward the war is simply not supported by recent polling data. For example, a new CNN/Opinion Research Poll finds opposition to the war at an all-time high of 68 percent. The latest NBC/Wall Street Journal Poll finds that 27 percent of Americans approve of the president's handling of the war, down 3 points from September and almost identical to the levels of support from the first half of the year. This same poll finds that the war remains easily the most important issue in the minds of Americans--26 percent named the war as the most important problem for the federal government to address with health care a distant second at 16 percent. 46 percent of the respondents chose Iraq as the first or second most important problem compared with 34 percent who chose health care as the first or second most important problem. Other issues including terrorism, illegal immigration, and global warming finished far behind.

The war remains enormously unpopular and major political liability for the Republican Party. The new ABC-Washington Post Poll finds Democrats favored over Republicans on the war by a 16 point margin, slightly higher than the Democratic margin earlier this year and last year.

The claim that public opinion has shifted on the war appears to be based almost entirely on a small uptick on one measure--opinion about how the war is going. There has been a small improvement on this question, presumably in response to reports of decreasing violence and, most importantly, decreasing U.S. casualties. But this shift is not indicative of any broader shift in public opinion toward the war. Opposition to the war remains as high as ever as does support for a withdrawal timetable. And Iraq clearly remains the most salient issue in the 2008 election.

--Guest Pollster on November 19, 2007 in Iraq | Permalink | Comments (3) | Trackback (0)

October 18, 2007

GUEST POLLSTER: SCHIP - Gallup Responds

[This Guest Pollster's Corner contribution comes from Lydia Saad, Senior Editor of The Gallup Poll, responding to criticism posted earlier today by Alan I. Abramowitz.]

Alan, I see your point about how Gallup's question explaining the difference between Bush's income threshold and the Democrats' threshold could have confused respondents. You overlook the fact that we set up the question with this introduction to the series: "As you may know, Congress is considering a bill that would increase the number of children eligible for government subsidized health insurance, but the Democrats in Congress and President Bush disagree on how much to increase the program." But your point is well-taken.

However, our question measuring concern about the Democrats' bill being a step toward socialized medicine isn't "biased" -- it was intentionally written to convey Bush's counterargument. That was the intent -- to test the strength of socialized medicine as a counterargument. And indeed we found a slim majority willing to say they are concerned. We didn't conclude from this that Americans think the bill WILL lead to socialized medicine. As you note, we merely said that Americans are sympathetic to the argument: "Americans are also generally sympathetic to Bush's concern about the program leading to socialized medicine."

It is always a challenge to write clear and balanced questions about complex policy issues. Along those lines, I would go further than your critique of Gallup, and submit that all of the public polling on SCHIP I've seen thus far can be criticized in one regard or another. Yesterday I saw this question by CNN which seems to suggest that SCHIP is a new $35 billion program for children in middle income families, and that Bush opposes the program. I see what CNN was trying to do (isolate the question to the program expansion) but their wording just doesn't succeed at accurately describing what the veto/override conflict is all about.

26A. As you may know, President Bush vetoed a bill passed by Congress that would create a program to spend 35 billion dollars to provide health insurance to some children in middle-income families. Do you think Congress should vote to create that program by overriding Bush's veto, or do you think Congress should vote to block that program by sustaining Bush's veto?

The ABC question you applaud is another reasonable attempt, but still conveys the sense that the policy choice is between supporting the Democrats' plan and not providing any insurance coverage for "millions of low-income children." And since they ask if Congress should vote to override Bush's veto, they should have a follow-up asking what should happen if that override fails: i.e. Now that Bush has vetoed the bill, should the Democrats and Republicans in Congress work together to pass a new compromise bill, or should they let the program expire?

Otherwise we are just falling into the same political traps the Democrats and Republicans are setting for each other, and not really finding out what kind of government sponsored children's health care coverage Americans want for the country.

Americans probably don't have a great command of the details of the SCHIP debate, but half say they are paying very or somewhat close attention to it. That's about the midpoint for public attention to policy-debates in Washington. Americans clearly have some opinions worth tapping, and the challenge is to probe further for a more thorough and accurate understanding of whether Americans would rather have the existing program that covers families earning up to twice the poverty level, or whether the program should be expanded to include families earning more than that. Separately, we can find out who Americans would blame if the program expires: Bush for vetoing the congressional bill, or the Democrats for not being willing to pass a compromise bill.

None of the polling I've seen thus far -- neither on a question by question basis, nor in its totality -- answers those questions for me.

Lydia Saad

Senior Editor, The Gallup Poll

--Guest Pollster on October 18, 2007 in Polls in the News | Permalink | Comments (10) | Trackback (0)

October 18, 2007

GUEST POLLSTER: The Problems with Gallup's SCHIP Poll

[Today's Guest Pollster's entry comes from Alan I. Abramowitz, the Alben W. Barkley Professor of Political Science at Emory University in Atlanta, Georgia. He has also been a frequent contributer to the blog Donkey Rising.]

The Gallup Poll has just released a report on public attitudes regarding President Bush's recent veto of a bill expanding the State Children's Health Insurance Program (SCHIP). Gallup reports that a majority of Americans trust the Democrats in Congress more than President Bush on this issue. That's consistent with what other polling organizations have found. However, Gallup reports that a majority of Americans support the President's position on where to set the income threshold for SCHIP eligibility and that "Americans are also generally sympathetic to Bush's concern about the program leading to socialized medicine." A close examination indicates however, that the questions on which these conclusions are based are clearly biased.

Here's the question concerning the income threshold:

As you may know, the Democrats want to allow a family of four earning about $62,000 to qualify for the program. President Bush wants most of the increases to go to families earning less than $41,000. Whose side do you favor?"

The problem, of course, is that the question implies that the Democrats, in contrast to President Bush, do not want most SCHIP funds to go to families earning less than $41,000. But this is not true. In fact, under the legislation passed by Congress, the large majority of SCHIP funds would go to families earning less than $41,000.

The second question is even worse:

How concerned are you that expanding this program would create an incentive for middle class Americans to drop private health insurance for a public program, which some consider to be a step toward socialized medicine? Are you very concerned, somewhat concerned, not too concerned, or not concerned at all?"

Here the respondents are presented with a totally one-sided argument using the loaded term "socialized medicine." It is actually surprising, given this wording, that a substantial proportion of the respondents indicated that they were not concerned.

Given the biased wording of this question, it is not surprising that the results are out of line with those of other recent polls on this topic based on a more balanced wording. For example, an ABC-Washington Post Poll conducted from September 27-30 of this year asked the following question:

There's a proposal to increase federal spending on children's health insurance by 35 billion dollars over the next five years. It would be funded by an increase in cigarette taxes. Supporters say this would provide insurance for millions of low-income children who are currently uninsured. Opponents say this goes too far in covering children in families that can afford health insurance on their own. Do you support or oppose this increased funding for this program?"

Seventy-two percent of the respondents in this survey favored the proposed expansion of the SCHIP program.

Update: Gallup's Lydia Saad responds

--Guest Pollster on October 18, 2007 in Polls in the News | Permalink | Comments (3) | Trackback (0)

July 20, 2007

Giuliani Campaign's Analysis of Recent Trends

Brent Seaborn, Director of Strategy for the Giuliani campaign, offers the following response to Charles Franklin's recent post on the trend in support for Giuliani.

By Brent Seaborn
Director of Strategy
Rudy Giuliani Presidential Committee

Despite assertions to the contrary on this site, the Giuliani campaign is in a very strong position at this point and is clearly best-positioned to win the primary. Let me point out a few differences between the McCain and Giuliani trend line:

• When Mayor Giuliani first announced his candidacy for president, he received a considerable bounce in the polls. We anticipated that the race would close after our initial bounce -- in memos written on March 22 and June 22. I wrote, at the time, we should expect polls to tighten, as they have.

• As the race developed early in the spring, the race quickly but briefly, developed in to a two-way race, and our initial bounce extended into the beginning of this two-way race. The two-way race divided most of the Republican primary vote between 2 major candidates -- the nature of a two-way race generally forces undecided or leaning voters to make a choice between the leading candidates and many broke our way.

• As McCain's trend line declined Mitt Romney's slowly rose and Fred Thompson entered the race. Senator McCain is still a candidate for President and continues to receive a substantial vote share.

• Fred Thompson now seems to be the beneficiary of an announcement (or pre-announcement) bounce. And Fred Thompson's entry to the campaign has effectively made this now a four-way race.

• After months as the frontrunner and the addition of a fourth candidate to the GOP primary it is notable that we are in roughly the same spot we were in before our bounce and when this was still a three-way race. In a four-way (or as your graph suggests a five-way race), a trend line from the first of the year until now, excluding our "announcement" bounce, is virtually flat.

• I also note this paragraph:
  
  "Are there any bright spots for Giuliani, other than money? Yes. There is a hint in the data that his decline may have slowed and support stabilized in the last month. In the first plot above, the blue line is my standard trend estimator which is rarely mislead by "blips" in the polls, but which is also a bit slow to be convinced that a change of trend has occurred. The red line in the plots is my more sensitive estimator-- quicker to notice a change, but also more easily fooled by "changes" that turn out to be phantoms. The red estimator has flattened out recently for Giuliani, and currently sees relative stability at about 26%. The blue estimator instead sees continued decline and a current level just under 25%. If the red estimator is right (and it often isn't) then perhaps the worst days of declining support are now behind Giuliani, at least nationally. If so, his campaign can try to get the trend moving up instead of down, but at least the decline has stopped. Unlike McCain, Giuliani has the money to try to make the numbers turn up."
  
  I believe this is more than a blip. The red trend line will begin to pull the blue line up to meet it. In fact, if one looks at major media polls over the last month most of them show Rudy Giuliani receiving 30% or more of the GOP primary vote. In fact the mean of the major media polls over the last month is 29%.
  
  Newsweek 6/20-6/21: RWG 27%
  CNN 6/22-6/24: RWG 30%
  FOX 6/26-6/27: RWG 29%
  CBS 6/26-6/28: RWG 34%
  USA Today/Gallup 7/6-7/8: RWG 30%
  AP 7/9-7/11: RWG 21%
  Gallup 7/12-7/15: RWG 33%
  FOX 7/17-6/18: RWG 27%
  
  Major media polls actually show Rudy Giuliani ahead of even the red tend line. And I would add that the red trend line, at this point in the race, disproportionately accounts for the Rasmussen Polls that by their regularity are a drag on Mayor Giuliani's trend line.

• Overall, we are very pleased with our performance in national polls. We are aware we will continue to face challenges and the race will likely continue to close. But we believe we have a real and solid base of support and we will remain competitive as this race evolves.

--Guest Pollster on July 20, 2007 in The 2008 Race | Permalink | Comments (16) | Trackback (0)

December 21, 2006

The 2006 Exit Polls: How Did They Perform?

Today's Guest Pollster's Corner contribution comes from Mark Lindeman, assistant professor of Political Studies at Bard College.

In the wake of allegations that the 2004 U.S. presidential exit polls pointed to a stolen election, many observers wondered how the 2006 exit polls would turn out. One widespread rumor asserted that no exit poll results whatsoever would be made public until after the polls had been forced to match the official vote counts. But in fact, CNN.com once again posted a preliminary national House tabulation a bit after 7 PM Eastern, and posted tabulations in state races soon after the polls closed in each state. (Other outlets may have done so as well: at the time I had my hands full with just one.) These tabulations appear to show discrepancies fairly similar to the 2004 discrepancies, as I report below.

Using tabulations to estimate exit poll "red shifts"

The tabulations are not intended to project the final vote counts. Rather, they offer crude but useful insights into why voters voted as they did. Nonetheless, each tabulation is based on a particular vote estimate made at a particular time. The exit pollsters use different estimates for different purposes. Before vote counts begin to arrive, the pollsters can refer to at least three estimates. These are (as described in the post-election evaluation report on the 2004 exit polls):

• The "best survey" or "Best Geo" estimate -- based on interview data (from exit polls and, in some states, telephone surveys of early and absentee voters), and also incorporating data on past results from the exit poll precincts
• The "prior" estimate -- based primarily on public pre-election surveys (something like the averages posted on Pollster.com)
• The "composite" estimate -- a hybrid which combines the interview data (Best Geo estimate) and pre-election polls (prior estimate).

The initial tabulations posted by CNN.com are based on the composite estimate -- not just on interview data. Therefore, they probably tend to understate the disparity between the exit poll results and the vote counts. For instance, the initial 2004 "screen shot" of Pennsylvania indicates that Kerry had about 54% of the vote, and the evaluation report confirms that the composite estimate was 54.2% [p. 22]. But the report also reveals that the interview-only Best Geo estimate gave Kerry almost 57% of the vote, or a 13.8-point margin [p. 22]. The official result -- Kerry won by 2.3 points -- constitutes an 11.5-point "red shift," or reduction in Kerry's net margin, compared to the Best Geo estimate. Because pre-election polls showed a very tight race in Pennsylvania, the composite estimate gave Kerry "only" an 8.5-point margin, or 6.2-point red shift. Overall in 2004, the average discrepancy was a 5.0-point red shift in the Best Geo estimate, but "only" a 3.6-point red shift in the composite estimate.

(Once vote counts start to arrive, the pollsters continually generate a variety of estimates that incorporate vote count data at both precinct and county levels. These dynamic estimates are used to inform the decisions to "call" -- or not to call -- each race. Intermittently the pollsters also generate new tabulations based on a current vote estimate. Updating the tabulations has been described by critics as replacing "pristine" exit poll results with "soiled" ones. [*] Actually, if "pristine" means "based on interviews only," none of the tabulations is pristine.)

To "estimate the estimates" from the early tabulations, I use each table in a tabulation to figure approximate party or candidate shares, then take the median of the differentials across all the tables. For instance, take this snippet of the preliminary national House poll:

We can use these percentages to estimate that 49% * 53% (or about 26%) of voters were men who voted for Democrats, and 51% * 57% (or about 29%) were women who voted for Democrats. So, based on this table, apparently Democrats got about 26% + 29% = 55% of the vote. Applying the same logic, apparently Republicans got about (49% * 45%) + (51% * 42%) = 43.5% of the vote, for about an 11.5% Democratic margin. However, other tables imply somewhat larger or smaller margins, due to the influence of rounding error. Using a median of estimates from all the tables reduces this rounding error, and a computer program interpreting the HTML tables can do the calculations almost instantly. (Because of mistakes I made on election night, I have cruder approximations for two uncompetitive Senate races -- Minnesota and Utah -- and no data for the gubernatorial races in Illinois and Tennessee.)

What I found

Overall, I estimate that the initial national House tabulation gave Democrats an 11.3% margin in total vote. The final tabulation currently available, weighted to the pollsters' vote estimate at that time, gives Democrats a 7.6% margin, so these figures imply a 3.7-point "red shift" -- close to the 3.6-point average in the 2004 presidential composite estimates. If the final official margin is closer to 7 points, as Mark Blumenthal has estimated, the red shift may be above 4 points. [*] However, the vote proportions are influenced at the margins by uncontested races, which appear on the ballot in some states and not in others. Without knowing exactly how NEP handles these uncontested races (nor whether voters accurately report their votes and non-votes in these races), it is unclear what vote totals we should compare to the exit poll estimates.

(Note also that the House tabulation is not quite like the state-level tabulations I discuss next. The state-level tabs, posted as the polls closed in each state, should incorporate all the interview data. The House tabulation, posted long before the polls closed in many states, relies on partial data from much of the country. I have no reason to think that the complete results would be much different.)

State-level races yield broadly similar red shift estimates. In the Senate races, I estimate that the average red shift was 2.3 points, and the median red shift was 2.8 points. In races for governor, I estimate that both the average and the median red shift was 4.0 points.

As in 2004, most of the largest exit poll discrepancies were in uncompetitive races. Some observers have cited (here , here, and here) the red shifts in the Virginia and Montana Senate races as pointing to vote miscount favoring the Republican incumbents -- who nonetheless lost both races. But since those two races have near-average red shift, there is little reason to single them out. Perhaps the most striking discrepancy is in the Minnesota governor's race. Democratic challenger Mike Hatch appeared to have an 8-9% lead in the initial exit poll tabulation, but lost to incumbent Tim Pawlenty by about 1%. The pollster.com 5-poll average gave Hatch a narrow 2.6-point margin, so the election result was closer to expectations than the exit poll result was. Minnesota also experienced one of the largest "red shifts" in 2004.

The House red shift has also been cited as evidence of vote miscount, most elaborately in a paper issued by the Election Defense Alliance (EDA). The paper argues that respondents' reports of their presidential votes in 2004 can be used as an "objective yardstick" to evaluate the 2006 poll. In the final House tabulation, (self-reported) Bush voters outnumber Kerry voters by 6 percentage points, more than double Bush's popular vote margin. EDA's analysts reason that the exit pollsters in effect had to invent millions of Bush voters (and/or delete Kerry voters) in order to match the House vote counts -- which, therefore, must be wrong. The basic flaw in this argument is that reported past vote is not an objective yardstick. On the contrary, as I have noted elsewhere, exit polls and other polls often -- even usually -- overstate past winners' vote shares. Worse, because the authors believe that Kerry won the popular vote and that Democrats had higher turnout in 2006, they end up conjecturing in a footnote that Democrats actually won the House vote by 23(!!) percentage points, a double-digit deviation from the initial tabulation. So much for defending the reliability of exit polls!

Comments

After the events of election night 2004, the NEP pollsters (Edison Media Research and Mitofsky International) announced efforts to reduce exit poll bias. Among other things, Edison/Mitofsky planned to improve interviewer training in order to minimize any selection bias on the part of interviewers. (See, for instance, Joe Lenski's interview with Mark B.) Despite the strong evidence of red shift in the 2006 data, we cannot conclude that these efforts were ineffectual. Participation bias easily could have been larger than ever in 2006. As Mark Blumenthal has noted, a Fox News/Opinion Dynamics pre-election poll found that 44% of Democrats, versus only 35% of Republicans, said that they would be "very likely" to participate in an exit poll. Differences in levels of concern about electronic voting and election fraud may (or may not) contribute to that disparity. In any case, no methodological refinement can force Democratic and Republican voters to participate at equal rates.

Interestingly, the pilot "Election Verification Exit Polls" (EVEP) conducted by Steve Freeman and Ken Warren reported similar or larger red shifts. Freeman's initial report indicates red shifts ranging from 5 to 8 percentage points in four distinct races (two House races, Senate, and governor) in the 28 Pennsylvania precincts surveyed. Freeman argues that the survey "eliminated most of the potential sources of error" (7), presumably through careful training of the interviewers. However, Freeman also reports that several interviewers who obtained relatively low completion rates "subjectively felt that Republicans were disproportionately avoiding participation" (8).

A first glance at the EVEP data files shows at least one case of large red shift where the exit poll result seems implausible. In this precinct, the exit poll registered a 63% majority for House Democratic challenger Lois Murphy (PA-06), while the official returns gave her just 44% of the vote. Registration statistics for the precinct (Chester County precinct 021, East Bradford North 2) show that registered Republicans outnumber Democrats by more than 2 to 1 (57% to 26%). If we concede Freeman's premise that the EVEP methodology was close to ideal, this result hardly inspires confidence in exit polls' inherent accuracy.

--Guest Pollster on December 21, 2006 in Exit Polls, The 2006 Race | Permalink | Comments (1) | Trackback (0)

December 15, 2006

Amy Simon: Random Digits or Lists

Today's Guest Pollster's Corner contribution comes from Amy Simon, a partner at Goodwin Simon Victoria Research.

News media and academics hold up Random Digit Dialing (RDD) sampling methodology as the gold standard for survey samples for elections. Meanwhile, many top notch political pollsters have been serving their clients well for years by instead using samples selected from the official list of registered voters (the statewide voter file), often called Registration-Based Sampling (RBS).

RDD samples are created when a computer randomly generates the last four digits of a phone number. The advantage of RDD is that everyone with a working landline phone is included in the sample - it doesn't matter if your phone service was just turned on that morning or if your number is unlisted, since the sample isn't generated from a list of actual phone numbers. An obvious disadvantage is that an RDD sample also includes business numbers, fax numbers, disconnected numbers, and even numbers that have never been connected - so the costs of administering an RDD sample are higher since the built-in inefficiencies bring down your contact rate.

An RBS sample draws a sample from a list of registered voters. The obvious advantage of using voter files for survey samples - one that has been noted for years - is that voter file studies are cheaper to administer than RDD studies. RDD surveys have to churn through not only bad numbers but also have to bear the cost of screening out the large portion of adults who are not registered voters, in order to find their real interview targets: respondents who self report as registered voters and who, after applying their own likely voter models, the pollsters define post-interview as likely voters.

With RBS surveys, when you do reach an actual person on the phone, you already know -- since you ask for them by name - that you have a real live actual registered voter on the line and therefore have a better production rate. (The cost difference between the two methods is even more significant in a primary or other low-turnout election scenario, but the debate about using RDD versus RBS samples in low, medium, and high turnout elections is another topic requiring its own separate discussion.) In states that have high quality voter history showing which registrants have actually voted in different types of elections, pollsters can use a likely voter screen to draw the sample in the first place, further ensuring that they are interviewing people most likely to vote in the kind of election they are attempting to measure.

Yet the news media and academics engaged in polling question whether RBS studies can be as accurate as RDD studies, since no voter registration list is 100% up to date, nor does any voter file include 100% of the phone numbers of voters. In fact, the phone match rate for a voter registration list is not only less than 100% but it can vary significantly across a state based on geography, with suburban areas showing a higher match rate than either urban or rural areas. So drawing an RBS sample requires special expertise in terms of controlling for this and other issues about who is potentially over-or under-represented in your sample. So why is it that so many experienced political pollsters continue to use RBS samples despite these concerns about its accuracy? We do so because we find that in many instances (though certainly not all) it is just as accurate, or even more so, than RDD studies.

In fact, some academics and media outlets have been experimenting with voter file survey samples and have found this to be the case. Several have publicly shared at least some of their findings about the ways in which the results do or do not differ when using RDD versus voter file samples. Several studies worth reviewing are by Mitofsky, Lenski and Bloom, by Gerber and Green in Public Opinion Quarterly and the online archive of Gerber and Green's work maintained by the list vendor Voter Contact Services (VCS). These studies have largely shown that RBS studies can be just as accurate and in some cases, more accurate, than RDD studies. One hypothesis offered is that samples drawn from voter registration lists by definition consist of actual voters, while RDD studies rely entirely on respondents' self-reporting about whether they are in fact registered to vote. Given the larger and larger portion of the adult American population that is not registered to vote, the potential for survey error when relying on self-reported behavior may be introducing larger error than carefully designed RBS studies contain.

In one recent example, we saw virtually no differences between the results of an RDD and an RBS study. We provide here just one example from our own work as the polling firm for Ned Lamont for U.S. Senate in Connecticut. In the course of the general election, at one point in September we simultaneously conducted both an RDD study and an RBS study. The results were dramatically in sync, with a margin of error of +/- 4.0 percent on the n=600 RBS study and a margin of error of +/- 3.5 percent on the n=800 RDD study. Considering the far higher cost of using RDD samples as compared to RBS samples, these results certainly give weight to the common practice among political pollsters of using voter file samples instead of RDD samples in general election campaigns.

--Guest Pollster on December 15, 2006 in Pollsters, Response Rates, The 2006 Race | Permalink | Comments (3) | Trackback (0)

December 12, 2006

Gallup's Newport & Jones on The Gallup Panel

(A post by Mark Blumenthal last week discussed results from a Gallup "panel" survey. Today, Frank Newport and Jeff Jones of the Gallup Organization respond with a Guest Pollster Corner piece that provides more information on the Gallup panel.)

The Gallup Poll Panel is a very large probability-based random sample of 18+ adults, used as the basis for a number of commercial and public opinion research projects. All members are recruited using the same random digit dial (RDD) methodology that is the basis for all "normal" national Gallup polls. The panel currently consists of over 37,000 households and over 54,000 individual members. Recruitment is conducted on an ongoing basis.

While the Gallup Poll Panel was developed primarily for commercial purposes, it is used as the basis for conducting occasional Gallup Poll studies. These Gallup Poll studies in essence are based on a random sample of a larger random sample. All interviewing is conducted by phone using standard Gallup Poll techniques. National samples drawn from the Gallup Poll panel are selected randomly, are demographically representative of the U.S. adult population, and are weighted on the back end to adjust to known population parameters, as we would any RDD poll. The Gallup Poll Panel sample is thus a projectable random sample of the adult population of the United States.

Before initiating the use of the Panel for Gallup Poll studies, Gallup conducted several carefully controlled experiments, fielding the same exact surveys at the same time using the panel sample and the standard RDD sample. Analysis showed very few substantive differences in the frequency distribution of results between RDD and panel. To the extent that there were differences, they were almost always within the margin of error (the one exception concerns asking knowledge questions; panel members seem to consistently score higher on these than RDD respondents). (For more information).

Gallup monitors very carefully the use of the Panel as the basis for nationally projectable samples, and will continue to do so in the future.

Frank Newport and Jeff Jones
The Gallup Poll

--Guest Pollster on December 12, 2006 in Pollsters | Permalink | Comments (0) | Trackback (0)

November 30, 2006

Why Polling for the 2008 Presidential Race is Premature

(Today's Guest Pollster's Corner contribrution comes from Greg Smith, president of Greg Smith & Associates, which conducted public polls in Idaho in the 2006 races for governor and congress.)

But, what the heck, it's still fun to talk about it! For pollsters, an objective look at a future scenario such as the 2008 U.S. Presidential race is always appealing. Further, the thought of having insight and uncovering variables contributing to the outcome of just such a scenario is downright enthralling! As pollsters, we need to not be overanxious and in too much of a hurry to examine the 2008 election, despite our zeal. For example, the majority of polling at this point seems to be of the "head-to-head" variety" (e.g., "if the election were held today, ....."). Further, and at this point in time, findings are so much a function of such interrelated factors as current levels of name awareness, exposure by the media, etc. We all can cite recent examples tending to say that Republicans favor Senator John McCain of Arizona, whereas Democrats would choose Senator Hillary Clinton of New York.

I'm not saying these polling efforts are a waste of time. Instead, one should take them simply for what they are -- largely reflective of name awareness. Remember, (1) the election itself won't be for 23 more months. Now is the time for campaigns to build effective organizations, raise money, etc. There will be plenty of time to do more meaningful polling. And, (2) we determine our nominees on a state-by-state basis, not in one national vote. So, even in later polling (i.e., mid- to late 2007), we would generally want to talk to voters in Iowa, then New Hampshire, etc.

However, given our overall interest in the subject, we political pollsters enjoy discussing the 2008 Presidential election, the developing attitudes and perceptions of the electorate, and certainly a brief look at potential campaign scenarios:

One scenario suggests that neither McCain nor Clinton will ever again be as popular as they are now. In McCain's case, he talks a good conservative game, but his actions and votes suggest that he is not conservative enough to win the GOP nomination -- certainly a requirement in recent years (or, at least to campaign as if he is a conservative). And, there is nothing to suggest that likely Republican primary voters in 2008 will be any less conservative in 2008. Clinton will be the target of every conceivable Democratic opponent as the front-runner. Her stances on Iraq are not of help: She has certainly flip-flopped on the Iraqi conflict, which will ultimately bring into questions of credibility on other viewpoints and issues, not to mention the fact that those who comprise the Democratic party at the national level are increasingly liberal.

In my opinion, Al Gore is the most likely 2008 Democratic Presidential nominee. He has "been to the mount", has and will have tremendous financial backing, and can make the argument that he won the 2000 Presidential election and we wouldn't be in the national and international messes we are in today had he "been rightfully installed" in 2000. There are no Democratic governors currently on the horizon (with the exception of Tom Vilsack of Iowa, who has virtually no base of support outside of Iowa), although you never know. Witness Bill Clinton and Jimmy Carter, who both went from single digits early on to become President. And, John Edwards has yet to demonstrate a support base of sufficient magnitude, although this could change quickly, since he represents a fresh face. Barack Obama, the junior senator from Illinois, certainly deserves a mention, although in all likelihood needs a few "gray hairs" (read, more experience and "seasoning") before he deserves serious consideration. Should he decide to run, he will likely not significantly impact the eventual outcome. John Kerry ran a poor 2004 campaign, and continues to commit fax paus after fax paus.

Regarding the 2008 Republican ticket, both McCain and former New York mayor Rudy Guiliani are admirable, although neither sufficiently possess "the right (conservative) stuff" to win the nomination. Then, whom? I personally feel that, at this point, it will be one of three people: Condoleeza Rice, Newt Gingrich, and (most likely) Gov. Mitt Romney of Massachusetts. Ms. Rice brings a high level of credibility and intelligence to the race, but we pollsters will both quantitatively and qualitatively detect some degree of anti-black sentiment (unfortunately). Further, her positions on issues are largely still unknown. We pollsters will reveal a perception of Newt Gingrich relating to a high degree of intellect, but his messy marital breakup, assertive personality, and "being yesterday's news" will create some angst among certain elements of the electorate. Gov. Romney is very well positioned as a strong conservative with specific accomplishments, certainly in the area of health care. Further, his background is squeaky clean. Public perception, however, will ultimately reflect hesitation toward (1) his religion, the LDS faith ("What is a Mormon?"), and (2) his relatively late transformation into a "pro-life" advocate. At this point in time, his Mormonism is not yet akin to JFK's Catholicism in 1960.

We pollsters can and will play an important role in revealing voter attitudes and preferences as they arise. We simply need to exercise good judgment, however, so that our work later in the election cycle is not adversely affected.

Greg Smith & Associates is a marketing/public opinion research and consulting firm headquartered in Eagle (Boise), Idaho, with a variety of clients in both the private and public sectors within Idaho, regionally, and nationally. Greg Smith, the president of the firm, is well known within and outside of Idaho for his political survey research, analysis and commentary. Smith received his M.B.A. from Northwestern University.

--Guest Pollster on November 30, 2006 in Interpreting Polls, The 2008 Race | Permalink | Comments (8) | Trackback (0)

November 15, 2006

Partisan Composition of Samples in 2006 Generic Congressional Ballot Surveys: Greater Discolsure, Less Controversy

Today's Guest Pollster Corner Contribution comes from Alan Reifman of Texas Tech University, who takes a closer look at this fall's pre-election polls.

In the months leading up to the 2000 and 2004 general elections, presidential election polls showed considerable variation -- both across different pollsters and within the same pollster at different times -- in the percentages of self-identified Democrats, Republicans, and Independents comprising the samples. Sample composition itself probably would not concern many people, but when these sampling variations seemed to affect the polls' candidate vs. candidate "horse race" numbers, people got agitated.

Discrepancies in polls' partisan compositions almost inevitably raise the issue of whether survey samples should be weighted (i.e., post-stratified) to match party ID figures from sources such as previous elections' exit polls, much like polls are weighted to match gender and other demographic parameters from the U.S. Census. Underlying the question of whether pollsters should weight by party ID lies another question: How fixed and enduring are voters' identifications with a party? Again, experts differ. Zogby was the first major pollster to weight on party ID, with Rasmussen following suit later. Most, if not all, of the remaining pollsters do not weight by party.

I track polls' partisan compositions at my sample weighting website. I am neither a pollster nor a political scientist, but I am a social scientist who teaches research methods and statistics, and I've spent much time studying and collecting data on party identification. I also took a graduate statistics class many years ago from Pollster.com contributor Charles Franklin.

If I had to summarize developments on the sample composition/weighting front for 2006 (where the main point of interest was the Generic Congressional Ballot), I would identify two trends:

1. Thanks to the efforts of the Mystery Pollster himself and others who raised the issue over the past few years, full "topline" documents (also known as polls' "internals"), which included party ID numbers, were freely accessible via the web for most of the national pollsters during this past election season.

2. The margin between the percentages of self-identified Democrats and Republicans (D minus R) comprising most national polls over the final two months of the campaign season was pretty stable. As a consequence, questioning of polls' partisan breakdowns was relatively rare this year.

On my website, I used Rasmussen's party ID readings as my benchmark for comparison, due to the large numbers of interviews involved (500 daily interviews, aggregated over the 90 days preceding the start of each new month). Most of the time, Rasmussen had the D-R margin at roughly 4.5 percent. As shown in the major chart on my website, when multiple independent polls that were in the field during roughly the same time frame (and which released the necessary party ID numbers) were available, I averaged their partisan percentages. Four polls (not including any from Rasmussen) taken from October 18-22 inclusive showed averages of D 34.4 and R 29.8, well in line with Rasmussen's margin. (The average of five polls from an earlier period, October 5-8 inclusive, had a wider Democratic margin: D 36.9, R 29.6.)

In the chart, I also provided brief verbal commentary on how each poll's partisan breakdown matched up with Rasmussen's. As can be seen, polls' D-R margins were sometimes described as "about right," with instances of "D edge understated" and "D edge overstated" almost perfectly balancing out over the final two weeks.

In the end, the New York Times exit poll (N = 13,251) showed the national electorate for U.S. House races to consist of 39% Democrats and 36% Republicans. This 3-point difference is slightly smaller than would have been anticipated from some of the late polls, but only slightly. It should also be noted that, even with its huge sample size, the Times exit poll still is a sample survey and thus carries a small margin of error (about +/- 1).

One final note: As animated as I get by party ID percentages, I must acknowledge that they are not the whole story. For example, among the final batch of polls, FOX, Pew, and Time all had Democratic respondents outnumbering their Republican counterparts by either 3 or 4 percent. Yet these polls differed widely in their Generic Ballot readings, with FOX and Time having Democrats up 13-15 percent (with FOX's sample explicitly described as consisting of "likely" voters), whereas Pew had them up only 8 (among registered voters) or 4 (among likely voters). Other traditional issues of survey methodology -- such as question wording and order effects -- thus have to be examined for their possible role in these polls' varying D-R margins on the Generic Ballot.

--Guest Pollster on November 15, 2006 in Exit Polls, The 2006 Race | Permalink | Comments (0) | Trackback (0)

November 11, 2006

Out of Step or Out of Office? (Or Just a Bad Election for Republicans?)

Today's Guest Pollster Corner Contribution comes from Simon Jackman of Stanford University, who takes a closer look at Tuesday's Senate election results. 

Two interesting questions to ask after Tuesday's election are (1) Were the six defeated Republican senators particularly "out of step" with their respective states?; (2) What will be the effect of the Democratic pickups on the look of the new, 100th Senate?

To answer this question I first assigned a liberal-to-conservative voting score to each senator based on an analysis of the 530 non-unanimous roll call votes cast in the 109th Senate. The resulting scores are scaled to have mean zero and standard deviation 1, with lower (negative) scores reflecting a more liberal voting history, and positive scores reflecting a more conservative voting history (the details of this scoring procedure appear in my 2004 article with Josh Clinton and Doug Rivers in the American Political Science Review. A familiar story results, with Democrats on the left, Republicans on the right, with virtually zero no overlap between the parties. Lincoln Chafee is estimated to be the most liberal Republican with a voting score of about zero, while Ben Nelson (NE) is the most conservative Democrat (again, with a voting score close to zero). The usual suspects anchor the extremes of both parties: Barbara Boxer (CA) and Ted Kennedy (MA) for the Democrats (scores of -1.9), and Inhofe (OK) and Demint (SC) for the Republicans (scores of 1.3).

To gauge each state's political complexion, I use a simple and convenient proxy: Bush's share of the 2004 presidential vote in each state, which ranges from a high of 71.5% in Utah, to a low of 36.8% in Massachusetts, with a median of 52.7% (bracketed by Florida's 52.1% and Missouri's 53.3%).

The graph below shows a scatter-plot of voting score against 2004 Bush vote. Each point corresponds to a senator (red for Republicans, blue for Democrats, with senators running for re-election given a heavier shading), with 2004 Bush vote on the horizontal axis, and the roll call voting score on the vertical axis (higher is more conservative, lower is more liberal). The gray line is a regression fit to the data, not to be taken too seriously, but rather more as a rough guide as to how "out of step" the senator may or may not be. Republicans running for re-election are all numbered: Republican senators losing their seats are numbered 1 through 6, the other Republican senators who were re-elected are numbered 7 through 14. Chafee (RI), Santorum (PA) and Allen (VA) seem to be the only Republican losers who are obvious candidates for a "out of step" with their state kind of story (along the lines proposed by Canes-Wrone, Brady and Cogan in a 2002 article in the American Political Science Review, lying relatively distant from the regression line. DeWine (OH) seems to have been caught in what was an extremely difficult election for Republicans in Ohio, and neither Talent (MO) nor Burns (MT) appear to have been particularly "out of step". And keep in mind that there are several Republican senators just as apparently "out of step" as Santorum or Chafee who did not lose their seats: e.g., Jon Kyl (AZ), who faced no Democratic opposition in 2000, but won 53-44 in 2006; or John Ensign (NV), who by almost identical margins in both 2000 and 2006.

It is interesting to speculate on shape of the new, 110th Senate. Chafee goes, replaced by a Democrat, leaving the Maine senators (Olympia Snowe, re-elected with a 74-21 margin in 2006, and Susan Collins) as the most moderate Republicans. It remains to be seen just how liberal or moderate the new Democrats will be. Given their states and the narrow margins with which they are projected to win, it is tough to imagine Webb (D-VA) or Tester (D-MT) being particularly liberal, perhaps voting more like relatively conservative Democrats from the plain states (e.g., Nelson, NE; Conrad and Dorgan from ND) or the other Montana senator (Baucus).

--Guest Pollster on November 11, 2006 in The 2006 Race | Permalink | Comments (7) | Trackback (0)