December 17, 2006 - December 23, 2006


Daily Roundup 12.22.06

  • A new Research 2000 survey of likely Democratic Iowa caucus voters shows Sens. John Edwards and Barack Obama in the lead (both at 22%) with Sen. Hillary Clinton in 4th place at 10%. Among Republicans, Sen. John McCain leads former Mayor Rudy Giuliani 27% to 26%. (story, results)

  • A new AP-AOL poll shows "many people [42%] long for the days when businesses routinely told customers "Merry Christmas" rather than the more politically correct, "Happy Holidays" or "Season's Greetings." But nearly half [48%] are not bothered by the broader greeting."

  • A new Rasmussen Reports automated survey reveals that with four days to go, 17% still haven't started their Christmas shopping, down 3% since Thursday.

  • A new Gallup Poll asked adults "to name what they enjoy most about the holiday season." 61% named spending time with family and friends, 5% named giving gifts.

The 2006 Exit Polls: How Did They Perform?

Topics: 2006 , Exit Polls , The 2006 Race

Today's Guest Pollster's Corner contribution comes from Mark Lindeman, assistant professor of Political Studies at Bard College.

In the wake of allegations that the 2004 U.S. presidential exit polls pointed to a stolen election, many observers wondered how the 2006 exit polls would turn out. One widespread rumor asserted that no exit poll results whatsoever would be made public until after the polls had been forced to match the official vote counts. But in fact, CNN.com once again posted a preliminary national House tabulation a bit after 7 PM Eastern, and posted tabulations in state races soon after the polls closed in each state. (Other outlets may have done so as well: at the time I had my hands full with just one.) These tabulations appear to show discrepancies fairly similar to the 2004 discrepancies, as I report below.

Using tabulations to estimate exit poll "red shifts"

The tabulations are not intended to project the final vote counts. Rather, they offer crude but useful insights into why voters voted as they did. Nonetheless, each tabulation is based on a particular vote estimate made at a particular time. The exit pollsters use different estimates for different purposes. Before vote counts begin to arrive, the pollsters can refer to at least three estimates. These are (as described in the post-election evaluation report on the 2004 exit polls):

  • The "best survey" or "Best Geo" estimate -- based on interview data (from exit polls and, in some states, telephone surveys of early and absentee voters), and also incorporating data on past results from the exit poll precincts
  • The "prior" estimate -- based primarily on public pre-election surveys (something like the averages posted on Pollster.com)
  • The "composite" estimate -- a hybrid which combines the interview data (Best Geo estimate) and pre-election polls (prior estimate).

The initial tabulations posted by CNN.com are based on the composite estimate -- not just on interview data. Therefore, they probably tend to understate the disparity between the exit poll results and the vote counts. For instance, the initial 2004 "screen shot" of Pennsylvania indicates that Kerry had about 54% of the vote, and the evaluation report confirms that the composite estimate was 54.2% [p. 22]. But the report also reveals that the interview-only Best Geo estimate gave Kerry almost 57% of the vote, or a 13.8-point margin [p. 22]. The official result -- Kerry won by 2.3 points -- constitutes an 11.5-point "red shift," or reduction in Kerry's net margin, compared to the Best Geo estimate. Because pre-election polls showed a very tight race in Pennsylvania, the composite estimate gave Kerry "only" an 8.5-point margin, or 6.2-point red shift. Overall in 2004, the average discrepancy was a 5.0-point red shift in the Best Geo estimate, but "only" a 3.6-point red shift in the composite estimate.

(Once vote counts start to arrive, the pollsters continually generate a variety of estimates that incorporate vote count data at both precinct and county levels. These dynamic estimates are used to inform the decisions to "call" -- or not to call -- each race. Intermittently the pollsters also generate new tabulations based on a current vote estimate. Updating the tabulations has been described by critics as replacing "pristine" exit poll results with "soiled" ones. [*] Actually, if "pristine" means "based on interviews only," none of the tabulations is pristine.)

To "estimate the estimates" from the early tabulations, I use each table in a tabulation to figure approximate party or candidate shares, then take the median of the differentials across all the tables. For instance, take this snippet of the preliminary national House poll:


We can use these percentages to estimate that 49% * 53% (or about 26%) of voters were men who voted for Democrats, and 51% * 57% (or about 29%) were women who voted for Democrats. So, based on this table, apparently Democrats got about 26% + 29% = 55% of the vote. Applying the same logic, apparently Republicans got about (49% * 45%) + (51% * 42%) = 43.5% of the vote, for about an 11.5% Democratic margin. However, other tables imply somewhat larger or smaller margins, due to the influence of rounding error. Using a median of estimates from all the tables reduces this rounding error, and a computer program interpreting the HTML tables can do the calculations almost instantly. (Because of mistakes I made on election night, I have cruder approximations for two uncompetitive Senate races -- Minnesota and Utah -- and no data for the gubernatorial races in Illinois and Tennessee.)

What I found

Overall, I estimate that the initial national House tabulation gave Democrats an 11.3% margin in total vote. The final tabulation currently available, weighted to the pollsters' vote estimate at that time, gives Democrats a 7.6% margin, so these figures imply a 3.7-point "red shift" -- close to the 3.6-point average in the 2004 presidential composite estimates. If the final official margin is closer to 7 points, as Mark Blumenthal has estimated, the red shift may be above 4 points. [*] However, the vote proportions are influenced at the margins by uncontested races, which appear on the ballot in some states and not in others. Without knowing exactly how NEP handles these uncontested races (nor whether voters accurately report their votes and non-votes in these races), it is unclear what vote totals we should compare to the exit poll estimates.

(Note also that the House tabulation is not quite like the state-level tabulations I discuss next. The state-level tabs, posted as the polls closed in each state, should incorporate all the interview data. The House tabulation, posted long before the polls closed in many states, relies on partial data from much of the country. I have no reason to think that the complete results would be much different.)

State-level races yield broadly similar red shift estimates. In the Senate races, I estimate that the average red shift was 2.3 points, and the median red shift was 2.8 points. In races for governor, I estimate that both the average and the median red shift was 4.0 points.


As in 2004, most of the largest exit poll discrepancies were in uncompetitive races. Some observers have cited (here , here, and here) the red shifts in the Virginia and Montana Senate races as pointing to vote miscount favoring the Republican incumbents -- who nonetheless lost both races. But since those two races have near-average red shift, there is little reason to single them out. Perhaps the most striking discrepancy is in the Minnesota governor's race. Democratic challenger Mike Hatch appeared to have an 8-9% lead in the initial exit poll tabulation, but lost to incumbent Tim Pawlenty by about 1%. The pollster.com 5-poll average gave Hatch a narrow 2.6-point margin, so the election result was closer to expectations than the exit poll result was. Minnesota also experienced one of the largest "red shifts" in 2004.

The House red shift has also been cited as evidence of vote miscount, most elaborately in a paper issued by the Election Defense Alliance (EDA). The paper argues that respondents' reports of their presidential votes in 2004 can be used as an "objective yardstick" to evaluate the 2006 poll. In the final House tabulation, (self-reported) Bush voters outnumber Kerry voters by 6 percentage points, more than double Bush's popular vote margin. EDA's analysts reason that the exit pollsters in effect had to invent millions of Bush voters (and/or delete Kerry voters) in order to match the House vote counts -- which, therefore, must be wrong. The basic flaw in this argument is that reported past vote is not an objective yardstick. On the contrary, as I have noted elsewhere, exit polls and other polls often -- even usually -- overstate past winners' vote shares. Worse, because the authors believe that Kerry won the popular vote and that Democrats had higher turnout in 2006, they end up conjecturing in a footnote that Democrats actually won the House vote by 23(!!) percentage points, a double-digit deviation from the initial tabulation. So much for defending the reliability of exit polls!


After the events of election night 2004, the NEP pollsters (Edison Media Research and Mitofsky International) announced efforts to reduce exit poll bias. Among other things, Edison/Mitofsky planned to improve interviewer training in order to minimize any selection bias on the part of interviewers. (See, for instance, Joe Lenski's interview with Mark B.) Despite the strong evidence of red shift in the 2006 data, we cannot conclude that these efforts were ineffectual. Participation bias easily could have been larger than ever in 2006. As Mark Blumenthal has noted, a Fox News/Opinion Dynamics pre-election poll found that 44% of Democrats, versus only 35% of Republicans, said that they would be "very likely" to participate in an exit poll. Differences in levels of concern about electronic voting and election fraud may (or may not) contribute to that disparity. In any case, no methodological refinement can force Democratic and Republican voters to participate at equal rates.

Interestingly, the pilot "Election Verification Exit Polls" (EVEP) conducted by Steve Freeman and Ken Warren reported similar or larger red shifts. Freeman's initial report indicates red shifts ranging from 5 to 8 percentage points in four distinct races (two House races, Senate, and governor) in the 28 Pennsylvania precincts surveyed. Freeman argues that the survey "eliminated most of the potential sources of error" (7), presumably through careful training of the interviewers. However, Freeman also reports that several interviewers who obtained relatively low completion rates "subjectively felt that Republicans were disproportionately avoiding participation" (8).

A first glance at the EVEP data files shows at least one case of large red shift where the exit poll result seems implausible. In this precinct, the exit poll registered a 63% majority for House Democratic challenger Lois Murphy (PA-06), while the official returns gave her just 44% of the vote. Registration statistics for the precinct (Chester County precinct 021, East Bradford North 2) show that registered Republicans outnumber Democrats by more than 2 to 1 (57% to 26%). If we concede Freeman's premise that the EVEP methodology was close to ideal, this result hardly inspires confidence in exit polls' inherent accuracy.

Campaign Dynamics for the Holidays

Topics: The 2004 Race


(Click the image for a full resolution version.)

I've had so much fun with the new oversize color laser printer that I felt the need to share another political graphic suitable for framing for the holidays. This one compares the 2000 and 2004 presidential campaign dynamics. I'll let you add your own commentary, suitable for either party.

An 11 x 17 version is available here for download (in .pdf format) for the political junkie on your shopping list. (If you lack a large color printer, I bet your local copy shop can print this for a modest fee.)

Cross-posted at Political Arithmetik.

Does Spelling Count?

Topics: Internet Polls

"To retain respect for sausages and laws, one must not watch them in the making." Unfortunately, that quotation (widely attributed to Otto von Bismarck) may sometimes also apply to surveys. It is all too easy to put out percentages on a press release calculated to one decimal place that add to 100.0%. More often than not, the factors that distinguish excellence in survey research are invisible to the consumer. Of course, with an online survey the respondent does have the ability to save the verbatim questionnaire, and that brings us today's story about the remarkable number of spelling and grammatical errors in two recent surveys conducted online by Zogby International.

I should say up front that yours truly is in no position to cast the proverbial first stone when it comes to spelling and typographical errors. In my first week of blogging, Mickey Kaus observed that the Mystery Pollster "also seems to be a bit of a Mystery Speller." While I have tried to have others proof my copy before it goes online, I am painfully aware that typos probably remain in my archives. Alas, proofreading is not my strong suit.

However, all pollsters understand the critical importance of proofing our work product -- be it a draft questionnaire or the final results of a poll -- before it goes into the field or out the door. In a business where consumers expect precision, evidence of sloppy work takes on added significance. Just one careless, trivial mistake can undermine a client's confidence in an otherwise brilliantly executed survey.

The importance that most pollsters place on proofreading makes the email I received last week from a long time reader, someone with considerable training and experience in survey research, so remarkable:

As you may recall I'm on Zogby's Interactive panel. In the past the surveys often contained oddly worded questions, but at least they were copy-edited, spell-checked, proof-read, etc. Lately? Not so much. A few days ago I took a survey that included a number of topics, including the Israeli-Palestinian situation. I was struck by a number of misspellings. Here are the items that included simple misspellings of the type that are caught by spell-check (some are the actual question; some are response options) [Misspelled words in bold type]:

The U.S. Constitution says that a president can be impeached for high crimes and misdeameanors. Knowing this, would you favor or oppose impeaching George W. Bush?

  • Have you ever visted Bethlehem?
  • How interested are you in visting Bethlehem again or for the first time?
  • Do you support or oppose the contruction of this wall?
  • The wall seperates some Bethlehem families from one another [repeats twice]
  • The wall cuts through a historic Christian Diocese, seperating Bethlehem from Jerusalem [repeats twice]
  • The wall seperates Bethlehem and Jerusalem - two cities that have been histoircally interlinked and interdependent through community ties, trade and religious traditions [repeats twice]
  • Do you think that the land that is being confiscated by Israel in order to build the wall is primarly taken from. . .?
  • Chrisitan residents of Bethlehem
  • Of those listed, which of the following would be the greatest deterent to visiting Bethlehem?
  • Peacful Coexistence

That's bad enough, but today I received [another Zogby online] survey that was so awful it was painful. After an annoying but not terrible section on moral choice situations, they started with some civil lawsuit scenarios and then started using the word "libel" all over the place. At first I'm trying to figure out what libel has to do with anything, but then I realize that they actually mean "liable." They do this not just once, but for several scenarios, each involving several questions. It's really awful!

My reader goes on to provide examples of questions. The survey presented a series of factual "scenarios" involving two fictional physicians, Dr. Smith and Dr. Jones, then asked questions like the following:

13. On a scale of one to five, with one being not libel and five being libel, how likely are you to find Dr. Smith libel in this case?

Not libel
Probably not libel
Probably libel
Not sure

14. On a scale of one to five, with one being not libel and five being libel, how likely are you to find Dr. Smith libel if you knew that he had never had a child suffer the damage Jane did, even though he has had several other children also complain of a stinging feeling while receiving the same vaccination?

Not libel
Probably not libel
Probably libel
Not sure

My reader continues:

That's just the first of 21 questions like that, all of which make the same repeated mistake. All in all, the word "libel" shows up 137 times in the survey! This is indeed a train wreck.

But that's not all. With my jaw still on the floor, on the fourth or fifth scenario I noticed something else. They keep on asking "on a scale of one to five" but there's no scale, no numbers, just the response options. This error is made 30 times in the survey.

I emailed Zogby's Director of Communications for comment on Friday of last week and have not received any response. Zogby posted results of the Bethlehem survey yesterday.

Readers may ask whether these misspellings affect the results of the survey. Of course, spelling errors matter less when a survey is conducted by telephone (assuming that the interviewers pronounce misspelled words like "visted" correctly rather than phonetically). The two Zogby surveys discussed here were conducted online, yet even when online respondents notice the misspellings of words like "seperate" and "historically," they can presumably guess at the intended meaning. But the libel/liable mix-up raises more troubling questions. If respondents did not know the difference between the two words, did they understand the issues raised by the questions well enough to provide meaningful answers? If they knew the meaning of "libel" but did not guess it really meant "liable," how did they interpret the question? And if they deciphered the error, as my friend did, how many times could they tolerate its repetition before bailing out of the survey altogether?

But mostly, this sort of error should make consumers wonder. If a pollster neglects something as simple as spell checking a questionnaire before fielding it, what other mistakes is that pollster making that are not quite so obvious? Oh wait...

Daily Roundup 12.20.06

  • A new Gallup Poll shows former Mayor Rudy Giuliani and Sen. John McCain tied for the Republican Presidential nomination (28% each). Sen. Hillary Clinton leads Sen. Barack Obama for the Democratic nomination (33% to 20%).

  • CNN/Opinion Research Corporation released more results from their recent national survey testing a generic ballot and hypothetical match-ups for the 2008 presidential election.

  • The latest Rasmussen Reports automated survey says 78% of U.S. workers expect their job prospects and the employment market for 2007 to be "as good, if not better" than 2006.

  • A new Quinnipiac survey shows that 57% of Florida voters rate termed-out Gov. Jeb Bush as either a "great" or "good" Governor.

Pres 08: The Republican Primary Race

Topics: 2008 , The 2008 Race


(Click the figure for full resolution. For a full resolution collector's edition 11 x 17 version of this graph in .pdf format, click here. They make great holiday gifts.)

The 2008 presidential races are off to a running start, with candidates announcing, dropping out, changing their minds. And one of the leaders in the polls says she isn't running (Condoleezza Rice).

In this first post of the primary season, we look at the Republicans.

The data here represent EVERY person who has been included in ANY poll's list of potential candidates (and in the case of California Governor Arnold Schwarzennegger isn't even legally eligible for the office.) I've thought about editing the graph to remove those who have not become serious candidates, or who have tried and failed. But there is some virtue in a comprehensive look at the entire field as defined by pollsters' questions.

I plot the trend line only after 10 polls are available for a candidate. The poll results are represented by gray points in each graph regardless of how many polls are available, including those candidates who have appeared in only a single poll.

One notable result stands out. Former New York City Mayor Rudolf Giuliani continues to hold a small but reliable lead over Arizona Senator John McCain. Of 39 polls with both names in the list of candidates, Giuliani leads McCain in 30 with four more ties. McCain leads in only 5 polls.

That doesn't seem to me to be the message I've been getting from the media, so I did a little research. I searched Lexis/Nexis for "McCain" and "frontrunner" in the same paragraph, but "Giuliani" NOT in the paragraph. I then did the opposite. And finally for both mentioned in the same paragraph with "frontrunner". The results don't seem to reflect the polls. Using "American Newspapers" as a broad sample of papers, and 5/1/2006-12/19/2006 as the date range, I found:

McCain, frontrunner, NOT Giuliani: 270 articles
Giuliani, frontrunner, NOT McCain: 28 articles
Both with frontrunner: 76 articles

So I was relieved to find that my impression of media coverage as MUCH more favorable to McCain was not just a delusion. It also appears at strong odds with the polling data. Giuliani leads McCain by an average of 3 points in the 39 polls. Not a huge lead, but a very consistent one.

Much of the reporting seems to downplay Giuliani's chances of winning the Republican nomination, but given the negative opinion of McCain among many conservative Republicans, I wouldn't assume McCain can win easily either.

Former Speaker of the House Newt Gingrich is usually in third place in the polls, showing considerable strength for someone who has been out of office for as long as he. Gingrich is, however, far behind both Giuliani and McCain.

Massachusetts Governor Mitt Romney has shown slow but steady growth in support, though he remains far behind as well. Not surprising at this point when name recognition counts for a great deal in these early polls.

None of the other potential candidates are so far generating much growth in support. In fact, none of these candidates outside the top 4 (top 5 if we count Rice) are averaging support of as much as 10%, and most are struggling to get to even a 3% average. Despite the dismal numbers I don't think that counts out these candidates at this point. There are substantial reasons to wonder if ANY of the top 4 of Giuliani, McCain, Gingrich or Romney can capture the Republican nomination, and that implies that there will be an opportunity for one of these "virtual unknowns" to emerge as a serious contender. It is a very long time until the Iowa caucus.

I'll take a look at the Democrats tomorrow, and we'll put it all in historical perspective soon.

Cross-posted at Political Arithmetik.

Daily Roundup 12.19.06

  • The latest CNN/Opinion Research Corp national survey (summary, full results) finds Bush's job approval for handling Iraq has dropped from 34% to 28% since November.

  • This morning's Gallup Poll release shows Congressional job approval now at 21% (26% in November).

  • The latest Rasmussen Reports automated survey says 45% of Americans consider Global warming a "very serious" problem while 46% of Americans believe Global Warming is caused primarily by human activity. Sen. John McCain also "crushes" Sen. John Kerry and Gen. Wesley Clark in a presidential match-up.

  • UPDATE: Full press releases for the recent Newsweek national survey are available via PR Newswire in two parts. (12/9, 12/17; second release includes primary and presidential match-up results)

"Is America Ready?"

Topics: 2008 , The 2008 Race

"Is America Ready?" That's the question posed on the cover of this week's Newsweek featuring Hillary Clinton and Barack Obama, a recent Newsweek poll, and other recent national surveys conducted by Gallup, Fox News, NBC/Wall Street Journal, Rasmussen Reports and Cook/RT Strategies. The question of whether U.S. voters are ready to elect a woman, an African American - or a Mormon for that matter - is something that political junkies will presumably continue to ponder for the course of the 2008 campaign. For those pondering such a question now, let me suggest a resource (and interview with Democratic pollster Celinda Lake) and one possibly overlooked point (about the candidacy of Harold Ford, Jr.).

In an interview that aired over the weekend (via an AAPOR member), my long-ago boss Celinda Lake spoke with NPR's Scott Simon about the challenges of asking the kinds of questions included in most of the recent surveys. Although she was understandably vague, given the nature of the brief interview, about how she arrived at her conclusions, Lake argued that 5 to 10 percent of voters lie on such questions, giving "tell the interviewer what they believe is the politically correct thing to say." She also made this notable observation:

It's very hard to poll now because people are reading in partisanship. So where Republicans and Democrats used to be equally supportive of a woman for president, when you ask that now, Republicans are less supportive because they assume you mean Hillary Clinton.

The same question - Is America ready for a woman or an African American president - also came up at the post-election conference I attended last week, and the four pollsters generally agreed that a race involving Clinton or Obama was not likely to be about race or gender. Many pointed to the Tennessee Senate race as evidence that the race of Democrat Harold Ford Jr. did little to limit his appeal. Several panelists throughout the day said that given the partisanship, Ford did as well as any Democrat could have, adding that a candidate like Jim Webb won in Virginia only because Virginia is less Republican than Tennessee.

The exit polls for Tennessee and Virginia tend to support that point. Both Democratic candidates received exactly the same percentage of support from Democratic partisans. Ford's race was certainly no unique barrier to those that identified as Democrats, although Webb did slightly better among independents (though one might quibble about the statistical significance of that difference. Similarly, in the Maryland Senate race, African American Republican Michael Steele did precisely as well among Republicans (94%) as Corker in Tennessee and Allen in Virginia.


Of course, one point made by both Celinda Lake in her NPR interview and Democratic pollster Harrison Hickman at the Cook Conference is that people think of presidents, and evaluate potential presidents, differently than Senators and Governors. As with all of the most interesting questions, we will have to wait and see what the political future holds.

Daily Roundup 12.18.06

  • Newsweek's latest national survey says 86% of registered voters would vote for a qualified woman candidate, 93% would vote for a qualified African-American.

  • A new Monmouth University survey found that New Jersey Republicans are more likely than Democrats to be talk radio listeners (50% to 38%).