Mark Blumenthal | November 7, 2006
Topics: Exit Polls
Yes, the television networks will be conducting exit polls today. But if you are looking for the leaked exit poll estimates that typically appear online on Election Day, you are probably out of luck at least until later tonight. More on that below. But as long as you are here, let me tell you a little bit about how exit polls are conducted, how they will be different this year, and why it is probably best to try to ignore the exit poll estimates that will inevitably leak later tonight.
I have always been a fan of exit polls. Despite their shortcomings and the inevitable controversies, the final network exit polls remain our best source of data on who voted and why. Having said that, exit polls are still just random sample surveys, possessing the usual limitations plus some that are unique to exit polling.
A quick summary of how exit polls work: The exit pollster begins by drawing a random sample of precincts within a state, selected so that the odds of any precinct being selected are proportionate to the number that typically vote in that precinct. The National Election Pool (NEP) consortium, which is conducting the exit polling for the six major networks, will send exit pollsters to more than 1,000 precincts across the country today.
One interviewer will typically report to each sampled precinct. Each interviewer will stand outside and attempt to randomly select roughly 100 voters during the day as they exit from voting. The interviewer will accomplish this task by counting voters as they leave the polling place and selecting every voter at a specific interval (every 3rd or 5th voter, for example). The interval is chosen so that approximately 100 interviews will be spread evenly over the course of the day. As we learned following 2004, random selection of voters at the polling place is the most important part of the process, and arguably the most susceptible to problems.
When a voter refuses to participate, the interviewer is instructed to record their gender, race and approximate age. These data allow the exit pollsters to do statistical corrections for the bias in gender, race and age that might result from refusals.
The selected voters receive a one-page paper questionnaire to fill out. In the past, the questionnaire included approximately 25 questions (see an example from the 2004 exit polls), although this year the exit pollsters have worked to prepare a shorter questionnaire. Respondents fill out the survey privately then place it in a clearly marked "ballot box" so they know their identities cannot be tracked and their answers remain confidential.
The logistics of transmit all the results to a central location quickly and accurately provides the biggest challenge. To facilitate the process, interviewers will take a 10 minute break during the day to tabulate responses. Interviewers have typically stopped to call in their tabulations at three approximate times during the day: 9:00 a.m., 3:00 p.m. and shortly before the polls close.
Once the polls close, the interviewer will attempt to obtain actual turnout counts, and if possible, actual vote returns for their precinct from polling place officials. One of the unique aspects of the NEP exit poll design is the way it gradually incorporates real turnout and vote data as it becomes available once the polls close. The exit pollsters have developed weighting schemes and algorithms to allow all sorts of comparisons to historical data that supports the networks as they decide whether to "call" a race for a particular candidate. When all of the votes have been counted, the exit poll is weighted by the vote to match the actual result.
Throughout the day and after the polls close, the exit polls are a part of a larger process that provides network analysts with various "estimates" of the vote in each state. They have esoteric names like "Best Survey Estimate" and "Composite Estimate" (a post-election report on the 2004 exit polls had some rare definitions and examples, see p. 7). Before the polls close, those estimates derive mostly from exit poll data, but as actual returns become available, the computer models gradually replace the exit poll tallies in each precinct with hard vote counts.
After the polls close, the network analysts also begin to look at estimates based on larger samples of approximately 2,000 randomly selected precincts where NEP personnel obtain official vote tallies as they become available. The networks typically use this larger system of tabulations and estimates -- rather than the exit polls alone -- to call close contests after the polls close. But when the vote is really close, as it is likely to be tonight in states like Missouri and Virginia, even the larger samples of hard returns will be inadequate. They will have to watch the complete vote count, just like the rest of us.
In years past, hundreds of producers, editors and reporters would have access to the mid-day estimates starting at about noon, and preliminary exit poll cross-tabulations (like this one) would inevitably leak out.. This year the networks have taken steps to prevent such leaks, creating a "Quarantine Room" where, as the Washington Post's Howard Kurtz reports, "two people from each of the networks and the Associated Press" will be allowed "entree to a windowless room in New York [where] all cellphones, laptops and BlackBerrys will be confiscated. The designated staffers will pore over the exit polls but will not be allowed to communicate with their offices until 5 p.m."
So there you have it folks. At 5:00 p.m. the data will be flowing to newsrooms as it always does and the usual leaks will presumably commence shortly thereafter. One rationale for the delay is that the data available at that hour will be more representative and not just based on those who vote in the morning.
So if this process is so sophisticated and the late afternoon numbers are better, why can't we rely on the leaked "numbers" that will undoubtedly leak after 5:00 p.m. today?
1) It is still just a survey -- Even when complete, an exit poll still has the same random variation as any other survey. NEP says typical state exit polls have a sampling error when complete of 4% at a 95% confidence level, and 3% for the national exit poll.
But that is the margin of error recommended for reading the cross-tabulations. The exit networks pollsters require far more statistical confidence before projecting winners. As Joe Lenski, whose company directs the exit poll operation, told me in an interview last week, the "rough rule of thumb" is to require "three standard errors, which would be 99.5% confidence" before projecting a winner. While the precise numbers will depend on the circumstances, that implies a much larger "margin of error."
2) Early or absentee voting -- The ABC News/Washington Post survey released on Sunday reported that roughly 12% of all registered voters nationally had already cast absentee or early ballots. Obviously, these voters will not be available to interviewers standing outside polling places. To incorporate early voting, the National Election Pool is doing telephone interviewing in many states to sample the votes of those who voted early (although they will not do telephone interviews in Maryland, where doubts about electronic voting have caused absentee requests to skyrocket). Will these results distributed to NEP subscribers after 5:00 p.m. include the telephone results for early voters? Who knows? I wouldn't count on it.
3) Past Errors favored Democrats -- In past elections, the results have slightly favored Democratic candidates. As we learned after the 2004 elections, the exit poll estimates have had at least some small error favoring the Democratic candidate for president in every election since at least 1988. In 1992 and 2004, the two elections with the highest levels of voter interest and turnout, those errors were more acute.
Since 2004, the exit pollsters have taken steps to try to eliminate those problems. As Joe Lenski explained, they have shortened the questionnaire, improved interviewer training and made an effort to hire older interviewers (because of problems that younger interviewers had in past years gaining cooperation from older voters).
They have also gone to court to allow more interviewers to near polling place exits. Hopefully, these efforts will reduce or eliminate the problems experienced in 2004, but we will have no way of gauging their success at 6:00 p.m. tonight.
4) Limited Data on the House elections -- While the exit polls will produce vote estimates for the major contests for Senate and Governor, they will not make projections on individual House seats. The networks will do that based on actual votes. The national exit poll sample will include an estimate of the national vote for Congress by party, but as Joe Lenski points out, "because of Gerrymandering, we know that popular vote for the House does not translate into House seats directly." (See also Professor Franklin's post on the imperfect relationship between the national vote for Congress and seat gain or loss).
Of course, we are all intensely curious about the outcome of today's elections, and should the exit poll numbers leak again, most of us will find a way to peek at them. I just want you to know that those leaked exit polls really don't tell us much more about the outcome of the race than the telephone polls we have been happily obsessing over at Pollster.com all fall. Even if we wanted to call a race on unfinished, late-day exit polls alone (something the networks will not do), we would need to see differences of at least 8-10 points separating the candidates to be statistically certain about the outcome.
So look at them if you must, but please, don't go leaping to conclusions based on a leaked exit poll margins showing one candidate "leading" by a few percentage points. Those numbers are not magic. You would be better off flipping a coin -- or better yet, checking the final public poll averages here on Pollster -- to determine the outcomes of the Senate contests in Missouri, Virginia and Montana.
PS: All of the above applies to true "exit polls," those that intercept voters as voters exit the polls. In the vacuum created by the "Quarantine Room," I would not be surprised to see someone release results from a "day of" telephone poll billed as an exit poll. Don't let the name fool you. If the network exit polls are of little value in close races, imagine the potential problems facing a telephone survey, conducted during the day when most voters are still at work. If one of these pops up, I'll be back to blog about it.
I will be online all day. If you have questions about exit polls or anything else, please send them my way.
The following are links to the NEP network web pages that will post exit poll results at some point. Most of these are either not yet displaying results page or all also include actual vote returns:
- ABC News - Vote 2006 / Hometown Races
- CBS News - Live Results
- CNN - Senate / Governor
- Fox News - Balance of Power
- MSNBC - Senate / House / All (nice map on the "live results" link)
You might also want to check some of the following web resources for more information:
- The Frequently Asked Questions (FAQ) page at the National Election Pool web site.
- My interview with exit pollster Joe Lenski, Part I & Part II
- My own Exit Poll FAQ on Pollster.com.
- Elizabeth Liddle's exit poll primer on DailyKos
- "Exit-Poll Secrecy Measures Aim to Plug Leaks to Blogs," by Amy Schatz in today's Wall Street Journal (free to non-subscribers).
- "Making the Call" on the CBS News blog "Public Eye"
Update: I'm live blogging election night in the next post.