November 5, 2006 - November 11, 2006
Today's Guest Pollster Corner Contribution comes from Simon Jackman of Stanford University, who takes a closer look at Tuesday's Senate election results.
Two interesting questions to ask after Tuesday's election are (1) Were the six defeated Republican senators particularly "out of step" with their respective states?; (2) What will be the effect of the Democratic pickups on the look of the new, 100th Senate?
To answer this question I first assigned a liberal-to-conservative voting score to each senator based on an analysis of the 530 non-unanimous roll call votes cast in the 109th Senate. The resulting scores are scaled to have mean zero and standard deviation 1, with lower (negative) scores reflecting a more liberal voting history, and positive scores reflecting a more conservative voting history (the details of this scoring procedure appear in my 2004 article with Josh Clinton and Doug Rivers in the American Political Science Review. A familiar story results, with Democrats on the left, Republicans on the right, with virtually zero no overlap between the parties. Lincoln Chafee is estimated to be the most liberal Republican with a voting score of about zero, while Ben Nelson (NE) is the most conservative Democrat (again, with a voting score close to zero). The usual suspects anchor the extremes of both parties: Barbara Boxer (CA) and Ted Kennedy (MA) for the Democrats (scores of -1.9), and Inhofe (OK) and Demint (SC) for the Republicans (scores of 1.3).
To gauge each state's political complexion, I use a simple and convenient proxy: Bush's share of the 2004 presidential vote in each state, which ranges from a high of 71.5% in Utah, to a low of 36.8% in Massachusetts, with a median of 52.7% (bracketed by Florida's 52.1% and Missouri's 53.3%).
The graph below shows a scatter-plot of voting score against 2004 Bush vote. Each point corresponds to a senator (red for Republicans, blue for Democrats, with senators running for re-election given a heavier shading), with 2004 Bush vote on the horizontal axis, and the roll call voting score on the vertical axis (higher is more conservative, lower is more liberal). The gray line is a regression fit to the data, not to be taken too seriously, but rather more as a rough guide as to how "out of step" the senator may or may not be. Republicans running for re-election are all numbered: Republican senators losing their seats are numbered 1 through 6, the other Republican senators who were re-elected are numbered 7 through 14. Chafee (RI), Santorum (PA) and Allen (VA) seem to be the only Republican losers who are obvious candidates for a "out of step" with their state kind of story (along the lines proposed by Canes-Wrone, Brady and Cogan in a 2002 article in the American Political Science Review, lying relatively distant from the regression line. DeWine (OH) seems to have been caught in what was an extremely difficult election for Republicans in Ohio, and neither Talent (MO) nor Burns (MT) appear to have been particularly "out of step". And keep in mind that there are several Republican senators just as apparently "out of step" as Santorum or Chafee who did not lose their seats: e.g., Jon Kyl (AZ), who faced no Democratic opposition in 2000, but won 53-44 in 2006; or John Ensign (NV), who by almost identical margins in both 2000 and 2006.
It is interesting to speculate on shape of the new, 110th Senate. Chafee goes, replaced by a Democrat, leaving the Maine senators (Olympia Snowe, re-elected with a 74-21 margin in 2006, and Susan Collins) as the most moderate Republicans. It remains to be seen just how liberal or moderate the new Democrats will be. Given their states and the narrow margins with which they are projected to win, it is tough to imagine Webb (D-VA) or Tester (D-MT) being particularly liberal, perhaps voting more like relatively conservative Democrats from the plain states (e.g., Nelson, NE; Conrad and Dorgan from ND) or the other Montana senator (Baucus).
Despite exhaustion and sleep deprivation, we want to take a few minutes today to a very quick and very preliminary look at how the preelection polls did as compared to yesterday's results. Since some precincts are still out and some absentee and provisional ballots are still being coutned, this quick looks is inherently preliminary and subject to change, but at the statewide level, the average of the last five polls in each races did reasonably well. In every case that we have examined so far, the leader in the average of the preelection polls was the leader on election day.
The following table includes only the most competitive Senate races that we tracked for the Slate Election Scorecard. It shows the curernt unofficial result in each state as compared to our final last-5-poll average. Since the preliminary results we gathered had been rounded to the nearest whole digit, we did the same with the final average. Again, every leader in the polls ran ahead yesterday.
[Note: For brevity's sake, the table above displays the results for Joe Lieberman in the Republican column, although Lieberman ran under the "Connecticut for Lieberman" party and has pledged to caucus with the Democrats in the Senate].
The list of the most competitive Gubernatorial races shows the same pattern. While the averagse did not predict the winners perfectly, the leader in the prelection polls was the leader on election day in every case.
[Update: The original version of the above table omitted the Minnesota Governor's race, which as several commenters noted, is the one state where the nominal leader in the averages was not the winner on election day. My apologies for the omission -- more details in a comment below.
Averaging results is obviously an imperfect solution to pre-election poll variation. The outcome in many races was off the "last-5-poll" average by as much or more than the Minnesota Governor's race: The Pennsylvania Senate race, the Maryland races for Senate and Governor, and the races for Governor in Alaska and Michigan all featured results differing from the final average that were as large as Minnesota].
We hope to have a far more comprehensive analysis in a few days looking at more races and using vote return data that is closer to complete. And these comparisons obviously make no effort to allocate undecided voters or use any of the more sophisticated measures of survey error. But for now, the bottom line is that the last-five-poll averages gave a pretty good impression of the likely outcomes of each of these competitive races.
1:40 am One more "alert reader" emails:
Actually, "alert reader MW" has it exactly wrong. If you look on the details page for Isle of Wight County, there are 22,861 registered voters in the county. With the count at 6,984 for Allen, 5,050 for Webb, and 163 others, the vote total of 12,197 gives a turnout of 53.35%. If the Webb number were actually 9,050, the turnout would be an absurdly high 70.84%. In addition, all the other races in the county are showing vote totals in the neighborhood of 12,000. So it seems unlikely that the Virginia site is incorrect. Still, with the more complete Virginia numbers (but without the benefit of the AP's extra 4,000 votes), Webb seems to be leading by about 1,500 votes with only a handful of heavily Democratic precincts yet to be counted. This one appears headed for the D column.
I for one, am not nearly alert enough to sort this one out. Probably a clue to say goodnight and get some sleep. Thanks to all.
1:15 am I have been reviewing the House races paying particular attention to the races I rated as toss-ups based on the surveys conducted in October. Of the pure toss-ups, those that have declared winners so far are splitting about evenly between Democrats and Republicans. As of now, I see eight toss-up races going to Democrats and seven to Republicans. Of course, many more are still being counted
One interesting result involves three districts that showed Democratic challengers leading by a significant margins only when we included the automated Majority Watch surveys in the averages: New York-25, New York-29 and Ohio-15. The Republicans incumbents in these three districts were all reelected.**
**Well, not quite. Apparently, with 1% of the vote uncounted and a two percentage point lead, Pryce has not been declared the winner in Ohio-15. Thought I saw a check mark next to that one. My error -- apologies.
1:03 am Alert reader MW writes:
In the last mid-term election Isle of Wight county had 44.5 percent voter turnout. The 9050 tally for Webb would put it closer to that number, around 48% turnout, based on a 2004 population of 32,774. The 5050 number would put turnout closer to 39%. Not that this means anything per se, but the talk has been of higher voter turnout hasn’t it?
11/8 12:28 am Back home and checking House races. Some very interesting comments by Pollster readers are ahead of something I just heard on CNN. The AP results are not matching the official numbers on the Commonwealth of Virginia State Board of Election web site. This sort of error is not unusual.
Specifically, this is from reader Jeremy Pressman:
What is with Isle of Wight county?
The state says Webb has 5050 votes while some news organizations say 9050 - obviously a huge difference in that race.
Compare: Virginia State Board of Election and CBS News
It has happened before. But whose typo?
10:52 pm I am going to need to relocate and will be offline for about 45 minutes.
10:45 pm In the comments, Gary Kilbride suggests a great site to look at where the outstanding vote is in Virginia. As of when I last updated the page, 143 precincts were uncounted and about 40 of those precincts come from four jurisdictions (Arlington, Fairfax, Norfolk and Richmond City) that Webb is carrying by margins of 59% or better. Another 30 are in Loudoun, which Webb is carrying 51% to 48%.
10:17 pm While my computer was slooowly rebooting ("virtual memory low" I hate that!) reader VZ emailed to remind me that the Montana tabulations are now online at CNN. An extrapolation on these numbers (which reflects the estimates applied as the polls closed) shows Tester leading by six (52 percent to 46 percent). Obviously, as the polls closed 15 minutes ago, this margin is not sufficient to call the race.
10:10 pm Promoting a comment from Mark Lindeman:
I've tried to estimate exit poll margins from a few of the tabulations for the 22 Senate polls I have so far (all of which I think I saved before they were updated). Those tabulations presumably are based on composite estimates incorporating pre-election returns.
When I compare the margins to the Pollster.com pre-election average margins, the exit polls appear to be running about 3.8 points more Dem than the pre-election polls -- which suggests that the actual gap could be wider. Several caveats on that: (1) I can already tell that my eyeballed margin in Missouri is about a point too large, so the gap could narrow. (2) Exit poll discrepancies have generally run high in the Northeast, which is overrepresented. The biggest discrepancies so far appear to be in non-competitive races, with the possible exception of CT. (3) Some part of this may be attributable to Democratic surge, and I don't have enough info yet to estimate that possible effect.
9:45 pm That last question really gets at something important. In some ways, it is a bad idea to think of the estimates we can extrapolate from the exit polls cross-tabulation as "exit polls." That may sound crazy, but the tabulations in states like Virginia, Missouri and Tennessee are now being weighted (or statistically adjusted) to reflect NEP's best estimate of the outcome at any given moment. Those estimates are gradually being updated to reflect more of the actual vote from the sampled precincts. That makes these estimates worth looking at -- the network decision desks certainly are.
On the other hand, the big risk in extrapolating from the exit poll crosstabulations is the considerable lag since they were run. Right now, the time stamps are 8:38 for Virginia, 8:49 for Tennessee and 8:10 for Missouri. So take these extrapolated estimates with a big grain of salt: McCaskill up by 3 in Missouri, but Webb and Ford down by 2 in Virginia and Tennessee respectively.
If we could look over the shoulders of the decision desk analysts right now, we would probably be seeing different numbers. Oh to be a fly on the wall in that room.
9:26 pm Very alert reader BM emails with a question: "It looks like the exit poll you quoted in the VA senate race has changed and would indicate that Allen has a majority. Am I missing something?."
Nope, you're not missing a thing. The tabulation has been updated and Allen now leads by slightly more among both men (55% to 44%), and Webb leads by slightly less among women (53% to 46%). Notice that the time stamp is 8:38 pm. What is happening is something I described in this morning's post: The NEP analysts are gradually replacing the exit poll talleys in each precinct with actual votes from that precinct. They are also beginning to fill out a second and larger sample of precincts from which they gather hard votes At any given time, they adjust the exit poll tabulations (displayed on CNN) to match the current estimate considred most reliable. And that process appears to have shifted the Virginia estimate -- for the moment -- in Allen's favor (roughly 50% to 49%)
8:56 pm Gary Kilbride has a very good catch in the comments. The Missouri exit poll is up on CNN. Those who decided in the last three days (who were 10% of all voters) went for McCaskill 57% to 38%. Earlier decideds split nearly evenly with 50% for McCaskill, 49% for Talent. The overall margin in the tabulation is far, far too close to tell us who will win, but given how close the pre-election poll looked, a late break if real would be decisive for McCaskill.
8:45 pm Not sure what to make of this: CBS News seems to be calling races much more readily than the other networks. The just called New Jersey, and an extrapolation on the currrent exit poll tabulation on CNN (with an 8:21 timestamp) shows Menendez with a roughly 10 point lead (54% to 44%). CNN just called it also.
8:16 pm LS asks a good question in the comments: "Why are your blog entries showing a TEN+ MINUTE lag time?" LS, we have "cached" our servers to handle the very heavy traffic today, and as I understand it, the cache only updates every ten minutes or so. So, unfortunately, these posts are updating on a ten minute delay. Also (as with the last update), I'm guessing wrong about how long it takes to write these updates.
8:10 pm Ok, here's another one. Extrapolate from the vote by gender tabulation now available on CNN and you get a 16 point lead for Democrat Bob Casey (58% to 42%). CBS has apparently called both Pennsylvania and Ohio for the Democrats, although the other networks I've been monitoring have not. This should tell us something important: The analysts are being very cautious about calling the result on exit polls alone. And these are states with candidates with double digit leads in the estimates applied to the CNN crosstabulations. For the states with closer margins, those exit polls aren't telling us much.
7:55 pm Polls close in five minutes in a bunch of states with closely watched Senate contests, including Connecticut, Maryland, Missouri, Pennsylvania and Tennessee.
7:43 pm Interesting. Want to see the difference between a margin big enough to "call" and election and one that isn't? Look at Ohio. Doing the math (all in my spreadsheet this time) the CNN tabulations show Democrat Strickland leading Blackwell in the Governor's race by roughly 26 points (62-36), but Democrat Sherrod Brown leading by 16 (58-42). The polls have been closed for ten minutes in Ohio and they haven't called it for Sherrod Brown yet. That should tell you what to think about a margin of less than 3 or 4 points.
7:20 pm Ok..here's the way you do the math, and these are not "leaked" results. From CNN's tabulation. Virginia: 49% men, 51% women; Allen-Webb 53-46% among men, 43-56% among women. Allen's approximate number from this tabulation is (.49*.53)+(.51*.43)= .479
.485. Webb's number (.49*.46)+(.51*.56)=51.1 50.5.
That's a 3.2
two point margin for Webb which is [still] (a) way too close to call on the exit poll estimate alone and (b) for what it's worth, narrower than the leaked number I saw about an hour ago. And not surprisingly, we continue to watch. [Sorry about the bad math]
7:05 pm Right now, if you got to the CNN exit poll page they are reporting the current "cross-tabulations" for each state where the polls are closed. They do not show the current vote estimate, but they do show the vote by gender, as well as the percentage male and female in each state, and it is not exactly rocket science to do the math.
6:57 pm Something else to remember: One of the things the network analysts are doing right now is comparing the exit poll results with averages of preelection polls -- averages not unlike those we have posted here on Pollster.com. If the exit poll result in a state looks out of line with the preelection result, they will not call the election even if the exit poll lead looks statistically significant. So if you see a "big" lead in a leaked exit poll, but the networks don't call that state when the polls close, you can assume that they are waiting to see hard data to confirm the exit poll result.
6:40 pm Something to remember about those leaked numbers you may or may not be seeing. First, if I say it once, I'll say it a thousand times: A "lead" of 2 or 3 points isn't much of a lead in an exit poll. We are seeing leaked numbers but we are not seeing the current "status" assigned to that state by the exit pollsters -- whether the lead is statistically significant enough to call the race. My guess, looking at the leaked numbers, is tha the networks will need hard vote data to call the Senate races we have long considered "toss-ups."
Second, we all need to remember that in 2004, the exit polls had an average error favoring the Democrats of about 5 or 6 points on the margin. In other words, if 2006 turns out like 2004, a 6 point lead may not be a lead.
6:30 pm - So what is this live blogging thing about? Two years ago, I vowed not to post leaked exit polls, and kept to that pledge, but in so doing opted out of the opportunity to comment on all the leaked data flying around the blogosphere. Also, we had just brought our three-day-old son home from the hospital, and so any excuse to avoid the computer was worthy. Tonight I want to do something different. The first wave of leaked estimates is now out, and I want to say a few things about it. I won't post the numbers here (and I'm not sure this will work) but I will try to offer some advice about how to read what you may be seeing.
Sound crazy? Maybe, but bear with me. I'll keep posting to the top of this entry.
6:22 pm - For those who may have missed it at the bottom of the last post, here are the best links I have for the NEP network sites both reporting vote results and (eventually) displaying exit poll cross-tabulations. When those will appear are anyone's guess.
Yes, the television networks will be conducting exit polls today. But if you are looking for the leaked exit poll estimates that typically appear online on Election Day, you are probably out of luck at least until later tonight. More on that below. But as long as you are here, let me tell you a little bit about how exit polls are conducted, how they will be different this year, and why it is probably best to try to ignore the exit poll estimates that will inevitably leak later tonight.
I have always been a fan of exit polls. Despite their shortcomings and the inevitable controversies, the final network exit polls remain our best source of data on who voted and why. Having said that, exit polls are still just random sample surveys, possessing the usual limitations plus some that are unique to exit polling.
A quick summary of how exit polls work: The exit pollster begins by drawing a random sample of precincts within a state, selected so that the odds of any precinct being selected are proportionate to the number that typically vote in that precinct. The National Election Pool (NEP) consortium, which is conducting the exit polling for the six major networks, will send exit pollsters to more than 1,000 precincts across the country today.
One interviewer will typically report to each sampled precinct. Each interviewer will stand outside and attempt to randomly select roughly 100 voters during the day as they exit from voting. The interviewer will accomplish this task by counting voters as they leave the polling place and selecting every voter at a specific interval (every 3rd or 5th voter, for example). The interval is chosen so that approximately 100 interviews will be spread evenly over the course of the day. As we learned following 2004, random selection of voters at the polling place is the most important part of the process, and arguably the most susceptible to problems.
When a voter refuses to participate, the interviewer is instructed to record their gender, race and approximate age. These data allow the exit pollsters to do statistical corrections for the bias in gender, race and age that might result from refusals.
The selected voters receive a one-page paper questionnaire to fill out. In the past, the questionnaire included approximately 25 questions (see an example from the 2004 exit polls), although this year the exit pollsters have worked to prepare a shorter questionnaire. Respondents fill out the survey privately then place it in a clearly marked "ballot box" so they know their identities cannot be tracked and their answers remain confidential.
The logistics of transmit all the results to a central location quickly and accurately provides the biggest challenge. To facilitate the process, interviewers will take a 10 minute break during the day to tabulate responses. Interviewers have typically stopped to call in their tabulations at three approximate times during the day: 9:00 a.m., 3:00 p.m. and shortly before the polls close.
Once the polls close, the interviewer will attempt to obtain actual turnout counts, and if possible, actual vote returns for their precinct from polling place officials. One of the unique aspects of the NEP exit poll design is the way it gradually incorporates real turnout and vote data as it becomes available once the polls close. The exit pollsters have developed weighting schemes and algorithms to allow all sorts of comparisons to historical data that supports the networks as they decide whether to "call" a race for a particular candidate. When all of the votes have been counted, the exit poll is weighted by the vote to match the actual result.
Throughout the day and after the polls close, the exit polls are a part of a larger process that provides network analysts with various "estimates" of the vote in each state. They have esoteric names like "Best Survey Estimate" and "Composite Estimate" (a post-election report on the 2004 exit polls had some rare definitions and examples, see p. 7). Before the polls close, those estimates derive mostly from exit poll data, but as actual returns become available, the computer models gradually replace the exit poll tallies in each precinct with hard vote counts.
After the polls close, the network analysts also begin to look at estimates based on larger samples of approximately 2,000 randomly selected precincts where NEP personnel obtain official vote tallies as they become available. The networks typically use this larger system of tabulations and estimates -- rather than the exit polls alone -- to call close contests after the polls close. But when the vote is really close, as it is likely to be tonight in states like Missouri and Virginia, even the larger samples of hard returns will be inadequate. They will have to watch the complete vote count, just like the rest of us.
In years past, hundreds of producers, editors and reporters would have access to the mid-day estimates starting at about noon, and preliminary exit poll cross-tabulations (like this one) would inevitably leak out.. This year the networks have taken steps to prevent such leaks, creating a "Quarantine Room" where, as the Washington Post's Howard Kurtz reports, "two people from each of the networks and the Associated Press" will be allowed "entree to a windowless room in New York [where] all cellphones, laptops and BlackBerrys will be confiscated. The designated staffers will pore over the exit polls but will not be allowed to communicate with their offices until 5 p.m."
So there you have it folks. At 5:00 p.m. the data will be flowing to newsrooms as it always does and the usual leaks will presumably commence shortly thereafter. One rationale for the delay is that the data available at that hour will be more representative and not just based on those who vote in the morning.
So if this process is so sophisticated and the late afternoon numbers are better, why can't we rely on the leaked "numbers" that will undoubtedly leak after 5:00 p.m. today?
1) It is still just a survey -- Even when complete, an exit poll still has the same random variation as any other survey. NEP says typical state exit polls have a sampling error when complete of 4% at a 95% confidence level, and 3% for the national exit poll.
But that is the margin of error recommended for reading the cross-tabulations. The exit networks pollsters require far more statistical confidence before projecting winners. As Joe Lenski, whose company directs the exit poll operation, told me in an interview last week, the "rough rule of thumb" is to require "three standard errors, which would be 99.5% confidence" before projecting a winner. While the precise numbers will depend on the circumstances, that implies a much larger "margin of error."
2) Early or absentee voting -- The ABC News/Washington Post survey released on Sunday reported that roughly 12% of all registered voters nationally had already cast absentee or early ballots. Obviously, these voters will not be available to interviewers standing outside polling places. To incorporate early voting, the National Election Pool is doing telephone interviewing in many states to sample the votes of those who voted early (although they will not do telephone interviews in Maryland, where doubts about electronic voting have caused absentee requests to skyrocket). Will these results distributed to NEP subscribers after 5:00 p.m. include the telephone results for early voters? Who knows? I wouldn't count on it.
3) Past Errors favored Democrats -- In past elections, the results have slightly favored Democratic candidates. As we learned after the 2004 elections, the exit poll estimates have had at least some small error favoring the Democratic candidate for president in every election since at least 1988. In 1992 and 2004, the two elections with the highest levels of voter interest and turnout, those errors were more acute.
Since 2004, the exit pollsters have taken steps to try to eliminate those problems. As Joe Lenski explained, they have shortened the questionnaire, improved interviewer training and made an effort to hire older interviewers (because of problems that younger interviewers had in past years gaining cooperation from older voters).
They have also gone to court to allow more interviewers to near polling place exits. Hopefully, these efforts will reduce or eliminate the problems experienced in 2004, but we will have no way of gauging their success at 6:00 p.m. tonight.
4) Limited Data on the House elections -- While the exit polls will produce vote estimates for the major contests for Senate and Governor, they will not make projections on individual House seats. The networks will do that based on actual votes. The national exit poll sample will include an estimate of the national vote for Congress by party, but as Joe Lenski points out, "because of Gerrymandering, we know that popular vote for the House does not translate into House seats directly." (See also Professor Franklin's post on the imperfect relationship between the national vote for Congress and seat gain or loss).
Of course, we are all intensely curious about the outcome of today's elections, and should the exit poll numbers leak again, most of us will find a way to peek at them. I just want you to know that those leaked exit polls really don't tell us much more about the outcome of the race than the telephone polls we have been happily obsessing over at Pollster.com all fall. Even if we wanted to call a race on unfinished, late-day exit polls alone (something the networks will not do), we would need to see differences of at least 8-10 points separating the candidates to be statistically certain about the outcome.
So look at them if you must, but please, don't go leaping to conclusions based on a leaked exit poll margins showing one candidate "leading" by a few percentage points. Those numbers are not magic. You would be better off flipping a coin -- or better yet, checking the final public poll averages here on Pollster -- to determine the outcomes of the Senate contests in Missouri, Virginia and Montana.
PS: All of the above applies to true "exit polls," those that intercept voters as voters exit the polls. In the vacuum created by the "Quarantine Room," I would not be surprised to see someone release results from a "day of" telephone poll billed as an exit poll. Don't let the name fool you. If the network exit polls are of little value in close races, imagine the potential problems facing a telephone survey, conducted during the day when most voters are still at work. If one of these pops up, I'll be back to blog about it.
I will be online all day. If you have questions about exit polls or anything else, please send them my way.
The following are links to the NEP network web pages that will post exit poll results at some point. Most of these are either not yet displaying results page or all also include actual vote returns:
You might also want to check some of the following web resources for more information:
Update: I'm live blogging election night in the next post.
My estimate of net national forces is moving again, but is it signal or noise? We'll know after tonight. With the last polls, through 11/6, the Senate estimate has turned back up. I HASTEN to add that this estimator is sensitive to the last points of data and can take some time to make up its mind about the trend. We have no more time, so here is what we've got. Perhaps the Republican surge ended over the weekend, and perhaps the upturn here is just a fluke of the last polls that came in. I HAVE made this a pretty conservative estimator, not TOO sensitive to short term noise, but with no more data after today it will be hard to know about the last couple of days of th campaign. Perhaps I'll be able to do more on that after the election, when only academics like me care about such things.
The House does not show the same upturn, but does show a change in the slope of the decline, perhaps reflecting a similar upturn in pro-Dem forces, but with fewer House polls we can't detect it.
The Governors races (bottom figure) don't seem to show any change in national forces.
Make of this what you will. And have fun with election day and election night.
The last Gallup pre-election poll puts approval of President Bush at 38% (and disapproval at 56%.) As the figure above shows, this is the second lowest approval at midterm of any president since World War II. In previous years, such low approval would be expected to lead to major losses in congress. Whether due to redistricting or something else, few forecasts anticipate losses of the magnitude such an approval rating alone would suggest. Current estimates range from the 20s to the high 30s, with a few above and (even fewer) below that range.
Back in May I wrote about approval and midterms and estimated a forecast of approval in the last pre-election poll based on approval in May. The plot below is from that post, with the actual approval rating added. Not bad for predicting the future more than 5 months out.
The last three national pre-election polls put the approval trend estimate at 37.8%. The polls by Fox, CNN and Gallup find approval at 38%, 35% and 35% respectively. All three polls completed interviewing on 11/5.
There isn't anything new to say. This approval level is terrible in historical context for a midterm election. What effect? We'll see tomorrow.
Our next to last Slate Election Scorecard reaps where things stand in the closest Senate races: Maryland, Missouri, Montana, Tennessee and Virginia. Meanwhile I wanted to give one last update on the overall "mashed-up" margin across all the Senate races on the Slate Scorecard. Consistent with the trend on Professor Franklin's Senate "national forces" charts (which have their basis in the same underlying data), the average Democratic margin across the turned downward over the last week -- for the first time in seven weeks.
One interesting twist to these findings is that the Republican Bob Corker's gains in Tennessee explain virtually all of the Democratic decline. The last 5-poll average in Tennessee went from a dead-even tie to a 7.4 point Corker lead in just a week. If we remove that race from the overall average, there is still a leveling off of the six week Democratic trend but virtually no decline. Republicans saw gains on the averages in some states over the last week, but so did Democrats and, except for Tennessee, the changes cancel out.
Democracy Corps, the project of Democratic pollster Stan Greeberg and Democratic consultants James Carville and Bob Shrum, just released their final tracking survey (memo, results) conducted among voters in 50 competitive districts currently held by Republicans. To be clear, they do not conduct 50 surveys in 50 districts, but one sample of 1,201
600 or so likely voters spread out across the 50 districts. While this approach does not allow for district-by-district projections, it is the only public survey available that has tracked attitudes on a weekly basis in the most competitive congressional districts (our massive collection of public polls, by comparison, provides what amounts to as a "time lapse" snapshot of these districts taken over the course of October).
There is evidence of a slight shift of the playing field to the Republicans at the end of last week, fully reflected in the first half sample of the survey on Thursday night. That shift includes perhaps a 2-point gain in party identification advantage amongst these likely voters (with and without leans) in the Republican districts; a 3-point rise in "right track" (though only 34 percent), a 3-point gain in Congress job approval (though only 34 percent) and a 4-point rise in "warm" reactions to Republican Congress (though only 38 percent).
Republican congressional voters' high interest in the election is up 4 points, but still lags 11 points behind that of Democratic voters. Together, that has likely cut the Democrats' margin by 2 points -- and that is not trivial in districts where Republicans are near 50 percent. But more striking is how stable is this race and how endangered the incumbents are. While the voting electorate has become marginally more Republican, it has not moved key indicators,
Of course, the polling company that conducted the survey -- Greenberg, Quinlan Rosner Research -- is a Democratic firm. So take these results with whatever grain of salt you deem appropriate.
The memo also makes some interesting observations about their year-long experiment with a "named" congressional ballot question. It is well worth reading in full.
The usual way to look at poll accuracy is to subtract the poll result from the vote result. But an alternative is to look at how the probability that a candidate wins depends on the margin they have in the pre-election polls. Since American elections are "winner-take-all" within districts, this is a good way of looking at the practical power of polls to predict winners.
After all-- a statistician would say a poll was better that predicted 51% for the loser who actually got 49% than a poll that predicted 51% for the winner who got 55%. That's right from one point of view, but not from the perspective of predicting winners right. Here I take a look at the latter view of what is important.
The data are from all statewide polls for Senate, Governor or President from 2000 and 2002.
The figure above plots results by poll margin. The x-axis shows the Dem minus Rep margin in the polls. The y-axis plots the percent of races the Dem ACTUALLY won for each margin we saw in the polls. So imagine I take all polls that found a 5-point lead for the Dem. The y-axis plots the proportion of those polls with a 5-point lead in which the Dem actually DID win. I do this separately for each race, Gov, Sen and Pres. The dots show there is a lot of variation, but the pattern of points, and the black trend line through the data show how the predictive accuracy varies over margins from -30 to +30.
One interesting feature is that a margin of zero (a tied poll) produces a 50-50 split in wins with remarkable accuracy. There is nothing I did statistically to force the black trend line to go through the "crosshairs" at the (0, .5) point in the graph, but it comes awfully close. So a tied poll really does predict a coin-flip outcome.
The probability of a win rises or falls rapidly as the polls move away from a margin of zero. By the time we see a 10 point lead in the poll for the Dem, about 90% of the Dems win. When we see a 10 point margin for the Rep, about 90% of Reps win. That symmetry is also not something I forced with the statistics-- it represents the simple and symmetric pattern in the data.
More practically, it means that polls rarely miss the winner with a 10 point lead, but they DO miss it 10% of the time.
A 5 point lead, on the other hand, turns out to be right only about 60-65% of the time. So bet on a candidate with a 5 point lead, but don't give odds. And for 1 or 2 point leads (as in some of our closer races tomorrow) the polls are only barely better than 50% right in picking the winner. That should be a sobering thought to those enthused by a narrow lead in the polls. Quite a few of those "leaders" will lose. Of course, an equal proportion of those trailing in the polls will win.
So read the polls-- they are a lot better than nothing. But don't take that 2 point lead to the bank. That is a failure to appreciate the practical consequences of the margin for error.
Three of the last six national polls have found sharp downturns in the Democratic lead on the congressional generic ballot. After rising steadily since the week before the Foley scandal, the Democratic advantage has now begun to turn down. USAToday/Gallup, ABC/Washington Post and Pew Research Center all find substantial drops. Newsweek, Time and last week's CBS/New York Times polls do not find that decline, but rather show stability at around a 15-point Democratic lead.
While these shifts this MAY signal a sharp change of opinion going into the weekend, the magnitude of the drop is quite uncertain with only three polls. We routinely see lots of variation across polls, especially when looking at the generic ballot margin. Nonetheless, the shifts have been enough to convince my "local regression" estimator (the blue line in the figure) to turn down for the first time in a while. Since the blue trend line considers ALL the polls, it is not overly sensitive to single polls, though the combined weight of Gallup, ABC/WP and Pew is enough to move it down about 4 points, from +15 to +11 for the Dems. It is likely that the individual polls are overstating the extent of the downturn. The trend estimator captures the "signal" among all the "noise" from poll to poll. It would take more polls to "know" how much this downturn really represents. But the "poll" taken on Tuesday will answer the question for sure.
However, the current estimate of the Democratic lead based on the trend of all recent surveys remains at roughly +11. While down from the peak of early October, check my post and comparison graphic from earlier this week. The final Democratic advantage has not been over 10 points (or even close) in the last 12 years.
President Bush's approval rating has leveled out after a month of sharp decline. Four new polls, by ABC/Washington Post, Pew, Newsweek and Time find approval at 40%, 41%, 35% and 37% respectively. This puts the trend estimate at 38.1, a small upturn from the last estimate at 37.4%.
While stopping the decline, approval well below 40% is historically quite low and is almost certainly taking a toll on Republican candidates.
The trends of the four polls are shown individually below.
Across the board, in Senate, House and Governor's races, the wave boosting the Democrats crested about 10 days ago. Since then the advantage Democrats have built throughout the year has been reduced by from 1.5 to 3.5 percentage points. While forces are still a net positive to the Democrats, these forces are weaker than they were during the week before Halloween. This implies that the most competitive races will now be harder for Democrats to win and easier for Republicans to hold. This implies that the anticipation of a major surge to Democrats now needs to be reconsidered. While race-by-race estimates still show an 18 seat Democratic gain, and 27 seats as tossups (see our scorecard at Pollster.com here), this reduction in national forces makes it less likely the Democrats sweep the large majority of the tossup seats and could result in total gains in the 20s rather than the 30s or even 40s that looked plausible 10 days ago.
This cresting of national forces has taken place across Senate, House AND Governors races and occurred essentially simultaneously around October 25th. The estimators here, plotted as the blue line in the figures, is a measure of national effects that are common across all races. The estimate uses all polls in all races, but estimates the Senate, House and Governors races independently, yet produce similar results for each in terms of timing, though with some variation in magnitude. (For more on the estimation method, see this earlier post.)
As of last Thursday's data, the downturn was clear for the Senate but no indication of change had appeared for the House. Adding the polling data from Friday, Saturday and Sunday, the downturn is now apparent there as well. I did not do the Governors last week.
The congressional generic ballot does NOT show any such change (yet!). In my earlier post I cautioned that the generic ballot might not be reflecting a realistic assessment of the Democratic advantage. It may also not be reflecting the last minute dynamics of the campaign this year.
So what does this mean? The House still looks likely to go Democratic, but probably by a smaller margin that it might have a week ago. For a while, the Senate looked to come down to who won two of VA, TN and MO. Now MT must be added to that, and TN moved to lean Rep, perhaps requiring a Dem sweep of VA, MO and MT. (Momentum in VA remains pro-Dem, while MO is completely flat and MT is strongly trending Rep.) Possible but more of a trick that 2 of the former 3 states. The shrinking margin in MD may well end with a Dem win, but clearly some races that were viewed as likely Dem pickups or holds are now somewhat more in doubt that before, possibly including RI.
With two days to go there is a time limit on this dynamic. Reps may not have time to profit greatly from this trend, and we've seen sharp changes before so Dems may be able to recover (Republicans had a bad end of the week last week, after John Kerry and the Dems had a bad first of the week.) So no firm prediction here, but the evidence is that the Dems are falling back from their best chance of large gains.
As should be evident by our "most recent polls" box on the right, the Mason-Dixon organization released their (presumably) final round of statewide surveys today. We just updated our charts and the new polls helped push both the Maryland Senate and Maryland Governor's races into the "toss-up" category.
Joe Lenski is the co-founder and executive vice-president of Edison Media Research. Under his supervision, and in partnership with Mitofsky International, the company of his later partner Warren Mitofsky, Edison Media Research currently conducts all exit polls and election projections for the six major news organizations -- ABC, CBS, CNN, Fox, NBC and the Associated Press. In Part I of his interview with Mark Blumenthal, he spoke generally about how the networks conduct exit polls and how they use them in their system to project winners on Election Night. The interview concludes with a discussion of the problems the exit poll experienced in 2004 and what will be done differently this year.
I want to ask more generally about how things will be different this year. First, let's talk about the issue of when and how you will release data to members of the National Election Pool (NEP) consortium and other subscribers. In the past, and please correct me if I'm wrong, hundreds of producers, editors and reporters had access to either the mid-day estimates or early versions of the crosstabulations that you would do, and the top-line estimate numbers would inevitably leak. How is that process going to be different this year?
The news organizations are really taking this challenge seriously on how to control the information for a couple of reasons. First, each of these news organizations have made a commitment to Congress over the years that they would not release data that would characterize the winner or leader in a race before the polls have closed. So in essence, by this data leaking, it was undermining that promise that they had made to Congress.
The other thing is that we know these are partial survey results. No polling organization leaks their partial survey results. If it's a four-day survey they don't leak results after two days. Similarly if it's a twelve-hour exit poll survey in the field you're not going to release results after just three hours of interviews. So the data will not be distributed to the news organizations until 5:00 p.m. in 2006, and that's a change from all the previous elections. The goal is that this will be more complete data and also we will have more time to review the data and deal with any issues in the data that look questionable that we need to investigate. It will still give news organizations time for their people to look at the data before the polls start closing.
In 2004 at least one network started posting the demographic cross-tabulations online for specific states. I believe these started appearing almost as soon as the polls closed, maybe shortly thereafter. Do you have any idea if they are planning to repeat that of if they will hold off on posting tabulations until most of the votes have been counted?
Again, that's an editorial decision by the news organizations, but they are well within their rights, as soon as the polls close within a state, to publish those results.
Let's switch gears a bit. There are a bunch of stories out now about the increase in demand for early voting or absentee ballots. In Maryland, where there are concerns about the voting equipment stemming from problems that had during the primary there, there are apparently so many requests that they are running out of ballots. What are you doing to cope with the fact that fewer and fewer voters are voting at polling places?
In the states with significant amounts of absentee or early voting, we are doing telephone surveys the week before the election of voters who have either already voted absentee or have their absentee ballot and plan to send it in or vote in person before Election Day.
Have you had time to adjust for the demand in Maryland? Is Maryland going to be one of those states?
No, Maryland will not. Most of these decisions were based on the share of the vote that was absentee in 2004 and Maryland in 2004 had a relatively small number. Yes, there will be big increases in several states. Maryland is one. Ohio may be another. We saw in 2004 really large increases in states like Florida and Iowa in terms of the people who voted early and absentee. So as absentee and early voting increases the need for us to do more of these telephone survey supplements to the exit poll is going to continue and we will need to budget for more of them.
As long time readers of my blog will certainly know you and Warren Mitofsky co-authored an analysis in January 2005 of the problems experience with the 2004 exit polls and the overall system. In particular there appeared to be a problem with interviewers either deviating from the random selection procedure or when they faced greater challenges in completing interviews. What will you be doing differently this year to try to fix those problems?
Well a lot of it has already been done. We sat down with all of the NEP members after that report came out and did a thorough review of all the recruiting and training procedures for the exit poll interviewers and we got a lot of input from all of the professionals that work at the news organizations that do their own surveys. They looked at the materials we were using. We had discussions and we came up with an improved training manual. We also prepared and filmed a training video that all interviewers are required to watch. We developed a new more rigorous training script and a quiz or evaluation at the end of that script to make sure that the interviewers understand the important facets of their job.
In addition to all that we have the input rehearsals that we have done every year, where we have two days in which we act like its Election Day. People call in with test results using the same phone numbers, the same questionnaires they will use on Election Day just to make sure they understand how the process works. So all of that has already gone into effect and I think our interviewers are much better trained this year than they were in 2004.
Another factor that came into play is that we found the error rates tended to be higher on average in precincts where there were younger interviewers, especially interviewers under the age of 25. This isn't to malign the abilities of interviewers under the age of 25, there just seems to be an interaction between older voters and younger interviewers that make older voters less likely to fill out surveys that are presented to them by younger interviewers, so we've also made a concerted effort to increase the average age of our interviewers.
I wrote just the other day about a question buried at the end of a recent Fox News poll that showed Democrats were significantly more likely to agree to participate in exit polls than Republicans. I'm wondering if you think that differential willingness to participate is worse now than in 2004? And how can you do an unbiased random sample at the precinct level under those conditions?
Well, typically in past non-presidential off-year elections there hasn't tended to be large exit poll biases. I have a feeling though, 2006 has, like 2000 and 2004, more passion than the typical off-year presidential election. So that does worry me to some extent. Again, the more we can train our interviewers to follow the proper sampling procedures the more we can eliminate a good bit of the bias that comes in from people in essence volunteering to take an exit poll when they weren't randomly selected to participate in an exit poll.
It still doesn't correct for the differential non-response that might exist even within the sampled voters. So you could properly select every fifth voter say as they are leaving a polling place but if 55 percent of the Republicans fill out a questionnaire but only 50 percent of Democrats fill it out you are still going to have a bias from non-response even if your sampling is absolutely perfect. What we are going to do, and what all the decision teams looking at this data are going to do, is know that the possibility for that type of bias exists and we'll be careful especially in states where we have seen that type of survey bias before projecting any winners.
The NEP exit polls are designed for a variety of different purposes, to provide an analytical tool to help people understand why people voted the way they did, to help assist with the projections, but they are not designed, at least as I understand it, to help detect fraud. So my question is, if you were in the business of designing an exit poll to do that in the United States, how would you design it differently than you do for NEP?
In answering this question I want to be careful not to malign the design of the exit polls as they now exit because they really serve the purpose of the news organizations and what they need. What they need is a lot of information about who voted, how they voted, why they voted, and to be able to present that information as quickly as possible on Election Night.
So there are some things that we do in designing these exit polls that we wouldn't do in designing an exit poll whose sole purpose would be to validate the results precinct by precinct, county by county, state by state. One is we interview on average about 100 voters per polling location. That's basically because there are several costs that come in. There are printing and shipping costs to get the questionnaires to polling places. Two, there is the interviewer cost -- you would probably need to hire more than one interviewer per location if you were doing more interviews. And three, we need to get all of this data into our system by the time the polls close in that state in time so that it can be reported on Election Night. So there would be the time cost -- it would be cost prohibitive to get all that data into the system on Election Day.
So if you were going to design a system solely for the purpose of validating election results, one, you would try to interview everyone, or at least approach everyone at the polling locations where you are trying to do that validation. Two, you would have a shorter questionnaire. You would not have the twelve to twenty five questions that are being asked in our questionnaires for analytical purposes. You wouldn't be asking a lot of the questions about the important issues in the race or be asking questions about religion, whether people are married, income, education, etc. The studies I've seen -- and there have not been enough of them and some of them are fairly old -- tend to conclude that you get the highest response rate and the lowest error from exit poll questionnaires that are about six to seven questions long. That would give you very little data to analyze the election. And that's why the NEP questionnaires are longer than that. But if you were trying to increase response rates and decrease bias and within-precinct-error, you would have shorter questionnaires as well.
The other thing you would do is to interview for all the hours in which the polls are open. Because of the time constraints, we tend to stop interviewing at most polling places about 30 to 60 minutes before the polls have closed, so we can get all the data for that day into our system. If you were going to do a survey that would validate the entire day's voting you would do interviewing from poll opening all the way to poll closing, including that last hour. So those are some of the things you would change in the design if you were designing exit polls solely to validate the voting results at the precinct level.