August 13, 2006 - August 19, 2006


More on the CT Exit Poll Experiment

Topics: Exit Polls

Today I want to catch up and fill in a few details provided by our intrepid reader/reporter Melanie about that experimental CBS/New York Times exit poll conducted last week in Connecticut.   As regular readers may recall, Melanie first brought the exit poll to our attention after being interviewed last Tuesday.   I asked her what she remembered about the experience, starting with the interviewer.  Here is her report: 

The poll person was a young guy, maybe 20 - looked like a college student (tall, a little messy, scraggly hair, very diffident). I think he had some sort of ID around his neck or clipped to his pocket, but I didn't notice what it said  (he was in the midst of a discussion with the woman running the voting site about where he could be set up - she said she had just gotten a faxed letter from someone giving this young guy permission to be there, wherever he wanted to be as long as he didn't interrupt anyone, so she told him he could stay out of the sun - he was about 5 feet from the door of the site).

Melanie's experience gives us a unique window into the real world challenge of trying to select voters randomly as they exit a polling place.  She happened to overhear the interviewer's most important interaction of the day, the one that enabled him to stand just outside the door of the polling place.  Had he been forced to stand farther away, his ability to sample exiting voters randomly would have been severely compromised.  The post-election report provided by the two companies that conducted the 2004 exit polls (Edison Research and Mitofsky International**) showed that errors in John Kerry's favor were more than twice as large (-12.3) when interviewers were forced to stand 100 feet or more from the door of the polling place than when they could stand right outside the door (-5.3, p. 37).

At first I assumed this conversation occurred first thing in the morning, just as the polling place opened.  But I asked her to clarify and she said:

I got there about 9, which is early for me, but not for them (they had opened at 6 - the folks inside told me about 10% of the possible voters had already been in by then).

So put it all together and notice the reference to a faxed permission letter that the polling place official "had just gotten."  From the description, it appears that when the interviewer first arrived, the polling place officials would not let him stand near the exit door.  So he presumably called his supervisor and they found a way to fax a letter to the polling place official, and just as Melanie was voting, they relented and allowed the interviewer to stand near the door.  Up until that time, the interviewer had to try to intercept 10% of the day's sample from a less advantageous position. 

Second, notice the clear impression that the interviewer's appearance made on Melanie.  He "looked like a college student" with "messy scraggly hair" and a "diffident" attitude, yet she did not notice what appeared on his ID.  Now dear reader, ask yourself what you might guess about the politics or personality of that interviewer.  How would you react to an approach from such a person?  Is it possible that your choice -- whether you make eye contact, approach with interest or walk briskly in the opposite direction -- might have some relationship to your politics?  The data in the Edison/Mitofsky report from 2004 strongly suggests that it does

I will let Melanie continue with her description of how she happened to be interviewed: 

He was alone -- he actually didn't approach me -- I saw a couple of other folks doing his poll and when I walked over to see if he'd ask me to do it too he just handed me the device.  (My voting site is pretty small -- just one voting machine -- with about 700 registered democrats -- in a part of a smallish suburb east of New Haven -- we never get exit polls here). 

An exit poll interviewer is usually instructed to select exiting voters at random using a predetermined selection rate.  In other words, they are given a number (usually between 1 and 10) and told to select every third voter or every fifth.  Some are told to select every voter that exits the polling place.  We do not know the interval used here, although with 700 voters likely a number greater than one -- that is, not every voter should have been selected.  We will also never know if Melanie was one of the voters who should have been selected, but it is awfully interesting that in this instance she approached the interviewer rather than the other way around.

The data from the Edison/Mitofsky report on the 2004 exit polls showed that errors in Kerry's favor were nearly three times greater when where interviewers had to approach every tenth voter (-10.5), than in smaller precincts where interviewers were instructed to interview every voter or every other voter (-3.6, p. 36).   The clear implication is that where there was more potential for the random selection procedure to break down (either because of more chaos at the voting place or just more room for error), Kerry voters were more likely than Bush voters to volunteer to participate. 

So here we have another anecdote that helps convey the most important challenges in conducting an exit poll.  Some argued the collection of "almost perfect" random samples outside a polling place is easy.  It is not. 

Now let's hear about the most novel aspect of the CBS/New York Times Connecticut experiment, as per Melanie's report, that they used an "electronic device" to conduct the interviews.  Exit polls have traditionally been conducted with a paper "secret ballot" given to respondents on clipboard so they can fill it out and drop it into a "ballot box" without revealing the answers to the interviewer.  In Connecticut, CBS and the Times experimented with something new: 

The device was sort of like what the UPS guys use, except horizontal -- about 11" wide and 7" high. There was a LCD screen making up the middle of it, and the questions were in the center of the screen -- you just tapped the answers with a pencil to register your choices. He didn't get involved at all -- just handed it to me and took it back when I was done.  He seemed to have 2 or 3 of them.  He also had a little thing that looked sort of like a Palm pilot or blackberry that (I think) he inserted into the back of the poll device after it was filled out -- I assumed that was the way each set of info was transmitted to the collection folks, but I could be completely wrong about that.

The downside of the paper ballot is that it forces the interviewer to stop intercepting voters several times during the day to talley up responses and call in respondent level data, reading off each answer to someone on the other end of the line that enters everything into a computer.  The NEP exit polls (and those done by their forerunner, VNS) typically had interviewers call in only half of the respondent level data in order to cut down on phone call bottleneck.  This new technology automates both the tabulation and transmission so interviewers can cover the polls constantly all day, and 100% of their collected data gets transmitted immediately. 

How hard was it to use the electronic device?  Melanie continues:

The last question on the poll was about how easy or difficult the text on the screen was to read, so clearly they are experimenting with that part of it too (it was a little tough to read, especially with sunlight reflecting off the LCD screen -- or maybe that's just my aging boomer eyes).  But it was very user-friendly -- I think it was much quicker than a paper ballot would have been.

One last footnote:  In my speculation about the exit poll problems in 2004, I have given great weight -- probably too much, in retrospect -- to the presence of the network logos on the survey questionnaire, the exit poll "ballot box," and on the interviewer name tags.  Note that Melanie did not notice that network identification until she was holding the device in her hands.  I asked her specifically when she first noticed the logos. Her answer:

I noticed the logo after I had started to read the questions on the device -- it was about 2-3" square, at the top left of the device -- he didn't say anything about it and I just happened to notice it after I had read the first question or 2.  It was the usual CBS logo, but it wasn't a particularly bright color -- it sort of blended in to the dark gray or black color of the device, so it wasn't splashy. I assumed that was on purpose, as was his failure to tell me whose poll it was.

Of course, Melanie was just one respondent and amounts to a sample size of one.  Of course, as a colleague of mine once told me, the plural of anecdote is data.

**An MP source reports that Edison-Mitofsky, the company that does the pooled exit polls for the NEP consortium did not conduct the CBS/New York Times exit poll in Connecticut last week. 

Incumbent Rule Redux

Topics: 2006 , Incumbent Rule , The 2006 Race

Time to revisit "incumbent rule," thanks to Mickey Kaus who highlighted this observation last week by Michael Barone's column in U.S. News & World Report:

It may be time to revise one of the cardinal rules of poll interpretation--that an incumbent is not going to get a higher percentage in an election than he got in the polls. Lieberman was clocked at 41 and 45 percent in recent Quinnipiac polls; he got 48 percent in the primary election. The assumption has been that voters know an incumbent, and any voter who is not for him will vote against him. But the numbers suggest that Lieberman's campaigning over the last weekend may have boosted his numbers-or that the good feelings many Democratic voters have had for him over the years may have overcome their opposition to his stands on Iraq and foreign policy.

I wrote about the incumbent rule quite a bit in the run-up to the 2004 elections (especially here and here), applied it the polls in Ohio and then considered how the rule came up short (here and here).  Reconsidering the rule has been buried on my MP to-do list for some time, and while I lack the data to provide conclusive answers, today is as good as any to think out loud about some of the key issues involved.

The best known empirical assessment of this "cardinal" rule was written by Chicago pollster Nick Panagakis for the Polling Report in 1989.  He gathered 155 final polls spanning the period from the 1970s to 1988 (though most came from 1986 and 1988) and found that for 82% of the polls, the majority of the undecided broke to the challenger.  Note, that this statistic tells us how many polls showed undecideds breaking for challengers, not the proportion of the undecided voters that broke that way.

In September 2004, MyDD's Chris Bowers persuaded Panagakis to share his database and updated it with polls conducted from 1992 to early 2004.  Bowers took the process a step further, calculating the average split of the undecided vote over all the elections.  He noticed something obviously important in retrospect.  The incumbent rule seemed to be weakening (although he had little data from 1996):  80% of the undecided vote broke to challengers in the poll Panagakis collected between 1976 and 1988, but only 60% went to the challenger in the polls Bowers gathered between 1992 and the summer of 2004.   And challengers did worst of all in the polls in 2002 and the spring/summer of 2004 (42% to the incumbent, 58% to the challenger).

I have not attempted the same sort of comprehensive review of all of final polls from the fall of 2004, but on the final national presidential surveys an average of roughly 40% of the undecided vote broke toward challenger Kerry.  And the break of undecided voters in battleground states looks closer to 50/50.  "According to the exit polls," as Slate's David Kenner and Will Saletan pointed out, "Bush got 46 percent of those who made up their minds in the last week of the campaign and 44 percent of those who made up their minds in the final three days."

One question I have wondered about is whether the apparent weakening between the 1980s and 1990s could have been an artifact of the changes in the nature of pre-election polling or the particular races included in the database.  For example, did the 1990s see more polling in contests for Senate, Governor and local offices and less in presidential races?   Did long term changes in the timing or volume of pre-election polling affect the statistics? 

The more important question is why undecided voters have stopped breaking toward challengers in the final week of the campaign.  There are many theories. 

  • One possibility is that post 9/11 politics makes voters more reluctant to take a chance on challengers.   Are undecided voters more averse to change given the current emphasis on war and terrorism in our campaigns?   Some of the high profile Senate and Gubernatorial races saw a break favoring in incumbents in 2002 (though the incumbents were not exclusively Republican).  Consider also this bit of purely anecdotal evidence from MyDD's Matt Stoller:

I phone-banked a bunch of undecideds who in all likelihood flipped to Lieberman in the waning days of the campaign.  "I hate the war, I hate Bush, but I'm just not sure we can pull out right now" was the way they put it.

  • There is also the alternative theory Barone articulated in his column last week:

The left is noisy, assertive, in your face, quick to declare its passionate support. Voters on the right and in the center may be quieter but then stubbornly resist the instruction of the mainstream media and show up on Election Day and vote Republican, as they did in 2004, or for Lieberman, as some apparently did this week.

  • Or could this change reflect a change in the nature of campaigning?  Negative television ads were a rarity in the 1970s, but have grown increasingly commonplace in the years since.  Has the willingness of incumbents to "go negative" limited the ability of challengers to make the race a referendum on the incumbent and shifted the attention of late breaking voters to the alleged shortcomings of the challengers? 

Unfortunately, I have no answers tonight.  What is clear is that past trends are not much help in interpreting the pre-election polls of 2006.  How the undecideds will "break"in the final days of the 2006 campaign is anyone's guess.

UPDATE 8/15:   Readers have made a number of points worth reviewing in the comments section about possible shortcomings in the speculation above, as well as with the previous analysis of the incumbent rule.   One thing worth noting is that academic political scientists and survey researchers have devoted  little  if  any attention to the incumbent rule.  We certainly have a lot to learn about this "cardinal rule," despite its past popularity with campaign pollsters including yours truly.