Pollster.com: Tennessee Polls: Not Created Equal

October 04, 2006

Instapundit Glenn Reynolds asked a good question yesterday:

THE LATEST POLL shows a Ford-Corker dead heat.

Hmm. Just yesterday we had one with Ford up by 5; not long before that there was one with Corker up by 5. Is it just me, or is this more variation than we usually see? Are voter sentiments that volatile (or superficial)? Or is there something about this race that makes minor differences in polling methodology more important? Or is this normal?

At the moment at least, I agree with the answer he received later from Michael Barone that the poll numbers in Tennessee do not appear unusually volatile. Barone pointed out that the results of nearly all the Tennessee polls this year appear to fall within sampling error of the grand average. That point is worth expanding on, but it is also worth noting that the averages conceal some important differences among the various Tennessee surveys.

First, let's talk about random sampling error. If we assume that all of the polls in Tennessee used the same mode of interview (they did not), that they were based on random samples of potential voters (the Internet polls were not), that they had very high rates response and coverage (none did), that they defined likely voters in exactly the same way (hardly), that they all asked the vote question in an identical way (close, but not quite) and that the preferences of voters have not changed over the course of the campaign (no again), then the results for the various polls should vary randomly like a bell curve.

Do the appropriate math, and if we assume that all had a sample size of roughly 500-650 voters (most did) than we would expect these hypothetically random samples to produce a results that falls within +/- 4% of the "true" result 95% of the time. Five percent (or one in twenty) should fall outside that range by chance alone. That is the standard "margin of error" that most polls report (which captures only the random variation due to random sampling. But remembering the bell curve, most of the polls should cluster near the center of the average. For example, 67% of those samples should fall within +/- 2% of the "true" value.

Now, let's look at all of the polls reported in Tennessee in the last month, including the non-random sample Zogby Internet polls:

TN%20polls%20since%209-1.jpg

As it happens, the average of these seven polls works out to a dead-even 44% tie, which helps simplify the math. In this example, only 1 the 14 (7%) results falls outside the range of 40% to 48%~~44%~~ (that is 44%, +/- 4%.). And only 4 3 of 14 (28%~~21%~~) fall outside the range of 42% to 46% (or 44%, +/- 2%). So as Michael Barone noted, the variation is mostly what we would expect by random sampling error alone. Considering all the departures from random sampling implied above, that level of consistency is quite surprising.

These results may seem more varied than in previous years partly because the samples sizes are considerably smaller than the national samples of (typically) 800 to 1000 likely voters that we obsessed over during the 2004 presidential race.

The confluence of the averages over the last month (or even over the course of the entire campaign, as Barone noted) glosses over both important differences among the pollsters and some real trends that the Tennessee polls have revealed. Charles Franklin helped me prepare the following chart, which shows how the various polls tracked the Ford margin (that is, Ford's percentage minus Corker's percentage). The chart draws a line to connect the dots for each pollster that has conducted more than one survey. The light blue dots are for pollsters that have done just one Tennessee survey to date.

The chart shows a fairly consistent pattern in the trends reported by the various telephone polls, both those done using traditional methods (particularly Mason-Dixon) and the automated pollster (Rasmussen). Franklin plotted a "local trend" line (in grey) that estimates the combined trend picked up by the telephone polls (both traditional and automated). The line "fits" the points well: It indicates that Ford fell slightly behind over the summer, but surged from August to September (as he began airing television advertising).

As Barone noticed, the five automated surveys conducted since July (including one by SurveyUSA) have been slightly and consistently more favorable to Ford than the three conventional surveys (to by Mason-Dixon and one by Middle Tennessee State University). But the differences are not large.

The one partisan pollster - the Democratic firm Benenson Strategy Group - released two surveys that showed the same trend but were a few points more favorable to Democrat Ford than the public polls. This partisan house effect among pollsters of both parties for surveys released into the public domain is not uncommon.

But now consider the green line, the one representing the non-random sample surveys of Zogby Interactive. It tells a completely different story: The first three surveys were far more favorable to Democrat Ford during the summer than the other polls, and Zogby has shown Ford falling behind over the last two months while the other pollsters have shown Ford's margins rising sharply.

This picture has two big lessons. The first is that for all their "random error" and other deviations from random sampling, telephone polls continue to provide a decent and reasonably consistent measure of trends over the course of the campaign. The second is that in Tennessee, as in other states we have examined so far, the Zogby Internet surveys are just not like the others.

UPDATE: Mickey Kaus picks up on Barone's observation that the automated polls have been a bit more favorable to the Democrats in Tennessee and speculates about a potentially hidden Democratic vote:

Maybe a new and different kind of PC error is at work--call it Red State Solidarity Error. Voters in Tennessee don't want to admit in front of their conservative, patriotic fellow citizens that they've lost confidence in Bush and the GOPs in the middle of a war on terror and that they're going to vote for the black Democrat. They're embarrassed to tell it to a human pollster. But talking to a robot--or voting by secret ballot--is a different story. A machine isn't going to call them "weak."

Reynolds updates his original post with a link to Kaus and asks whether the same pattern exists elsewhere.

Another good question, although for now our answer is incomplete. We did a similar "pollster compare" graphic on the Virginia Senate race over the weekend. The pattern of automated surveys showing a slightly more favorable result for the Democrats was similar from July to early September, but the pattern has disappeared over the last few weeks as the surveys have converged. In Virginia, the most recent Mason-Dixon survey has been the most favorable to Democrat Jim Webb.

While we will definitely take a closer look at this question in other states in the coming days and weeks, it is worth remembering that most of the "conventional surveys" in Tennessee and Virginia were done by one firm (Mason-Dixon), while most of the automated surveys to date in Tennessee have been done by Rasmussen. As such, the differences may result from differences in methodology other than the mode of interviewer among these firms (such as how they sample and select likely voters or whether they weight by party as Rasmussen does).

[Missing "+/-" signs restored]

-- Mark Blumenthal

October 04, 2006 in Divergent Polls, IVR Polls, Internet Polls, Sampling Error, The 2006 Race

TrackBack

TrackBack URL for this entry:
http://www.pollster.com/cgi-bin/mt/mt-tb.fcgi/3071.

Listed below are links to weblogs that reference Tennessee Polls: Not Created Equal:

» Tennessee Senate Race from Sensible Mom
Instapundit posed a question about the last few polls of the Tennessee Senate race asking, Is it just me, or is this more variation than we usually see? Are voter sentiments that volatile (or superficial)? Or is there something about [Read More]

Tracked on> October 4, 2006 9:38 AM

Comments

Anonymous:

The other issue is that because of the norm of racial equality, voters may tell real people surveying them that they will vote for a black candidate, but when they actually vote, they do not. Example: Dinkins v. Giuliani mayoral race in NYC. This may hurt Ford esp. in a Southern state.

Posted on October 4, 2006 8:33 AM

____________________

Nick Panagakis:

"As it happens, the average of these seven polls works out to a dead-even 44% tie, which helps simplify the math. In this example, only 1 the 14 (7%) results falls outside the range of 40% to 44% (that is 44%, 4%.). And only 3 of 14 (21%) fall outside the range of 42% to 46% (or 44%, 2%). So as Michael Barone noted, the variation is mostly what we would expect by random sampling error alone."

If I am reading this correctly, 4 of 14 polls fall outside the margin error. But at the 95% level of confidence, the odds are that only 1 in 20 polls should be expected to fall outside the margin error. Therefore, the variation is *not* what we would expect by random sampling error alone.

Comment?

Nick Panagakis

Posted on October 4, 2006 8:44 AM

____________________

Mark Blumenthal:

Sorry Nick (and all):

A sloppy error (or two) and some missing "+/-" signs in that paragraph probably confused. It was 4 of 14, but the +/- 2% margin inolved a 67% level of confidence.

Posted on October 4, 2006 8:54 AM

____________________

Mark Lindeman:

Ah, not to repeat Mark B.: so, 4 of 14 polls are outside 1 standard error, which is close to the expected proportion (32% or so), and 1 of 14 is outside 2 standard errors, which is close to the expected proportion (5% or so).

I would be startled if the underlying reality is quite that neat, however.

Posted on October 4, 2006 8:59 AM

____________________

HJE:

I would think that automated polling would be likely to show racism. People are more willing to tell the machine that they are "voting against the black man"...... So to me, I would think that automated pollster would be less favorable to Ford. Interesting that result is reversed. Another thing to take into consideration is the fact that the undecideds are much higher on the old-fashioned pollsters.... Finally, anyone else notice that the Tenn state poll was done over 11 days and by students. TO me that screams inaccuracy. Just me though.

Posted on October 4, 2006 11:02 AM

____________________

Scott:

>>The other issue is that because of the norm of racial equality, voters may tell real people surveying them that they will vote for a black candidate, but when they actually vote, they do not. Example: Dinkins v. Giuliani mayoral race in NYC. This may hurt Ford esp. in a Southern state.

I live here (Knoxville, TN). Comparing a racial issue in an election here with one in New York City is laughable. No one in this state is going to say they will vote for a black candidate to a surveyor and then vote for his opponent out of a sense of "racial equality"! People here wear their racist views proudly on their sleeve. They'll tell a pollster they're supporting Corker, period.

Posted on October 4, 2006 11:48 AM

____________________

James:

If this race is so remarkable, what are we to make of Connecticut, where Zogby and Rasmussen keep showing a split inside the margin of error, while Quinnipiac keeps showing Lieberman leading by double digits?

Posted on October 4, 2006 12:21 PM

____________________

TNRep:

I think that you would also need to think about where the polls took there samples from. Tennessee, probably not unlike some other states, has large areas that are common to one party. For instance, Memphis - Ford's hometown is heavily Democratic inside the city limits. If you move out into Shelby County and look at only voters that live outside Memphis' city limits the trend is totally opposite with the county voters leaning heavily Republican. There are fewer people outside the city limits than inside by about 250,000-500,000. That makes for some very interesting races in W. Tenn. and the slight dominance in, at least local, politics of Democratic candidates. Where places like Florida, California, and New York may be heavily weighted into one party with small blocks of the opposite party in specific areas, Tennessee seems to have larger more equal blocks of both parties and that may lend itself to larger variations in the results. Maybe the net is just not cast wide enough.

Posted on October 4, 2006 12:45 PM

____________________

HJE:

Rasmussen now has connecticut +10 for Lieberman....

Though I am biased, I think the afraid to say i'm voting blue might actually be at work

Posted on October 4, 2006 2:11 PM

____________________

Anonymous:

Re: white voters who say "they will vote for a black candidate, but when they actually vote, they do not". This theory surfaced in the mid-90s and persists today. I don't quarrel with it because I am not familiar with those polls and races. (Allthough one was an *exit poll* in VA which seems like a stretch. Why would voters lie on a "secret ballot" exit poll questionnaire?)

I have covered over a dozen races with black candidates over 25 years or so, both phone and exit polls. Exit polls are a non-issue here but they do provide race composition of who voted on election day. These races were elections in Chicago, Cook County, statewide Illinois and Milwaukee city and county. So my experience is geographically limited.

I have never seen any indication that white voters give socially acceptable answers. I do sometimes see the opposite - higher undecided levels among blacks in early polls which could mean reluctance to state preference for a black candidate, at least in early polls. Again, sometimes.

An example from 2004 in a jurisdiction (Milwaukee) with high black composition - higher than Tennessee..
"The sharp difference between black and white voter preferences means that the final outcome on April 6 will depend on the final composition of the electorate. Below, with 31% black voter composition, Barrett leads by 4 points. With 35% of all voters African-Americans, Pratt leads by 2 points."

The variation in Tennessee polls could be due to variation in black composition. That is the first place I would look.

Voting Registration & Voting Census reports show that non-Hispanic blacks ranged from 11.7% to 15.8% in TN from 2000 to 2004. Huge sample errors in these estimates. Not a lot of help here, not even from the Census. It is difficult to estimate what the *actual* composition will be in TN until after the election - after the exit poll.

Sample composition could be the reason for variation between polls.

Census Voting & Reg. data here. Select year, select detailed tables for excel files, select State reports.
http://www.census.gov/population/www/socdemo/voting.html

Posted on October 5, 2006 9:09 AM

____________________

Jeff Z:

The first question I would ask you is, "What types of voters are most or less likely to answer a robot poll?" Strictly anecdotal, but as soon as I hear that artificial voice, I hang up.

Another question I've always had is to what extent the time of day plays in predetermining poll repsondents. I'm familiar with the notion that whether the call is on the weekend or weekday may have an effect, but what about early evening calls, which is a chaotic time in a family household (slightly more Republican demo, as I understand it) and when no one has time to answer a phone survey.

Posted on October 5, 2006 11:57 AM

____________________

enough:

Nice answer. Thanks for taking the time.

Posted on October 5, 2006 5:25 PM

____________________

Brandi:

>>>I live here (Knoxville, TN). Comparing a racial issue in an election here with one in New York City is laughable. No one in this state is going to say they will vote for a black candidate to a surveyor and then vote for his opponent out of a sense of "racial equality"! People here wear their racist views proudly on their sleeve. They'll tell a pollster they're supporting Corker, period.

I live in Knoxville, TN too and Scott is exactly right!!! I think Ford could make it, but it will come from West and Middle Tennessee period. It's not just Knoxville, where Ford will have a racial problem. It is the entire East Tennessee region period.

Posted on October 23, 2006 6:57 PM

____________________