May 08, 2008

Poblano's Model

My latest NationalJournal.com column, about the remarkable success of the non-poll statistical model created by Poblano of FiveThirtyEight.com, is now online.

--Mark Blumenthal on May 08, 2008 in The 2008 Race | Permalink | Comments (10) | Trackback (0)

May 07, 2008

"Outliers" from the IN/PA and Beyond

[I fell behind a bit on his feature over the last hectic week, so some of these are a little stale].

Kathy Frankovic considers the reasons for conflicting national numbers from CBS/NYT and Gallup earlier this week (see also my post on this topic); last week she had a terrific review of how question order can affect survey results.

Carl Bialik revisits the primary math post IN/NC and considers Brian Schaffner's superdelegate projection model.

CJR's Clint Hendler takes an in-depth look at the various popular vote counts.

David Hill says "every superdelegate can find one survey that confirms the outcome he or she intuitively prefers for the Obama-Clinton fight.

Jay Cost crunches the exits from IN and NC

John Cohen finds that most Republicans voting in IN and NC express "little other than a sincere preference for Clinton over Obama."

Brian Schaffner notes that turnout in IN and NC "exceeded the number of votes Kerry won in the states in the general election"

Tom Schaller thinks Hillary could have done better among African Americans (via Smith).

PPP's Tom Jensen tips his hat to SurveyUSA's Jay Leve

--Mark Blumenthal on May 07, 2008 in Miscellanous, The 2008 Race | Permalink | Comments (8) | Trackback (0)

May 07, 2008

NC-IN Follow-Up Links

Here are links to exit poll and related analysis from:

• ABC News
• Associated Press
  
• CBS News
• CNN
• Wall Street Journal
  
• Washington Post

Other pollster related news:

• Brian Schaffner plots pollster performance
• SurveyUSA report cards for NC and IN
• PPP takes a victory lap
• Mickey Kaus (8:45 entry) vents about hidden exit poll estimates

If you've seen other exit poll analysis or pollster related news, email me a link and I'll add it to the list.

--Mark Blumenthal on May 07, 2008 in Exit Polls, The 2008 Race | Permalink | Comments (30) | Trackback (0)

May 06, 2008

Live Blogging North Carolina and Indiana Election Night

I will be live-blogging here tonight on what we might learn -- and what we might do better to ignore -- from the exit poll in Indiana and North Carolina. More details to follow, but please feel free to use this as an open-thread on what is appearing on the net and elsewhere on the exit polls. Here are the links where official exit poll tabulations will appear shortly after the polls close at 8:00 p.m.:

• CBS: North Carolina, Indiana
• Fox News: North Carolina, Indiana
• MSNBC: North Carolina, Indiana
• CNN: North Carolina, Indiana]

Comments will appear in reverse chronological order -- all times Eastern.

12:30 - And finally...while some important votes are still being counted n Indiana, at least some conclusions about pollster performance are inescapable. First, it does appear that most of the polls significantly understated Barack Obama's percentage of the African American vote, especially in North Carolina, as they have in other states. According to the most current exit poll tabulations (as of this writing), Obama won over 90% of African Americans in both North Carolina and Indiana.

Second, it seems clear that in terms of the overall result, the winner** among the pollsters tonight was Zogby International. Their final polls had Obama ahead by 14 in North Carolina and up by 2 in Indiana -- a closer margin in Indiana than any survey reported in the final week. [Update: Not so fast. Another pollster -- PPP -- did as well or better, depending on the yardstick. The SurveyUSA report cards for Indiana and North Carolina, as well as the Brian Schaffner's graphic, show that Zogby and PPP both scored well, with PPP doing slightly better on 10 of 16 rankings].

I have certainly been critical of Zogby over the years, but credit is due. Pollster reader political_junkie was right with this comment earlier tonight: "It took a lot of courage for them to publish their 'outlier' results last night, one night before primary."

Third, the non-polling based statistical model developed by Poblano at FiveThirtyEight.com outperformed most of the polls. His models predicted a narrow (51.0% to 49.0%) margin in Indiana and a 17 point margin in North Carolina (58.6% to 41.4%).

10:16 - Better late than never, answering the question raised at 8:11: I am told that Edison/Mitofsky conducted a telephone poll of early voters in North Carolina to gather data to combine with the interviews completed at polling places. In Indiana, it was polling place interviews only.

10:11 -- Just posted by Poblano, who is doing his own modeling of the vote count:

I'm now showing Clinton winning Indiana by 1.8 percent, or about 23,000 votes. And one thing to remember about Indiana is the provisional ballot issue -- people who were rejected at the polls because they did not meet the state's ID requirements could still cast provisional ballots and prove their identity later. It's possible that we'll still have a hanging chads type of situation.

9:39 - Thatcher asks: "Did I just see MSNBC change their call from 'Too early' [to call] to "Too close'"? Yes you did.

9:32 - A friend passes along this blog post:

I am watching MSNC coverage of the Indiana and North Carolina Democratic Primary results and I am struck by how profligate the network is with almost all of the exit poll results for Indiana (which has not yet been called) except the aggregate percentages for each of the candidates! The race is "too close to call," but that doesn't explain this.

Why don't the networks tell us who "won" according to the exit polls? The polls have closed, so they can't affect the results? Are the exit polls so lacking in trustworthiness that they aren't really informative? If that is the case, why are the demographic breakdowns from the exit polls thrown around and talked as if they were gospel? (They don't even mention margins of error when talking about them!)

Here's bottom line: When they say a race is "too close to call" or that it's "too early to call," they don't "know" who won yet. They have estimates, and they have a good sense of who is likely to win, but not enough statistical confidence in those estimates to project a winner with complete confidence. No one wants to repeat the mistakes made in projecting Florida eight years ago.

On the other hand, I tend to agree with those who wish the networks would make more of their estimate data available via the web on election night. The various estimates that the exit poll operation generates -- including the levels of statistical confidence in each measure and the real-time estimates of the precinct level error -- are truly fascinating.

8:25 - From Nora O'Donnell's report on MSNBC: In Indiana, Clinton carries whites without a college degree 63% to 34%; Obama ahead by two points (51% to 49%) among college educated white voters.

8:22 - Via Ambinder: CBS News has projected Hillary Clinton the winner in Indiana (thanks PHGrl).

8:15 - Reader Thatcher asks: "So does every network use the same exit poll info?" Yep.

8:11 - Reader RS asks a really good question: "Maybe this question has been answered earlier, but: Do the exit polls account for early voters?"

The networks typically do a pre-election telephone interviews among those who say they have already voted in states that typically get (or expect) a very large proportion of early voters, so I assume that they did one in North Carolina. But to be honest, I'm not sure.

7:55 - MSNBC's Nora O'Donnell just read results from two subgroups of North Carolina voters we've watched carefully in other states. Among white college educated voters, Clinton is leading by 7 (52% to 45%). Among white voters without a college degree, Clinton leads 68% to 26%.

7:54 - The North Carolina tabulations just updated with an additional 554 exit poll interviews not included in the first batch. My extrapolation of the underlying estimate now gives Obama a 14-point lead, 55% to 41%.

7:50 - Just a note on the mechanics of the exit poll updates and the network projection process. The numbers we have seen so far are -- presumably -- based on the exit poll interviews weighted based on some hard counts of turnout and (probably) weighted to the "composite estimate" which splits the difference between the exit poll tallies and pre-election polls. Right now, however, exit poll interviews and other results "reporters" are obtaining actual vote counts for the sampled precincts and these are being gradually incorporated into the estimates that the network "decision desks" look at to make their projections.

Unfortunately, If tonight's updates follow the pattern of recent election nights, the cross-tabulations we can see will not be updated until much later in the evening.

7:38 - Clinton again does better among the late than early deciders in North Carolina though not by as much as in Indiana. She won those deciding in the last three days by six points more (48%) than those who decided earlier (42%).

7:36 - The exit poll estimates 33% of North Carolina's Democrats as African American, and Obama is winning then 91% to 6%. Clinton is holding a 59% to 36% margin among white voters.

7:32 - CNN and MSNBC project Obama the winner in North Carolina. My extrapolation of the initial vote estimate used to weight the current exit poll tabulations shows Obama leading 55% to 41%, with 4% choosing "no preference."

7:27 - Polls close in NC shortly. Here's a CBS summary

7:19 - Once again, Hillary did better in Indiana among those who made up their minds in the last three days (+9) or within the last week (+8) than those who decided earlier (tie). See my Pennsylvania election night post (6:26 update) for comparative data from past primaries).

7:17 - Demographics: African Americans are 15% of the current estimate in Indiana. Obama is winning 92% of black voters, Clinton 60% of white voters. Compare to the pre-election polls here.

7:13 - Interesting that the MSNBC anchors have a new phrase that better fits the past problems with these numbers: "Too early to call."

7:12 - While I was typing, the Indiana table updated slightly but the overall estimate still rounds to 52-48% in Clinton's favor.

7:09 - Time for the usual caveats, with a twist. My usual election night helper Mark Lindeman had a social engagement tonight, so his spiffy extrapolation program is unavailable. I will be doing simpler extrapolations off of the vote by gender cross tab the old-fashioned way (which may mean a tad more rounding error).

Again, these initial tabulations are weighted to an estimate of the result that is usually a mashup of pre-election polls and the interviews exit polls conducted at polling places and over the phone (with early voters) by the networks. These estimates improve, becoming more accurate over the course of the night. Click here for more detail on how these numbers are derived and how they improve over the course of the evening.

7:04 - The exit poll tabulations now available on the network sites (links above) show an initial 52% to 48% estimate favoring Hillary Clinton. Please note that these initial estimates are usually a cross between the interviews conducted at polling places and an average of pre-election polls and, more importantly, have often been quite different than the final result.

6:55 - From the ABC summary:

These preliminary results indicate that a third of North Carolina's voters are African-Americans, in line with the norm, e.g. 31 percent in 1992, the last primary there for which exit poll data are available.

6:51 - Here (via The Page) are official early exit poll summaries, focusing mostly on results on questions other than vote preference from CNN, ABC, Fox, ABC, CBS, AP and MSNBC.

5:50 - Almost forgot. Halperin has these words of wisdom about "what makes it tough to produce good models for the exit polls in North Carolina and Indiana:"

1. They are not closed contests open only to Democrats.
2. Turnout is going to be huge (probably record breaking).
3. The absence of recent competitive primaries.

So let’s all be patient, shall we?

5:45. First out of the block in the leaked exit poll derby (at least that I'm aware of) is Huffington Post. Now before you click that link, please read my column on the problems exit polls have had this primary season (paying close attention to the table) and the follow-up this week on Pennsylvania. I'll be relocating to the Pollster.com "home office" over the next 30-40 minutes...see you back at 7:00.

--Mark Blumenthal on May 06, 2008 in Exit Polls, The 2008 Race | Permalink | Comments (72) | Trackback (0)

May 06, 2008

NC-IN Predictions Roundup

A typically busy election day here, but I want to quickly link to some interesting vote predictions popping up in the blogosphere:

• Tom Holbrook, a professor of political science at the University of Wisconsin-Milwaukee, has used our final standard trend estimates for Clinton and Obama for the past primary states to create a statistical model that projects forward. His projection of Hillary Clinton's share of the two-party vote is 53% in Indiana and and 47% in North Carolina.

Holbrook first tested (and explained) the workings of his model in Pennsylvania on primary day, and nailed the result. He projected that Clinton would get 54.8% of the two-candidate vote. She won 54.6% of the vote. In the craziness of the Pennsylvania election day, I neglected to post a link to Holbook's projection -- apologies to all for that.

• Brian Schaffner has updated his delegate projections for tonight's two primary states also based on our polling trend estimates: a 38-34 split in Clinton's favor in Indiana and a 62-53 split for Obama in North Carolina.

• FiveThirtyEight's Poblano has used a regression model based not on poll results but on census and other "hard" population data at the Congressional District level. The model projected a 53.7% to 46.3% Clinton win in Pennsylvania. In Indiana, his model predicts a narrow (51.0% to 49.0%) Clinton win in Indiana that translates into a delegate split of 36 to 36. In North Carolina, the model predicts a very large (58.6% to 41.4%) Obama victory, with Obama gaining 66 delegates and Clinton 49.

Using a similar approach, Poblano has also generated "scorecards" for both Indiana and North Carolina that attempts to project results by county to match the final RealClearPolitics statewide polling averages (Clinton +5.0 in Indiana and Obama +8.0 in North Carolina). The idea is to create statistics to use to follow the election returns. If Clinton or Obama is outperforming his projections in the various counties, it would suggest a performance better or worse than the statewide assumptions.

I will predict with high confidence that others have posted similar models somewhere on the web. If you know of one worth adding, please post a comment or send us an email (questions at pollster dot com).

--Mark Blumenthal on May 06, 2008 in The 2008 Race | Permalink | Comments (5) | Trackback (0)

May 06, 2008

Vote by Race in NC and IN

I thought it might be helpful, given our focus on the racial composition in the electorates for Indiana and especially North Carolina, to post a summary of results by race in each state. Fortunately, just about all of the pollsters have released results by race on their final polls. A summary of each follows.

Let's start with North Carolina, where the ultimate margin will be extremely sensitive to the racial composition, as these numbers will make clear.

Most of the surveys have had the Clinton percentage of the white vote at roughly 61%, and -- with one notetable exception -- at about the average of 10% of the black vote. Much of the variation in the overall margin comes from the percentage of African Americans in the full sample. Most of the pollster have been reporting an African-American percentage of 32% to 33%. But as FiveThirtyEight's Poblano points out about his very helpful North Carolina prediction spreadsheet,

[A]n increase of 1 percent in the fraction of the electorate that is African-American translates to roughly a 1-point increase in Barack Obama's margin over Hillary Clinton. So -- if the pollsters are assuming 33% black turnout when it will actually be 40%, that would add 7 points to Obama's margin -- putting us in the 13-14 point range.

In the same post, Poblano makes the case that the African American percentage in North Carolina is likely to be higher than the 32% reported by most pollsters.

Another huge issue in trying to use these findings as the basis for vote projections is deciding what to do about undecided voters. In South Carolina pre-election polls vastly underestimated Barack Obama's vote, mostly among African-Americans. Polls conducted with an automated methodology in South Carolina showed Obama doing better among black voters than those using live interviewers, and PPP's Tom Jensen and FiveThirtyEight's Poblano 'have both noted that (as Poblano put it) "polls have significantly underestimated Barack Obama's margin of victory in Southern states with substantial black populations." This pattern in past primaries argues for assuming that Obama's support among African-Americans will be higher tonight than the 82% average noted above.

On the other hand, TNR's Noam Scheiber noticed that Clinton does slightly better among IVR pollsters in North Carolina, and he wonders if the automated surveys are "picking up on a queasiness [about Obama and Wright that] black voters are less comfortable sharing with human interviewers." Perhaps, but notice that much of the difference comes from the InsiderAdvantage poll.

What's your guess? You can plug numbers into Poblano's spreadsheet to test your theories.

The most consistent aspect of the Indiana results is Obama's percentage among white voters, which nearly every pollster pegs at or near the average of 39%. If Clinton gets 60% or more of the white vote tonight, the only question will be the size of her margin given the percentage of African-Americans. To win, Obama will need to hold Clinton a few points under 60 and boost the African American share to roughly 15%.

One striking inconsistency in Indiana is the result among African-Americans. However, we should keep in mind that the sample sizes are usually very small (probably in the range of 50 to 100 interviews), so we should not be surprised to see big variation. Moreover, the pattern is not consistent across pollsters by mode.

Update: I neglected to include explicit links above, but I did posts yesterday on what the composition by all of the North Carolina and Indiana polls by race and other demographics.

--Mark Blumenthal on May 06, 2008 in The 2008 Race | Permalink | Comments (3) | Trackback (0)

May 06, 2008

Final Polls: About The Undecided?

Well, for the umpteenth time this primary season, we wake up to wide variation on the final polls for the day's primaries. Today we have polls showing Barack Obama leading Hillary Clinton by anywhere from 4 to 14 points in North Carolina. Meanwhile, poll in Indiana show everything from a 2-point Obama edge to a 12-point Clinton blowout. One big question in looking at the variation is whether the pattern of variation suggests a pattern of a larger undecided translating into a hidden Clinton vote. The evidence on this question is mixed, however, and relies mostly on the results from just one pollster.

First, rather than listing the polls, lets do a chart of the final results in the last week or so from each pollster. Start with North Carolina. The following chart simply plots each result on a grid with the Obama percentage on the vertical axis and the Clinton percentage on the horizontal axis. All of the points on the North Carolina chart are above the blue diagonal line indicating an Obama lead.

Many -- such as one of Josh Marshall's readers -- think they see a pattern (which yours truly also saw at first in Pennsylvania but that subsequently disappeared) suggesting a coming "break" of undecided voters to Clinton. Such a pattern would imply a horizontal pattern to the dots above, with all of the variation in the Clinton number and little in the Obama number. That pattern holds only with respect to the Zogby poll, the one showing Obama leading by the biggest (51% to 37% margin), but also the poll with the most respondents categorized as either undecided or as choosing "someone else."

The pattern of the other points in North Carolina is mostly circular, about as varied as we would expect given sampling error and centered around a roughly seven point Obama advantage (50% to 43%) with 7% left over as undecided or "other."

Now Indiana:

In Indiana the dots are slightly more dispersed, with Zogby again the showing the best result for Obama, in this case a 2-point Obama advantage (45% to 43%), with 12% categorized as either undecided or "other." In this case, however, two polls have shown roughly as many voters choosing an option other than Obama or Clinton, although both were about a week old: One from TeleResearch (showing Clinton leading by 10 points with 14% undecided/other) and the other from Rasmussen Reports (giving Clinton a 5-point lead with 13% undecided/other).

Again, if we set the Zogby result aside, we get most of the polls forming a circular, mostly random pattern around an average advantage of 7 points (50% to 43%) with 7% undecided or "other."

In thinking about what to make about the difference between Zogby and the other polls, it may be useful to think about what it means for a respondent to tell a pollster they are "undecided" in the days leading up to the election. There are at least three possibilities:

1. They are going to vote but are still uncertain about which candidate to support
2. They are going to vote, have decided which candidate they support but are not willing to share their preference with the person (or computer) on the other end of the phone line
   
3. They are not going to vote but were mistakenly identified as a "likely voter" by the pollster

Pollsters understand that many voters hover somewhere between a final decision and being totally undecided. So most consider it good practice to "push" uncertain voters, especially near election day, as the candidate they lean to supporting is almost always the candidate they ultimately support.

So I tend to agree with Pollster readers who have expressed frustration in comments with pollsters reporting a large undecided preference. What is especially puzzling about the Zogby result, however, is the very large percentage that they have reported as favoring "someone else" -- 4% in North Carolina (down from 8% over the weekend) and 5% in Indiana (down from 7%). What does that mean? Are respondents expressing a preference for John Edwards? John McCain? Are those really non-primary voters?

At any rate, given that the demographics of Zogby's samples are not radically different from the other pollsters in the two states, there is certainly a good possibility that a harder "push" would benefit Clinton. We also have seen in exit polls that late deciders have favored Clinton in most of the primaries since Super Tuesday. More on both states -- and particularly the issue of late deciders favoring Clinton -- later today.

Update: Just noticed this helpful information posted by Dick Bennett on the ARG web site:

The Democratic ballot in Indiana has two lines (Hillary Clinton and Barack Obama), while the Democratic ballot in North Carolina has four lines (Hillary Clinton, Mike Gravel, Barack Obama, and No Preference).

Democratic primary voters in our surveys in Indiana were asked just the two candidate choices. If voters said someone other than Clinton or Obama, our interviewers were instructed to inform the voters that there are only two choices on the ballot. In most cases, voters then selected Clinton or Obama instead of saying they were undecided.

In North Carolina, our surveys gave the four choices on the ballot. Democratic primary voters selecting the "no preference" line also told us that they would never vote for Clinton or Obama. Our results combine the no preference with someone else (even though no preference will get more votes than Mike Gravel).

In watching the results tonight, be aware that "someone else" is not on the ballot in Indiana and some voters in North Carolina will vote the no preference line.

--Mark Blumenthal on May 06, 2008 in Divergent Polls, The 2008 Race | Permalink | Comments (2) | Trackback (0)

May 06, 2008

What's Up With The National Polls?

Last week, Democratic pollster Mark Mellman used his column in The Hill to argue that one could use "same-poll to same-poll comparisons" to argue that the controversy over Barack Obama's "bitter" comments had either no impact or a great deal of impact on the Pennsylvania primary. "In addition to providing jobs for pollsters," he wrote, "proliferating polls now give everyone the evidence to prove their favorite theory." He drew specific examples from the lesser known pollsters active in statewide contests that are typically derided by the polling establishment and the mainstream media.

Over the last 48 hours, however, we have seen Mellman's point proved by the country's most respected and well established polls and media organizations. As Gallup editor in chief Frank Newport noted in his Gallup Guru blog:

Two different headlines today in the New York Times and USA Today about the impact of the Jeremiah Wright controversy came to different conclusions about Jeremiah Wright and Barack Obama. The Times’ headline: “ In Poll, Obama Survives Furor; but Fall Is the Test”, while USA today headlines: “Flap over pastor pulls Obama down, poll finds”.

And for those who missed it, ABC's Gary Langer briefly summarized the conflicting numbers:

Briefly: Times/CBS has Barack Obama +12 vs. Hillary Clinton, with a headline saying Obama “survives furor” over the Rev. Jeremiah Wright. USAT/Gallup has Clinton +7, saying the flap over Wright “pulls Obama down.” Adding to the mix is Gallup’s daily poll, which has Obama +4.

These polls also differ in their general election match-ups: Times/CBS has Obama +11 and Clinton +12 vs. John McCain, while USAT/Gallup has them basically tied. Gallup daily has Clinton-McCain tied, McCain +5 vs. Obama.

And just in overnight: AP/IPSOS weighs in with a national poll showing Clinton +7 over Obama, with Clinton +5 and Obama +4 over McCain, spawning a thousand different headlines, no doubt (as AP stories always do).

Why all the confusion? Langer has a nice summary of the various methodological quirks that may or may not explain the differences between the Times/CBS and Gallup polls, though his bottom line is the most important take-away point. This episode, he writes,

[Is] a reminder that all polls – even good-quality ones – are done differently, and don’t always get the same results or engender the same analysis. And that horse-race results, in the midst of a close and unsettled campaign, may be particularly vulnerable to these kinds of influences.

Another way to make the same point. Look at our chart for the national Democratic primary polls as captured this morning It plots points for every survey released since January 2007 (captured this morning).

Let your eyes focus, for a moment, not on the the lines but on the cloud of dots surrounding each line. Each dot represents an individual poll. The dots are a bit more dense lately, a pattern explained mostly by our inclusion of daily tracking polls by Gallup and Rasmussen Reports since January 2008 (note: we plot only ever third or fourth day of each survey so we their rolling average samples do not overlap). However, the pattern of variation -- the spread of points around each line -- is considerable but not any wider now than at any other point over the last 17 months.

What is different right now is that the gap between Obama and Clinton is very close. So some polls show Clinton ahead, some show Obama ahead and given the sate of the race and we are paying much more attention to the small differences in individual polls than we usually do. Random error may explain some of the variation, small differences in methodology (question wording, order, the particular sub-population that answered he question) explains the rest. Either way, the cloud of variation -- bringing with it sometimes odd and conflicting "same-poll to same-poll comparisons" -- of "is an inherent part of political polling.

The variation is also the reason why we favor looking at all the polls in this "mashed up" graphical form rather than debating endlessly over which individual poll is closest to "right" at any moment.

--Mark Blumenthal on May 06, 2008 in Divergent Polls, The 2008 Race | Permalink | Comments (2) | Trackback (0)

May 05, 2008

The Demographics of the Indiana Surveys

And following-up on my post this morning on the demographics for the North Carolina polls, here is the same set of statistics, when available, for the recent surveys of Indiana. Since African-American's are a much smaller share of the population in Indiana, the Clinton-Obama results are not quite as sensitive to their percentage of the Democratic electorate as in North Carolina. However, should the Indiana result be close, the size of the African-American population will be important. Also, the tables show that the polls vary on age as much as in other states.

The following table shows demographic composition statistics for those pollsters that have released them. Click on the table to display a larger version that also includes the vote preference results for reach poll.

The table excludes statistics from pollsters that have not publicly released demographic information for their North Carolina surveys (or perhaps more accurately, have not published anywhere I could find it): LA Times/Bloomberg, Indianapolis Star/WTHR/Selzer, Howey-Gauge (and thanks again to Pollster reader jac13 for sharing the demographic profile data that Zogby makes available to paid subscribers).

In Indiana we see the same wide variation in the age distribution among pollsters seen elsewhere: Even The percentage of 18-to-29-year-olds varies from 8% to 22%, the percentage 18-to-44 varies from 26% to 51%.

With the exception of one pollster, the variation in racial composition is smaller. Most show an African-American percentage of somewhere between 9% and 12%, with Research2000 (13%) and Suffolk University (15%). The most extreme value is the Howey-Gauge survey, which reported a much higher percentage of African-Americans (20%) among likely primary voters.

Brian Schaffner noticed last week that larger percentage of African-Americans in the Howey-Gauge poll explained how they showed Obama with a two-point advantage while other firms showed Obama trailing by seven or more percentage points. He has some interesting speculation about the composition of the Indiana electorate, but ultimately I have to agree with his bottom line conclusion: Given the lack of an Indiana benchmark for past Democratic presidential primaries, "we don't really know what to expect in terms of African American turnout."

[Updated table to include new surveys from PPP and InsiderAdvantage]

--Mark Blumenthal on May 05, 2008 in Divergent Polls, The 2008 Race | Permalink | Comments (12) | Trackback (0)

May 05, 2008

The Demographics of the North Carolina Polls

Time for another round-up of available poll demographics, this time from North Carolina. The most important variable in this state is the African American percentage of likely Democratic primary voters. The most recent polls -- at least among those that have disclosed their demographics -- have converged around a black percentage of 32-33%. Needless to say, given the near monolithic support that African Americans have given Barack Obama, that percentage will ultimately be critical to his share of the vote on Tuesday.

The following table shows demographic composition statistics for those pollsters that have released them. Click on the table to display a larger version that also includes the vote preference results for reach poll.

The table excludes the pollsters that have, as of yet, not publicly released demographic information for their North Carolina surveys: Mason-Dixon, Rasmussen Reports, and LA Times/Bloomberg (special thanks to readers Paul and jac13 for sharing the demographic profile data that Zogby shares with paid subscribers).

As in previous states, we see considerable variation in the kinds of voters selected as "likely primary voters." Easily the most variant likely voter sample on the list is the one from the Civitas Institute from early April, with a composition of just 28% African American and 17% under the age of 45. However, even if we set that survey aside, we still see considerable variation: from 51% to 58% female, from 39% to 55% age 18-to-44 and from 25% to 37% African American (and those last extremes come from a single pollster -- more below).

A quick review from my post on the demographics of the Pennsylvania surveys:

It is important to remember that pollsters come to these composition statistics through different paths. Some interview samples of adults, weight those demographically to match census estimates of Pennsylvania's adults, then select "likely voters" and let their demographics fall where they may. Others will weight their "likely voter" samples directly to pre-determined demographic targets. Some pollsters will not set weights or quotas for demographics, but will set such weights or quotas for geographic regions (based on past turnout and their assumptions about what might be different this time).

With that in mind, note two very striking changes from two pollsters that set pre-determined demographic targets, Public Policy Polling (PPP) and InsiderAdvantage:

• The first three surveys released in April by PPP had an African American composition of 36% or 37%. Their most recent survey, fielded last Sunday and Monday evenings, had a black composition of just 33%.

• The gyrations in the weighting by InsiderAdvantage are even more dizzying. Their first North Carolina survey in late March was 37% African American. Their next two surveys in April were only 25% African American, and their most recent poll last week bumped the black percentage back up to 33%. Notice that none of their percentages for women, 18-29-year-olds, 18-44-year-olds or those 65+ changed by a single digit, despite a 12-point variance in the black percentage.

Both pollsters put out written summaries of their results, but neither made any reference (that I could find) explaining or justifying their changing assumptions about the racial composition of the North Carolina electorate. [Update: On their final poll, PPP upped the black share to 35%, but explained their rationale]. By the way, we know that these two pollsters set predetermined demographic targets, because both have confirmed as much to me in previous communications (here for InsiderAdvantage and here for PPP).

The change in the PPP poll is important -- they should have noted it -- but relatively modest compared to the astonishingly large, significant and unexplained shifts in the African American composition in the InsiderAdvantage polls. InsiderAdvantage's Matt Towery likes to brag of his "significant experience" as a pollster, but after a number of curious episodes over the last few months, it is getting very hard to take those claims seriously.

It's also worth pointing out the relative stability in the racial composition of the SurveyUSA results, given that they do not force their samples to a pre-determined demographic profile (details on their procedures here). The percentage of African Americans in their four surveys since March have remained relatively stable, falling within the range of 30% to 33%.

Finally, one caution about the percentage reported as "unaffiliated" (having no party affiliation). Only PPP includes the full text of their party question, and it is possible other pollsters are asking about party identification (whether respondents "consider themselves" as partisans) rather than party registration.

Update: Almost forgot. Fivethirtyeight's Poblano posted a handy spreadsheet that can help you see just how much small changes in the racial composition of the North Carolina electorate can affect the potential margin between the candidates. It's well worth the click.

Update II: In posting this last night, I neglected to point out that North Carolina has been releasing reports on the demographics of early voters. As North Carolina is one of nine southern states still required by the Voting Rights Act of 1965 to track voter registration by race, racial tallies among early voters are also available. The demographic composition of early voters have been analyzed by Brian Schaffner, DailyKos diarist dean4ever and noted in comments by many of our readers over the weekend.

Overnight, GMU Professor Michael McDonald, whose academic focus is voter turnout, posted the following comment:

North Carolina is an exceptional state in that it provides near real-time updates of its voter registration file. Indeed, you can download the entire file of absentee and early in-person voters directly from the state's ftp site.

North Carolina is also an interesting state because race and gender are recorded on the voter file (birthdate appears to be supressed in the absentee file). When I crunch the numbers, out of the 397,850 persons who are listed as returning a Democratic Party ballot as of 5/03:

39.9% are African American
60.8% are women

Note, a small percentage (less than 1%) of records have missing data.

Will these percentages hold for Tuesday? That is hard to say, mostly because people who study early voting (myself included) don't know much about the characteristics of early primary voters. Added is the confounding factor that one-stop registration and voting is permitted for in-person early voters only and not for Election Day voters. Providing little further clues, African-Americans are only slightly more likely to vote early in-person, 40.6%, and women slightly less, 60.7%.

The fact that nearly 400,000 early votes have been cast so far is remarkable given past primary turnout in North Carolina. The state held a caucus in 2004 (due to a redistricting battle that delayed the primary), but 544,922 Democrats voted in the largely uncontested primary in May 2000, and 691,875 voted in May 1992 (statistics I gathered for a column noting that pollster PPP has been sampling from a total universe of 874,222). The record was 961,000 in 1984, according to the Charlotte Observer, which cites "long time N.C. political observers" guessing that "as many as 1.5 million" may vote this year. So this early vote will be a significant portion of the total votes cast, but as McDonald points out, no one knows exactly how big.

It is also worth pointing out that the Obama campaign has made early voting drives a focus of their field organizing, so it is certainly possible that the ranks of early voters are disproportionately swollen with Obama voters. Last week's poll from SurveyUSA showed Obama leading by a 18 points (57% to 39%) among early voters, but that subgroup was just 2% of their total sample. Thus, one key result to watch in the final poll releases today -- among those far sighted enough to track and report it -- will be the size and preference of the early voters.

--Mark Blumenthal on May 05, 2008 in Divergent Polls, The 2006 Race | Permalink | Comments (9) | Trackback (0)

May 04, 2008

POLL: National Daily Tracking (through 5/3)

Gallup Daily

National
Obama 49, Clinton 45
McCain 47, Obama 42... McCain 46, Clinton 45

Rasmussen Reports

National
Clinton 44, Obama 45
McCain 47, Obama 44... Clinton 46, McCain 44

Favorable / Unfavorable
McCain: 52 / 46
Clinton: 48 / 50
Obama: 50 / 48

--Mark Blumenthal on May 04, 2008 in Poll Update | Permalink | Comments (5) | Trackback (0)

May 04, 2008

POLL: CBS/Times National (5/1-3)

CBS News/New York Times
(CBS story, Wright/Obama/Campaign results, Economy/Gas Tax results; Times story, results, methodology)
n=671 adults, 601 registered voters, 283 Democratic primary voters

National
Dem primary voters: Obama 51, Clinton 38

Registered voters:
Obama 51, McCain 40, Clinton 53, McCain 41

--Mark Blumenthal on May 04, 2008 in Poll Update | Permalink | Comments (6) | Trackback (0)

May 02, 2008

North Carolina: Trend Sensitivity

Time for another update looking at the "sensitivity" of our trend lines for North Carolina. As Professor Franklin is tied up with his day job this morning, I will be your guide. But let's go immediately to the chart that Franklin just generated.

The solid red and blue lines represent our standard trend estimates of support for Barack Obama and Hillary Clinton, respectively, in North Carolina. The trend lines are based, not on simple averaging, but on a local-regression based trend. We deliberately set the standard estimate to be conservative in that it takes a good bit of evidence of "real" change (i.e. more than one or two contrary polls) before the trend will show a sharp turn. As we have noted before, with lots of polls, this more conservative estimator has an excellent track record of finding real turning points of opinion while not chasing outliers.

In this case, however, the standard line appears to be too conservative. The last nine polls have produced results below the solid trend line for Obama and all but one above the trend line for Clinton. So in the graph above, we have also included trend lines based on a more sensitive estimator -- the dotted lines -- for both candidates. The sensitive estimator uses the same local regression methodology as our standard approach, but sets the degree of smoothing to about half that of the standard blue estimator. The sensitive estimator should detect short term change more quickly than "blue", but it will also sometimes chase phantom changes due to flukes of a few polls that happen to be too high or too low.

In this case, for the moment at least, the sensitive estimator (which shows Obama leading by a 7.7 point margin or 50.3 to 42.6) fits the most recent data better than the standard estimate (which shows Obama leading by 13.7, or 52.7 to 39.0).

Update: In the hour since Charles generated this chart, we have added new polls from Rasmussen Reports and ARG that change the numbers for the standard estimates on on our North Carolina chart slightly from what appears in the graph above. We'll try to post an update of the more sensitive estimate trend later this afternoon.

--Mark Blumenthal on May 02, 2008 in The 2008 Race | Permalink | Comments (0) | Trackback (0)

May 01, 2008

Exit Poll Potpourri

My NationalJournal.com column, with more on the performance of exit polls in Pennsylvania, is now posted online.

--Mark Blumenthal on May 01, 2008 in Exit Polls, The 2008 Race | Permalink | Comments (1) | Trackback (0)

April 30, 2008

Age or Education?

Amidst the personal craziness last week, I neglected to link to two columns from network pollsters that provide some valuable data from the exit polls on the Obama-Clinton race tabulated by race, education and income. Interest in this issue peaked last week after Barack Obama, said the following after his loss in the Pennsylvania primary:

I have to say if you look at and I know my staff has talked about this: If you look at the numbers, in fact, our problem has less to do with white working class voters. In fact, the problem is that, to the extent there is a problem, is that the older voters are very loyal to Senator Clinton.

ABC's polling director Gary Langer combined data from exit polls to look at support for the two candidates among white voters by age and income. "Age clearly is a factor," he concludes, "but it’s equally clear that socioeconomic status, as measured by the education and income alike, is independently a factor, and a big one."

Langer's column has tables with all the data. To make the patterns easier to see, I created two charts I created using only the percentage supporting Obama.

Here is Langer's analysis:

Look just at seniors, for instance: Across all primaries to date, among less well-off white seniors (those with less than $50,000 in household incomes), Clinton has beaten Obama by 70-22 percent. Among white seniors with more than $100,000 in household incomes, by contrast, Obama’s actually run ahead, by 50-45 percent.

Put another way, Obama’s support from high-income white seniors has been 28 points higher than it’s been among working-class white seniors. That isn’t just a senior problem. [...]

The relationship is weakest in Obama’s best age group, under 30s, but it’s still there. He’s won under-30 whites in $100,000+ households by 65-33 percent; he’s won young whites in under-$50,000 households by a much closer 53-42 percent.

The results are similar by education – Obama does 21 points better with white seniors who’ve earned college degrees than with those who haven’t. College-educated white seniors have favored Clinton by just 8 points, 50-42 percent; those without degrees have backed her by a whopping 48 points, 69-21 percent.

Kathy Frankovic, polling director at CBS News, looks at the same exit polling data (or presumably the same -- she explained that she combined exit polls "weighted to total votes...excluding Florida and Michigan") and adds a little more granularity for the youngest voters:

Among white voters with a college degree, Obama and Clinton have run almost even so far this year - 49 percent for Obama, 47 percent for Clinton. The results are very different by age within this group - those under 45 have given Obama a lead, and those over 45 have chosen Clinton. This does seem to support Obama’s claim that older, better-educated Democratic voters are staying with what they know, keeping on “track.”

White voters without a college degree, however, vote differently. This year, they have voted for Clinton over Obama by almost two-to-one - 61 percent to 33 percent. And the age of the voter matters less. Clinton leads decisively with just about all age groups of these voters - as long as they are over 30. She even edged Obama, 48 percent to 47 percent, among non-degreed voters under 30, but over 24 years old. Only the white non-college graduates younger than 25 have favored Obama so far this primary season. They voted for him 59 percent to 38 percent.

Frankovic's column also draws on an innovative survey released last week conducted among college students in Pennsylvania in partnership with the website Uwire (another survey I neglected to link to last week). College students have always been notoriously difficult to survey, and the ubiquity of cell phones among students has made it even worse. In this case, CBS sampled and interviewed students online using email lists of all students, presumably obtained directly from the universities. The full results from CBS include more methodological details.

For those of us that have been following trends in the Obama-Clinton contest by race, education and income, these two columns from Langer and Frankovic are invaluable. Both are worth reading in full. Also, be on the lookout for analysis of this data and more by my colleague Ron Brownstein in National Journal on Friday.

--Mark Blumenthal on April 30, 2008 in Exit Polls, The 2008 Race | Permalink | Comments (4) | Trackback (0)

April 30, 2008

A Rasmussen Poll That Wasn't

Two week ago, we linked to a survey result for the Louisiana Senate race released by Rasmussen Reports. The survey purported to show Sen. Mary Landrieu with a 16-point lead over challenger John Kennedy. Apparently, according to this report that appeared on April 20 in the New Orleans Times-Picayne, that initial result was in error:

According to one news account of a new poll by Rasmussen Reports, Sen. Mary Landrieu, D-La., had a 39 percent to 55 percent lead over her Republican challenger, state Treasurer John Kennedy. According to another account, that same poll had Kennedy ahead by a statistically insignificant 46 percent to 47 percent. Who was right? As it turns out, no one. The first poll results showing Landrieu ahead were posted on the Rasmussen Web site and then pulled after the firm realized it had confused the results with polling done in the Virginia Senate race, which showed Democrat Mark Warner ahead by that same 39 percent to 55 percent margin. Rasmussen later posted the 46 percent to 47 percent results, and then quickly removed that from its Web site. A company spokesman confirmed that the first results were wrong, but could not explain what happened with the second posted results. All he would say is that Rasmussen doesn't have any current polling data in the Louisiana Senate race. For Landrieu campaign staff, the sudden fall from 16 points ahead to one point behind, followed by a "never mind" from Rasmussen, was softened by the release last week of another poll, this one by Southern Media & Opinion. It showed Landrieu running ahead of Kennedy 38 percent to 50 percent.

Needless to say, after learning about the Rasmussen error this afternoon, we immediately removed the erroneous result from our chart of the Louisiana Senate race. Apologies for not doing so sooner.

--Mark Blumenthal on April 30, 2008 in The 2008 Race | Permalink | Comments (0) | Trackback (0)

April 29, 2008

Message Testing: Hear It For Yourself

Late last week, a North Carolina musician named David LaMotte received a survey call from Garin-Hart-Yang, the firm of Clinton pollster Geoff Garin. The call, as he reported to HuffingtonPost blogger and DailyKos diarist Paul Loeb, "started out normal enough" but soon "turned to long Hillary-praising and Barack bashing policy statements" with response options that asked him to evaluate each statement. At the end of the call, they asked, "now based on everything we've discussed, who would you vote for?" LaMotte used his telephone answering machine to record the latter half of the call, and as a result was the transcript that Loeb posted at DailyKos and later as streaming audio posted by Loeb, Politico's Ben Smith and ABC's Jake Tapper.

Not surprisingly, much of the commentary about this call focuses on whether the Garin survey meets the classic definition of a "push poll." It does not, at least as far as I can tell.

The call in question was long, included dozens of question that seemed "normal enough" to LaMotte and, as he confirmed to me via email, concluded with a set of demographic items that LaMotte deleted from the audio recording in order to protect his own privacy. This call has none of the hallmarks of the classic, so-called "push poll" intended only to spread a negative message under the false guise of a survey.

It was, rather, a "message testing" survey, albeit one that tested a highly negative and -- to many -- objectionable message. It was not measuring "public opinion" as it exists now but rather voter reactions to a series of positive statements about Hillary Clinton and negative attacks directed at Barack Obama. Garin asked respondents to react to each statement, and subsequently asked a second vote question ("Now based on everything we've discussed, who would you vote for?"), in order to identify the most effective attack and the voters most likely to be swayed by it.

Like it or not, this sort of testing is common in most campaigns, and almost none of the results ever see the light of day. Full disclosure: As a campaign pollster, I helped design hundreds of surveys with similar tests of messages. (I have written previously about the differences between message testing and "push polls,' see also the commentary by Roll Call's Stu Rothenberg and the recent statement on "push polls" and message testing by the American Association for Public Opinion Research-AAPOR).

Of course, simply labeling this survey as "message testing" does not absolve the pollster of ethical constraints. The pollster still has an obligation to tell the truth and treat respondents with fairness and respect. Did this survey do that? LaMotte's audio has the interviewer reading five statements that he describes as "criticisms that opponents might make about Barack Obama." After each of the statements below, the interviewer asks "if they would give you very major doubts, some doubts or no real doubts about supporting Obama."

At a time when we need leaders who are clear, strong and decisive, Obama has been inconsistent, saying he would remove all troops, but then indicating that he might not, and pledging to renegotiate NAFTA, but then sending signals that he would not actually do so as president.

He supported George W. Bush's 2005 energy bill which payed six billion dollars in subsidies to the oil and gas industry, nine billion dollars in subsidies to the coal industry and twelve billion dollars in subsidies to the nuclear power industry. It was called 'a piñata of perks' and 'the best energy bill corporations could buy.

He leads the committee with oversight on Afghanistan but failed to hold a single committee meeting or hearing on Al Qaeda in Afghanistan or anything else.

He sided with the credit card companies voting against the bill that would cap interest rates at 30 percent.

While he talks about universal health care he has failed to make the hard choices that would truly get us to universal coverage and lower health care costs for all. His plan would leave 15 million Americans uninsured.

Let's stipulate up-front that the Obama camp vigorously contests these arguments, with some support from journalists. While by no means a complete listing, here are links to reports that provide more context on the NAFTA, energy, credit card, health care and Afghanistan issues. Readers are encouraged to add more in the comments, if warranted.

However, here is the non-rhetorical question that interests me most: How much do these statements differ from those included in Clinton mailers on NAFTA, the energy bill, the credit card bill, health care or Hillary Clinton's statements on the stump about the Afghanistan oversight committee? And if they are essentially the same, why would testing these assertions in the context of a survey be any more or less objectionable than making the same assertions in a debate, a speech, a television ad or a campaign mailer?

[The original version of this post included some extraneous verbiage in the third paragraph that I've cleaned up]

--Mark Blumenthal on April 29, 2008 in Push "Polls", The 2008 Race | Permalink | Comments (7) | Trackback (0)

April 28, 2008

Novak and "The Bradley Effect"

Robert Novak's column last week led with this reference to the Pennsylvania exit poll results:

When Pennsylvania exit polls came out late Tuesday afternoon showing a lead of 3.6 points for Hillary Clinton over Barack Obama, Democratic leaders who desperately wanted her to end her candidacy were not cheered. They were sure that this puny lead overstated Obama's strength, as exit polls nearly always have in diverse states with large urban populations. How is it possible, then, that Clinton, given up for dead by her party's establishment, won Pennsylvania in a 10-point landslide? The answer is the dreaded "Bradley effect."

Prominent Democrats only whisper when they compare Obama's experience, the first African American with a serious chance to be president, with what happened to Los Angeles Mayor Tom Bradley a quarter-century ago. In 1982, exit polls showed Bradley, who was black, ahead in the race for governor of California, but he ultimately lost to Republican George Deukmejian. Pollster John Zogby (who predicted Clinton's double-digit win Tuesday) said what practicing Democrats would not: "I think voters face to face are not willing to say they would oppose an African American candidate."

Unfortunately, Novak confounds two issues, and Zogby's contribution confuses things further. The "Bradley effect" (also called the "Bradley/Wilder effect," the latter based on the 1989 election of Doug Wilder in Virginia by narrower margins than indicated by pre-election polls) pertained less to exit polls but to pre-election telephone surveys. The underlying theory was that white respondents were sometimes unwilling to reveal their preference for the white candidate in a bi-racial contest when they felt some "social discomfort" in doing so. That is, respondents would be less likely to reveal their true preference in a telephone interview if they believed the interviewer supported a different candidate. The most important evidence was an observed race-of-interviewer effect: Support for Doug Wilder in one 1989 survey (pdf) was eight points higher when the interviewer was black than when the interviewer was white.

The problem with extending this idea to the 2008 exit polls is that -- contrary to the apparent assumptions of both Bob Novak and John Zogby -- exit polls do not involve a "face to face" interview. Rather, the exit poll interviewer's task is to randomly select and recruit respondents, hand them a paper questionnaire, a pencil and a clipboard and allow the respondents to privately fill out the questionnaire and deposit it into a large "ballot box."

The more likely explanation for the consistent Obama skew in the exit polls this year is likely less about "voters not willing to say they would oppose an African American candidate," than about the relative youth of the interviewers, and the well established problem that the typically younger exit poll interviewers have in winning cooperation from older respondents. Here is a summary I wrote two years ago about information included in the official, post-election report on the 2004 exit polls:

The [National Election Pool] NEP exit polls depended heavily on younger and college age interviewers. More than a third (36%) were age 18-24 and more than half (52%) were under 35 years of age (p. 43-44). These younger interviewers had a much harder time completing interviews: The completion rate among 18-24 year olds was 50% compared to 61% among those 60 or older. The college age interviewers also reported having a harder time interviewing voters...The percentage of interviewers who said "the voters at your location" were "very cooperative" was 69% among interviewers over 55 but only 27% among those age 18 to 24 -- see p. 44 of the Edison/Mitofsky report.

Given the huge differences by age in both pre-election and exit polls -- Obama wins those under 30 while Clinton dominates among those over 60 -- an age-related selection bias is not surprising. And the issue may not be about simply getting the age mix right in the exit poll. The issue may also be related to the "social discomfort" theories behind the Bradley-Wilder effect.

Respondents may be making judgements about the exit poll interviewers based on their appearance (age, gender and race) that influence whether they agree to participate or avoid the interviewer altogether. Similarly, while exit poll interviewers are supposed to be carefully counting exiting voters and sticking rigidly to instructions that they select every fourth voter (or whatever interval they are instructed to select) anecdotal evidence suggests that those with less experience often deviate from the procedure and "take who they can get." So less experienced, overburdened interviewers are probably making judgments about which respondents (based on their age, gender and race) might be most likely to cooperate.

Again, quoting from my own summary two years ago:

"It's not that younger interviewers aren't good," as Kathy Frankovic puts it (slide #30), "it's that different kinds of voters perceive them differently." Put all the evidence together, we have considerable support for the idea "that Bush voters were predisposed to steer around college-age interviewers" (Lindeman, p. 14) or, put another way, that "when the interviewer has a hard time, they may be tempted to gravitate to people like them" (Frankovic, slide #30).

It is not at all surprising that this same mix of issues -- younger interviewers who have trouble winning cooperation with older respondents and a huge age differential in the results -- produces a consistent skew to Obama in the context of 2008.

--Mark Blumenthal on April 28, 2008 in Exit Polls, The 2008 Race | Permalink | Comments (7) | Trackback (0)

April 27, 2008

POLL: National Daily Tracking for 4/27

The Gallup Daily

National
Obama 47, Clinton 47
McCain 45, Obama 45... Clinton 47, McCain 44

Rasmussen Reports

National
Obama 48, Clinton 42
McCain 46, Obama 46... McCain 47, Clinton 45
Article

--Mark Blumenthal on April 27, 2008 in Poll Update | Permalink | Comments (9) | Trackback (0)

April 26, 2008

POLL: Newsweek National

Newsweek
Survey of 1,203 registered voters, 592 registered voters that identify or lean Democratic, interviews conducted 4/24-25 (article, results).

National

Among Registered Voters:
Vote Preference:
Obama 47, McCain 44
Clinton 48, McCain 45

Favorable/Unfavorable:
Obama 53/40
Clinton 47/49
McCain 51/41

Among Registered/Dem-Dem Leaners:
Obama 48, Clinton 41

--Mark Blumenthal on April 26, 2008 in Poll Update | Permalink | Comments (7) | Trackback (0)