March 23, 2008 - March 29, 2008
Kathy Frankovic explains the utility of panel-back surveys for measuring change, and shares more data on their follow-up survey on Obama's speech on race.
Jennifer Agiesta digs deeper into the recent national surveys and sees "potential negatives on both Democratic candidates... percolating beneath the stable horse-race numbers."
Frank Newport sees potential positive and negative for Democrats in the Gallup "defector" data.
Hart and McInturff explain what pollsters mean by "oversample."
Mark Mellman wonders if the reduced press coverage of the Iraq war explains Americans' "more benign picture of the situation" there.
David Hill uses poll data on Americans leaving "the faith of their youth" to argue that Barack Obama should leave his church.
Todd Domke proposes a cross between "a pollster's focus group, Lincoln-Douglas-style debate, jury trial, and a secret ballot" to resolve the Florida/Michigan impasse.
Gary Andres sees "some positive views" about George Bush beyond his job approval numbers.
Ken Walsh sees the same in some polling from GOP Pollster Ed Goeas.
Marc Ambinder plugs some numbers into Jay Cost's popular vote projection spreadsheet.
Mickey Kaus has been arguing over the last week that the greater emphasis on racial issues in the Clinton-Obama nomination contest may have caused a return of the so-called Bradley-Wilder effect. The term refers to a pattern observed in the 1980s and early 1990s when the white opponents of African American candidates would do better on election day than indicated in pre-election polls (explained in more detail here, links to other sources here). Kaus thinks we may have evidence of a re-emergence of Bradley-Wilder in the results from the Gallup Daily and the Rasmussen Reports automated tracking surveys. Last Sunday, he wrote:
Gallup's national tracking poll has Obama retaking the lead over Hillary after bottoming out on the day of his big race speech. Rasmussen's robo-poll, on the other hand, shows Obama losing ground since last Tuesday. True, even Rasmussen doesn't seem to be putting a lot of emphasis on his survey's 6-point shift. But isn't this week's primary race exactly the sort of environment--i.e.., the issue of race is in the air--when robo-polling is supposed to have an advantage over the conventional human telephone polling used by Gallup? Voters wary of looking like bigots to a live operator--'and why didn't you like Obama's plea for mutual for understanding that all the editorial pages liked?'--might lie about their opinions, a phenomenon known as the Bradley Effect. But they might be more willing to tell the truth to a machine. ...
On Tuesday, I noted that the overall results from Gallup and Rasmussen were not that different when looking at data collected from March 14 (the day the Wright story broke) through the previous day:
- Live Interviewer Gallup Daily: Clinton +2 (47% to 45)
- Automated Rasmussen Reports: Obama +1 (45% to 44%)
Kaus updated his original post and responded that we should be focusing on the trends:
[I]f you look at the trend since Obama's 3/18 speech--which is what arguably charged the campaign with high-minded condemnation of racism and MSM sympathy for Obama of the sort that might produce a Bradley Effect--Obama gains 6 points in Gallup and loses 6 in Rasmussen through last Friday (and he's since lost one more on Rasmussen). That seems like a non-small difference. ...
He has continued to note the difference:
[3/25] Obama has now lost a net of 8 points on Rasmussen since the 18th, and 11 points since the 14th. On Gallup, he's gained several points.
[Yesterday] Bradley still in the race: Gallup (telephone poll) and Rasmussen (robo-poll) continue to diverge.
I have been puzzling over the trend and thought it would be helpful to post a chart of the data in question. In the chart below (click for a larger pop-up version), Kaus is focusing on the trends since the Obama/Wright speech on March 18. The dates on the chart are end-dates for each survey release. Keep in mind that Gallup reports a three-day rolling average, and Rasmussen reports a rolling four-day average, so the trend line reactions would theoretically lag slightly behind events.
If you overlook today's release, the chart does show a largely divergent trend, though most of the difference occurs in the three to four days after the speech. However, if you step back and look at the complete time series, the Gallup and Rasmussen lines are no more divergent now than they have been all along. In fact, if you remove three days of live-interviewer Gallup data -- the March 16-18 release which had Clinton (the white candidate) leading by seven points -- the divergent trend largely disappears.
Yes, for the last week, Obama has done better in the Rasmussen (automated) survey than the Gallup (live interviewer) data, but the difference is on the same scale as similar gaps since January that have see-sawed back and forth, favoring neither candidate consistently. So call me crazy, but I just don't see compelling evidence of a return of the Bradley-Wilder effect in these data, especially keeping in mind the potential for random variation in the trends.
One interesting pattern here -- and to be honest, I'm not sure what to make of it -- involves large gaps opposite of what we would expect from the Bradley-Wilder effect throughout much of January and again briefly centered on February 5/6 and March 18: At those times, the white candidate (Clinton) does better on the live-interviewer (Gallup) surveys than on the automated (Rasmussen) surveys. Also, as the charts below demonstrate, most of that difference occurs in the percentage supporting Clinton, not the percentage supporting Obama.
The difference in January may have had something to do with how the two surveys asked about other candidates still in the race, and the gaps afterward purely random, although that's a pure guess. Anyone have any better theories?
[Today's Guest Pollster contribution comes from Alex Lundry, research director at the Republican polling firm, TargetPoint Consulting.]
TargetPoint Consulting recently partnered with the Cook Political Report and RT Strategies, adding a new political research question called the Net Promoter Score (NPS) to their most recent national omnibus survey (March 6-9, N=802). This measure, adapted from the world of consumer research, attempts to measure voter enthusiasm and passion for a candidate. The results provide some new understanding to how the general election for President could shape up given either an Obama or Clinton candidacy.
First introduced by Frederick Reichheld in the Harvard Business Review and since popularized in his book, "The Ultimate Question," the NPS is used in the business world as a customer satisfaction metric, measuring a customer's likelihood to recommend a product, brand or company to someone else. This captures a number of difficult-to-quantify emotions, attitudes and preferences, by posing it as a recommendation. A recommendation is the ultimate endorsement, showing just how passionately you feel about a particular company or product. A recommendation means putting your own reputation on the line; an indication of loyalty, passion, and even the latent potential for word of mouth buzz.
The question is simple:
On a 0 to 10 scale, with 0 being "not at all likely" and 10 being "extremely likely," how likely is it that you would recommend voting for [INSERT CANDIDATE NAME] in the next election to a friend or colleague?
The NPS is calculated by subtracting the number of detractors (ratings of 0-6) from the number of promoters (ratings of 9 and 10). In the business world, +16 is the median score of more than 400 companies across 28 industries; CostCo has one of the highest known scores at +81. (See the NPS website for more details and similar statistics).
Studies have shown a direct and significant correlation between a business' score and company growth - specifically, a 7 point increase in overall NPS or a 2 point reduction in the percentage of detractors can each account for one percent of positive growth, thus indicating the potential electoral consequences of this measure once adapted to political polling.
To be fair, the NPS is not without it's own set of detractors and it's validity in the political world remains to be seen. For now we can only speculate about any correlation with electoral outcomes. Nonetheless, there is some promising historical data: TargetPoint began tracking the NPS on a generic congressional ballot in September of 2005 through August of 2006 and the results did seem to forebode the Republican fall from favor and the impending Democratic advances of that November. During that time the GOP NPS had a distinctively downward slope, falling from a high of +56 to a low of +11; meanwhile, the generic Democratic NPS trended upwards from a low of +32 to a high of +56.
But what about this year's election? In an Obama/McCain match-up McCain leads 45-43, but the NPS indicate some form of an "enthusiasm advantage" for Obama: among Obama voters, the NPS is +28 (53% promoters minus 25% detractors); while 48% of McCain voters are promoters and 31% detractors for a NPS of +17. Hence an Obama advantage of 11 points.
The Clinton/McCain ballot (McCain leads 47-45) again shows a Democrat enthusiasm advantage, though a slightly smaller one of 8 points (McCain: 44% promoter, 33% detractor, +11 NPS; Clinton: 48% promoter, 29% detractor, +19 NPS).
Though there is little surface difference between the candidates, deeper analysis indicates two critical demographic differences: enthusiasm among youth and Independent voters. The NPS among Independents voting for Obama (+30) is a stunning forty-nine points higher than the score among Independent McCain voters (-19). Interestingly, McCain actually wins Independents against both Clinton and Obama, but his Indy voters are much less enthusiastic than either Obama's or Clinton's. Clinton's NPS among her independent voters is also negative (-5), and a full 45 points short of Obama's. It appears that a Clinton candidacy would remove any passion or enthusiasm among Democrat-voting Independents.
Finally, we see nearly identical performance among 18-40 year olds. McCain's NPS among this age group is +7 and -5 against Obama and Clinton respectively; Clinton actually performs worse than McCain here, with a negative score of -13, while Obama dominates at +32. Again, we are left wondering what would happen to this youth enthusiasm should Clinton become the nominee.
Keep in mind that these are scores among people already voting for that particular candidate. While a vote is still all that matters on Election Day, a recommendation driven campaign can produce new votes faster, cheaper and in a more trustworthy and impactful way than traditional campaign appeals of advertising, direct mail and robo-calls.
Brian Schaffer, who on Wednesday reminded us of some Pew Research Center results from March 2000 on defectors from presidential candidates supported in primaries, has done one better. He went back to the data from the Annenberg National Election Study (NAES) in 2000 and checked the defection rates of supporters of John McCain and Bill Bradlee during the fall campaign:
How common were defections? Fairly common, actually. Even in October, only 49% of former McCain voters intended to vote for Bush and 29% were planning on casting their ballot for Gore (in March of 2000, a Pew Survey reported that 51% of McCain supporters planned to vote for Gore). McCain supporters were also far more likely to be undecided late in the race as 11% of this group reported that they still did not know who they intended to vote for.
Former Bradley supporters were also divided. While 52% of this group planned on voting for Gore, another 28% intended to vote for Bush.
Perhaps unsurprisingly, both former Bradley and former McCain supporters appeared more likely to prefer 3rd party candidate Ralph Nader. About 10% of Bradley supporters and 7% of McCain supporters expressed their intent to vote for Nader.
Schaffner also notes that since "since turnout in primaries is usually far less than it is in general elections," the defectors made for relatively small portion of those that ultimately supported Bush or Gore:
Based on this survey, former McCain voters accounted for 4% of those who intended to vote for Gore while former Bradley supporters accounted for 2.6% of those who voted for Bush. (Interestingly, citizens who voted for Gore in the primaries accounted for 1.8% of those who intended to vote for Bush and Bush primary supporters were 1% of those who intended to vote for Gore).
Of course, as both Schaffner and many of our readers point out, 2000 is not 2008. Both party primaries were wrapped up fairly quickly. How big a factor defections will be in 2008 may depend on how long it takes to resolve the Democratic nomination fight.
American Research Group
Clinton 52, Obama 39
Clinton: 35 / 42
Obama: 38 / 40
McCain 34 / 39
Bush: 22 / 61
Cheney: 15 / 55
Spitzer: 4 / 50
Pew Research Center
Obama 49, Clinton 39 (was Obama 49, Clinton 40 in late February)
Obama 49, McCain 43 (was Obama 50, McCain 43)
Clinton 49, McCain 44 (was Clinton 50, McCain 45)
The survey also tested reactions to Rev. Wright, the economy, and the traits associated with each party and candidate.
Gallup follows up on their recent survey which looked at the percent of Obama and Clinton primary supporters who would vote for McCain if the other became the Democratic nominee.
"Black Democratic voters, regardless of whom they support, seem prepared to remain quite loyal to the Democratic Party. Fifteen percent of blacks who support Clinton would vote for McCain if Obama is the nominee, and only 10% of blacks who support Obama would vote for McCain if Clinton is the nominee. In other words, there is little apparent risk of losing a substantial proportion of black voters regardless of who the nominee is."
"The data do not address the issue of motivation or turnout, which could be lower among blacks if Obama is not the nominee, nor do the data address the implications of the precise way in which Clinton might win the nomination."
Read the full analysis here.
My NationalJournal.com column, on those wildly variant automated polls in North Carolina from Public Policy Polling (PPP), is now online.
A few additional pieces of the story: First, I get a lot of email asking about the firms like PPP. Who are they? Who pays for the polls? PPP was founded by a North Carolina businessman named Dean Debnam, and its clients are mostly Democratic candidates holding or seeking local office in North Carolina. According to Tom Jensen, PPP's communications director, Debnam founded the company to help provide "low cost, high quality polling" to candidates for local offices who "could never afford a $12,000 poll."
PPP is a good example of a growing trend that the automated (or interactive voice response - IVR) technology makes possible. It is easier than ever for organizations with little prior experience in survey research to make calls, ask questions, tabulate the results and disseminate them via the Internet. Where my pollster colleagues disagree -- often vehemently -- is whether the new firms like PPP are delivering the "high quality" polling they promise. For example, one campaign pollster friend I talked to this week said he had a "hearty laugh" about the change in the sample selection methodology I describe in the column because, "I doubt seriously that they had one in the first place."
I should say that Jensen has been very responsive on behalf of PPP and as transparent about their methods as any pollster we have dealt with. On the other hand, PPP made no reference to their changed sample selection in their most recent releases (here and here, though they did note the change in a separate blog entry). They also neglected to extract the relevant vote history data from the sample that would have allowed a simple tabulation of the results from the latest survey using the older, narrower universe of past primary voters. Those are the kinds of mistakes that fuel skepticism among experienced pollsters.
Professor Franklin and I share an attitude about these sorts of surveys that sometimes puts us at odds with many of our colleagues in survey research. We believe we ought to judge all surveys by their performance rather than simply dismissing them by their methodology alone. Skepticism is certainly appropriate for newcomers using relatively unproven methods, but we will continue to track and follow the results from companies like PPP in order to evaluate their ultimate success or failure in achieving their stated goals.
Over at RealClearPolitics, Jay Cost has done an invaluable service by posting a spreadsheet that allows you to create your own popular vote projection for the Democratic presidential race by entering assumptions about turnout and vote shares in the nine remaining contests and using whichever counting method you prefer.
Of course, delegates will choose the Democratic nominee, not popular votes. But as just about every delegate counter has concluded, neither candidate is likely to attain the necessary majority on pledged delegates alone. The ultimate decision will be in the hands of the unpledged, so-called "super delegates." Whatever the end result, the two campaigns are likely to spar over the the outcome of the popular vote as a debate point in winning over uncommitted super-delegates. But how do you count the popular vote in a system designed to select and count delegates?
As Cost explains, the task is not easy:
First, there are many reasonable ways to count the popular vote. None is obviously superior to the rest. Of course, it does not matter which we think is most appropriate. What matters is what the superdelegates think, as they will be the "tie-breakers" in the nomination battle.
They could approach it in many ways. They could take the basic vote count and choose to exclude or include Michigan, Florida, or caucus estimates. Assuming they want to include the Michigan results and the caucus estimates (for IA, ME, NV, and WA, whose state parties do not supply actual vote totals), they could account for them in different ways. With Michigan, they could (a) give Obama the "unaffiliated" vote, (b) not give Obama the "unaffiliated" vote, or (c) reallocate the vote based upon whom voters claimed in the exit poll they would support if all candidates had been on the ballot. If they include caucus estimates, they could (i) count the non-binding Washington primary instead of the caucus, or (ii) count the Washington caucus instead of the primary.
In his spreadsheet, Cost provides 15 different ways to count popular votes and encourages his readers to predict the race for themselves. I have to agree with his main underlying theme, that "a prediction like this must be very imprecise" given the ongoing debate about the most appropriate way to count the votes cast and the many unknown variables involving contests still to come. His full write-up is worth a read before tinkering with the spreadsheet.
The spreadsheet also helps illustrate the potential significance of Puerto Rico in projecting popular votes:
The biggest problem is with Puerto Rico. We are literally without precedent there. It's never voted in a presidential election of any kind. It is therefore extremely difficult to get an idea of who will win, let alone by how much. An even bigger question with Puerto Rico is turnout. Puerto Ricans are some of the most active voters in the world, and turnout could be very high. But how high? 100,000, 500,000, 1 million, 2 million? Again, we have no precedent for it.
In a post a few weeks ago, Cost noted that "about 2 million Puerto Ricans voted in 2004, or about 52% of the public," but those contests involved contests for governor and resident commissioner. Puerto Rico casts no votes in the general election for President, Also, as Cost points out, Puerto Rico's political parties do not systematically align themselves with the stateside Democratic or Republican parties. So projecting turnout and vote shares there is truly a shot in the dark, but potentially crucial in determining the ultimate popular vote leader.
Public Policy Institute of California
Obama 49, McCain 40... Clinton 46, McCain 43
NBC News/Wall Street Journal
(March 24-25, n=800 adults, 700 registered voters; NBC story, video ; WSJ story, results)
Clinton 45%, Obama 45% (Clinton led 47% to 43% on 3/7-10 survey)
Obama 44%, McCain 42%
McCain 46% Clinton 44%
The lead of Jackie Calmes' Wall Street Journal poll story:
The racially charged debate over Barack Obama's relationship with his longtime pastor hasn't much changed his close contest against Hillary Clinton, or hurt him against Republican nominee-in-waiting John McCain, according to a new Wall Street Journal/NBC News poll.
Democratic pollster Peter Hart, who conducts the Journal/NBC polls with Republican pollster Bill McInturff, called the latest poll a "myth-buster" that showed the pastor controversy is "not the beginning of the end for the Obama campaign."
And from Chuck Todd's must read analysis on First Call:
As for the damage [the Wright] controversy did or didn't do to Obama, it's a mixed bag. Yes, Obama saw some of his numbers go down slightly among certain voting groups, most notably Republicans. But he's still much more competitive with independent voters when matched up against John McCain than Hillary Clinton. And he still sports a net-positive personal rating of 49-32, which is down only slightly from two weeks ago when it was 51-28. Again, the biggest shift in those negative numbers were among Republicans.
The survey also includes questions on the Wright controversy and Obama's response. Todd's write up includes this passage, especially relevant to the previous post on defectors:
One thing about these head-to-head matchups: our pollsters found that for the second poll in a row, more than 20% of Clinton and Obama supporters say they would support McCain when he's matched up against the other Democrat. There is clearly some hardening of feelings among some of the most core supporters of both Democrats, though it may be Obama voters, who are more bitter in the long run.
Why? Because among Obama voters, Clinton has a net-negative personal rating (35-43) while Clinton voters have a net-positive view of Obama (50-29). Taken together, this appears to be evidence that Obama, intially, should have the easier time uniting the party than Clinton.
Brian Schaffner provides some helpful context for the Gallup "Democratic defector" results that we linked to earlier today. He notes a Pew Research Center survey from March 2000 that suggested similar fallout for George W. Bush as a result of his primary against John McCain.
Here are the key passages from the Pew report, starting with the lead paragraph:
The presidential primary season may prove to be a decisive factor in Campaign 2000, not only for who won, but for the way the winners emerged from the process in the eyes of the voters. Al Gore was clearly helped, and George W. Bush was just as clearly hurt. The vice president has improved his personal image, while making gains among two key groups whose support had eluded him last year, independents and men. In contrast, many people have come to dislike Bush personally, especially former supporters of John McCain. As a consequence, the Texas governor now trails Gore for the first time in a nationwide Pew Research Center survey, by 49%-43%....
Later, the report turned to the impact of the primaries:
Primaries Costly for Bush
Moreover, Gore leads Bush by a 51%-44% margin among voters who say they backed McCain during the primary process.(1) These McCain supporters are especially vocal critics of Bush as a person -- nearly half (48%) of those who support Gore point to Bush's personality as the thing they like least about him.[NOTE 1: Unless otherwise noted, former McCain and Bradley backers/supporters are those who say they strongly supported McCain/Bradley for their parties' nomination.]
But Gore's most important gains from supporters of McCain and Bradley come among independents who now disproportionately favor the vice president. In contrast, the party regulars have largely returned to the fold, with Republicans supporting Bush and Democrats supporting Gore.
Needless to say, that early Gore advantage did not persist. I'll let Schaffner blog the rest:
Eventually, many of those McCain backers likely returned to vote for Bush and most of the Bradley backers likely returned to vote for Gore. The hard feelings that existed shortly after the end of the primary eventually subsided as the party unified for the general election. It is likely the case that Obama and Clinton supporters would eventually return to the fold and support the Democratic nominee in the Fall as well. However, the key difference between 2000 and 2008 will be the timing. When McCain lost the nomination, Bush had between 7-8 months to court McCain's old supporters. The Democratic nominee will have less time to do the courting this year. The critical question is how much time will he or she have?
In the Gallup analysis, Frank Newport makes a similar point:
[I]t may be normal for some voters to claim early on in the process -- perhaps out of frustration -- that they will desert their party if certain things do not happen to their liking. And it may be equally likely that they fall back into line by the time of the general election. It is worth noting that in Gallup's historical final pre-election polls from 1992 to 2004, 10% or less of Republicans and Democrats typically vote for the other party's presidential candidate.
Incidentally, for those looking to test electability with these early snapshots, keep in mind that the Gallup analysis focuses solely on self-identified Democrats that say they vote in primaries. It does not cover to the ability of the two Democrats to attract independent or cross-over support from those who say they do not vote in Democratic primaries. On their late February survey, Pew observed that "Obama has much greater personal appeal to independent voters than does either McCain or Clinton," and Pew's Scott Keeter reported that roughly equal numbers of voters are Obama-not-Clinton or Clinton-not-Obama in matchups against John McCain. It would be interesting to replicate those calculations using the Gallup Daily data, although the fact that Obama gets 44% and Clinton 45% against McCain suggests that the rough parity in these defector/cross-over groups persists.
Of course, the larger point of the eight year old Pew numbers is that snapshots from March have a short half-life, so speculate with caution.
New analysis from Gallup's daily tracking data (compiled from 3/7 through 3/22) finds 28% of those who support Clinton in the Democratic primary would vote for McCain if he were pitted against Obama in the general election, while 19% of those who support Obama in the primary would vote for McCain if he were pitted against Clinton (video).
Update: In the comments, Chris G argues that I am "way off" to conclude that "there has been far more stability than change in the national Obama-Clinton vote preference since Super Tuesday." He writes:
[T]hat simply does not follow from the simulations. the only thing that can be inferred is that if we're looking at these 2 time series alone, any meaningful changes in support are swamped by the noise. that's all we can conclude.
Since I may have been unclear, let me try to clarify: I am not arguing that the Gallup Daily and Rasmussen Reports tracking data proof the complete absence of change in candidate preference since Super Tuesday. Chris is absolutely right: No survey can do that. The best we can do is conclude that changes have been too small to be detected with confidence.
The point I was trying to make is that the changes since Super Tuesday have been (a) short lived, (b) small enough that they are indistinguishable from random noise, or (c) both. I do not consider changes of that sort to be very meaningful substantively, though your definition of "meaningful" may differ.
I am also not arguing that we should ignore the Gallup Daily. We just need to be patient and wait to see big, persistent changes. Look back at the numbers they reported in January through early February and you can see a very large, sustained and meaningful trend toward Obama:
In the midst of writing this update, I discovered that Gallup's Frank Newport made essentially the same point in his daily video report today:
As I look at the Gallup Daily election tracking, I am struck by the fact that neither candidate, Hillary Clinton or Barack Obama has been able to move ahead to a sustained and significant lead over the other [emphasis added].
Public Policy Polling (D)
North Carolina 3/24
Obama 55, Clinton 34
NOTE: PPP's latest survey now includes those who voted in the 2006 general election. In previous North Carolina primary surveys, their universe only included those who voted in either the 2004 or 2006 primary. More from PPP's Tom Jensen here.
In case you missed our update, the most recent Gallup Daily result on the Democratic race shows a near dead-heat, with Barack Obama ahead of Hillary Clinton by a single percentage point margin not nearly large enough to attain statistical significance (47% to 46%). That one point lead is somewhat apropos, since it is virtually identical to the average of all of Gallup's Daily releases since February 8 (Obama 46%, Clinton 45%). So the question for the day: How much of the daily variation over the last six weeks has been real and how much is random noise?
Let's start with the chart of the Gallup Daily results since their three-day track completed on February 8 (and released on February 9). That was the first three-day result collected entirely after the results from the Super Tuesday primaries were known.
While the Gallup trend has shown several "figure eights" over the last few weeks (as reader "emcee" put it), most of that variation occurs within the range that we should expect from a survey with a +/- 3 point margin of sampling error.
To illustrate that point, consider the hypothetical possibility that the preferences among Democrats have remained perfectly stable for the last six weeks. Let's assume that the average result since February 8 -- 46% to 45% favoring Obama -- has been the unchanging reality. What sort of random variation should we expect from taking a sample rather than interviewing the entire population?
First, remember that the so-called "margin of error" applies to the individual percentages, not the margin between the candidates. So under our hypothetical "no change" scenario, we would expect the the Obama percentages to fall somewhere between 43% and 49% (46% +/- 3) and the Clinton percentages to fall somewhere between 42% and 48% for Clinton (45% +/-3).
Since February 8, the results of the actual Gallup Daily have fallen outside that range on just three days:
- March 1, when Obama led 50% to 42%
- March 13, when Obama led 50% to 44%
- March 18, when Clinton led 49% to 42%
But wait. As some of you may remember, most political surveys (including Gallup) calculate the margin of error using a 95% confidence level. That assumption means that we should expect results slightly outside the margin of error for one poll in twenty.
Unfortunately, at this point our story gets a little bit more complicated, because the "one in twenty" assumption applies to statistically independent measurements. Since each Gallup Daily release is based on a three-day rolling average, there is overlap in the sample on successive days. So only the results from every third day are truly "independent." 'll skip over some even more confusing explanation and get to the bottom line: Since February 8, roughly one-in-seven independent samples from the Gallup Daily series has produced a result outside the margin of error from my hypothetical, no-change, 46-45 scenario. That's a little bit more than we would expect by chance alone, but not much more.
Having said all that, my explanation still oversimplifies. It ignores the possibility for meaningful change within the standard "margin of error" -- subtle shifts that might not attain statistical significance in a single three-day sampling, but might over the course of a week or more.
A better way to distinguish the meaningful patterns is to compare Gallup's results to those from another pollster or two. Let's start with a chart of the Rasmussen Reports daily tracking poll over the same six week period. Not surprisingly, the average of the Rasmussen data gathered since February 8 also shows Obama leading by a single percentage point (45% to 44%).
Compare the two charts (or look at the chart below, which plots a Clinton-minus-Obama margin for both polls) and you will see several features in common:
- Both show a shift from Clinton to Obama between Super Tuesday and mid-February
- Both show Obama maintaining a low single-digit lead from mid to late February
- Both show Clinton rising a few days before the March 4 primaries and falling a few days after
And yet, at about the time the news surrounding Jeremiah Wright became a full-blown media obsession (March 14), the results of the two polls appear to diverge. Why is that?
We should keep in mind that Gallup and Rasmussen collect their data differently (and ask slightly different questions -- see the postscript). Gallup uses live interviewers, makes repeated call-backs to unavailable respondents, samples cell phone numbers, and routes calls to Spanish speaking interviewers when they reach a Spanish speaking household. Rasmussen uses an automated system and recorded voice to conduct interviews, a slightly tighter screen for "likely voters," yet (as I understand it) makes no calls backs, does not call cell-phones and makes no provision for bilingual interviewing.
Some, I am sure, will readily conclude that one or more of these characteristics (or perhaps others that I've omitted) provide "obvious" explanations for the discrepancies. I am reluctant to make too much of these differences. The reasons be clearer after we look at data from a third source. I obtained it earlier today from an anonymous but trusted pollster that I'll call "Polimatic." Here is a chart of the Polimatic's tracking data for the last six weeks:
Those who notice the greater stability in the Polimatic data as compared to Gallup and Rasmussen are on to something important. Next consider how the Clinton-minus-Obama margin from the Polimatic data compares to the other pollsters:
See some interesting patterns? Starting to form theories about what type of poll Polimatic is, or how their methodology might influence their results?
Well, before you go too far, I should fess up. I fibbed. "Polimatic" is not a pollster at all. The data are based on a simulation run by our friend Mark Lindeman. Mark created a spreadsheet that generates random results consistent with a thee-day rolling average tracking sample of 1,260
40 interviews and the assumption that the "true" population value remains an unchanging 46% to 45% Obama lead.
The Polimatic line is more stable, suggesting that the consistently highest highs and lowest lows of the blue and red lines probably represent real divergence. However, the purely random variation of the simulated poll trend line is frequently hard to distinguish from the real surveys.
To generate the results above, I closed my eyes and clicked the mouse to let the spreadsheet recalculate. As such, the "Polimatic" line illustrates one potential trend showing nothing but random noise around a 46% to 45% margin. I'll say it one more time to be clear: All of the variation in the Pollmatic trend lines is based on purely random chance. Any resemblance to real changes as measured by Gallup or Rasmussen is entirely coincidental.
So what can we conclude from all this?
First, there has been far more stability than change in the national Obama-Clinton vote preference since Super Tuesday, and that includes the period of last ten days. To the extent that we have seen real changes, they are barely bigger than what we might expect by chance alone.
Second, if you look closely, you will notice that the seemingly odd divergence between Gallup and Rasmussen since the Wright story broke is really not that unusual. It is comparable to similar separations in the trend lines that occurred around February 13 and February 29. Random variation will do that.
Third, and probably most important, it is far too easy to look at these rolling average tracking surveys and see compelling narratives and spin interesting theories from what is often little more than random noise.
PS: Yes, as a few readers have already suggested in prior comments, some of the stability in national Democratic vote preference may stem from the fact that most states have already held their primaries and caucuses. We had some discussion about a month ago about how Gallup alters its screen slightly to accommodate states that have already voted. However, neither Gallup nor Rasmussen alters their vote question for those who have already voted. Here is the text used by each:
Gallup: Which of these candidates would you be most likely to support for the Democratic nomination for president in 2008, or would you support someone else? [ROTATED: New York Senator, Hillary Clinton; Former Alaska Senator, Mike Gravel; Illinois Senator, Barack Obama]
Rasmussen: If the Democratic Presidential Primary were held in your state today, would you vote for Hillary Clinton or Barack Obama? [options are rotated]
PPS: While I was writing this post, Mickey Kaus blogged a theory for the divergent Gallup and Rasmussen trend lines:
The 'Bradley Effect' is Back? Gallup's national tracking poll has Obama retaking the lead over Hillary after bottoming out on the day of his big race speech. Rasmussen's robo-poll, on the other hand, shows Obama losing ground since last Tuesday. True, even Rasmussen doesn't seem to be putting a lot of emphasis on his survey's 6-point shift. But isn't this week's primary race exactly the sort of environment--i.e.., the issue of race is in the air--when robo-polling is supposed to have an advantage over the conventional human telephone polling used by Gallup? Voters wary of looking like bigots to a live operator--'and why didn't you like Obama's plea for mutual for understanding that all the editorial pages liked?'--might lie about their opinions, a phenomenon known as the Bradley Effect. But they might be more willing to tell the truth to a machine. ...
Or more likely, the apparent differences between are about random variation in one or both polls. If you average the results from data collected since March 14 (the day the Wright story exploded) they are not very different:
- Live Interviewer Gallup Daily: Clinton +2 (47% to 45)
- Automated Rasmussen Reports: Obama +1 (45% to 44%)
Kaus also links to an automated PPP survey in North Carolina that fielded on the evening of March 17, the night before the Obama speech. As such, it is consistent with Gallup's "bottoming out" for Obama, not contradictory. The SurveyUSA results I blogged about on Friday were also collected from March 14 to March 16, just after the Wright story broke but before Obama's speech.