August 17, 2008
By Mark Blumenthal
"Bounce" or "Bump?" The terminology is up to you, but this is certainly the season to consider the short term changes in polling numbers that frequently result in the wake of national political conventions.
A useful first stop would be the 2004 analysis from Gallup's Jeff Jones. It includes the post convention "bounce" numbers from Gallup back to 1964 that are the primary source of the conclusion that average post convention gain for candidates has been six or seven percentage points.
Last week, Tom Holbrook posted a more a thorough review of the past "before and after" data and it's implications. He looks at the gain for each candidate by taking their "average share of the two-party vote in trial-heat polls conducted six days to two weeks prior to the start of the convention" an subtracting that from the candidate's "share of the two-party vote in polls conducted during the seven days following the close of the convention." He does not list the polls used for each year, but presumably his data looks at much more than the Gallup time series for more recent elections.
Holbrook's post is worth reading in full for the lessons he draws from the considerable variation in past convention bumps, although probably the most important is his caution that "the magnitude of the convention bump is not a great predictor of election outcome." Still, he sees a pattern to the past variation that "should be a useful guide to what to expect" from the conventions and promises to update later this week with a prediction for each candidate.
But before reading two much into the twitches in the daily tracking polls over the next three weeks, please read the latest column from CBS News director of surveys Kathy Frankovic. She reminds us that the gap between the Democratic and Republican conventions is just three days -- much shorter than in past elections -- and will coincide with the Labor Day weekend:
Will we even be able to measure whatever impact the Democratic Convention has on Obama before it’s time to measure the GOP convention’s impact on John McCain? And will we be able to sort out what has caused what?
Will we actually discover a “bounce” or a “bump?”
Probably not. Polling over Labor Day Weekend is always a problem. We confront more than the usual number of people who don’t respond or can’t respond. People are away from their homes, heading back from summer vacation, or preparing their children for the start of the school year. In addition, the focus will shift so quickly from the Democrats to the Republicans that whatever opinions might be expressed over Labor Day Weekend might not last too long.
I thought it would be helpful to look at some past elections, not in terms of the immediate "before and after" averages, but rather where the conventions fit into the longer arc of the trend in the vote preference over the course of the election year. I spent some time gathering past polls from a variety of sources and asked Charles Franklin to create some charts matching the format he used to look at the trends of the 2000 and 2004 elections a few weeks back.
I had hoped to use those charts for a series of posts. Unfortunately, I got delayed, so I will post the charts along with a some very compressed discussion after the jump.
Continue reading "Convention "Bumps" in Context"
By Mark Blumenthal on August 17, 2008 4:49 PM
| Permalink
| Comments (2)
August 4, 2008
By Charles Franklin
The most common description of polls is that they are snapshots, not predictions. A good way to look at that in the 2008 election is to compare the '08 campaign with the two that came before.
The chart above shows the trend estimates for each of the last three presidential campaigns. I'm plotting the estimated margin between the two candidates, Dem minus Rep, for each year.
With 93 days to go until the 2008 election, Obama holds a 3.3 point advantage over McCain, though that has been eroding over the past six weeks. If we put a confidence interval around today's estimate, we get a race that is just barely leaning Democratic.
But what about the future? The dynamics of the next 92 days are all important for where we stand on November 4. Since we can't foresee those 92 days yet, let's see what happened during the same time in 2000 and 2004. That gives us a better idea how much change we might anticipate in the next three months.
In 2004, Kerry slowly built a 2 point lead by this time, and held a small lead through much of the summer. But then the race took a sharp turn, with Bush making a 6 point run, taking a four point lead with 50 days to go. Kerry gained back 3 points of that in the polling, but less than 2 points of it in the actual vote, losing by a 2.4 point margin.
In 2000, Bush led in most of the early polls, holding a 6 point lead with 107 days to go. Then Gore moved sharply up, erasing Bush's lead and then adding a 3 point lead for Gore with about 56 days left. Bush promptly reversed Gore's gains with a six point move in the GOP's direction, and led by about 3 points over the last three weeks of the campaign. Of course, the 2000 polls were misleading in predicting a Bush win. Gore won the popular vote by 0.6 points.
So far in 2008, Obama has enjoyed a run up of 5.5 points since his low point in late March. That run is on a par with Bush's in 2004 but still a bit less than Gore's 9 point run in 2000, and on par the Bush's 6 point rebound that year.
Judging from the dynamics we've seen in the past it is quite reasonable to expect the current trend to shift by half-a-dozen points. August and the conventions have been periods of substantial change in both previous elections, so if history repeats itself the next 4 or 5 weeks should be pretty interesting.
The bottom line is neither campaign should be complacent or despondent. There is a lot of time left and recent history shows that both up and down swings of 6-9 points are entirely plausible.
As a P.S. here are the three campaigns with educational confidence intervals around them.
The current 2008 estimate is just barely inside the "lean Dem" range, and will move to toss up if the current trend continues for another couple or three polls.
The 2004 estimate was pretty close to the outcome which was well within the 68% confidence interval around the trend.
The polls in 2000 were troubling for having the wrong popular vote winner, but even there the outcome was inside the 95% confidence interval. With races as close as the last two, it is worth appreciating just how wide those confidence intervals are.
Our efforts to characterize races rely on the best estimates of those confidence intervals, but it is all too easy to focus on who's ahead and not remember how much uncertainty there is. That uncertainty is both about where the current estimate says the race stands today and about how the race may change in coming weeks. The data here show that unless one candidate builds a bigger lead than either has held so far, the uncertainty remains pretty big.
Note: My trend here is slightly different from the Pollster National trend because I'm working off the difference between candidates, not each trend separately, and because I've made 2008 comparable to 2000 and 2004, just a slightly different amount of smoothing compared to Pollster's standard estimator this year. None of those differences change the qualitative picture or shift the magnitude of changes I cite above.
Cross posted at Political Arithemik.
By Charles Franklin on August 4, 2008 5:49 PM
| Permalink
| Comments (3)
| TrackBacks (0)
June 17, 2008
By Mark Blumenthal
If you read about today's Washington Post/ABC News poll online, you probably missed the "Act II, Scene I" sidebar graphic from the Post's print edition that is also available online. The gist if the graphic is that while the McCain-Obama vote looks now looks identical to Bush vs. Kerry preferences four years ago, the larger political terrain as defined by the Bush job rating, the perceived direction of the nation and views about the Iraq War are very different.
One piece of context worth considering, however, is that the Post/ABC poll of June 2004 had Kerry doing slightly better than other polls taken at about the same time. The following list -- which comes from the RealClearPolitics listing from 2004 -- shows that the Post/ABC survey was the only one conducted during the latter half of the month showing Kerry with even a "numeric" lead. The average for the month had Bush ahead by a single percentage point (45% to 44%).
Regardless of whether the current national horse-race is exactly the same as at this point in 2004, or a few points better for the Democrats, the larger point of the Post graphic still holds: We have seen far bigger changes in the percentage of Americans that disapprove of George Bush's performance as president (+17 percentage points since June 2004 in the ABC/Post poll), that say things in the country "have gotten pretty seriously off on the wrong track" (+27) and that conclude the war in Iraq was "not worth fighting for" (+11).
And as long as we are on the subject, we cannot repeat it often enough: Polls are, at best, a measure of where the race stands "if the election were held today. " It isn't held today. The 2004 race aside, polls in June are historically poor predictors of the ultimate outcome of the presidential election in November.
Nate Silver did a nice round-up over the weekend of how June polls since 1988 compare to November outcomes and concluded:
So in four out of the last five elections, an average of June polls would have incorrectly picked the winner of the popular vote. That's kind of a problem for anybody who is overly confident about how this election is going to turn out.
By Mark Blumenthal on June 17, 2008 11:58 AM
| Permalink
| Comments (1)
December 29, 2007
By Charles Franklin
As we pour over the latest data, it is worth taking a look back at a previous year in Iowa.
The Democratic contest in 2004 was strikingly dynamic, with a sudden surge for Kerry and Edwards as Gephardt and Dean slumped. (And note the gap in polling. Those are the holidays we are in right now when polling wasn't done.) By the last polls, Kerry had established a clear lead, but Dean, Edwards and Gephardt were within four points of each other. The polling got the trends pretty close to right. Edwards was clearly on his way up through the last 12 days of the race, as was Kerry. Dean may or may not have reversed his fall in the last week, and the polls said Gephardt was coming down. So far, so good.
But the Democratic caucus process does a lot to change the outcomes, with supporters of non-viable candidates joining forces with their second choices. That process is likely to boost the well off candidate, while robbing the struggling campaigns. And of course there is always the issue of which candidate's supporters actually turn out on caucus night.
The entrance poll is the best measure we have of whose supporters actually show up on caucus night, even with all the appropriate cautions about the entrance poll itself. Comparing the entrance poll to that last estimated trend value for candidate support from pre-caucus polling, we can see how the two rising campaigns did even better than the polling predicted. Kerry's last poll trend estimate was 25.9%, good for first place. But his entrance poll support was 34.8%. For Edwards, his final trend estimate was 21.4%, but the entrance poll found 26.2% supporting him.
For Gephardt the story was the opposite. His supporters stayed home on caucus night. The pre-caucus trend had Gephardt falling but at 17.8%. But the entrance poll found only 10.3% to be Gephardt supporters.
Dean was the only top candidate whose preelection and entrance poll numbers match closely. His trend estimate was 20.3% and the entrance poll put his support at 20.5%.
Caucus night had the effect of stretching out the differences between the candidates, advantaging the top two while damaging the fourth place candidate. (Dennis Kucinich was the exception in the back of the pack, with a trend estimate of 1.7 but an entrance support of 4%.)
Then they vote. And form coalitions with non-viable supporters. And weight the delegates in a complex formula and finally there is an allocation of delegates to the state convention. That allocation is the best we can do to consider a "final" outcome of this process.
The results there further favored Kerry and Edwards. Kerry moved up to 38% of delegates, from 34.8% in the entrance poll and 25.9% in the pre-caucus poll trend. Edwards got 32% of delegates, up from his 26.2% in the entrance poll and 21.4% in the poll trend.
Dean ended up with 18% of delegates, down a bit from the 20.5% in the entrance and 20.3% in the poll trend. And Gephardt's delegates just about matched his entrance poll, 11% of delegates and 10.3% in the entrance poll. A disappointment from his 17.8% final trend estimate.
So let's take one important lesson away from this Caucus Past. The pre-election poll trend got the order of finish right, if only by a point separating Edwards and Dean. But the process of caucus night that makes it tough to come out means that enthusiastic supporters are more likely to turn out than those who are discouraged by recent slippage. That is probably true for both parties. On the Democratic side, the complex voting and coalition formation further exaggerates the lead of the top candidates and diminishes the showing of marginal ones. And that process is seen even in comparison with the entrance polls, let alone the pre-election polling.
In Iowa the "outcome" is quite a few steps removed from the simple balance of preferences among the population. The mechanisms themselves intervene to affect the final delegate counts. So don't expect to see the pre-election polls hit the final delegate percentages very closely. If the polls get the order right, that will be good enough.
For a fair test of how good the polling is this year, wait for New Hampshire where pollsters poll, voters vote, and they just count up the results.
Cross-posted at Political Arithmetik.
By Charles Franklin on December 29, 2007 6:26 PM
| Permalink
| Comments (1)
| TrackBacks (0)
August 30, 2007
By Mark Blumenthal
So today we have another installment in that pollster's nightmare
known as the Iowa
caucuses: Two new polls of "likely Democratic caucus goers" conducted over the
last ten days that show very different results. The American Research Group
(ARG) survey
(conducted 8/26-29, n=600) shows Hillary Clinton (with 28%) leading Barack
Obama (23%) and John Edwards (20%). And a new survey from Time/SRBI (conducted 8/22-26, n=519, Time story,
SRBI results) shows
essentially the opposite, Edwards (with 29%) leading Clinton (24%) and Obama (22%).
Is one result more trustworthy than the other? That is
always a tough question to answer, but one of these polls is considerably more
transparent about its methods. And that should tell us something.
While I have been opining lately about both the
difficulty in polling the Iowa Caucuses and the remarkable lack of disclosure
of methodology in the early states (especially here
and here and all the posts here),
the new Time survey stands out as a model of transparency:
The sample source was a list of
registered Democratic and Independent voters in Iowa provided by Voter Contact Services.
These registered voters were screened to determine their likelihood of
attending the 2008 Iowa
Democratic caucuses.
Likely voters included in the
sample included those who said they were
-
100% certain that they would attend the Iowa caucuses, OR
-
probably going to attend and reported
that they had attended a previous Iowa
caucus.
The margin of error for the
entire sample is approximately +/- 5 percentage points. The margin of error is
higher for subgroups. Surveys are subject to other error sources as well,
including sampling coverage error, recording error, and respondent error.
Data were weighted to approximate
the 2004 Iowa
Democratic Caucus "Entrance Polls," conducted January 19, 2004.
Turnout in primary elections and
caucuses tends to be low, with polls at this early stage generally
overestimating attendance.
The sample included cell phone
numbers, which, to the extent SRBI was able to identify them, were dialed
manually.
I emailed Schulman to ask about the incidence and he quickly
replied with a "back of the envelope" calculation: Their sample of 519 likely
caucus goers represents roughly 12% of eligible adults in Iowa (details on the
jump), exactly the same percentage as obtained by the recent ABC
News/Washington Post poll, but
higher than the reported 2004 Democratic caucus turnout (5.5% of eligible
adults). Keep in mind, however, that the ABC/Post poll used a random digit dial
methodology and screened from the population of all Iowa adults.
The Time/SRBI survey started with a list of registered
Democrats and independents - so theoretically did a better job screening out
non-registrants and Republicans. On the Time survey, 92% of respondents report having
"ever attended" Iowa
precinct caucuses (see Q2)."
On the Post/ABC survey, 68% report having "attended any previous Iowa caucuses" (see
Q12). Readers will notice that on the 2004 entrance
poll, 55% of the caucus-goers said they had participated before.
What is the American
Research Group Methodology? All they tell us on the website is that they
completed 600 interviews and that respondents were asked:
Would you say that you definitely
plan to participate in the 2008 Democratic presidential caucus, that you might
participate in the 2008 Democratic presidential caucus, or that you will
probably not participate in the 2008 Democratic presidential caucus?
Blogger speculation alert: If this was the only question used to
screen, it is likely that ARG's incidence of eligible adults was much higher. Such a difference likely explains why they show Clinton
doing consistently better
in Iowa than other pollsters, but that is just an educated guess. [Update: A guess that turns out to be wrong....]. We owe
Dick Bennett the opportunity to respond with more details. I have emailed him
with questions and will post a response when I get it. [Update: Details of Bennett's response here. They ask four questions to screen for likely voters and their Democratic sample in this case represented roughly 12% of adults in Iowa. Apologies to ARG].
I suspect that if we could know all about every pollsters'
methods in Iowa, we would see evidence of a disagreement about how tightly to
screen and about what percentage of the completed sample should report having
participated in a prior caucus.
The resolution of that argument is neither simple nor obvious,
but seems to have a profound impact on the results. Surveys that appear to
include more past caucus goers (Time,
Des Moines Register and One
Campaign survey -- see our Iowa compilation) tend to favor John Edwards, while Hillary Clinton
does better on surveys that define the likely caucus-goer universe more
broadly. [Update: The disagreement may have more to do with the appropriate number of self-reported past caucus goers].
Details on Time's "back of the envelope" incidence
calculation after the jump...
Continue reading "Iowa: A Tale of Two New Polls"
By Mark Blumenthal on August 30, 2007 8:20 PM
| Permalink
| Comments (7)
| TrackBacks (0)
July 30, 2007
By Charles Franklin
A quick reminder not to assume what is today will be so tomorrow.
The 2004 Democratic primary race as of late July 2003 showed a continuous first place held by Lieberman, though with a slow but steady erosion. Kerry and Gephardt locked in a long running tie, and Howard Dean a rising 4th place at about 12% support. Clark's late entry and sharp rise hadn't happened. Edwards looked like a goner as his initial 9% had sunk to about 5%. So from this, who would be the candidates left standing after Iowa?
But of course the dynamics changed. Between summer and late fall, Dean became the "inevitable" nominee, sparking talk of running mates and gaining Gore's December endorsement. Kerry by that point was under 10%.
And then Iowa happened, and support shifted dramatically to Kerry and Edwards and away from Dean. (And Gephardt and Lieberman were gone.)
To those who say this is clear evidence that early polling is useless, I'd say no-- early polling is reflecting the dynamics of the race. But the dynamics are highly fluid and the point of the polling is not to predict the winner from today's polls, but to understand how the race is moving and ultimately to look back at how we got to the final outcome.
The great mistake analysts make is to look at current polls and conclude from them that the dynamics are fixed. That Dean can't rise. That Dean is a lock. That Kerry was inevitable after all.
The current Democratic race appears, as of last week's polling, to be relatively static. And compared to the Republicans that is certainly true. But let's not jump to the conclusion that the polls after Labor Day have to look like today's just because today's look like June 1. The polls are of interest for what they show about the history of the race so far and how it stands today. Not for their ability to predict what happens in a month or two.
Cross-posted at Political Arithmetik.
By Charles Franklin on July 30, 2007 8:13 PM
| Permalink
| Comments (11)
| TrackBacks (0)
March 27, 2007
By Mark Blumenthal
Onward with my Primary Polling Primer. In the first
two
installments, we talked about turnout, or rather, how national polls tend to sample
a broader population than actually participates in the presidential primaries
and caucuses. Today, I want to turn to something even more important to the
"accuracy" of those national vote preference polls: Timing.
The presidential primaries and caucuses are a dynamic
process. Unlike virtually every other election that pollsters ask about, the
selection of presidential nominees does not occur on a single election day. Rather,
the nominating process consists of a series of statewide primaries and caucuses
that plays out over the first few months of every presidential election year.
Here is the critical point: A few early primaries,
especially the Iowa caucuses and the New Hampshire primary,
play a huge role in influencing voter preferences in the states whose primaries
and caucuses follow.
Go back and read that last sentence again, because the basic
idea is hard to overstate, especially in years (like 2008) without an incumbent
president seeking reelection. Moreover, since the outcome of the Iowa caucuses and New
Hampshire primary tend to reshuffle voter
preferences, they can render the "standings" of earlier horse race polling more
or less moot.
Over the next few posts of this series, I will try to
provide some data to demonstrate the influence of those events and how they
often cause a significant reshuffling of voter preferences.
For today, consider the last Democratic nomination battle in
2004, which produced one of the most dramatic shifts. Fortunately, for those of
us who obsess about such things, we have an incredible resource in the National
Annenberg Election Study (NAES), which
conducted a rolling tracking survey from October 2003 through November 2004. Kate Kenski, now an
assistant professor at the University
of Arizona, complied over
7,000 interviews conducted among respondents who were planning to vote in the
Democratic presidential primaries and caucuses in 2004 and prepared the
following graphic:
As the chart makes clear, the Iowa
caucuses and New Hampshire
primary turned the 2004 Democratic contest upside-down. Just before Iowa, according to the
ANES data, only 9% of those planning to vote in the Democratic primaries or
caucuses nationally supported Kerry. Kerry shot up after his Iowa
caucus victory, approaching 50% in the week before New Hampshire and hitting 68% by the end of
February. At the same time, Howard Dean's support collapsed, falling from 31%
in early January to near zero by March, while John Edwards, who finished a
surprisingly strong second in Iowa,
saw his national support grow from 5% to 25%.
Kenski's analysis (which she presented at the 2004 meeting
of the American Political Science Association) indicates that the impact of Iowa and New
Hampshire that year had less to do with increased name
recognition and more with changing voter evaluations of the candidates. Nationally,
Kerry experienced a name recognition gain of roughly ten percentage points nationally
after Iowa,
but he was already known to more than 80% of Democratic primary voters
beforehand. Kenski's analysis showed more dramatic shifts in voter ratings of Kerry and Dean. In other
words, voters who already knew the candidates changed their minds about them
after Iowa and New Hampshire. The NAES surveys also showed
a dramatic shift in perceptions of Kerry and Dean's chances, respectively, of winning
the Democratic nomination that neatly paralleled shifts in voter preference.
The 2004 Democratic race - as illustrated by the NAES data -
is arguably one of the most dramatic examples of a phenomenon typical in the
presidential primary season. Those who win early contests get a boost that
helps them in subsequent contests. Those who study primary elections may disagree
about the reasons for that "momentum" (see this review
just last Sunday by the Boston Globe's
Drake Bennett), but my sense is that it is more than just a blind "bandwagon"
effect. Here are two reasons the early primaries shake up the race:
First, they effectively winnow the field. While news
accounts may emphasize the large field of candidates before the early primaries
(with an emphasis on front-runners), the coverage afterwards focuses far more
intensely on the early winners. The result is that also-ran candidates either
drop out entirely, or effectively drop out of site, a process that simplifies voter
choices. If a voter had been torn between two candidates, and one of those
candidates wins an early primary while the other finishes far back in the pack,
that voter's decision gets much easier.
Second, and probably far more important, the horse race
nature of the coverage leads voters to more positive evaluations of the winners
and more negative evaluations of the losers. Media accounts portray the winners
and their campaigns as competent and able, while the losers look hapless and
faltering. Which set of characteristics would you want in a president? Not
surprisingly, voters readily make the connection between winning and
competence.
Either way, the implication is that the current horse-race
preference numbers are not particularly meaningful as predictors of the
outcome. An early loss is not necessarily fatal. Many early front runners
(Reagan, Mondale, Clinton, both Bushes) have lost early primaries and bounced
back to win their nominations, but the early primaries almost always change
voter preferences.
So what effect will the especially "front-loaded" primary
calendar have in 2008? I'll take that up in the next installment.
By Mark Blumenthal on March 27, 2007 12:50 PM
| Permalink
| Comments (3)
| TrackBacks (0)
February 5, 2007
By Mark Blumenthal
I need to take a moment to note something I overlooked during
the very busy final days of last fall's campaign. In late October, the National Annenberg Election
Survey (NAES) released its complete respondent level data collected during
the 2004 campaign, which can now be purchased for $35 on CD-ROM along with a textbook on how to analyze the data,
Capturing Campaign
Dynamics 2000 & 2004.
For those unfamiliar, NAES is an academic survey project
that conducted a nightly telephone tracking survey throughout both the 2000 and
2004 campaigns. The 2004 tracking started in October 2003 and continued nightly
through mid-November 2004, completing 81,422 interviews as part of their
national "rolling cross sectional" design. The details are explained on their methodology
page, but the gist is that they applied the most rigorous methods of
telephone survey research, and the net result is (as the book jacket claims) "the
largest studies ever undertaken of the American electorate. It averaged more
than 160 interviews per day (about 5,000 per month) during the primary season,
and 300 per day (nearly 9,000 a month) during the final two months of the
campaign.
The 2004 project also includes additional data sets: NAES interviewed
special "over-samples" among voters in New
Hampshire just before the primary and in the households
of in active duty military personnel in the fall. They also conducted panel
studies (which interview the same individuals before and after an event) to
track changing opinions during the political conventions, the debates and the
general election itself.
And to be clear, the CD that comes with the new book includes
all of the data from both the 2000 and 2004 surveys, over 200,000 interviews in
all. As a tool for academics and political junkies that know their way around
statistical software, there is nothing else quite like it.
By Mark Blumenthal on February 5, 2007 2:28 PM
| Permalink
| Comments (0)
| TrackBacks (0)
December 21, 2006
By Charles Franklin
(Click the image for a full resolution version.)
I've had so much fun with the new oversize color laser printer that I felt the need to share another political graphic suitable for framing for the holidays. This one compares the 2000 and 2004 presidential campaign dynamics. I'll let you add your own commentary, suitable for either party.
An 11 x 17 version is available here for download (in .pdf format) for the political junkie on your shopping list. (If you lack a large color printer, I bet your local copy shop can print this for a modest fee.)
Cross-posted at Political Arithmetik.
By Charles Franklin on December 21, 2006 12:11 PM
| Permalink
| Comments (0)
| TrackBacks (0)
