August 27, 2008
By Mark Blumenthal
Following up on yesterday's post on the Nielsen ratings for the first night of the convention. A.C. Nielsen sends a press release pointing to their new blog site "with background information on political, Olympics and other viewing information."
Here's the latest:
1) Hillary's night (26.0 million viewers) had higher ratings than Michelle's night (22.3 million viewers).
2) African Americans continue to watch the convention in a higher proportion than the rest of the population (the African American rating, or percentage of the African American population watching, was 12.7 vs. a 9.0 for the population as a whole)
3) Almost five times as many people (26.0 million) watched Day Two coverage in 2008 vs. Day Two in 2004 (5.9 million) when only the cable networks covered the convention.
By Mark Blumenthal on August 27, 2008 5:07 PM
| Permalink
| Comments (1)
| TrackBacks (0)
By Mark Blumenthal
How is the Obama campaign using surveys and other data to guide their strategy? What do they think about national polling generally, and the Gallup Daily tracking in particular? This morning, I got an earful on both subjects at an on-the-record briefing by Obama campaign manager David Plouffe, communications director Dan Pfeiffer and campaign advisor Anita Dunn for a dozen or so editors and executives of the Atlantic Media company and the National Journal Group.
My colleague Marc Ambinder has already blogged some highlights from the session. Let me fill in a few more details that touched on the campaigns use of polling, research and targeting.
First, Ambinder reported on this exchange on national polling:
We tried to get Plouffe to react to a spate of national polls showing a tightening race.
"All we care about is these 18 states," he said. He repeated, with emphasis, that the campaign does not care about national polling. Instead, the campaign's own identification, registration and canvassing efforts provide the data he uses to determine where to invest money and resources.
Plouffe also emphasized that the internal polling the campaign does is focused on those same 18 states, and that their real concern is not the horse race results but the "data underneath." Later, he added, "the top-line [polling data] doesn't tell you anything." Rather, they focus on who the "true undecideds" are, "how they're likely to break," and what messages will best persuade them.
The Gallup Daily tracking poll is apparently a particular sore point. When asked whether they were unhappy that the Biden announcement had not produced a bounce in national polls, Plouffe shot back: "How do you determine a bounce. . . from the Gallup Daily?" The Gallup Daily, he added is "something we don't pay attention to," he said again.
Communications director Dan Pfieffer later put it more bluntly, expressing unhappiness with the "inordinate focus on bad polling" by the media and also in the routine misinterpretation of sampling noise in the Gallup Daily poll. "The Gallup Daily is the worst thing that's happened in journalism in 10 years," he said.
Plouffe also warned against "making too big an assumption" based on focus groups when asked about the Frank Luntz group of undecided voters that received a fair amount of attention this week. "We certainly don't use [focus] groups to make assessments of swing voters," he said. They conduct focus groups, mostly "to hear people talk" about the issues and candidates, but when it comes to identifying "true undecided" voters, their emphasis is on quantitative data, including traditional surveys and data on registration and vote history collected from lists and supplemented with information gleaned through direct voter contact.
I asked about Marc Ambinder's report of the "data" collected by the Obama campaign Monday night that left them with a "high degree of confidence" that Michelle Obama's speech went over well in their 18 target states. Marc had inferred that "the campaign ran several focus groups" Monday might, but I'm skeptical given the logistical challenge of doing traditional focus groups in 18 states in one night. My guess is that they ran some sort of online test, and asked Plouffe if he could add more detail:
That information "will have to be a mystery," enjoying the play on my nomme de Internet. He did say that their efforts to monitor undecided voters features "not through the traditional methods of quantitative and qualitative research. We also have hundreds of thousands of contacts [made] every night."
Much of the briefing covered specifics on the focus on turnout by the Obama campaign and their massive effort to "adjust the electorate" to their benefit. He cited several examples, including Florida where he claimed that roughly 600,000 African Americans that were registered but did not vote in 2004, with more than half of that group coming from African Americans under 40 years of age. "If we just execute on turnout" in Florida, he said, "we're going to be bumping up on our win number." They also believe they can keep states like Virginia and North Carolina competitive if they "blow the doors off turnout."
The briefing included much more that my National Journal and Atlantic Media colleagues will be reporting on later today and this week, and I will try to add links here as they become available. I'm also likely to say more at some point about the Obama campaign's overall approach to research and strategy based on these comments.
Finally, please note that the verbatim quotations above are from my notes. We are hoping to post a full transcript later in the week. [Update/Correction: With a transcript in hand, I have corrected a few minor wording errors. In the original version, I erroneously quoted Dan Pfeiffer describing the Gallup Daily as "the worst thing that's happened in journalism in 20 years" -- he actually said 10 years.]
Update - Here's a quick response via email from Gallup's Frank Newport:
These are the same types
of sentiments that have been expressed since George Gallup's first
presidential polls in 1936. Campaigns like to control the narrative, and don't
like outside intrusion in their story lines. Bottom line: The American
public is vastly interested, and always has been, in where a presidential race
stands during a campaign. Gallup (and others) can help provide a
scientific answer to that question, using careful methodology and deliberate
analysis. Without independent polling, the public would be reliant on
campaign operatives' self-promoting insights on where the race stands, or
on journalists' guesses. And, of course, polling provides a vast
array of insights into the dynamics and currents of a campaign and represents
the voters' views, thoughts, and wishes.
By Mark Blumenthal on August 27, 2008 1:53 PM
| Permalink
| Comments (28)
August 26, 2008
By Mark Blumenthal
The Page is linking to a report from the LA Times blog on the Nielsen ratings from last night:
The opening night of the Democratic National Convention drew more than 22 million viewers, a 20% larger audience than in 2004, according to Nielsen Media Research.
If you want even more detail on the numbers (and I know you do) check out this post on the very helpful blog, TV By The Numbers.
To keep all of these numbers in perspective, consider that 22 million is double the 10.7 million that watched ABC's broadcast of the April 16 Clinton-Obama debate in Pennsylvania and five to twenty times the audience size of the many debates broadcast in 2007.
On the other hand, 22 million is still a far cry from the 62 million that watched the first Bush-Kerry debate in 2004.
These numbers show us that while the conventions are the most watched political events so far this cycle, they are still not quite the voters-as-jury experience that we sometimes assume. The news coverage that excerpts speeches and convention "moments" reaches a far bigger cumulative audience. Those of us interested in measuring the impact of the debates need to allow time for Americans to view that coverage, absorb it and sleep on it for a few days.
In other words, be patient and stop worrying about the "bounce" (or lack thereof) in daily tracking. Conventions matter, but the response we are interested is not necessarily instant.
By Mark Blumenthal on August 26, 2008 9:38 PM
| Permalink
| Comments (1)
By Mark Blumenthal
My second National Journal column for the week (which will appear in tomorrow's Convention Daily) will be posted within the next few hours. Since it is all about the opportunities presented by Hillary Clinton's speech tonight, I'll post a key block quote now and add the link later. Update: the full column is now live.
The gist is that I disagree with the "Hillary can't win," damned if she does, damned if she doesn't them of Marie Cocco's column in this morning's Washington Post (echoed to some extent by Todd, et. al. in FirstRead). I think the speech presents Clinton with a huge opportunity, both for her own long term interests and for the Obama-Biden ticket. I make the case with survey data in the column. While an Obama-Clinton ticket would have come with risks to offset benefits, the same cannot be said for tonight's speech. Quoting myself:
And the decision by the McCain campaign to release (if not air) three different television advertisements this week invoking Clinton's criticisms of Obama during the primaries provides her with a huge tactical opportunity to create one of the convention's most memorable moments.
"I'm Hillary Rodham Clinton, and I do not approve of that message," she told the New York state delegation yesterday. In her speech, she can do more. I am not a speechwriter, but the "truth hurts" tagline of the first of these spots seems like an obvious opening for a riff on the records of McCain and President Bush.
We'll see how it turns out.
PS: Nate Silver made a very similar point about the Clinton-quoting McCain ads earlier in the week:
I could see the ad being very effective. But it also tosses a big
softball to Hillary Clinton, who will speak to a national audience on
Tuesday. The risk to the Republicans can be summarized in five words:
"Shame on You, John McCain". A finger-wagging, how-dare-you moment by
either of the Clintons at the convention -- but especially Hillary --
could be both effective and therapeutic, especially when coupled with a
reminder that McCain voted against measures like SCHIP (and voted to
impeach her husband).
I prefer big "hanging curve ball," but I'll defer to the baseball guy.
PPS: I'm catching up on my RSS feed while listening to the speeches. This post yesterday from Marc Ambinder seems relevant to what Clinton can help accomplish (emphasis added):
They are, yes, Hillary supporters, but a certain type of Hillary
supporters: mainly white voters without college degrees. Ron Brownstein
has noted
that in four polls taken before the convention, Obama sits at 38% with
this group. These voters, as pollster Stan Greenberg's new data shows,
have a panoply of concerns. Unquestionably, some are racist. But a
majority of them worry about Obama's credentials, his liberal positions
on national security issues, and whether he truly understands their
economic insecurities.
It is much easier to convince these
voters to vote for Obama when they see Obama as the antidote to the
Bush presidency, and when they see McCain as a Bush Republican. SO --
you will hear and see speaker after speaker portray McCain as a Bush
Republican. Polling shows that even when recalcitrant Democrats learn
about Obama's middle class roots, they're still skeptical. It is MUCH
harder to convince them to vote for Obama because they LIKE him. It is
much easier to convince them to vote for Obama because they think
McCain represents a continuation of President Bush's policies. (Obama's
campaign has polling data suggesting that an unusually large number of
pro-choice Democrats don't know that McCain is pro-life.)
By Mark Blumenthal on August 26, 2008 6:47 PM
| Permalink
| Comments (5)
August 25, 2008
By Mark Blumenthal
Here's the link Convention Daily edition of my National Journal column for today, which covers the difficulty pollsters will have measuring the "bounce" that Barack Obama gets this week. The short version: There will be many polls this week but no way to measure the bump or bounce that is comparable to past years. So it's probably best to dispense with metaphysical comparisons to years past and just focus on what surveys tell us about what voters are learning and what conclusions they are coming to.
Incidentally, I will be writing two Conventional Daily columns for the National Journal this week (the second will appear on Wednesday morning), and two next week. In exchange for this contribution, I get to be on hand in Denver this week and in Minneapolis next week.
I've been traveling most of today and just got to our work space. Soon I'll be off to wander the hall, and seek out pollsters and their wisdom...
By Mark Blumenthal on August 25, 2008 7:47 PM
| Permalink
| Comments (2)
August 22, 2008
By Mark Blumenthal
So I went off the the beach with spouse and kids a few hours ago ("vacation," remember?), assuming that I'd come back to news on Obama's running mate. But as of the moment I clicked "publish" on this entry, nothing had been announced.
The reason a lot of us assume the announcement is imminent is the report that has been airing on CNN all day that a "highly placed Democratic source" who says Obama "called some people on his shortlist for the vice presidential slot Thursday night to tell them he had not selected them as a running mate." Usually, when the phone calls start, the news leaks out almost immediately.
My colleague Marc Ambinder confirms that Obama has called some "who were vetted by didn't quite make it." He adds: "Maybe these aren't the short-listers. Maybe these are the long-listers."
"What the hell is taking so long," Noam Scheiber asks? In retrospect, I think the reasons for the timing seem obvious to me. The slow drip of "news" is entirely consistent with maximizing response to their "Be the First To Know" email/text message campaign.
Think about it: The week starts with a leak to the New York Times generating a front page story that tells us:
Senator Barack Obama has all but settled on his choice for a running mate and set an elaborate rollout plan for his decision, beginning with an early morning alert to supporters, perhaps as soon as Wednesday morning, aides said.
Somehow, Matt Drudge hears about the article the day before and gives the news the full Drudge treatment. Meanwhile, the Obama campaign runs banner ads on web sites all over the Internet promoting the email text message alerts. Then after days of speculation, Obama confirms yesterday that he has made a decision. Today his campaign confirms to reporters that some potential running mates have been called. And nearly every story features some reference to the fact that the campaign will share its news via email or text messaging.
Coincidence? I think not.
PS: One of Marc Ambinder's readers points out the irony of CNN "begging viewers to stay tuned so CNN can bring them coverage of a text message." Another "triumph of new media" is the timing itself: Major campaign news timed not for the evening news, but (perhaps) for prime text messaging time.
By Mark Blumenthal on August 22, 2008 6:11 PM
| Permalink
| Comments (32)
By Mark Blumenthal
Here's a lesson: If you work in politics (a) don't try to take a vacation in August of an even numbered year and (b) if you do, leave your laptop behind. Having made both mistakes, and waiting like everyone else to see who Barack Obama has selected as his running mate, I have a reaction to the one piece of real news we got yesterday about the choice.
In interviews published in the last 24 hours, Barack Obama has implied that his choice leans toward someone who will balance him ideologically. He has decided on "somebody who's independent," he told USA Today, "somebody who can push against my preconceived notions and challenge me so we have got a robust debate in the White House."
The conventional wisdom about vice-presidential choices shifted a bit in 1992 when Bill Clinton picked Al Gore, arguably the most successful vice presidential selection of the last several decades. Gore's selection was widely viewed as reinforcing Clinton's key strengths rather than providing geographic or ideological balance. Clinton, a young, Southern centrist Democrat bucked the conventional wisdom about ticket balancing and picked another young, Southern centrist politician. The combination reinforced the central "change" message of the 1992 campaign and helped provide a huge and sustained boost to the Clinton-Gore ticket.
But if you look back, Clinton had a real need to reinforce his core "change" message. Before the 1992 Democratic convention, Clinton had net negative favorable ratings and was running behind George H.W. Bush and (at on some polls) independent candidate Ross Perot. Voters had been introduced to Clinton during the primaries through news about an alleged affair and efforts to avoid the draft while a student at Oxford and Yale. In his book, Middle Class Dreams , Clinton pollster Stan Greenberg recounts learning from his research that doubts about Clinton focused on the perception that he was a typical politician from a privileged background.
To address the perception of privilege, the Clinton campaign used the convention to emphasize the "Man from Hope" story of Clinton's modest upbringing. My sense is that the Gore selection helped counter the perception of Clinton as a younger, but otherwise typical pol. Rather than making the predictably "political" choice (an grey eminence with years of Washington experience), he picked another young Southerner (albeit one with considerable Washington experience). So in picking Gore, Clinton was, in a sense, shoring up a weakness, making that case that his election really would be a break with politics as usual.
Now consider Obama. He owns "change." Between his age, his race, his name, his unusual background, his limited time in Washington and his campaign's exceptional message discipline, Obama has no need to convince anyone that his presidency will be different or that he "really likes change." What voters doubt most is whether he is prepared to be president, and perhaps whether he is a bit too taken with the "audacity" of his own candidacy. The NBC/Wall Street Journal poll shows that 22% of voters choose "arrogant and cocky" as the negative characteristic that best describes Obama.
So Obama reaches out to someone of considerable experience with whom he disagreed on the Iraq War, someone with a different political philosophy or someone with proven willingness to challenge him, he can help shore up a weakness with relatively little risk to his core brand. At least that strikes me is the logic behind the kind of pick Obama is telegraphing.
By Mark Blumenthal on August 22, 2008 9:03 AM
| Permalink
| Comments (9)
August 18, 2008
By Mark Blumenthal
Interrupting my "vacation" for this special announcement: My NationalJournal.com column, on the potential for a running-mate "bump" and whether we will be able to measure one, is now online.
By Mark Blumenthal on August 18, 2008 5:38 PM
| Permalink
| Comments (0)
| TrackBacks (0)
August 17, 2008
By Mark Blumenthal
I will be taking a break this week, although I filed a National Journal column for the week, which should appear in a day or two. Meanwhile, Eric will continue with poll updates and our other contributors should be active this week. See you next week!
By Mark Blumenthal on August 17, 2008 9:49 PM
| Permalink
| Comments (0)
| TrackBacks (0)
By Mark Blumenthal
"Bounce" or "Bump?" The terminology is up to you, but this is certainly the season to consider the short term changes in polling numbers that frequently result in the wake of national political conventions.
A useful first stop would be the 2004 analysis from Gallup's Jeff Jones. It includes the post convention "bounce" numbers from Gallup back to 1964 that are the primary source of the conclusion that average post convention gain for candidates has been six or seven percentage points.
Last week, Tom Holbrook posted a more a thorough review of the past "before and after" data and it's implications. He looks at the gain for each candidate by taking their "average share of the two-party vote in trial-heat polls conducted six days to two weeks prior to the start of the convention" an subtracting that from the candidate's "share of the two-party vote in polls conducted during the seven days following the close of the convention." He does not list the polls used for each year, but presumably his data looks at much more than the Gallup time series for more recent elections.
Holbrook's post is worth reading in full for the lessons he draws from the considerable variation in past convention bumps, although probably the most important is his caution that "the magnitude of the convention bump is not a great predictor of election outcome." Still, he sees a pattern to the past variation that "should be a useful guide to what to expect" from the conventions and promises to update later this week with a prediction for each candidate.
But before reading two much into the twitches in the daily tracking polls over the next three weeks, please read the latest column from CBS News director of surveys Kathy Frankovic. She reminds us that the gap between the Democratic and Republican conventions is just three days -- much shorter than in past elections -- and will coincide with the Labor Day weekend:
Will we even be able to measure whatever impact the Democratic Convention has on Obama before it’s time to measure the GOP convention’s impact on John McCain? And will we be able to sort out what has caused what?
Will we actually discover a “bounce” or a “bump?”
Probably not. Polling over Labor Day Weekend is always a problem. We confront more than the usual number of people who don’t respond or can’t respond. People are away from their homes, heading back from summer vacation, or preparing their children for the start of the school year. In addition, the focus will shift so quickly from the Democrats to the Republicans that whatever opinions might be expressed over Labor Day Weekend might not last too long.
I thought it would be helpful to look at some past elections, not in terms of the immediate "before and after" averages, but rather where the conventions fit into the longer arc of the trend in the vote preference over the course of the election year. I spent some time gathering past polls from a variety of sources and asked Charles Franklin to create some charts matching the format he used to look at the trends of the 2000 and 2004 elections a few weeks back.
I had hoped to use those charts for a series of posts. Unfortunately, I got delayed, so I will post the charts along with a some very compressed discussion after the jump.
Continue reading "Convention "Bumps" in Context"
By Mark Blumenthal on August 17, 2008 4:49 PM
| Permalink
| Comments (2)
August 13, 2008
By Mark Blumenthal
In the first two installments of this online dialogue, I asked a question we have heard from readers about why we choose the results for "likely voters" (LVs) over "registered voters" (RVs) when pollsters release both. Charles answered and explained our rationale for our "fixed rule" for these situations (this is the gist):
That rule for election horse races is "take the sample that is most likely to vote" as determined by the pollster that conducted the survey. If the pollster was content to just survey adults, then so be it. That was their call. If they were content with registered voters, again use that. But if they offer more than one result, use the one that is intended to best represent the electorate. That is likely voters, when available.
Despite my own doubts, I'm convinced by the rule for this reason: I can't come up with a better one. Yes, we would arbitrarily choose RVs over LVs until some specified date, but that would leave us still plotting numbers from pollsters that only release LV samples. And on which date do we suddenly start using the LV numbers? After the conventions? After October 1? What makes sense to me about our rule, is that in almost all cases (see the prior posts for examples) it defers to the judgement of the pollster.
Several readers posed good questions in the comments on the last post. Let me tackle a few. Amit ("Systematic Error") asked about how likely voters are constructed and whether we might be able to plot results by "a family of LV screens (say, LV_soft, LV_medium, LV_hard)" and allow readers to judge the effect.
I wrote quite a bit back in 2004 about how likely voter screens are created, and a shorter version focusing on the Gallup model two weeks ago. One big obstacle to Amit's suggestion is that few pollsters provide enough information about how they model likely voters (and how that modeling changes over the course of the election cycle) to allow for such a categorization.
"Independent" raised a related issue:
Looking at the plot, it appears that Likely Voters show the highest variability as a function of time, while Registered Voters show the least. Is there some reason why LVs should be more volatile than RVs? If not, shouldn't one suspect that the higher variability of the LV votes is an artifact of the LV screening process?
The best explanation comes from a 2004 analysis (subs. req.) in Public Opinion Quarterly by Robert Erikson, Costas Panagopoulos and Christopher Wlezien. They found that the classic 7-question Gallup model "exaggerates" reported volatility in ways that are "not due to actual voter shifts in preference but rather to changes in the composition of Gallup's likely voter pool." I also summarized their findings in a blog post four years ago.
Finally, let me toss one new question back to Charles that many readers have raised in recent weeks. The two daily tracking surveys -- the Gallup Daily and the Rasmussen Reports automated survey -- contribute disproportionately to our national chart. For example, we have logged 51 national surveys since July 1, and more than half of those points on the chart (27) are either Gallup Daily or Rasmussen tracking surveys. Are we giving too much weight to the trackers? And what would the trend look like if we removed those surveys?
By Mark Blumenthal on August 13, 2008 2:26 PM
| Permalink
| Comments (3)
| TrackBacks (0)
August 12, 2008
By Mark Blumenthal
I want to add one thought the chorus of commentary on Josh Green's Atlantic Monthly article on the Hillary Clinton campaign, based on a remarkable collection of email and memoranda he obtained from sources within the campaign. It concerns the first sentence in an April 25 email from newly installed pollster Geoff Garin to the Clinton high command:
Attached is the filled in questionnaire from the North Carolina survey.
Those ten words probably seem utterly mundane to the ordinary reader, even to the ordinary campaign consultant. Pollsters share results with their clients. It's a basic part of the job. Notice also that Garin sent his email at 7:25 a.m. on a Friday morning. The timing and content imply that he was sharing the most critical "top line" results of a tracking survey that had completed the night before.** Thus, this email shows us Garin passing along results as soon as he has them for review by other decision makers. Further analysis and internal discussion no doubt followed.
What makes Garin's ordinary act so remarkable is that Mark Penn, the original Clinton pollster and "chief strategist" rarely delivered a "filled in questionnaires" to the Clinton campaign's senior decision makers. I know this because I heard the story a few months ago from a Clinton staffer with first-hand knowledge of what Penn provided to the campaign (who agreed to share the story on condition of anonymity). My source said that Penn would routinely brief strategy sessions without providing the complete results of the poll in advance. Instead, he would present whatever results best made his case (as exemplified by the the smattering of numbers that appear in the Penn memoranda that accompany Green's article).
Perhaps Hillary and Bill Clinton received the full data, but the senior staff and consultants did not. Amazingly, I am told, Penn also initially refused to share the full cross-tabular reports (the reams of tables like this one showing results to every question by every subgroup of interest), as is also standard practice among campaign pollsters. It was not until relatively late in the campaign at the insistence of then campaign manager Patti Solis Doyle that Penn relented, sharing a hard copy of the cross-tabs on condition that Solis Doyle keep it locked up in a file cabinet in her office.
One can understand the temptation that a "chief strategist" might have to control the flow of data. If you are convinced you have the right strategy, and you make the final decision, why give others a tool to question your judgment?
The problem with that approach should be obvious. It poisons the environment within which functional campaigns privately hash out disagreements and reach consensus about strategy. The pollsters job in this process is to put the data on the table, to provide analysis and guidance about that data, but also to let other senior staffers examine and question it. When the pollster wears two hats -- pollster and "chief strategist" -- greater conflict, questioning of motive and campaign "dysfunction" are inevitable.
**One reason I'm confident that this email followed within hours after completion of calling is that one of the respondents later blogged about his experience (discussed here). The respondent reported having been called a night or two before Garin sent his email.
By Mark Blumenthal on August 12, 2008 4:25 PM
| Permalink
| Comments (7)
| TrackBacks (0)
August 11, 2008
By Mark Blumenthal
Since adding the maps and upgrading this site, we have received a number of good questions about how the charts and trend lines work and why we choose to include the poll results that we do. I want to answer a few of those questions this week before we call get swept up in the conventions and the final stretch of the fall campaign.
Our approach to charting and aggregating poll data follows the lead and philosophy of our co-founder Charles Franklin. And while I am tempted to describe that approach as well entrenched, the reality is that in many ways it has and will continue to evolve.
Since launching this site nearly two years ago, Franklin and I have continued to discuss (and occasionally debate) some of the technical issues offline. Most of the time we agree, but I tend to propose ways to change or tinker with our approach, and Franklin usually succeeds in convincing me to stay the course.
In considering some of issues that came up more recently, I thought it might be helpful to take this dialogue online. Hopefully, we can both answer some of the questions readers have asked and also seek further input on those issues we have not completely resolved.
So with that introduction out of the way, here is the first question for Franklin:
Over the last few weeks, in commenting on the "likely voter" subgroups reported by Gallup and other national pollsters, I have essentially recommended that we focus on the more stable population of registered voters (RV) now, and leave the "likely voter" (LV) models for October (see especially here, here, here and here). Yet as many readers have noticed, when national surveys publish numbers for both likely and registered voters, our practice has been to use the "likely voter" numbers for our charts and tables.
Why?
Like the other sites that aggregate polling results from different sources, we face the challenge of how to best choose among many polls that are not strictly comparable to each other. Even if we examine data from one pollster at a time, we will still see methodological changes: Many national pollsters will shift at some point from reporting results from registered voters to "likely voters." Some will shift from one likely voter "model" to another, or will tinker with the mechanics of their model, often without providing any explanation or notice of the change. And no two pollsters are exactly alike in terms of either the mechanics they use or the timing of the changes they make.
As such, two principles guide our practices for selecting results for the charts and tables: First, we want to defer to each pollster's judgement about the most appropriate methodology (be it sample, questionnaire design or the most appropriate method to select the probable electorate). Second, we want a simple, objective set of rules to follow in deciding which numbers to plot on the chart.
In that spirit, when pollsters release results for more than one population of potential voters, our rule is to use the most restrictive. So we give preference to results among "likely" voters over registered voters and to registered voters over results among all adults. In almost all cases, the rule is consistent with the underlying philosophy: The numbers for the more restrictive populations are usually the ones that the pollsters themselves (or their media clients) choose to emphasize.
But there have been some notable exceptions recently, of which, last month's ABC News/Washington Post poll provided the most glaring example. ABC News put out a report and filled-in questionnaire with two sets of results: They showed Barack Obama leading John McCain by eight points (50% to 42%) among registered voters, but by only three points (49% to 46%) among likely voters. Following our standard procedure, we included the likely voter numbers in our chart.
However, ABC News emphasized the eight-point registered voters numbers in the headline of their online story ("Obama Leads McCain by Eight But Doubts Loom"). Within the text, they first reported the registered vote numbers and then used the likely voter results to argue that "turnout makes a difference." The 8-point lead also made the headline of the Washington Post story, but they did not report the likely voter results at all, either in the text of the story on in their version of the filled-in questionnaire.
So in this case, the news organizations that sponsored the poll clearly indicated that the RV numbers deserved greater emphasis, yet we followed our rule and included the LV numbers in our charts.
Charles, in cases like these, should we find a way make an exception? And why not just report on "registered" voters until after the conventions?
Update: Franklin answers in Part II.
By Mark Blumenthal on August 11, 2008 5:59 PM
| Permalink
| Comments (3)
| TrackBacks (0)
By Mark Blumenthal
The punditry is crackling this morning over remarks by Howard Wolfson, Hillary Clinton's campaign communications director, over what might have happened had John Edwards' been forced out of the presidential race last year: "I believe we would have won Iowa, and Clinton today would therefore have been the nominee," Wolfson told ABC News.
Washington Post polling director Jon Cohen did the logical thing and checked relevant survey data from Iowa:
It is a pure hypothetical, of course, and the entire dynamics of the contest would have been different without Edwards. But the public data do not bolster the notion that Clinton would have won.
In the networks' Iowa entrance poll, 43 percent of those who went to a caucus to support Edwards said Obama was their second choice, far fewer, 24 percent said they would support Clinton if their top choice did not garner enough votes at that location. The remainder of Edwards' backers said they would be uncommitted under such a scenario, offered no second choice or said they preferred someone else.
Nor was Clinton the obvious second choice among Edwards supporters in Post-ABC pre-election Iowa caucus polls in July, November or December. In July, for their alternate pick, Iowans split 32 percent for Obama to 30 percent for Clinton. In November, Obama led 43 to 26 percent as backup pick, and he had a slight 37 to 30 percent edge in December.
Nate Silver echoes that last point point and notes that, looking at the trend lines in late January, "Barack Obama appeared to get the lion's share of Edwards supporters once Edwards dropped from the race."
By Mark Blumenthal on August 11, 2008 12:15 PM
| Permalink
| Comments (6)
| TrackBacks (0)
August 8, 2008
By Mark Blumenthal
Kathy Frankovic reminds us that despite "record levels of participation and interest...voters may be doing other things in July and August."
Mark Mellman sees the din of poll analysis drowning out an opportunity to educate.
Lydia Saad explains why the election is not just about Obama and why McCain is keeping it close.
John Sides reminds us that the forecasting models predict a close election.
Nate Silver examines whether Barack Obama has been "underachieving."
Jennifer Agiesta compares low-wage workers interviewed in cell-phone-only households with those from land-line telephone households.
Frank Newport says religious affiliation remains an important predictor of vote choice
Todd Domke reviews the factors that will make for unpredictable polls in 2008.
Andrew Gellman shows that from 1948 to 1992, the popular vote showed little "bias" in it relationship with electoral votes (and links to a paper finding much the same in 2000 and 2004).
Tom Jensen identifies the "tipping point" for Obama in North Carolina African American turnout.
Craig Garthwaite and Tim Moore [pdf] say Oprah Winfrey was responsible for "an additional 1,000,000 votes" for Barack Obama (via John Sides).
David Hill chides GOP consultants for second guessing the McCain campaign.
Jay Cost dives deep into the Pennsylvania numbers.
Josh Goodman has a polling wish list.
The New Yorker's Bruce McCall has the latest from the all-important New Orleans Times-Picayune/Bravo/Popular Mechanics poll..and more!
And from our regular contributors...
Charles Franklin compares polling trends in 2000, 2004 and 2008
Margie Omero finds Barack Obama's support from women anything but "lackluster."
Steve Lombardo says the race is closer than you think.
Kristen Soltis makes a case for weighting by party.
David Moore and guest contributor Nick Panagakis continue to debate how to best measure "undecided."
By Mark Blumenthal on August 8, 2008 5:33 PM
| Permalink
| Comments (1)
| TrackBacks (0)
By Mark Blumenthal
Here are a few short but relevant updates to topics covered earlier in the week that I did not want to get lost at the bottom of older posts:
- Yesterday, here and in my column, I looked closely at the small percentage (10%) of 18-to-29-year-olds among the "likely voters" in the most recent USA Today/Gallup poll. Later, I also noted the different approach to modeling likely voters taken by the recent Time/AbtSRBI poll that appears to reduce the volatility in these early numbers.
One thing we overlooked in the Time poll: The self-identified "registered" voters included an even smaller percentage of 18-to-29-year-olds (9% - see QF1) than the "likely voters" in the USA Today/Gallup survey (10%), and six points fewer than the self-ID'd registered voters in the Gallup survey (15%).
- Earlier in the week, I also pointed to some data from the News Index surveys by the Pew Research Center to make the point that most voters in July are not following the campaign as a jury follows a trial. This passage in the CBS News analysis of their follow-up survey of uncommitted voters makes the point even more clearly:
One possible reason the uncommitted voters haven’t changed much: they’re paying much less attention to the campaign in the last few weeks.
When asked in mid-July how much attention they’d been paying to the 2008 campaign, generally, 45% said they’d paid a lot and just 14% said not much or none. When asked in this poll how much attention they’d been paying in the last few weeks, only 18% reported paying a lot lately.
- Finally, University of Wisconsin-Milwaukee political science professor Tom Holbrook was thinking along the same lines as I was on Monday morning regarding the Washington Post-Kaiser-Harvard survey of low wage workers. Apologies to Tom for not linking sooner.
By Mark Blumenthal on August 8, 2008 1:04 PM
| Permalink
| Comments (4)
| TrackBacks (0)
August 7, 2008
By Mark Blumenthal
Are all "likely voter" models created equally? Not at all.
Case in point, the comment left yesterday by George Mason University political science Professor Michael McDonald about the latest Time/SRBI poll:
Continuing my war on likely voter models...
Here we have 808 "Registered Likely Voters." Q1 reports 100% of the sample is registered and Q2 reports 90% are "definitely" going to vote and 10% "probably." I guess this means that registered likely voters must have to respond affirmatively to being registered and "definitely" or "probably" to voting. This is different from Gallup, which requires likely voters to have a past history of voting and to express an interest in the campaign. There is no indication of weighting in this survey, so who knows what it going on there.
If I am correct, then this two-question likely voter model seems less biased against young voters and less volatile due to changing interest. This may explain the stability since June in this poll compared with the USAToday/Gallup poll.
Mike's theory seemed plausible, so I sent an email to Mark Schulman, CEO of Abt SRBI, the firm that conducts the Time poll. Here is his full response:
Mike, the Time sample is indeed weighted based upon the entire cross-section sample, as are most election surveys. We retain demographics for the entire sample, registered or not, and weight the entire cross-section sample on the usual Census demographic variables. The 100% you cite is the total of self-reported registered voters who are then asked about likelihood to vote. It does not include unregistered screen outs, who skip straight to the weighting demographics. I see that this can cause confusion. I'm glad that you requested this clarification.
You are correct in that we are not currently using past vote in our model. My objective in the pre-convention polling is to be fairly inclusive in the voter model until after the nominating conventions, when the campaigning starts in earnest. We're likely being a bit too inclusive with the light voter screen, but this still improves upon reporting based upon registered voters. Research on models which include "interest in campaign" and related questions finds variability in the composition of the likely voter profile during early campaign period, leading to some volatility in the estimate. This volatility is reduced as the election approaches.
We always tighten the model a notch after the nominating conventions. To be perfectly honest, I don't claim to have all the answers at this point on which approach we will use to tighten the model. I'm concerned about the likely influx of new voters, young voters, newly registered voters, newly activated voters. In 2004, we had an increase in turnout, even with an incumbent whose job rating was still just below 50% at that time. I don't have a fix at the moment on what to expect in 2008. Our plan is to consult with several leading experts in turnout models later this month and then make some decisions on which approach to take on our turnout model and targets. We're not wedded to any one approach. FYI, for internal purposes, we do break out our horse-race data by likelihood to vote to gauge the impact of smaller vs. larger turnouts.
I do wish to emphasize that we should not strictly abide by past turnout percentages reported by the U.S. Census. Our landline telephone universe is smaller than the Census CPS universe because of undercoverage. Therefore, our target turnout number will be higher than Census turnout trend data would suggest.
Thank you again for requesting this clarification.
If all of this detail confuses you, here is the short version: The Gallup Likely voter model, as applied to the last two USA Today/Gallup polls, uses self-reports of past voting and interest in the election to help identify "likely voters" (in addition to questions about registration and intent to vote). In surveys conducted before the conventions, the Time/SRBI poll does not -- it uses only questions about registration and intent to vote.
Update: Although the Time/SRBI poll uses a simplified likely voter model that should produce less volatility, their sample of registered voters managed to include an even smaller percentage of 18-to-29-year-olds (9% - see QF1) than the "likely voters" in the USA Today/Gallup survey (10%) discussed earlier, and six points fewer than the self-ID'd registered voters in the Gallup survey (15%).
By Mark Blumenthal on August 7, 2008 2:02 PM
| Permalink
| Comments (2)
| TrackBacks (0)
By Mark Blumenthal
My NationalJournal.com column for the week is now online. It revisits the nearly two-week old USA Today Gallup poll that showed a big difference between registered voters and those selected as "likely voters" with a focus on the age of the likely voter pool.
After you read the column, the following data may be of interest. First, notice that while the most recent, conducted in late July, showed a net shift of seven points between registered and likely voters, no such gap existed in the poll conducted just a month before. In mid-June, Obama led by six percentage points among both registered and likely voters.
What makes that difference interesting is the additional data generously provided by Jeff Jones of the Gallup organization showing how respondents in different age groups answered the four questions used to identify likely voters. As noted in the column, younger voters tend to score lower on all four questions. Notice that the percentage of 18-29-year-olds who said they had given "quite a lot of thought" to the election plummeted from June (60%) to July (45%). Similarly, the percentage who rated their chances of voting as a 9 or 10 on a 1-10 scale dropped ten points (from 69% to 59%).
Thoughts anyone?
Update: Nate Silver has additional thoughts. Note that the method he describes as "the most logical way to handle" the likely voter problem is, in essence, the way the CBS/New York Times poll will model likely voters in October. Their most recent release provides results for registered voters, but not likely voters.
Also, see the related comments we just posted from Time/SRBI pollster Mark Schulman.
By Mark Blumenthal on August 7, 2008 12:19 PM
| Permalink
| Comments (4)
| TrackBacks (0)
August 5, 2008
By Mark Blumenthal
Bear with me. The post that follows links to three seemingly unrelated items that will hopefully add up to a coherent point about our tendency to over-analyze day to day "change" in polls on the presidential race.
The first is Ellen Gamerman's recent Wall Street Journal feature on "whether people are telling the truth" to pollsters. She reviews the steps well known political pollsters are taking to check for the "Bradley Effect" (sometimes also called the Bradley-Wilder effect), "the idea that some white voters are reluctant to say they support a white candidate over a black candidate." The piece notes that both CBS and ABC News will be checking whether results vary with the race of the interviewer have an impact on vote preferences.
The article also includes a useful review of the work being done by academic survey researchers on whether respondents will be more honest on "self administered" surveys (those without a live interviewer). Don't miss the interactive graphic featuring audio commentary from the researchers on examples of other ways that respondents are sometimes less than honest on surveys. While there are some intriguing new findings (see especially those on "good TV" and "M&M's"), the bulk of the research on this subject warns us to watch out for circumstances where respondents tell us "what they expect [the interviewers] want to hear" (as Tim Byers, a researcher at the Colorado School of Public Health, puts it).
That possibility leads me to a second finding from last week's "News Interest Index" survey from the Pew Research Center. The result that caught my eye received no mention in their analysis, largely because it involved a question they have been tracking on a weekly basis since January that showed no meaningful change last week:
[In the past week] Did you follow news about candidates for the 2008 presidential election very closely, fairly closely, not too closely or not at all closely?
30% very closely
34% fairly closely
21% not too closely
15% not closely at all
<1% don't know/refused
So, taking these results at face value, we know that less than a third of Americans are paying "very close" attention to the presidential race. More than a third (36%) say they are following the campaign "not too closely" or "not closely at all." Now consider the 34% who say "fairly closely" in light of what survey researchers tend to take for granted: Respondents sometimes tell us things they think we want to hear. In the context of a survey about about how much attention people are paying to the news, some respondents may be exaggerating their attentiveness to news. I would take the "fairly closely" result with a grain of salt.
Next consider the results from the second and third questions asked on the same survey. Fifty-nine percent (59%) said they heard nothing about Barack Obama in the previous week that made them either more or less favorable to Obama, and 62% said they heard nothing about John McCain that changed their view of him.
These data paint a clear picture for me: Most Americans are paying far less attention to news about the campaign than most journalists, pundits and readers of this site. If we assume that all Americans are following the campaign as a jury follows a trial, we are in error.
Finally, consider something ABC's polling director Gary Langer wrote earlier today:
We too often expect knee-jerk reactions to events of the day; rarely, in fact, do we see them. With few exceptions public opinion proceeds, instead, by a process known as considered judgment: People obtain information as it develops, evaluate it, let it accumulate to the point that it warrants reconsideration of existing attitudes, and at that point re-evaluate and either maintain or change their views.
Attitudes, this means, are far less flighty or reactive to individual events than is commonly assumed; for the most part they are, actually, rational. Obama's trip, like everything else he's doing - and ditto for John McCain - are therefore about building a case, not about changing daily numbers (which, at this stage, are fundamentally silly).
Combine Langer's description with the fact that many Americans are "obtaining" information about the candidates at glacial pace, and we should be surprised to see much meaningful change in the polling numbers right now, especially those measuring vote preference.
Update: In the comments, Along makes a good point:
As Mr. Franklin posted on Monday, the polls in 2004 and 2000 began to shift quite a bit right around now--the 100 days out mark. In 1992 Bill Clinton also started rising significantly in mid-July, after the convention. Kerry's summer swoon started just after his convention, Gore's summer rise started just before his. So are you saying the summer movement in those races was not "meaningful change," or that it was, and we should simply be waiting till the conventions grab our attention to see meaningful movement this year?
I meant mostly the latter. The conventions were earlier in previous years. While convention "bumps" may not endure, voters do pay more attention and the information they receive during conventions is important. Nate Silver made a similar observation last night:
Pundits -- including yours truly -- generally exaggerate the speed with which political news reaches a saturation depth in the American electorate. There are a few exceptions -- debates, conventions, and major victories in the primaries can have measurable effects almost immediately, and certainly within the first 48-72 hours. So can DEFCON-2 level controversies like Jeremiah Wright. But most of the things we write about here, or the National Review talks about, or Keith Olbermann talks about, take a long time to penetrate the electorate if they do so at all.
By Mark Blumenthal on August 5, 2008 4:23 PM
| Permalink
| Comments (6)
| TrackBacks (0)
August 4, 2008
By Mark Blumenthal
How is Barack Obama doing among low income white voters? Quite well, says the headline and lead of a front page story in today's Washington Post. But before leaping to conclusions, we might want to take a closer look at the complete survey questionnaire.
Under the headline "Obama leads, Pessimism Reigns Among Key Group," The Washington Post tells us that Barack Obama "holds a 2 to 1 edge" over John McCain "among the nation's low-wage workers." The Post, in partnership with the Henry J. Kaiser Family Foundation and Harvard University, interviewed 1,350 randomly selected adults under 65 earning $27,000 a year or less and working at least 30 hours a week. Obama's margin was 56% to 27% among all adult respondents,, slightly more (58% to 28%) among registered voters. But the result getting much attention today came in the second paragraph (emphasis added):
Obama's advantage is attributable largely to overwhelming support from two traditional Democratic constituencies: African Americans and Hispanics. But even among white workers -- a group of voters that has been targeted by both parties as a key to victory in November -- Obama leads McCain by 10 percentage points, 47 percent to 37 percent, and has the advantage as the more empathetic candidate.
"Now this," writes TPM's Greg Sargent, "should put the "Obama's working class whites problem" meme to rest." Perhaps.
Taken at face value, Obama's margins do look strong, even stronger than what John Kerry received four years ago among similar voters according to exit polling. While I cannot precisely replicate the universe sampled by Post/Kaiser/Harvard study, the respondent level exit poll data from 2004 available from the Roper archive get us pretty close. I tabulated results for voters under 65 with incomes under $30,000 a year who said they were employed full time. Those voters supported Kerry by a nearly twenty point margin (59% to 40%), while the white voters in the subgroup divided almost evenly, 50% for Kerry to 49% for Bush.
So a survey showing Obama leading by 10 point among low income white voters would certainly represent an improvement.
But take a closer look at the complete questionnaire that -- to their credit -- the Post published online. The presidential vote preference question (#36) comes (by my count) 59 items and roughly 15 minutes into the interview. Before asking about presidential vote preference the survey probed respondents about their personal financial situation, the state of the American economy, their priorities for the things "the government might do to try to improve people's financial situation." They suggested seven different things "you or someone in your family done in the past year to make ends meet," and asked if any applied.
They also asked respondents if their "personal financial situation" has improved or declined since "George W. Bush took office in 2001" (48% said it had declined, 11% said it improved). And finally, immediately before asking the the presidential vote preference question, they asked:
During the past year, have you or has someone in your family had your overtime or regular hours cut back at work, or not?
During the past year, have you or has someone in your family been laid off or lost your job, or not?
And then they asked which candidate they would be most likely to support. Do we think that priming respondents for 15 minutes about the state of the economy and their own personal financial insecurities would have no impact on their vote preference?
Don't get me wrong. The researchers that designed this study are among the best in the field. The survey itself represents an extraordinary and unparalleled effort to "take a close look" the the lives of low wage workers "and try to understand how they are faring amidst all the economic changes around them," as Washington Post economics correspondent Michael Fletcher puts it in a companion video analysis. Among other aspects of their rigorous methodology, the pollsters used used cell phone interviewing to get an adequate representation of low wage workers living without land line phones.
However, if the pollsters wanted to measure where the presidential vote preferences stand now among low income workers, they should have asked the vote preference question up front. The results they obtained tell us something about how low wage workers might react to a campaign framed entirely on economic and pocketbook issues (a finding that presents an obvious strategy for Obama). As such, its measurement of vote preferences is hypothetical and possibly misleading.
Update: University of Wisconsin-Milwaukee political science professor Tom Holbrook was thinking along the same lines.
By Mark Blumenthal on August 4, 2008 2:58 PM
| Permalink
| Comments (2)
| TrackBacks (0)
August 1, 2008
By Mark Blumenthal
Andrew Gellman solves the "nonpuzzle" of close election polls.
Chris Cillizza notes an "attacking unfairly" gap in this week's CNN poll (Pollster reader Gary Kilbride caught it too).
Mark Mellman sees significant changes in Barack Obama's base of support, as compared to Kerry in 2004 and Gore in 2000.
Nate Silver sees similar patterns in the Gallup data.
Jennifer Agiesta delves deeper into how increased black turnout might have affected the 2004 outcome.
Allan and Sheri Rivlin think John McCain needs to articulate a credible economic plan.
Tom Jensen points out the online recording of an automated PPP interview call.
SurveyUSA rounds up their recent polling on the economy.
The Associated Press has questions and answers for polling skeptics.
Late update:
Kathy Frankovic says the economic issues remain dominant, as they were in 1992
By Mark Blumenthal on August 1, 2008 5:49 PM
| Permalink
| Comments (2)
| TrackBacks (0)
By Mark Blumenthal
"Numbers Guy" Carl Bialik devotes his Wall Street Journal column and a companion blog post today to the subject of the automated "interactive voice response" polling that has become such a staple of the current campaign. Both are well worth reading in full.
Bialik managed to interview most of the major players in the political IVR field, and had a reaction from our partner Charles Franklin, summing up our own philosophy regarding the automated polls (that use a recorded voice rather than a live interviewer, and ask respondents to answer questions by pressing keys on their touch-tone phones):
The automated-polling method, says Charles Franklin, professor of political science at the University of Wisconsin and co-developer of the poll-tracking site Pollster.com, "can prove itself through performance or it can fail through poor performance, but we shouldn't rule it out a priori."
The column notes that IVR pollster SurveyUSA ranks second most accurate among all pollsters during the 2008 primaries in the ratings compiled by Nate Silver and that IVR polling was indistinguishable during the primaries in terms of how the final poll compared
