My column this week looks at the controversy over a series of surveys conducted by SurveyUSA for the liberal web site Firedloglake. Please click through to read the whole thing.
Lost in the attack memos and other questions raised is an important question facing nearly every telephone survey conducted in House, Senate and Gubernatorial races this year: Are we at the point where the majority of true "likely voters" under the age of 35 are out of reach of landline telephone samples? And at what point is simply "weighting up" those younger voters that pollsters can still reach inadequate to solve the problem?
The table below, produced by the Pew Research Center and based on their national surveys, shows that by 2006 their unweighted landline samples were under-representing roughly a third of adults under age 35. And that was as of three years ago, when the percentage of all adults living in landline-only households was estimated at 12%, nine percentage points lower than the most recent estimate:
Now consider the estimated growth in the cell-phone-only population over the last three years. As shown in the chart below (which comes from a report last year by the National Center for Health Statistics of the Centers for Disease Control and Prevention), landline-only samples are most likely to miss voters under age 35.
Now consider this additional statistic reported on Pollster.com by Mike Mokrzycki in December. On the most recent CDC report covering the first half of 2009, nearly two thirds (63.5%) of people age 25-29 live in households with either no landline phone (45.8%) or in "cell-mostly" households (17.7%), those were "all or almost all calls are received on cell phones."
So what should a pollster do if they reach so few 18-to-34-year-old voters that they make up just 1% of the likely voters sample for an election where past turnout suggests that age group should make up roughly 10% of the electorate? If the pollster believes they have under-represented younger voters, can they simply weight to correct the problem? Not if the shortfall is that extreme. In a sample with only 400 or 500 completed interviews, such a weight would multiply 4 or 5 interviews by a factor of 10. As I wrote in the column, you don't need to be a statistician to imagine how those "super respondents" might crate greater error and volatility in the results, especially those produced by cross-tabulations of demographic subgroups.
Let's remember that we are able to pick at SurveyUSA because they were willing to disclose the weighted demographics of their sample and because they opted against any such extreme weighting in this case. So rather than beat up on SurveyUSA, we might do better to ask: How many polls have we seen in recent months that involved a similarly sparse number of younger likely voters and were simply weighted up by factors of 5 or greater to conceal the shortfall? How would we know?
Finally, whatever we want to make of the Firedoglake surveys, it is important to remember that SurveyUSA has maintained an outstanding record of final-poll accuracy, especially in U.S. House elections and in hard-to-model primary elections. For House races, the company's own scorecard -- which I have no reason to doubt -- shows that their average error on the margin in polling 27 House races in 2006 (3.4) was roughly half that of all other pollsters combined (6.3). Their error rate was also significantly lower than the three most prolific public pollsters that year, Research2000 (5.5), Zogby (5.9) and RT Strategies (5.9).
So since we have picked at their work mercilessly, I want to give SurveyUSA's Jay Leve the last word and reproduce the full email he sent me last week in response to my questions about the Firedoglake surveys:
In August 2002, SurveyUSA released a poll showing US Senator Robert Torricelli (D-NJ) trailing. No survey to that point had showed Torricelli trailing. An hour after the poll was released, SurveyUSA's client, CBS-TV in Philadelphia, called SurveyUSA and said, "Put your helmets on. The DSCC is coming after you." And the DSCC did. The DSCC found a journalist willing to write the smack that the DSCC was shoveling, and the message went forth: Nothing wrong with Robert Torricelli, plenty wrong with SurveyUSA.
A few weeks later, Torricelli dropped out of the race. Other polls had the same results as SurveyUSA.
Fast forward to today: In a poll conducted in January 2010, at a time the Democrats were losing the state of Massachusetts, SurveyUSA finds an incumbent Democrat in a tight fight in New York state. The DCCC is unhappy. Partisans start shoveling smack. "Sources" start providing willing journalists with leaked memos. Nothing wrong with Democrat Tim Bishop. Plenty wrong with SurveyUSA.
The highway to high office is littered with the road kill of political operatives who find it easier to campaign against a poll than an opponent.
Lost in the hurly burly is an opportunity for real reflection. To my knowledge, there has never (ever) been a publicly released telephone poll conducted in a U.S. congressional district that included a known subset of interviews with respondents who did not have a home (aka: landline) telephone. An acknowledged limitation of SurveyUSA's work in NY-01, and a known limitation to date of all congressional district polling, is that voters who do not have a home phone are under represented. At a statewide-level (in contrast to the CD level), only one pollster in the 2009 election included a known subset of cellphone-only respondents in its sample (at extraordinary expense, because of the theoretical justification), and that pollster's results were worse than many polling firms who did not include a known subset of cell-phone-only respondents. Whether one anticipates that in 2010 young voters will turn out in record numbers of stay home in record numbers, the problem of how to count those voters is real, and right before us.
Another update, this one on the volunteer exit poll conducted this week in Cook County Illinois by the recently launched Chicago Current. Current editor Geoff Dougherty posted a refreshingly candid postmortem on their efforts:
At 6:11 p.m. yesterday, before the polls closed, I wrote that our exit polling suggested Toni Preckwinkle had the Cook County Board president's race locked down.
And I was right. Our survey honed in on Preckwinkle's strong performance early in the day, and continued to highlight her lead as the election progressed.
And yet ... our poll was wrong. I predicted Preckwinkle would snag 69% of the vote, and noted that the poll had an 8% margin of error. Preckwinkle ended the day with 49% of the vote -- well outside that margin.
Such are the joys and pains of exit polling.
There's more, and it's worth clicking through to read the rest.
I would give the Current an "A" for effort and transparency, but we need to be realistic about the quality of the survey they ultimately produced. Dougherty says it cost just $200, "most of which went for a $100 rental car," and don't think he would argue with the conclusion that they got what they paid for. The poll managed to collect just 93 completed interviews at only 9 of 25 precincts (presumably) selected at random. As Dougherty reported at 1:32 p.m. on Tuesday:
So far we've got about 30 responses. We'll be taking a pause here as our field crew relocates to new spots and starts talking to voters.
We'd originally planned to survey 25 precincts, but logistics are interfering, and we'll probably wind up with about half that. We'd targeted 600 voters, but low turnout will probably leave us with about half of that count.
Never mind the very small sample size. How truly random was the sample? It's hard to tell from this description, but the execution clearly fell short of ideal.
Dougherty says that the "networks often pay tens of thousands of dollars for these things." That's not quite right. I'm not sure how it translates into a per-state cost, but the every-two-year National Election Pool (NEP) exit polling operation has a multi-million dollar budget (Voter News Services, VNS, the forerunner to NEP, operated in 2000 on a budget of over $35 million; my understanding is that current costs are much lower but still in the millions). Note that in most states of interest, NEP will sample 20 to 50 precincts. As the scale of what the Current was attempting in a single county was in line with the exit poll that NEP conducts in each state.
I write this post not to beat up on the Current -- again, I give them credit for enterprise and transparency -- but to remind my media colleagues that all "exit polls" are not created equal. Not by a long shot.
Update: The cost statistic I cited for VNS from 2000 is accurate but potentially misleading. VNS was responsible for both exit polls and reporting final vote counts for every race (the latter function is now provided by the Associated Press). The costs also vary considerably between presidential and off-year elections. Finally, the NEP exit operation still includes more than just exit polls, it also collects vote results at samples of key precincts and provides statistical modeling and analysis used to "call" races.
Today we have yet another odd epilogue to story of Strategic Vision, LLC. Apparently not satisfied with their history of setting the low bar for basic disclosure about the surveys they claim to have conducted since 2004, the company is now attempting something new: Attempting to retroactively withdraw previous disclosure.
Until a few weeks ago, the content published at the company's web site, strategicvision.biz, had been automatically archived by the non-profit Internet Archive along with hundreds of thousands of other web pages. In my December column, I linked to two such pages (displaying polls conducted during 2005 and 2007**). As of today, however, if you search the Internet Archive for strategicvision.biz or try either of the links I used previously (and be forewarned: their heavily trafficked site is notoriously slow), you will encounter this error message:
Robots.txt Query Exclusion.
We're sorry, access to http://www.strategicvision.biz has been blocked by the site owner via robots.txt.
What that means is that sometime in January, someone at Strategic Vision added some code ("User-agent: ia_archiver Disallow: /") to a file on their web site that specifically blocks the Internet Archive from searching and displaying pages from their company web site. Let's be clear that Strategic Vision is well within its rights in blocking such searches, and has done nothing illegal or particularly nefarious. As explained on their Frequently Asked Questions (FAQ) page, the Internet Archive "is not interested in preserving or offering access to Web sites or other Internet documents of persons who do not want their materials in the collection," and thus provide instructions on how to "exclude any historical pages."
That said, given the swirl of accusations about Strategic Vision arising from a failure to disclose basic information about their methods, this new effort to scrub previously disclosed information from what is essentially a public library for the Internet is more than a little creepy. Combined their apparent blocking of access to strategicvision.biz to me and my colleagues at the National Journal, and we get a story of a company that keeps digging a deeper and deeper hole for itself.
By the way, all credit for spotting this latest twist in the story goes to Michael Weissman, the retired University of Illinois physics professor who previously published a "Fourier analysis" of Strategic Vision's results on FiveThirtyEight.com. His son Jonathan realized that Strategic Vision might delete their archive, and thus downloaded everything he could before it disappeared. So the archived pages live on -- undoing previous disclosure is harder than it looks.
**As of this writing we were still able to load some of the 2005 page (sporadically), and if you experience as similar result it is probably because of something gone awry at archive.org. The code in the robots.txt file on the Strategic Vision site shows that they want Internet Archive to remove stop displaying their content.
I want to add a few thoughts to Emily's post earlier today on the DailyKos/Research 2000 poll of Republicans and how they might check for a skew in the sample that some argue would result from "sane Republicans" hanging up after taking offense to the questions. Another potential problem, called out today by Republican pollster Alex Lundry, is not as easy to check: The possibility of a skew in respondents' answers caused by what pollsters call "acquiescence bias."
Acquiescence bias is the tendency of some respondents to select affirmative answers where the choice is whether to affirm or reject the statement presented (including "agree or disagree," "favor or oppose" and "yes or no" formats). This topic has been the subject of decades of study and debate among social scientists, and even though pollsters continue to rely on agree-disagree questions, academic survey researchers mostly agree that this format tends to produce more apparent agreement than questions offer a choice between two competing statements.
Here is an example from Schuman and Presser's classic text, Questions and Answers in Surveys (p. 221), based on an experiment first conducted by the NORC General Social Survey in 1974: They asked a random half sample to agree or disagree with this statement: "Most men are better suited emotionally for politics than women." Slightly less than half (47.0%) agreed, 53.0% disagreed.
They asked the other random half-sample to choose between two statements (and included a middle choice):
Would you say that most men are better suited emotionally for politics than are most women, that men and women are equally suited, or that women are better suited than men in this area?
Fewer (33.1%) agreed that men were better, 4.3% said women were better suited than men, and 62.6% said they were both equally suited. Researchers at the University of Michigan's Survey Research Center replicated the experiment three times between 1974 and 1976, producing similar results. They produced consistently greater agreement that "men are better" using the agree/disagree format (ranging from 44.3% to 45.5%) than when using forced-choice format (ranging from 32.5% to 38.3%).
Another strategy to reduce this bias is to try to balance the direction of the statements, as recommended in Presser, et. al, Methods for Testing and Evaluating Survey Questions (p. 440):
Acquiescence bias can be reduced by balancing scales so that the affirming response half the time is in the direction of the construct and half the time is in the opposite direction (e.g. six agree/disagree items on national pride, with the patriotic response matching three agree and three disagree responses).
With those recommendations in mind, consider the questions asked on the DailyKos/Research2000 survey in the order in which they presented the results. The first eight present all of the more sensational, ludicrous assertions (most of which pertain to President Obama). Seven of eight ask respondents to affirm or reject the extreme statement:
- Should Barack Obama be impeached, or not?
- Do you believe Barack Obama was born in the United States, or not?
- Do you think Barack Obama is a socialist?
- Do you believe Barack Obama wants the terrorists to win?
- Do you believe ACORN stole the 2008 election?
- Do you believe Sarah Palin is more qualified to be President than Barack Obama?
- Do you believe Barack Obama is a racist who hates White people?
- Do you believe your state should secede from the United States?
They then ask 15 issue questions that do mix up the order somewhat. Eight questions -- ask respondents if they agree with a liberal policy position, five ask about a conservative policy position, and two (the questions about Christ and marriage as a partnership) force choices between two statements:
- Should Congress make it easier for workers to form and join labor unions?
- Would you favor or oppose giving illegal immigrants now living in the United States the right to live here legally if they pay a fine and learn English?
- Should openly gay men and women be allowed to serve in the military?
- Should same sex couples be allowed to marry?
- Should gay couples receive any state or federal benefits?
- Should openly gay men and women be allowed to teach in public schools?
- Should sex education be taught in the public schools?
- Should public school students be taught that the book of Genesis in the Bible explains how God created the world?
- Are marriages equal partnerships, or are men the leaders of their households?
- Should contraceptive use be outlawed?
- Do you believe the birth control pill is abortion?
- Do you consider abortion to be murder?
- Do you support the death penalty?
- Should women work outside the home?
- Do you believe that the only way for an individual to go to heaven is though Jesus Christ, or can one make it to heaven through another faith?
I don't want to overstate the consensus of pollsters -- academic or otherwise -- on this issue. Many highly regarded survey researchers continue to rely on agree/disagree questions, often because of their simplicity and brevity or because such questions are part of a long-standing time series that the pollster would rather not disrupt (good example of the latter here; for more discussion see Javeline, 1999).
So while it would be a bit unfair to condemn Research 2000 for relying on question formats that pollsters and academics continue to rely on, Lundry has a point. Acquiescence bias probably exaggerates the amount of agreement measured for some of the more ludicrous assertions about Barack Obama tested on the Kos poll.
Update: As Alex Lundry notes below, his comments about acquiescence bias earlier today came after reading a message sent by Stanford graduate student Josh Pasek to AAPOR's members only listserv. With Josh's permission, here is a portion of that message:
Given that 10-20% of respondents tend agree with any statement (likely
due to social norms), I went through the survey mentally subtracting 15
percentage points from every "yes" answer. That does leave some
shocking numbers -- particularly as acquiescence tended to indicate
support for gay rights, sex education, etc. -- but suggests that
Birthers, for instance, may be outnumbered in the party (a slight
consolation at best). I'm not saying this to suggest that the opinions
being expressed even with a correction are reasonable, but I worry that
not addressing this kind of issue is the reason so many people out there
are skeptical of survey results in the first place.
Nothing draws the Google traffic on an election day like the words "exit poll," and the enterprising folks at the Chicago Current (a political newspaper and website) are using their own reporters and journalism students from Northwestern University to conduct an exit poll in Cook County (i.e. not statewide) and appears to be asking about the Democratic contests only (via Christine Matthews).
The Current's Geoff Dougherty has a write-up of what they are doing that includes some worthy disclaimers. Short version: exit polls have random error like other surveys (actually, more given the need for a clustered sample) and is "not the final score" when based on early returns, even if flawless. They are posting initial impressions here.
I cannot say anything about the methodology for the Current poll because, frankly, I don't know a thing about it. For those who have never read it before, you might want to consult my classic primer on exit polls, though remember that my advice was based on the methods employed by the network exit polls done by the folks that invented them and have spent decades honing their techniques.
Without knowing more about the Current poll, I'd urge extreme caution in interpreting the results, especially if your goal is to "call" the result. Remember, exit polls are most useful after the votes are counted -- when results can be weighted to match actual turnout -- in helping to understand who voted and why.
If I learn more today, I'll post it here.
My column for this week looks at how Americans came to believe what they do about health care reform, with a focus on this puzzle: If American's followed news coverage about the health reform debate as closely as the Pew Research Center news interest surveys say they did, why are so many "unfamiliar with key elements of the major bills," as reported by the Kaiser Family Foundation survey last month?
The short version is that perceptions of health insurance reform appear to have been shaped by both the typically process-oriented coverage of the health care debate and the larger context of double-digit unemployment and massive increases in government spending over the last year. I end the column with the obvious conclusion: Months of more legislative wrangling on this issue is unlikely to change impressions or increase awareness of what's actually in the bill. Please click through for the whole thing.
But what about the question of greatest consequence right now? What if Congress were to quickly pass the existing legislation or, alternatively, just let it drop? How would voters react? This topic was the subject of a lot of discussion over the past week, from voices such as Megan McArdle and (since I filed the column) from Nate Silver and Jonathan Chait.
Looking forward, we are on much more shaky and speculative ground but I find Chait most persuasive in arguing that the rationale for Democrats to move forward and pass the bill is that they've already voted for it and thus "already own the downside." They will be attacked for "having voted for tax hikes and Medicare cuts and death panels" regardless of the outcome. He continues:
Suppose there's no upside at all to passing health care reform. McArdle assumes, without explicating her reasons, that walking away from the issue is a way for Democrats to cut their losses. Why, though, would that be the case? Passing the bill may or may not make it more popular, letting it die is surely going to make it less popular. If the bill dies, then it's the subject of lengthy, painful postmortem coverage detailing its flaws and mistakes. It becomes the symbol of big government run amok, and the 60 Senate Democrats and 220 House Democrats who voted for it will suffer politically all the more. Moreover, the already-demoralized liberal base would become apoplectic with the Democratic Party. 1994 was bad, but passing a bill through both chambers then sitting by and letting it die is the kind of behavior that makes even the most pragmatic Democratic voter want to punish his own party.
It's hard to guess at where public opinion will move next, but if I were still offering political advice to Democrats, I'd side with Chait.
From our Pollster-centric perch, this is a memorable comment. In answering the last question of the Q & A session with the House Republicans today in Baltimore, President Obama called out pollster Frank Luntz (emphasis added):
That's why I say if we're going to frame these debates in ways that allow us to solve them, then we can't start off by figuring out, (a), who is to blame, (b), how can we make the american people afraid of the other side? Unfortunately that's how our politics works right now. That's how a lot of our discussion works.
That's how we start off every time somebody speaks in Congress, the first thing they do, they stand up and all the talking points - I see Frank Luntz sitting in the front. He has already polled it and he said the way you're really going to -- I've done a focus group and the way we're going to really box in Obama on this one or make Pelosi look bad on that one -- I know Frank. I like Frank. We've had conversations between Frank and I, but that's how we operate. It's all tactics. It's not solving problems.
So the question is, at what point can we have a serious conversation about Medicare and its long-term liability or a serious conversation about Social Security or a serious conversation about budget and debt in which we're not simply trying to position ourselves politically? That's what I'm committed to doing. We won't agree all the time in getting it done, but I'm committed to doing it.
I'll update with video if and when I can find it (thanks to Marc Ambinder for noting this exchange via Twitter). Update: CSPAN has video here, but the video is not embeddable; the comments above begin at 84:21. Update 2: Thanks to MSNBC, here's the clip:
PS: Probably says more about the nature of inside-the-beltway chatter, but Mark Halperin apparently considered this exchange the most "extraordinary" aspect of the Q&A.
Update 2: Greg Sargent reports on a post question-time interview with Luntz (via Ben Smith):
[Luntz] conceded Obama had the advantage today -- but said he'd still
advise Republicans to debate him again, because it put them on his
"level."
Luntz also confided that Obama had approached him after
the event and joked with him about calling him out. "We had a laugh
about it," Luntz told me in an interview just now. "He said, `It's good
for business.'"
No, we are not doing any live-blogging tonight, though I might post a thought or two via Twitter.
Also, I will update this post later tonight with links to whatever quick reaction polls or focus groups get released.
Meanwhile, if you haven't yet, you might want to read my primer from earlier today on what to expect from post SOTU polls, plus my post from September on the widely varying methodology of the sort of quick reaction polls and focus groups we might say later tonight.
Update 1:
CNN's "Flash Poll" (story, results) -- CNN reported on air some results from their "flash poll" (typically conducted by re-interviewing respondents to an earlier poll that planned to watch the speech). Anchor Soledad OBrien said that, as usual, "more people who are watching the speech favor the party of the person who is giving the speech. That means, in short, more Democrats were being poll here." She did not give specific numbers, but as noted here earlier, that's a typical pattern.
The reaction was overwhelmingly positive -- 48% very positive, 30% somewhat positive, 21% negative -- although O'Brien noted that the very positive number was lower than following Obama's economic address (68% very positive) last year. It is also on the low end of the reactions recorded by Gallup during the late 90's and early 00s (see the second table here).
CNN typically posts results on CNN.com -- I'll add a link when it's up links now added.
CBS' Instant Poll (summary, report & results) - As in previous years, CBS conducted a representative online sample with Knowledge Networks among 522 speech watchers (more on the methodology here). They report that 83% of speech watchers approve of the proposals the president made in his speech tonight, but that just 42% of speech watchers think that Barack Obama will be able to accomplish all the goals he set out in his speech tonight (57% do not think he will be able to).
Democracy Corps Dial Group (
analysis,
pre/post-speech scoresheet) - Democratic pollster Stan Greenberg's Democracy Corps conducted a "dial group" -- a focus group where participants turn dials while watching the speech to indicate their reactions -- in Las Vegas tonight. Greenberg and Democracy Corps Senior Associate Andrew Bauman just provided some highlights via conference call. In addition to the dial meter readings, they also conducted before-and-after survey with the participants.
Greenberg reported a "very positive response," but warned that the "scale of shifts are always artificial" in a dial group because "people are watching him with such intensity." While they saw across the boards shifts in opinion on Obama, but the area with the biggest shift during the speech was on "bank reform and wall street and special interests."
The shifts there are very extraordinary. On the issue of whether he puts Wall Street ahead of the middle class, it was a 50 point shift on people saying that [doesn't describe him] well. There was a 40-point shift...on fighting special interests. On banking reform, on support, it was a 38 point shift in favor of that. And that's clearly, far and away the place where he showed the greatest strength and clarity.
I asked about the lack of State of the Union "bounces" for previous presidents, whether he has seen sustained movement on other internal measures following previous addresses and whether similar dial tests foretold any such shifts.
Greenberg said he could recall shifts in "thermometer approval" (favorable ratings) that held, specifically citing the movement for Obama following last year's joint session address on health care. With Obama's approval falling "within a band..averaging 48 or 49 percent," he considers big shifts unlikely. "Attributes are different," he said. "People who are not supportive don't feel they have to lock in, so there's more space there, and the view of the president is more complicated than just approval." Greenberg cited attitudes on Obama's orientation toward Wall Street as most likely to produce "sustained shifts" in opinion.
Democracy Corps plans to post a full report overnight or early Thursday (links now added above)..
Perhaps political journalists have all gotten the message by now that polling "bumps" from the annual State of the Union (SOTU) address are more myth than reality. If so, this post may be something of a cliche. But I'm not convinced, so I want to recommend this very helpful report published last night by Gallup based on their 30-year archive of pre and post SOTU polling.
If you report or comment on politics, it's a must read. If you are short on time, here are the two main points (with a little added value from our own posts over the last five years):
1) "These speeches rarely affect a president's public standing in a meaningful way, despite the amount of attention they receive."
Gallup's report includes a table showing the level of presidential approval measured immediately before and after the last 27 State of the Union addresses. "Across all presidents," they report, "the average change in approval has been less than a one percentage-point decline.
It is also keeping in mind, as I wrote on the old Mystery Pollster blog four years ago, that the one big exception to the rule -- the apparent 10 percentage point jump for Bill Clinton in 1998 -- was a very unique presidential address:
The Monica Lewinsky story had broken just a few days before. The day before that speech, Bill Clinton faced the cameras and delivered his infamous "I never had sex with that woman" quote. MP cannot find the ratings for that speech, but interest in the speech was certainly high. Ironically, the reaction to Clinton's performance - seemingly unfazed by the scandal erupting around him - help[ed] boost his numbers in a way that persisted until the impeachment trial ended with an acquittal.
So the one exception to the rule may have been less about perceptions of the speech itself and more about how the speech fit into the context of a larger event.
Bottom line: Don't expect a big bump tonight. My post from 2006 covers a lot of the same ground as the Gallup report, as does Charles' Franklin's commentary in 2007 on this graphic
2) "The audiences for the State of the Union tend to be heavily tilted toward the president's existing supporters."
This table from the report is in many ways the most useful for those of us who will be looking at the one-night-wonder polls conducted immediately after the speech.
In many ways, this is the one poll measurement that interests me tonight: Will the speech attract an audience heavily tilted toward Democrats, as is typical, or will the enthusiasm gap apparent in so many approval polls (Republicans report a greater likelihood to vote in 2010 than Democrats) make for a more balanced audience? In other words, will the audience look more like 1999 or 1995 (just days after Republicans took control of the House and Senate, installing Newt Gingrich in the seat just behind President Clinton)? While a typically Democratic skew will tell us little (notice the composition in 1994), it will be a very bad sign for Democrats if the audience consists of as many who tune in to jeer as to cheer.
On the subject of immediate reaction polls and focus groups, see my post from this past September which reviews their widely varying methodologies and the limitations inherent in this sort of survey.
Last week's Senate election in Massachusetts gave those of us who follow and report on politics an experience analogous to the movie It's a Wonderful Life. We now know what life is like without exit polls.
Unfortunately, by the time it became obvious that Scott Brown had a real chance of upsetting Martha Coakley, the lead-time necessary to conduct a true exit poll had passed. As a result, we were left with a handful of post-election telephone surveys. While the high quality survey conducted by the Washington Post and the Kaiser Family Foundation helped fill the gap, we were also treated to a number of post election polls conducted by partisans seeking to put their own spin on the results.
The day after the election, I got an email from a former network political producer who thought now would be a good time to restate the case for exit polls:
The absence of good data to help understand what happened in Mass and why is glaring...it drives me nuts that the Networks get bashed by print media and the pundits for the Exits not being perfect for prediction but never get credit for the service they provide for interpreting the results.
Very true. Too many of us assume that exit polls exist for the sole purpose of "calling" election results hours before the polls close -- an ability that has long been more myth than reality -- while the real value of the enterprise is helping us understand, once all the votes have been counted, who voted and why. Yes, we have all picked at the problems of exit polling, but if anyone has discovered a better method to survey actual voters while also correcting for apparent sampling bias, I am not aware of it.
A marginally related point: The networks do a great public service when they deposit the raw, respondent-level data in academic archives like those at the Roper Center and the Interuniversity Consortium for Political and Social Research (ICPSR). These data allow scholars to conduct all sorts of analyses that are impossible using only the simple cross-tabluations posted on election night.
Unfortunately, the raw data release takes time due to the slow process of creating accurate documentation of the samples interviewed nationwide and in all 50 states plus the District of Columbia. In the past, that process has taken many months (although the 2005 data release was expedited due to intense interest in the apparent exit poll miscues of 2004).
Scholars are still waiting on the release of the 2008 election, and the delay has produced considerable back-channel grumbling, some of which has reached me via email. Why has the release been delayed so long? That I cannot say, but I did contact the folks who are responsible for preparing the data, and they assure me that it is coming soon. The creation of documentation, they tell me, is now virtually complete, but "the files and the documentation are in the process of being reviewed [by the media sponsors], and this process should take a few more weeks to a month."
My column for this week examines an argument made last week by Mark Mellman, pollster for the Democratic Senatorial Campaign Committee (DSCC), that polls showing a close race between Scott Brown and Martha Coakley turned out to be a self-fulfilling prophecy.
The column ultimately addresses the question of why two automated polls showed a neck-and-neck race two weeks before the election, while some telephone surveys, including one conducted by Mellman's firm, showed a big Coakley lead. The column includes a statistic shared by Evan Tracey, founder of the Campaign Media Analysis Group, showing that Brown's television advertising exceeded a half million dollars while the first automated survey by PPP was still in the field.
After I filed the column on Friday, I heard from a Republican source who makes a point I did not address in the column: Brown's did not require television advertising to begin to gain on Coakley. His personal campaigning, as covered by Boston newspapers and television, helped boost his recognition and probably amplified the advertising that he ran in early January:
Massachusetts is the MOST politically aware state in the country. Behind the Pats and the Bosox, it's their blood sport. They FOLLOW it passionately. The number of verbatim comments from voters who brought stuff up that had never been advertised was amazing.
My source also passes along that by Saturday January 9, the day the first PPP poll finished interviewing, Brown's internal tracking showed that 65% of voters in the Boston market reported having seen the Brown truck ad.
The events of the week delayed my pointing to a terrific resource made available to pollsters and polling junkies over the weekend by Public Opinion Quarterly, the academic journal published by the American Association for Public Opinion Research (AAPOR). Their special issue on the 2008 Presidential Election is now posted online and access is totally free.
It leads with the article previously teased on Pollster.com by co-author Mike Mokrzycki on exit poll measurements of cell-phone only voters and implications for future polling. It also includes many more articles of interest to Pollster readers, including a review of poll performance in 2008 by Mike Traugott and Chris Wlezien, a look at how predication markets compare to polls in forecasting outcomes by David Rothschild and much, much more.
Again, POQ has made access open and free to all. Definitely worth a click.
On a morning full of big political news -- the apparent death of the health reform bill, a Supreme Court decision ending restrictions on corporate campaign spending and John Edwards admission that he fathered an out of wedlock child -- I spent the morning attending a briefing something even more exciting: What's in store for execution of the 2010 Census.
OK, maybe not that exciting, but the session sponsored by the DC chapter of the American Association for Public Opinion Research (AAPOR) and the Pew Research Center did provide a fascinating glimpse at one of the government's decennial effort to count and gather demographic data on its citizenry.
While newly installed Census Director Robert Groves did not break any news with his brief presentation, he conveyed a sense of the "massive undertaking" involved in executing the Census and passed along some information that would interest pollsters and data geeks:
- In March the Census will mail a census form to every residence in the United States, a procedure used by the Census since 1960.
- To improve the rate of response to the initial mailing, the Census will run a paid advertising campaign as it did 10 years ago. Groves added this statistic: "For every one percentage point that we raise the mail response rate through this advertising campaign, we will reduce the total cost to the Census by about $85 million," costs that they incur by sending follow-up mailings and in-person enumerators to gain full response (a video of highlights from the launch of the ad campaign is posted here).
- The Census is currently conducting a daily telephone tracking survey to monitor what Groves described as "key predictors" from prior research of what predicts likelihood of participating in the 2010 Census. They then run "predictive models" and "watch how the predictors are changing" overall and within key subgroups and, if necessary, tweak their advertising buy or messages appropriately Groves showed a tracking chart indicating that awareness started to climb following the New York City roll-out of the Census "Road Tour" (seen in the video below).
- As also indicated in the survey released by the Pew Research Center yesterday, younger Americans -- those between 18 and 29 years of age -- are the "laggard group" in terms of reported awareness and intent to participate. Groves, the noted expert on response rates in surveys, voiced a caution we rarely hear about telephone surveys: These data on younger Americans "are subject to great misinterpretation" since they are from the respondents of a random digit dial (RDD) telephone survey. "The proportion of all people sampled that became respondents is much, much, much, much, much, much lower than we'll ever get in the census, so these [younger respondents] are the cream of the crop" in terms of their willingness to participate in a survey, and thus perhaps, in the census itself. "The usefulness of this [survey]," he added, "is to watch this over time, to see if things are moving."
- Starting at the end of March, the Census will launch a tool on its web site that will allow anyone to monitor real time updates of the participation rate, featuring a thematic "heat map" that will display regional variation. You will also be able to "drill down" to see similar mapping for individual counties or zip codes, or create on screen comparisons between localities so "New York could compete with L.A." (or perhaps Washington DC with Dallas?).
- The Census is also undertaking a Census in the Schools initiative with the assistance of Sesame Street characters Rosita and Count von Count, who Groves describes wryly as his "senior technical adviser."
Not surprisingly, Groves now has a blog on the Census 2010 website.
Regular readers may recall my personal pet peeve about rushing to quick conclusions about the "most accurate pollster" in any given election. One of my objections -- all votes are typically not counted immediately -- is slightly less of a worry when applied to yesterday's Massachusetts race, as election officials there have produced an "unofficial" count based on 100% of the state's precincts. Still, the final certified count can sometimes differ slightly, sometimes enough to move the final count by a percentage point, so please take what follows as preliminary.
Another objection of mine, however, is even more valid in looking at a single state as it was back in February where we had several contests to consider:
[T[he whole notion of crowing a "big winner" based on a handful of polls in a handful of states is foolish. The final polls yesterday had random sampling error of at least +/- 3 percentage points. If a poll produces a forecast outside its margin of error, that's important. But if several polls capture the actual result within their standard error, chance alone is as likely as anything else to determine which one "nails it" and which miss by a point or two.
Let's use today's results to illustrate the problem. The following chart shows Brown's percentage of the vote as reported by the public polls conducted during the last 7 days of the campaign, with an error bar based the poll's reported margin of error. The horizontal bar represents Scott Brown's actual percentage of the vote.
What stands out most is that most of the polls produced an estimate of Brown's percentage of the vote within their own margin of error of the actual result. The exceptions are those on the left, which were conducted almost a week before the election, and if you followed our chart or read Charles Franklin's post on Monday, you know that Brown's support rocketed up over the course of January, so we should expect some of the earlier polls to show Brown's support lower than it turned out to be.
Another perennial issue with measuring the accuracy of pre-election trial heat questions is the issue of how to handle the undecided percentage. I did not allocate undecideds, and some polls had a bigger undecided percentage than others. Also, in this case, some pollsters included independent Joe Kennedy as an option, others did not. Kennedy ultimately received only 1% of the vote, but the Blue Mass Group/Research 2000 poll that missed the Brown percentage by 11 points had the biggest percentage either for Kennedy (5%) or undecided (5%).
So by and large, it is fair to say that all of the surveys conducted after Wednesday produced estimates of Brown's vote that were as accurate as a survey can be given the potential for random error.
You might reach a different conclusion, however, when you look at how they did forecasting Martha Coakley's percentage.
Here four surveys, all conducted after Wednesday, all significantly understated Coakley's percentage of the vote. Two were conducted for Pajamas Media by the Republican firm CrossTarget. The others were done by InsiderAdvantage and and by the Merriman River Group for InsideMedford.com. These four surveys are mostly responsible for the small understatement of Coakley's support in our overall trend estimate.
For what it's worth, all four used an automated IVR methodology and were completed in a single day, but all four also reported slightly higher undecided percentages than the others surveys conducted over the final weekend. So perhaps their significantly lower estimates of Coakley's support had something to do with their calling procedures, or perhaps they were not pushing undecideds hard enough.
I also produced the following table that calculates the error on each poll for each candidate and the error on the margin. The two surveys that missed the Brown's margin by the most were the Pajamas/CrossTarget and BlueMassGroup/Research 2000 polls conducted more than four days before the election - and they managed to miss in opposite directions.
Click table for full size version
A round-up of Pollster-centric reactions to the Massachusetts results:
This Boston Herald lead speaks volumes: "High turnout in Bay State 'burbs and among independent voters who flocked to the polls eclipsed a healthy turnout in staunchly Democratic Boston, fueling Republican Scott Brown's victory yesterday."
Brown pollster Neil Newhouse stays up late to offer his take on the message sent by Massachusetts voters, Brown's 12 keys to victory and a memo to Democratic leaders.
Coakley pollster Celinda Lake gives an interview to Ryan Grim defending their campaign.
The Note produces a December Coakley campaign polling memo.
Mark Mellman argues (pre-results) that early January automated (IVR) polls were a self-fulfilling prophecy.
The Republican National Committee releases a pre-election survey (via Memoli).
Rasmussen Reports conducts an election night survey.
PPP's Tom Jensen offers takeaways from Massachusetts.
Nate Silver allocates blame.
Washington Post pollster Jon Cohen reminds us of what we gets lost in the absence of exit polling and a proliferation of horse-race only polls.
Former AP polling director Mike Mokrzycki reviews what exit polls could have told us.
John Zogby predicts a Coakley win, then walks it back.
Political scientist Josh Tucker graphs the actual effect of the Massachusetts election, but Brenden Nyhan offers a different interpretation.
PS: Don't forget Charles Franklin's post below ("How Massachusetts Votes Shifted").
Late add: Research Rants adds some helpful perspective from a distance.
Other assessments of the Massachusetts polls and voter turnout from around the web:
Suffolk University shows momentum for Brown in a poll of bellwether counties.
Nate Silver sees Brown as a 3:1 favorite, gives Coakley a 25% chance of winning.
Chris Bowers gives Coakley a 35% chance to win (his assessment yesterday of the performance of polling in previous general, primary and special election is well worth reading).
Harry Enten shows Brown winning regardless of turnout.
Gallup finds the same percentage of Democratic identifiers in Massachusetts as the nation.
Martha Coakley says the polls are wrong.
So Does GregR at Blue Mass Group.
Jim Geraghty notes Brown's support among Democrats.
Reid Wilson looks at the stakes for pollsters.
WBZ reports that MA Secretary of State William Gallup expects a 40% to 55% turnout among registered voters, more reports on turnout from Ben Smith, TPM, Reid Wilson.
Patrick Ruffini sets up a Google Spreadsheet to track results at the town level (brilliant!).
PS: As I noted a few days ago on Twitter, the networks are not be conducting an exit poll in Massachusetts today. Network interest in doing one, I'm told, occurred too late for the long lead time necessary. We'll probably live-blog again tonight...check back in later for details.
Update: More...
The Cook Political Report's David Wasserman suggests some bellwethers to watch.'
Mark Ambinder posts a "what to watch for" cheat sheet with links to websites reporting results.
My column today considers the proliferation of inexpensive automated surveys and two developments that promise to fuel even more growth: Two companies that promise to offer the ability to conduct an automated telephone poll for less than a thousand dollars.
I filed the column on Friday afternoon, which in Massachusetts Senate time was seven polls ago, and those new polls underscore the message of the column. Nine of the 16 polls we logged on Pollster.com since January 1used an automated telephone methodology, including 5 of the last 7. Two of the surveys, those sponsored by Pajamas Media and and InsideMedford.com, used pollsters whose work we have not previously tracked.
Of those 15 polls, only two -- the surveys conducted by the University of New Hampshire and Suffolk University -- had traditional, mainstream media sponsors. The rest were conducted or sponsored by polling public relations, political partisans, "new media" web sites or some combination of these three.
So the vision of the future described in the column is, in many ways, already upon us.
Over the last few days, thousands of words have been written and many charts posted on Pollster and elsewhere, all trying to make sense of the sometimes divergent Massachusetts Senate polls. If you haven't yet, make sure you read Charles Franklin's tour de force review posted here a few hours ago, that walks through eighteen permutations of trend line models, all of which show Republican Scott Brown leading Martha Coakley, most showing him ahead by 4 points or more.
If you prefer a simpler summary, consider this:
- Of eight surveys completed and released since Wednesday, seven show Brown leading by at least a point. The one exception shows a dead heat Our chart of all polls shows a nearly seven point Brown gap between the trend lines for Brown and Coakley (51.2% to 44.3%).
- Browns' support on our standard trend estimate has increased by nearly twelve points (from 38.5% to 51.2%) in just the last two weeks.
A trend this strong is unusual, especially in a contest between a Democrat and a Republican. We do see such surges occasionally in primary elections -- the surprise victory by Creigh Deeds in last year's surprise victory in Virginia's Democratic primary being the most recent example -- but they are far more rare in general election contests . Over the weekend, I reviewed the most competitive contests we have tracked on Pollster.com since 2006 and found no race that produced a trend anywhere near this strong over the last few weeks of the campaign.
I am sure that there are other example, but the one that stands out for me is the victory of Democrat Harris Wofford in 1991. Wofford, appointed earlier that year to fill a vacant Senate seat, began as a virtual unknown and began trailing by more than 40 points against popular former Republican Governor Dick Thornburgh. Although the final round of public polls showed the candidates running about even, Wofford's momentum helped carry him to what turned out to be an eleven point victory margin (55 percent to 44 percent).
Of course, the same factors that make the trend toward Scott Brown so unusual also make the polling challenging and potentially misleading. Brown has moved up so rapidly partly because campaign has been truncated, but the rapid change also prompted a late avalanche of negative advertising by the Democrats directed at Brown. Because it is a special election being held on an usual date, Pollsters have no prior history to judge the size and demographics of the likely electorate. The likely voter problem is one reason why polling errors tend to be larger in special elections.
So while we have the Wofford experience on one hand, we have the lessons of the New Hampshire Democratic primary in 2008 and the special election in New York's 23rd District this past fall on the other. In both cases, candidates surged in the final polls, only to see their apparent leads disintegrate on Election Day. What those races had in common were huge surprise developments that occurred a few days days before the election (Barack Obama's Iowa victory and the withdrawal of Dede Scozzafava) that helped shake up the race, fuel the polling surge and -- perhaps -- provoke voters to focus more closely on their choices and rethink their preferences in the final hours.
Does the nationalization of the Massachusetts Senate race combined with the heavy negative advertising blitz qualify as the same sort of last minute surprise? Perhaps, but it seems like a stretch to me.
Some believe that non-response bias may have contributed to the errors in those two races, exaggerating the contribution of the most enthusiastic supporters of the surging candidate. Mike Mokrzycki developed that theory in the context of the Massachusetts Senate race here over the weekend. Some believe this phenomenon may be more acute in automated surveys, and we should not ignore that only two of the last seven public polls used live interviewers.
Yes, Coakley has done better on live interviewer surveys than the automated polls, but we saw a similar pattern in New Jersey last fall, and the robo-polls ultimately provided a closer forecast of the final margin.
Yes, the internal Coakley campaign poll numbers that have leaked out show a dead even race and perhaps a slight improvement over last week. However, there was more than one internal poll conducted by Democrats A little birdie tells me that the final tracking survey conducted by the Mellman Group for the Democratic Senatorial Campaign Committee had Brown ahead by five points.
So for me it boils down to this: I was a Democratic consultant for long enough to want to believe that Coakley can still prevail, and there is still a remote chance that the polls in this race will be as misleading as they were in New Hampshire. However, my head is not my heart. Barring another polling meltdown, Scott Brown is the likely winner.
This is the first of two posts that will wrap up my thoughts about the Massachusetts Senate race. This first one covers an admittedly narrow point. It is provoked by an email I received reacting to Nate Silver's post from Sunday -- "A Statistical Ray of Hope for Coakley" -- from Pollster reader Harry Enten (aka commenter Poughies):
[Silver's] piece makes the point that the margin between Martha Coakley and Steve Brown could be overstating Brown's lead. Silver points out that polling in close (margins of 10 or less in the polls) Senate elections since 2000 in deeply blue states (as measured by the Cook Political Partisan Index) has by an average of 3.4 points underestimated Democratic candidates' margin of victories. In deeply red states, on the other hand, polling has underestimated Republican margins by 1.9 points.
Though he used a Pollster.com average in 2008 and Real Clear Politics average in 2006 and 2004, Silver used a simple average of all non-partisan polls conducted in the final two weeks compiled by Pollingreport.com in 2002 and 2000. Perhaps, he did not know, but a Real Clear Politics average is also available for 2002 and 2000.
I was interested what, if any, effect substituting the Real Clear Politics averages in 2002 and 2000 would have on Silver's results. Therefore, I decided to create a new dataset modeled after Silver's, but using the Pollster.com data in 2008 and Real Clear Politics average* in 2006, 2004, 2002, and 2000.
Using these new rules, the underestimation of Democratic margins in blue states stays the same at 3.4 points. This result is not surprising, as the blue states part of the dataset is small, and only three results are available from 2000-2002. The underestimation of Republican margins in red states drops from 1.9 to 0.9 points.
The overall average of underestimation drops from 2.3 to 1.5 points. In only 7 of 23 contests was the underestimation above 3 points. In no contest did the polling average incorrectly predict the winner due to underestimation of Democratic candidates in blue states or Republican candidates in red states. The only incorrect winner chosen was in the South Dakota (a red state) 2002 race when the polls predicted a victory by Republican John Thune.
The bottom line is that perhaps the polls are overestimating Brown's margin in Massachusetts. The limited data involving closely polled elections in blue states suggest that Coakley might do better than the polls suggest, but when you look at the larger dataset of red states, Coakley should not expect a bump.
*In some cases (such as Alaska 2004), no Real Clear Politics "average" existed. I just averaged all the polls listed on Real Clear Politics (and in the case of Louisiana 2002) conducted in the final two weeks. I, unlike, Silver use internal polls in these cases... modeling myself after Pollster.com's inclusion of them.
P.S. The Real Clear Politics web pages can be hard to find [but are available at these links] for 2002 and 2000.
He also passed along a table (in Excel and PDF formats) with the numbers.
It's also worth taking a closer look at the six races -- there were only six -- that are the basis for the conclusion that polls understate support for Democrats:
Most interesting are the three races (Maryland and Rhode Island in 2006 and New York in 2000) that produced the biggest errors. Of these, the Rhode Island example is partly the result of an RCP average based on just two surveys conducted over the final weekend of the campaign. We used a simple last-5-poll average for Rhode Island that year, which showed Whitehouse winning by six, just one point off the actual margin. Just swapping the Pollster and RCP averages for that one race would cut the average variance to +2.5. Whatever the challenges of polling in Massachusetts, greater random error due to a shortage of final polls is not one of them.
That leaves two big errors affecting Ben Cardin in Maryland in 2006 and Hillary Clinton in New York in 2000. What's interesting about the errors in both contests was that the final round of polls had the percentage for the Republican candidate about right (within a point), but understated the support for the Democrat by about five percentage points. Even if we assume that a similar pattern will apply in Massachusetts tomorrow, the problem is that the six of the last seven surveys estimate Brown's support at 51% or greater.
One of the odd aspects of last week of the Massachusetts Senate campaign is the way Coakley "internal" polling numbers have leaked on a near daily basis, through blogger Steve Kornacki and others. Wednesday night, according to Kornacki, the Coakley campaign's own polling showed her "barely ahead, 46 to 44 percent." Thursday night's results showed her trailing, 47 to 44 percent, and conservative columnist Byron York added a quote from an unnamed but "well-connected Democratic strategist" who "heard" that "in the last two days the bottom has fallen out of her poll numbers." Then on Friday night, again according to Kornacki, Brown was ahead by just two points on Coakley's poll (47% to 45%), and a three-day average of the results from Wednesday through Friday night gave Brown the same two-point lead (47% to 45%).
These leaks produced some snickering: Via Twitter, PPP's Tom Jensen pronounced the leaks the "sign of a highly undisciplined campaign." Jay Cost asked "how lame is the Coakley campaign" to leak their internal tracking polls "EVERY DAY?" And a very smart reader emailed this morning with the observation that leaks mark Coakley's campaign "more undisciplined than a 4 year old at K-mart on a sugar high."
Let me be clear: The conclusion that Coakley's campaign -- her staff or the consultants she retains -- is responsible for these leaks is probably unfair and a bit naive. They were likely not the source.
Now, alas, I do not have any inside information and have not been the recipient of any such leaks (really, old consultant comrades, where is the love?). But I can say from my own experience as a Democratic campaign pollster that it's fairly standard practice for a Senate campaign like Coakley's to share their daily tracking results with the Democratic Senatorial Campaign Committee (DSCC), the White House political office and EMILY's List. It's really unimaginable that Coakely, or any other "disciplined" campaign in their position, would not. Also, either directly or through these organizations, the same are almost certainly going to the labor groups and other interests conducting their own campaigns on Coakley's behalf. Each of those organizations has its own pollsters, media and direct mail consultants. So the leaks could have come from damn near any "well connected strategist" in Washington.
The incentive to leak would be especially high for those who have parachuted in to help in the final week, those with great incentive lay the groundwork to take credit should Coakley "come from behind" to win. Consider that these incentives are even greater for the White House. Obama really had no choice but to come to Massachusetts to Coakley's aid (as he is doing today). A Coakley loss will be catastrophic for Obama's legislative agenda, and the White House will take some of the blame either way. So a mid-week decision to come to her assistance creates huge incentive to leak these numbers. Again, if she loses, well, the bottom had already "fallen out." If she wins, they claim credit for turning things around.
Now all of this probably speaks to a breakdown in team play or the sort of ugly finger pointing that always seems to accompany defeat. For those surprised by the wide dissemination of "internal" Coakley polling data, consider that in the fall of 2008, the Obama campaign shared polling numbers and a whole lot more on a daily consultant conference calls whose participants (I'm told) close to a hundred. Nothing of significance leaked from those calls before Election Day. One way or another, Martha Coakely and her campaign are worthy of much criticism, but piling on over these leaks is unfair.
By the way, it's more than a little crazy to be paying much attention to the random zigs and zags apparent in the relatively small one-night samples used in internal campaign polling. I certainly hope that the pollsters of record are not making decisions or recommendations on the basis of anything but the three-night rolling averages.
This morning, I'm sure of only one thing about the Massachusetts Senate race: The perception among Massachusetts voters that Democrat Martha Coakley is likely to defeat Republican Scott Brown, is not long for this world. Even the newly released Suffolk University poll shows that by a better than two-to-one margin (64% to 26%) Massachusetts still believe Coakley will win. But since Suffolk is a local University, uses live interviewers and has the Boston Herald and Boston's NBC affiliate as sponsors, their finding that Brown leads Coakley by a not quite statistically significant four points (50% to 46%) is huge news in Massachusetts this morning.
In addition to the Herald front page, the poll was also the big story on all of the local Boston television stations last night or this morning (online video available as of this writing at WCVB-5, WHDH-7, WFXT-25, WBZ-38). Watch those stories, and its hard to imagine that perceptions of a likely Coakley victory will survive the weekend. If you are a Democrat, that is probably the only silver lining in today's news.
On the other hand, the tone of these stories follows the all too typical pattern of political news coverage: an analytical focus on strategy and tactics: "What happened to Coakley's lead?" "Why is Brown surging?" In assessing tactics, they inevitably praise Brown's efforts ("Brown has been out-hustling Coakley on the campaign trail"), while dissecting the apparent failures of the Coakley campaign. If "momentum" is a factor in campaign politics, this sort of coverage is a big reason why.
Another troubling pattern for Democrats: To the extent that these stories discuss the negative advertising being run by Coakley and her Democratic allies this week, it is only as a possible explanation for her poll numbers. The WBZ story, for example, cites callers to local talk radio and emails to the station complaining about negative ads. The WBZ anchor then concludes:
That is the combination of [Coakley's] problems, visibility and negativity. You can go negative, political ads work, political consultants always say that, but only if the voter knows exactly who you are, so at this point, since there is this perception that she hasn't been out there hustling as much...since voters don't know who she is, all they see from her is negativity at this point, and at the 11th hour, that's tough to overcome.
That's not quite right. Negative advertising works when voters see its message as credible. Ad buys as heavy as the combined efforts of the Coakley campaign, the DSCC and SEIU have undoubtedly been noticed and, as such, will create some cognitive dissonance among voters still leaning to Brown. The big question is how those voters resolve the dissonance. If they come to accept the arguments the ads are making as valid, some may back away from supporting Brown. But cognitive dissonance theory says that denial and rationalization are more powerful instincts than acceptance, so it is easier for voters who already like Brown to dismiss the content of the ads as typical political "mudslinging."
The key to resolving that dissonance is the way the news media covers the campaign: If news stories focus on the substance of the ads and the debate between the candidates, there is a greater chance that the ads will have an impact. If coverage focuses on tactics alone -- as horse race stories inevitably do -- the ads are more likely to fail.
One last thing about the Suffolk Poll. One astute Pollster reader emails with a question: The poll asks "if you know when the election is (and terminates the interview if you don't have the right answer). Is that unusual for special elections?"
That question is a little unusual, in my experience, but in fairness to the Suffolk University pollsters, there really is no "usual" with likely voter screens, especially in special elections. Selecting likely voters is really where political polling is more art than science. To make matters worse, pollsters do not typically reveal the full text of their screen questions, so give the Suffolk pollsters credit for being fully transparent on that score.
I think their screen is reasonable. After all, you're not very likely to vote if you don't know the election is next week [UPDATE: but see the contrary view of reader Dan below]. The classic Gallup likely voter model includes a similar question about knowledge of your voting location (although that is just one item in a seven question scale). I would question the Suffolk screen if I believed that the Coakley campaign was poised to mount a massive weekend get-out-the-vote effort aimed to reminding identified supporters about where and when they vote. By most accounts, that is not likely.
Watching a clip of Glenn Beck interviewing Sarah Palin, Andrew Sullivan catches this comment from Beck: "I don't know yet if [Palin's] strong enough, if she's well-enough advised, or if she knows she can no longer trust anyone." Sullivan goes on to comment that "distrust of everything in politics, of every politician, of the 'system' that has been co-opted by mysterious and menacing elites, and a sense of total beleaguerment in the modern world" has become a familiar theme from "the far right."
For what it's worth, "distrust of everything" was also a prominent theme of a briefing I attended this morning on the new Allstate/National Journal poll. As my colleague Ron Brownstein put it this morning, in an economy where they are "more directly exposed to financial risk than earlier generations," Americans "don't have much confidence that any institution, government, business or the financial sector is doing much right now to help them" deal with that risk.
"As the recession has grown more prolonged," he added, "this alienation from institutions has only deepened and hardened." In follow-up interviews he conducted with survey respondents, Brownstein found "more of an edge of fear, even desperation. A sense of this economic weight settling on people and and being uncertain when the cloud is going to lift." (I've embedded the full video of the presentations below -- Brownstein's comments are at 9:55, the presentation by FD pollster Ed Reilly begins at 13:30).
The data that Brownstein had in mind came from a set of questions that asked respondents to report how much trust they have in a series of people and institutions "to help you manage the financial risks in your life." Most expressed "a lot" of trust in themselves (74%) and their spouse or family members (64%) but others ranked much lower:
Just 15 percent say they have "a lot" of trust in financial advisers to help them cope with financial risk. Labor unions ranked next in trustworthiness, at 12 percent.
Fewer than one in 10 express "a lot" of confidence in national banks, corporations, or the federal government. Strikingly, about half of those polled say they have no trust in any of those three institutions to help them. "These government plans aren't working out, and these corporations are using the government to their benefit," insisted Scott Holland, whose timber business in Garner, N.C., is bankrupt. "I really don't trust anybody at this point."
The flip side of this distrust is a sense that while Wall Street has benefited from government efforts to jump start the economy, ordinary people have seen little benefit:
At the root of the dissatisfaction crackling through the survey is the widespread belief that government's response to the economic crisis mostly benefited affluent individuals and powerful business and financial institutions--the very groups that many respondents blame for the upheaval. "There weren't enough consequences on some of the people who were part of the demise of the banks," said William Fields, a retired engineer and political independent from Villa Hills, Ky.
Who has "benefited most" from "the actions the federal government has taken to respond to the financial crisis over the last 12 months?" Three-quarters of those polled say it has been Big Business or the rich. Forty percent picked banks and investment companies, with another 20 percent identifying major corporations and 16 percent choosing wealthy individuals. Just 9 percent contend that the middle class has been the principal beneficiary, and 8 percent say that the poor have benefited most. African-Americans are more likely than whites to see middle- or lower-income families as the major beneficiaries. But in all demographic groups, the sense is widespread that over the past year most of the jam was placed on the top shelf, to paraphrase Texas populist Jim Hightower.
The survey also asked what "large financial corporations" could do to "increase the trust you have in them." The proposal that drew the most favorable response, by far, was "paying back the bailout money they received from the federal government as soon a possible." More than half (59%) said that would increase their trust "a lot," 28% said it would increase it "some" and only 11% said "not at all.".
An audience member asked about a tax increase on financial institutions proposed today by President Obama to help defray the costs of the financial bailout. Pollster Reilly responded (at 57:55 in the video) that the measure is "obviously very much aligned with where public opinion is" and that "in order to maintain any kind of a consensus moving forward politically, they have to demonstrate to these institutions that benefited from government assistance during the last year, that a fair deal has been struck and that they are paying back."
So responding to a sense of "distrust of everything in politics" is not just the province of the far right. It is a reaction to public opinion across the board.
[Special thanks to the National Journal's Ryan Morris and Reuben Dalke for sharing their beautiful graphics above that will appear in the print edition of the magazine this week].
Given the humanitarian crisis in Haiti, the internet has been buzzing with ways to help. I want to amplify this suggestion from The Nation's Peter Rothberg:
There are numerous ways to help groups already on the ground. One of the best, Partners In Health, has been operating in the country since 1987, originally to deliver health care to the residents of Haiti's mountainous Central Plateau region. PiH now also operates clinics in Port au Prince and other major Haitian cities. With hospitals and a highly trained medical staff in place, Partners In Health is already mobilizing resources and preparing plans to bring medical assistance and supplies to areas that have been hardest hit. Donations to help earthquake relief efforts will be quickly routed to the disaster.
You can make an online donation to Partners in Health on their website. According to this detailed update on what they are doing to help, their "greatest need is financial support."
I am seconding that recommendation because of a similar testimonial I heard this morning from my spouse, a physician and former colleague of Paul Farmer, Partner's executive vice president, when they both practiced at the Brigham & Women's Hospital in Boston.
Farmer's work in Haiti was the subject of Tracy Kidder's best-selling book, Mountains Beyond Mountains. To get a sense of the story, and why my wife has total confidence in Farmer's work, here are two brief excerpts of an interview Kidder gave last year:
I had met [Paul Farmer] in 1994 and found him intriguing, but I think the decisive moment was when I saw his health center in Haiti for the first time in 2000.You travel from the airport along this horrible road where you mostly notice the absence of things: Electricity, arable land, even trees. And after three hours of witnessing unremitting misery all around you --people without food, without shoes-- you come to this verdant citadel that provides high-quality medical services to everyone for miles around, regardless of their ability to pay. I remember feeling that if it was possible for this to be here, then anything was possible.
[...]
It's amazing: They now have nine sites in Haiti, including four hospitals complete with operating rooms, and they've got AIDS under control in the entire central plateau of Haiti. I mean Haiti is still in desperate shape, but this is something good that's happened there, and they're continuing to expand. They now have about three thousand staff members of Partners In Health in Haiti, almost all Haitian, and a total of about five thousand worldwide.
Take it from my family: If you want to make a donation that will be put to immediate use to help those in need in Haiti, you can do no better than Partners in Health.
Update - An even better testimonial from Kidder comes this morning in an op-ed in today's New York Times:
But there are effective aid organizations working in Haiti. At least
one has not been crippled by the earthquake. Partners in Health, or in
Haitian Creole Zanmi Lasante, has been the largest health care provider
in rural Haiti. (I serve on this organization's development committee.)
It operates, in partnership with the Haitian Ministry of Health, some
10 hospitals and clinics, all far from the capital and all still
intact. As a result of this calamity, Partners in Health probably just
became the largest health care provider still standing in all Haiti.
Fortunately,
it also offers a solid model for independence -- a model where only a
handful of Americans are involved in day-to-day operations, and
Haitians run the show. Efforts like this could provide one way for
Haiti, as it rebuilds, to renew the promise of its revolution.
The flurry of commentary on polling in the Massachusetts Senate race plus a handful of new polls -- some real, some rumor -- provide some new topics worth discussing. I'm going to take these in order, but none take away from the obvious conclusion that the race is likely to be a lot closer than most assumed a few weeks ago.
1) Recent Trend? On Sunday, I bemoaned the lack of apples-to-apples polling comparisons available over the last few weeks. Yesterday's update from Rasmussen Reports showing Scott Brown trailing Martha Coakley by just two percentage points (49% to 47%) appears to provide such a comparison, since Rasmussen's track a week ago showed Coakley leading by nine (50% to 41%). But read the fine print:
The results of this poll are not precisely comparable with last week's results because this poll includes the independent candidate by name while the previous poll simply offered the choice of "some other candidate." Additionally, the latest poll results include "leaners."
Leaners are those who don't initially have a preference for one of the major candidates but indicate that they are leaning in that direction. Without "leaners," Brown was actually ahead by a single percentage point.
The pressing of leaners is the bigger issue here, particularly given a pattern I'll discuss below. Rasmussen shows independent Joe L. Kennedy receiving just 3% of the vote (this Kennedy is unrelated to the more famous Kennedy family that includes the late Senator and his nephew Joseph P. Kennedy, who once represented Boston in Congress).
Now, I don't want to belabor this point. My colleague Marc Ambinder reported last night that the "internal Democratic tracking in MA last week had Coakley up by 15. Today, she's up by five." And there is no denying that Brown has significantly narrowed the gap since November, so it's likely that he has continued to gain in recent days.
Our chart above -- which happily compares apples and oranges without remorse, but does not include the rumored internal polling -- does indicate a narrowing margin since January 1.
2) Turnout or Persuasion? On Sunday, I argued that "turnout matters," mostly because cross-tabulations in both the Boston Globe/UNH and Rasmussen surveys show a much closer race among the most interested and likely voters than among other respondents. This latest Rasmussen survey (added to the bottom of the table below) confirms the trend:
We can assume that a special election will draw fewer voters than an off-year general election for Governor or Senator held in November, but it is not clear where to draw the line in defining the likely electorate. And efforts to increase turnout will help Coakley: And for every three previously disinterested voters who change their minds and decide to vote this week, two will be Coakley supporters..
In that context, I have to agree with Chuck Todd and company at NBC's First Read:
[I]t probably doesn't help Brown that the contest has been nationalized. All the ads Democratic and conservative groups are now airing, all the money that's now flowing into the race, and all the reminders about how health care hangs in the balance will likely boost Democratic enthusiasm.
That said, I agree with Nate Silver that the close nature of the race is "not just about turnout." As Pollster reader Harry Enten (aka Poughies) noted on Sunday, the voters identified as independent on the automated surveys support Scott Brown by margins of better than two-to-one, while the Boston Globe/University of New Hampshire poll shows a very slight Coakley advantage among both independent identifiers and the much larger group of those who report they declared no party affiliation when they registered to vote.
Having made these comparisons, let me offer some big caveats: In comparing "independents" we have two variables that might introduce house effects -- the likely voter screen and the way the pollster asks about party identification -- plus much larger random error (as these smaller subgroups typically involve much smaller sample sizes). That said, I can't account for all the difference between the Globe and the other polls on the basis of the LV screen and question wording alone. Given that, as Harry Enten notes, the surveys are much more similar in terms of the results they report for Democrats and Republicans, it looks like the differences separating the Globe and the automated surveys have a lot to do with those who consider themselves independent.
[An aside: it is likely that a more stringent likely voter screen would have a slightly disproportionate impact on independents. I checked some national pre-election polling data from 2004 and 2008 and found that while narrowing from the least restrictive to most restrictive likely voter models makes independents slightly more Republican leaning, it is hard to see how such a phenomenon alone explains the 20+ point differences in the table above].
If you accept the very close margins on the PPP and Rasmussen surveys as real, then Brown is successfully persuading a lot of non-Republicans to support him who typically vote Democratic.
Here's a hypothesis that might explain the pattern: if Brown ekes out a victory or comes within a few percentage points of winning, it will because he wins the support of a lot of voters -- most of them independent -- who typically vote Democratic. Brown has probably not yet closed the sale with these voters, given their prior vote history, but they are poised to support him. Perhaps it is harder for them to tell a live interviewer they are ready to vote Republican. Perhaps the more anonymous nature of the automated methodology better simulates the act of voting which will ultimately force a decision.
3) A 50% Coakley Ceiling? On Monday, pollster Scott Rasmussen noted that all three surveys available then "show Coakley right around the 50% mark....If Coakley is truly right around [50%], then the race is hers to lose, and Brown's best possible scenario is a very narrow victory." Rasmussen is correct about the consistency of of Coakley's support. It extends to his most recent survey and to the narrower cuts of likely and interested voters from both the Rasmussen and Globe surveys:
Consider again the theory I offered above. It may be that Brown is on the verge of winning the support of a lot of voters who typically vote for Democrats, so the differences in methodology -- how hard each pollster effectively pushes for a decision -- produce a much bigger variation in his measured support.
Sean Trende of RealClearPolitics has a different theory. He argues that the pattern evokes the much discussed "incumbent rule" (popularized by pollster Nick Panagakis) and thus expects a decisive break of undecideds likely for Brown. Trende concedes that the rule has weakened over the past decade, a development that he attributes to the faster flow of information in the internet/cable age, something Mickey Kaus has dubbed the Feiler Faster thesis.
Previously, incumbents generally enjoyed strong name recognition, while challengers were typically unknown. The reasoning behind the undecided rule," Trende says, "is that if voters haven't fallen in love with the incumbent by election day, they aren't going to vote for him (or her). The undecideds, therefore, can be expected to take a flier on the challenger." He argues that in a world where information flows faster, races for Governor, Senate and Congress now "receive a lot more scrutiny than they used to," so challengers are better known and the "rule" breaks down. At very least, the rule "is probably inapplicable as a predictive device...when you have two well-known candidates."
This special election, he says, is different, and Coakley is effectively the incumbent:
We have a sitting Attorney General who came out of a contested primary, going up against a more-or-less completely unknown state Senator. She's struggling to get above 50%. All of this points toward a very close final race -- potentially much closer than a week ago when I guessed at a 54-46 spread. Again, this is also consistent with what we're seeing in the variance in the Coakley/Brown numbers. Coakley should be worried.
Coakley should be worried -- and may well face a ceiling of support near 50% -- but I would not count on it. First, Brown is a long way from "completely unknown." Even the Globe poll, completed a week ago, found more than two thirds (69%) of likely voters able to rate Brown either favorably or unfavorably. Second, while Coakley is better known, the name recognition disparity between the two candidates is not unusual, even in the internet age.
Finally, I think Trende misses the best explanation for the incumbent rule, offered three ago by my old boss, Democratic pollster Harrison Hickman:
Hickman pointed out that in the 1980s, the conventional wisdom was to avoid mention of your opponent, a habit that helped explain why challengers won much of the late undecided vote. Now, he said, the general pattern is for incumbents to vigorously attack challengers throughout the campaign. "Incumbents put so much more pressure on challengers then they used to." (See this pre-election column by Dick Meyer of CBS News that includes data Hickman gathered showing the impact of negative advertising on candidate favorable ratings since 1986).
And of course, over the last 24 hours, the Coakley campaign, the DSCC and SEIU have all devoted significant media buys to negative attacks on Brown. Those attack ads may or may not persuade (a different issue), and will certainly be answered, but if we are truly living in a Feiler Faster world, expect the negative messages to disseminate rapidly.
Five years ago, a certain blogger noticed a similar pattern in the polling on the Bush-Kerry race in Ohio. Bush's numbers were amazingly consistent -- four polls had him at 47% and one at 46% -- while Kerry's numbers fluctuated between 45% and 50%. The blogger speculated that this pattern showed "the underlying principles of the Incumbent Rule in action" and boldly predicted that Bush "is likely headed for an Ohio defeat."
Bush carried Ohio, 50.8% to 48.7%. Incumbents did not break decisively to Kerry. If anything, Bush gained over the last two weeks, mostly because he made the race as much about Kerry as Bush. The big question hanging over the Massachusetts Senate race is whether Coakely and her allies can do the same to Brown over the next week.
An article yesterday by Larry Peterson of the Savannah Morning News rehashes much of the Strategic Vision, LLC controversy covered in my column a few weeks ago, but also features some new comments from Strategic Vision CEO David Johnson and this bit of news:
Some pollsters include cross-tabs with their results; others supply them only to paying clients. Until now, Johnson has said he's among the latter.
But [Johnson's] opted for a nod to critics who say his lack of transparency raises suspicions that he may have at least cut some corners.
He's provided the Morning News cross-tabs for a recent Georgia survey. And says he'll provide them to the news media with a survey his firm plans to conduct later this month.
No doubt, they'll be scrutinized by experts.
No doubt. If only someone would publish them. Mr. Peterson, if you have crosstabulations for a Strategic Vision poll, you may be the first, It would help if your newspaper could share with your readers whatever Johnson provided.
If Johnson's promise sounds familiar, it is probably because he made a similar pledge to the Atlanta Journal Constitution's Jim Galloway back in September:
Events of the last week have caused Strategic Vision to come to the same conclusion about its future polling.
"We're going to release all the crosstabs, and put an end to this right now," Johnson said. "That will squelch anybody from saying anything."
We're still waiting.
Finally, Peterson's article includes this curious reference to me under the heading "Partisan Flap:"
Johnson says the flap has a partisan dimension.
A conservative, he works mostly for Republicans; Blumenthal is an avowed Democrat and has served as a director of the opinion research group.
I'm not sure how an "avowed Democrat" differs from an ordinary "Democrat" (except that the former sounds more sinister), but the second half of that sentence is incorrect: I have never been the "director" or president of the American Association for Public Opinion Research (AAPOR) or of any other "opinion research group," as Peterson reports. I did serve for two years as a member of AAPOR's 15-member, all-volunteer Executive Council, something disclosed in almost every column or blog post I have written on this subject.
My column for this week looks at whether Sen. Christopher Dodd was right to insist that "any certain prediction" of his defeat in 2010, had he chosen to run for reelection, "would be absurd." While the odds of an incumbent Senator coming back from a double-digit polling deficit this early are very long, there are a few examples that I review in the column.
Thanks to the pollsters I emailed and my Twitter "tweeps" for digging into their collective memories of Senate campaigns in the not-so-recent past.
Note: most of the poll results for past races cited in the column come from the subscriber-only archives of The Hotline, thus the lack of links.
We have two new polls out in Massachusetts on the January 19 special election to fill Ted Kennedy's Senate seat, and their results could not be more different. The new survey conducted Saturday through Wednesday last week by the University of New Hampshire Survey Center on behalf of the Boston Globe shows Democrat Martha Coakley leading by 17 percentage points (53% to 36%), while a new automated poll conducted on Thursday and Friday by Public Policy Polling (PPP) shows a dead heat, with Brown one point ahead (48% to 47%). A third survey conducted on Monday by Rasmussen Reports has Coakley ahead by nine (50% to 41%).
The disparity of the results is likely to provoke the usual angst about inconsistent polls, debates about past pollster accuracy and the customary conspiracy theories about intentional bias. Forgive me if I don't join in, because as different as these results seem to be, I think the discrepancies actually add up to a consistent and important finding on the state of voter preferences this past week.
Here are three things to keep in mind about polls on the special election:
Turnout Will Matter -- The big spread in results among the polls, and differences apparent within two of them, are all consistent in supporting one finding: The lower the turnout, the better the odds for Scott Brown. These differences indicate that the voters most interested and most likely to vote are Republican, while Democrats are more blase.
Consider the differences in the table below from within Globe/UNH and Rasmussen surveys. Both show a dead even race among the most interested and certain voters, while Coakley leads by huge double-digit margins among all other voters.
Those differences mean the overall results reported by any poll are going to be very sensitive to the "tightness" of the screen or likely voter model used. The more restrictive the screen, the closer the result. My assumption is that the "if you do not intend to vote...please hang up" automated methodology employed by PPP produced an effectively tighter screen and, thus, a likely voter sample closer to the "certain" or "extremely interested" subgroups of the Boston Globe and Rasmussen polls.
Pollsters can't predict turnout - I have yet to see any poll or statistical model that can predict voter turnout with precision, especially in an oddly timed special election like the one in Massachusetts. What pollsters try to do is monitor self reported enthusiasm and interest as compared to previous, comparable contests and try to calibrate their screens and models appropriately (although there is much debate among pollsters about the accuracy of those calibrations and their necessity).
The bigger challenge in predicting turnout, however, has to do with something more fundamental: The size and makeup of the electorate will depend on decisions not yet made by those who may or may not vote on January 19. How many will become more interested and decide to vote over the next 9 days? I'm not sure any poll or methodology can predict that with confidence.
Keep in mind that as of this past week, most Massachusetts voters assumed that Coakley would win in a walk. According to Globe/UNH poll, nearly three quarters (74%) of Massachusetts voters believe Coakley will win, while only 11% say the same about Brown. In that sense, news of a narrowing race could work to Coakley's advantage if it convinces Democrats that their votes are needed and that Ted Kennedy's seat could be lost to the Republicans without their help.
Turnout differences complicate trend tracking - The big spread in these poll results complicates our ability to spot trends. For example, PPP's Tom Jensen last night noted that they fielded their poll on Thursday and Friday, while the Globe/UNH poll was fielded in the first part of last week (Saturday through Wednesday). The earlier start to the Globe poll, he wrote yesterday, "could make a diff[erence] when things are moving fast." That's true in theory but difficult to evaluate in this case because we have to assume we are comparing an apple (the Globe/UNH results) to an orange (PPP) in terms of their likely voter samples.
Now that we have more than five polls released for this race, we should have our tracking chart posted (along with the tracking table, probably later tonight), but be forewarned: The small number of polls and the big "house effects" among them mean that we will really need to limit ourselves to same-pollster comparisons to evaluate trends over the last week. Coakley lead by an average of 29 percentage points on three surveys conducted before the primary last year, but leads by an average of 8 point on the three surveys conducted this past week. So we will see narrowing of the margin between the trend lines on our chart. Has Brown continued to gain over the last week? To answer that questions, we will need o watch tracking polls conducted next week by the same pollsters in the field this week.
Do we have a clear picture today of who will win on January 19 and by how much? Probably not, but we do have a sense of the dynamics that will ultimately determine the outcome.
And one last thought for those covering and commenting on this race: please spare us the cliche about the outcome depending on which campaign's "troops" do the best job turning out their supporters. Field organizations can make a difference, especially when contests are close, but the discrepancies in enthusiasm we are seeing are unrelated to canvassing and phone banking. Conservative Republicans are angry and ready to walk on hot coals if necessary to register their discontent with the direction of government. If he enthusiasm gap narrows, it will be because Democrats come to believe that Martha Coakley shares their priorities, Scott Brown threatens those priorities and the outcome of the election is in doubt.
Update: Via Twitter, Alex Lundry notes that the Globe Poll tests independent Joseph L. Kennedy (no relation to the famous family), while the PPP poll does not. What's interesting about that is that the presence of a "Kennedy" on the ballot appears to cos Republican Brown more support than Democrat Coakley . Also, for what it's worth, roughly 90% of those who support "Kennedy" (4 of his 5 percentage points) have not yet "definitely decided on a candidate, and about the same number (90%) are voters that are less than "extremely interested" in the Senate race.
Update 2: Nate Silver reviews some of the other differences between the three polls.
Update 3: Our chart is now live:
My friend Chris Bowers posted a thoughtful response to my column from this week, which argued that only a small fraction of liberal Democrats have grown disillusioned enough with President Obama to express disapproval. Chris is smart and reality-based about poll data, so his arguments are worth considering.
Chris points out that the liberal Democrat subgroup that I focused on (17% of adults in February 2009 according to Gallup's data, 15% in December) overlooks the small portion of liberals that do not identify or lean Democratic . He estimates that liberal-non-Democrats (or LNDs as he calls them) are now roughly 6% of adults, since Gallup reports that 21% of adults self-identified as liberal during all of 2009. I'll spare the details, but he extrapolates from the available Gallup data to conclude that Obama's approval rating among liberal-non-Democrats is now hovering at just over 50% after falling roughly 15 percentage points since February.
I won't quarrel with Chris' central argument: I have no doubt that a group of liberal-non-Democrats exists that rates Obama less positively than the larger group of liberal Democrats and whose rating of Obama has fallen over the course of the year at a rate similar to independents and moderate Republicans.
I also have no doubt that some liberals are feeling disillusioned. Again, as I reported in the column, the size of the liberal Democrat subgroup fell from 17% in February to 15% in December (I did not ask Gallup for numbers for other months, but the massive sample sizes involved -- over 11,000 in December and over 14,000 in February -- make even a 2 point drop meaningful). So a small sliver of liberals may shifted their allegiance to independent over the course of the year.
My point was mostly that the number of disillusioned liberal Democrats is small relative to disillusioned conservative Democrats and pure independents. Chris argues that liberal-non-Democrats are a small but nonetheless critical swing constituency. That's an interesting but different argument, and I think we can agree that it's not an argument about an erosion of the Democratic "base."
Chris also compares Obama's current approval ratings by ideology to his vote by ideology as reported on the 2008 exit polls to argue that "the only people who have become disillusioned with President Obama are liberals." I know these sorts of comparisons are popular, but I'm not a fan: They compare apples to oranges. Some Republicans will express approval of a Democratic president they did not vote for (and may never support), while some Democrats will express disapproval of a President of their own party, even if they would never consider voting for a future Republican opponent. I would be more convinced by this sort of comparison if it involved a vote or "reelect" question about 2012, rather than presidential approval.
All of that aside, this is a good conversation to continue.
This certainly seems like a banner week for blogging about pollster Scott Rasmussen, as I count at least three entry-worthy topics on the automated polltaker: (1) the flurry of commentary surrounding the piece by Politico's Alex Isenstadt on attacks from the left on Rasmussen's credibility (2) reporting Monday by the liberal blog Think Progress showing that Rasmussen was paid $140,500 by the 2004 Bush campaign for survey research and the good question that raises about whether sites like ours should label Rasmussen as "[R]" for Republican (3) yesterday's new Rasmussen survey of likely voters in this month's special election in Massachusetts.
As commenting on all three at once exceeds both my time and mental bandwidth, I'm going to start with the third and most timely topic, but I will come back to the other two later this week.
Rasmussen's Massachusetts survey, consisting of 500 automated interviews of Massachusetts likely voters conducted in just one day (Monday), shows Democrat Martha Coakley leading Republican Scott Brown by just nine percentage points (50% to 41%). That's a surprisingly narrow lead in a heavily Democratic state that Barack Obama carried by more than 26 percentage points (61% to 36%). Even in 1994, a banner year for Republicans, Ted Kennedy defeated Mitt Romney in an unusually competitive reelection contest by 17 points (58% to 41%).
Nate Silver looked at the Rasmussen results by party and extrapolated that the survey consisted of 52% Democrats, 21% Republicans and 27% in the independent/other category:
Although there are lots of different ways to ask about party identification, typically that's not what we see in elections in the Bay State, as the number of independents is usually much higher (43 percent of Massachusetts voters were independent/other in 2008, and 51 percent are registered as independents). They're also showing an electorate that is 39 percent liberal, 34 percent conservative, and 27 percent moderate; that compares to 2008 exit poll demographics of 31 percent liberal, 19 percent conservative, and 49 percent moderate.
So Rasmussen's theory on this election, basically, is that the people in the middle won't bother to show up; there are many fewer independents and many fewer moderates in their sample than you usually get in Massachusetts. Instead, it will be a race between the bases.
My first reaction is that while there are indeed different ways to ask about party identification (and even more ways to ask about self-reported ideology), it's a bad idea to compare official party registration statistics (that tally how many voters check the box for Democrat or Republican when they register to vote) with survey questions about party identification (typically: "when it comes to politics, do you think of yourself as a Democrat, a Republican or an independent?"). Depending on the state, the two measures can produce very different sets of numbers.
Back in October, Monmouth University pollster Patrick Murray explained why officially "unaffiliated" voters in New Jersey are very different from the "independents" identified by pollsters. I suspected that Massachusetts, with its very high percentage of non-partisan registrants, might produce similar differences, so I emailed Andrew Smith, director of the University of New Hampshire Survey Center who frequently conducts surveys of Massachusetts for the Boston Globe. Here is his take:
[It's] important to point out that a high percentage of the registered Independents in MA (they're actually called unenrolled) are really either Democrats (36%) or Republicans (34%) when you look at PARTY ID. (We use the Univ of Michigan question and I recode leaners into the partisan buckets). Calling them "Independents" makes it look like there is a large pool of free-thinkers out there up for grabs, which is simply not the case ... not in MA, not in NH (a regular media story during the NH Primary is about the large number of Independents up for grabs, a story which sounds good, but has no basis in fact!), not anywhere!
Those who are registered Unenrolled in MA are less interested in elections and less likely to vote than are registered Republicans or Democrats. This phenomenon is consistent across the US (see "The American Voter Revisited" for the most recent in a long line of studies making this point). It's my sense that the 2010 MA special election will have low turnout and the percentage of voters who are registered as either Democrat or Republican will be higher than the percentage registered as such among all MA adults.
Let me explain that a little more slowly. The Boston Globe/UNH poll routinely asks respondents about both their party registration and their party identification. Early in their interview they ask: "Are you registered to vote as a Democrat, Independent, Republican or something else?" On their September survey, according to data Smith provided, 36% of all registrants said they were registered as Democrats, 14% said they were Republicans and the rest (50%) reported they were unenrolled.
Then at the end of their survey they ask: "GENERALLY SPEAKING, do you usually think of yourself as a Republican, a Democrat, an Independent or what?" To those who initially identify as independent, identify with another party or offer no preference they ask a follow-up: "Do you think of yourself as closer to the Republican or to the Democratic party?"
When Smith combined the initial identifiers with leaners in September, he found 50% were Democrats, 32% were Republicans and only 19% remained independent or without a preference. And a cross-tabulation of the UNH party registration and party identification questions shows that more than two thirds of the unenrolled voters identify or lean to either the Democratic (36%) or Republican (34%) parties.
So unaffiliated/independent was 50% on the party registration question, but only 19% on party identification (with leaners allocated). So again, it is a mistake to expect the party identification results produced by a poll to match the party registration statistics produced by the Secretary of State (and an even bigger mistake to weight results of a poll to match those statistics, but that's a topic for another day).
But wait, the party identification results I cited are for adults. Does the Democratic advantage narrow among "likely voters" in Massachusetts?
Yes, although the results that Smith provided from the September Boston Globe/UNH poll did not ask about the special election. They did ask, however, about interest and likelihood to vote in the general election for governor in November 2010, and thanks for Andrew Smith, I can provide those tabulations below:
As the table shows, if you narrow the survey based on interest in the gubernatorial election, the Democratic advantage narrows considerably, from 17 percentage points among all adults (49% to 32%) to just three percentage points among the 35% of adults that say they are extremely interested (45% to 42%). On the other hand, when the UNH pollsters asked voters if they were likely to vote in the November 2010 general election, the Democratic advantage actually grows slightly, to 19 points (51% to 32%). Which of these, or what combination, might provide the best "model" of a true likely voter? There's no obvious answer -- welcome to the highly varied "art" of likely voter modeling.
But wait...the point of all of this is how these numbers compare to extrapolated party numbers produced by Nate Silver for the Rasmussen poll, and at first blush, the Globe/UNH numbers are not that far off. If anything, Rasmussen's party results (52% Democrat to 27% Republican) are more favorable to the Democrats than the Globe/UNH numbers from September.
I had even considered including the Rasmussen numbers in the table above, but decided against because the comparison is a bit misleading. The problem is that Rasmussen asks a very different partisanship question:
If you are a Republican, press 1. If a Democrat, press 2. If you belong to some other political party, press 3. If you are independent, press 4. If you are not sure, press 5.
Does this question ask about party identification or registration? Given the absence of the "do you consider yourself" clause and the use of "belong to, a respondent might interpret it either way. And I'm assuming that since Rasmussen uses an automated method, the respondent can interrupt the question at any time to choose a selection, something I suspect they tend to do more readily toward the end of the interview (especially if they are feeling impatient to get off the phone).
Also, notice that unlike the Globe/UNH question, Rasmussen does not include a follow-up to press independents on how they lean. Yet if you click on a Rasmussen result on our national party ID chart, you will see that when asked of national adult samples, Rasmussen results tend to produce more partisans (about 12 percentage points more, on average) than other pollsters (roughly 9 points higher on Republicans and 3 points higher on Democrats).
Why is it different? Does the combination of a different question and mode effectively push some independents to say how they lean (with a bigger push toward the GOP)? Or do Rasmussen's sampling and calling procedures yield an adult sample that skews more partisan and Republican, even before they apply their likely voter screen and party weighting? You can make a case for either argument, but I don't have conclusive evidence to resolve this puzzle.
Also, Rasmussen typically weights their statewide pre-election samples by party to targets derived in a somewhat fuzzy process. How did they determine their party weighting targets for the Massachusetts survey? How much did the party weighting alter the results (as compared to weighting on demographics alone)? What percentage of Massachusetts adults passed the screen and qualified as likely voters?
And most important, why aren't answers to these questions disclosed on a routine basis on RasmussenReports.com? Keep in mind that Nate Silver had to extrapolate his estimate of Rasmussen's partisan balance, and even that came from crosstabs available to subscribers only.
I'll have more to say later about the questions of bias, intentional and otherwise, that have been swirling around Rasmussen this week. But until pollsters like Rasmussen start disclosing more about the numbers they produce, it is hard to do much more than speculate about whether poll like the one he did in Massachusetts are as representative as they should be. Is this new poll "horribly, terribly wrong?" With so little information to go on, it's hard to say.
Update: Harry Enten (aka "poughies"**), the Dartmouth student who wrote a guest contribution a month ago on modeling gay marriage referenda, takes issue with my conclusion with an intriguing comment below that argues that the Rasmussen poll "has too many Republicans and not enough independents." He reaches this conclusion by comparing the relationship between results for party ID and actual registration in a previous Massachusetts congressional district race polled by SurveyUSA (which asks a more traditional party identification question). It's worth reading.
**And yes, he gave me permission to reveal his identity.
Does Barack Obama have a liberal revolt on his hands? If you read progressive blogs, you would say yes, but if you talk to a representative sample of Americans who describe themselves as liberal and identify as Democrats, as the Gallup does on a daily basis, you come to a very different conclusion. See all the details in my weekly column.
Special thanks for Jeff Jones of Gallup and Jenn Agiesta and Jon Cohen of the Washington Post for providing additional data cited in the column. Also, note that the chart is the column is based on data that Gallup publishes in spreadsheet form so you can create your own charts galore (located on a standard web page showing data for the last few weeks). We all owe Gallup a big "thank you" for that.
Since I posted the National Geographic chart on global health spending and outcomes and included a link to commentary by Andrew Gelman that I characterized as "approving" of the chart, it's only fair to give equal space to an improved version that Gelman posted and explained late last week.
Here is the original version from National Geographic:
Here is Gelman's scatterplot:
And here is is explanation of why he considers the first one "somewhat misleading" (as noted this morning on FiveThirtyEight):
What the scatterplot really made me realize was the arbitrariness of the scaling of the parallel coordinate plot. In particular, the posted [National Geographic] graph gives a sense of convergence, that spending is all over the map but all countries have pretty much the same life expectancy--look at the way the lines converge to a narrow zone as you follow the lines from the left to the right of the plot.
Actually, though, once you remove the U.S., there's a strong correlation between spending and life expectancy, and this is super-clear from the scatterplot.
Of course, the key words above are "once you remove the U.S." The point of the original graphic, as explained by the NGM blog, is that the exceptionally high per-person health care spending is not translating into greater life expectancy as compared to "most other developed nations and many developing ones."
I just want to wish our readers a happy New Year on behalf of everyone at
Pollster.com. We appreciate your continuing support and look forward to
serving you again in 2010. See you Monday!
Barack Obama tops the list of USA Today/Gallup's most admired men, Hillary Clinton and Sarah Palin tied for most admired women.
The AP/GfK poll looks back at 2009, forward to 2010 (full results).
Frank Newport reviews Gallup's top 10 intriguing poll findings of the year; Lymari Morales aggregates links to many more.
A Financial Times/Harris online poll reviews expectations for the new decade.
Zogby Interactive finds terror defines the decade.
Rasmussen finds hardening attitudes on health reform and perceptions that it will raise costs.
Andrew Sullivan uses Pollster charts to call out Rasmussen's discrepancy, JunkCharts takes notice.
Rasmussen finds Ben Nelson trailing in a hypothetical 2012 matchup; Nate Silver opens up a big can of sarcasm.
Chris Bowers updates his 2010 Senate outlook ratings.
Desmoinesdem reports that a GOP Republican candidate Brandstadt is robocalling Democrats.
National Geographic posts a graphic on global health care spending and outcomes; Sullivan raves, Gelman approves:
Emily and I are both taking some time off this week, so I will be posting links to whatever polls are released in a daily 'outliers' entry that will appear in the middle column on the front page. We'll be back to our usual schedule on 1/4.
We hope you are enjoying whatever holiday's you are celebrating, thank you for your continuing support and look forward to updating you on all the polls in 2010!
Poll updates:
Rasmussen: 63% expect health reform to pass; support (chart) unchanged.
Gallup/ USA Today: 39% worry about terrorism in U.S.
Marist Poll: 48% are likely to make a New Year's resolution.
Quinnipiac: 63% of football fans want to junk the BCS in favor of playoffs.
DailyKos/Research 2000: 55% rate President Obama favorably (chart).
...and in other news:
Gallup asks about winners and losers in 2009; Steve Singiser links, feels dirty about it.
PPP releases results on a Webb-Allen matchup collected in August.
Gov. Dave Freudenthal polls on whether to seek reelection (via Memoli).
Eric Keelfeld quarrels with Jim Inhofe's reading of the ABC/Washington Post poll on global warming.
Timothy Geithner isn't afraid of doing "deeply unpopular" things.
Note: This week, we are posting poll updates and links in this abbreviated daily form. We'll be back to our usual schedule on 1/4.
Gallup says 78% identify as Christian.
ABC News asks the country what it wants for Christmas.
Rasmussen finds 66% rate Christmas our most important holiday.
Zogby finds 39% less in the mood to celebrate the holidays this year.
The Marist Poll finds 23% have no problem with re-gifting.
And in non-holiday news...
Frank Newport reacts to David Axelrod's comments on health care polling.
Mark Mellman does not expect a health care bump.
Bruce Drake says health care polls don't tell the whole story.
Michael Barone compares health reform to the Kansas Nebraska Act.
Chris Bowers thinks it's obvious when Lieberman's favorable rating took a hit.
The Census Bureau releases new population estimates; Rich Cohen and Reid Wilson have more in the likely impact on Congressional reapportionment.
Dale Kulp, rest in peace.
Some housekeeping notes: Emily and I will be taking some time off next week, so the pace of blogging and updates will slow considerably here until after the New Year. Hopefully, the pace of poll releases will slow as well. I will update the Obama job approval chart to the extent that the daily trackers release new data (Gallup Daily, for example, did not update today and will not update again until Saturday), and post roundups of new releases and 'outlier' links as warranted. Happy Holidays to all!
Reader DG emails with a reaction to my column and accompanying post on Monday about Strategic Vision LLC:
I am a huge fan of your work and deeply appreciative of all the effort you and your staff have put into making pollster.com one of the best political sites on the Internet.
I do have to confess, though, to being deeply disturbed by the debacle with Strategic Vision. The fact is that there have been problems with the shop for years, yet little attention was paid, even while respectable bloggers (such as electoral-vote.com) made the call in 2004 to stop reporting SV's numbers as they were consistently, and suspiciously, pro-GOP. SV appears to me to be a very bald-faced effort to gratuitously influence national and local debates through nefarious means, and could have seriously damaged the reputation pollsters have worked so hard to build over the preceding decades. Even worse, Strategic Vision was enabled by people who damn well should have known better, like yourself.
Your site is a one-stop shop for journalists, pundits, Administration officials, etc. and anything that gets reported by you is magnified because of that. Moreover, these people do not have the time or training to effectively evaluate polls. As such, you have a responsibility to ensure methodological rigor is adhered by the pollsters whose results you report, and you must begin to call out anything from consistently being an over-the-top outlier to having an uncommonly large (such as Kaiser) or uncommonly small (Fox) party ID spread. I am not even saying to stop reporting polls like Kaiser or Fox, simply make it clear that there are methodological hang-ups with the data that your readership should be aware of. Your "general philosophy" of reporting results as long as the pollster "purports" to adhere to methodological basics is at best lazy, at worst, dangerous. Like it or not, websites such as yours have become such powerful aggregators of information that you must impose some kind of control to limit the ability of the mendacious and malicious from having an undue influence. You must be a Wikipedia, not a Google.
I agree with DG's general argument: Sites like ours need to do more to help readers evaluate individual pollsters and their methods. That was the spirt of the three part series I wrote in August titled, "Can I Trust This Poll," and the reason why I want to use our site to actively promote better methodological disclosure by pollsters.
That said, I'll cop to "lazy" in just one respect: On Monday, I gave short shrift to our "general philosophy." I combines two goals, (1) making all poll results available and (2) providing an informed and critical context -- through interactive charts and commentary -- for understanding those results. The best examples are the interactive tools we built into our interactive charts (the "filter" tool and the ability to click on any point and "connect the dots" for that pollster) to make it easy to compare the results for any individual pollster to the overall trend. We have also devoted considerable time to commentary on pollster house effects both generally and for specific pollsters (like Rasmussen).
I'll also take issue with the idea that we "damn well should have known better" with respect to Strategic Vision. The evidence that they were a "consistently over-the-top outlier" relative to other pollsters is weak. This was Charles Franklin's take three years ago:
I tracked 1486 statewide polls of the 2004 presidential race, of which Strategic Vision did 196. The Strategic Vision polls average error overstated the Bush margin by 1.2%. The 1290 non-Strategic Vision polls overstated KERRY's margin by 1.3%. Further, the variability of the errors was a bit smaller for Strategic Vision than for all the other polls combined.
Try the connect-the-dots-tool on the 2008 Obama-McCain charts for Pennsylvania, Florida, Georgia and Wisconsin (the states where Strategic Vision released five or more "polls"), and make your own judgements for 2008.
But again, I tend to agree with DG's central thrust. We can do better. I am particularly intrigued by DG's comment about being "a Wikipedia, not a Google." What Wikipedia is about, for better or worse, is "crowdsourcing." A few weeks ago, the Wall Street Journal described crowdsourcing as the idea that "there is wisdom in aggregating independent contributions from multitudes of Web users." How might a site like ours help individuals collaborate on efforts to evaluate pollsters? If you have thoughts or suggestions on any of this, we would love to hear them.
So "what happens now" to President Obama's job approval rating, especially in the event that the health care reform legislation passes into law next month? That's the "big question" that my colleague Steve Lombardo asks below. I'd add one more that's just as big: Will passage of health care reform bill change attitudes about the legislation itself?
Let's start with the president's approval rating. Yesterday, Politico's Ben Smith noted his reporting over the weekend that an "administration official" had "predicted" that passage of the reform bill "will send Obama's approval rating up past 60 percent and restore his supporters' enthusiasm." Smith subsequently updated his post: "A senior White House official objects, and says the White House certainly doesn't expect a polling bounce."
Here's an educated guess: That "senior white House official" is someone far more familiar with trends in presidential approval ratings than the first "official." Our current trend estimate of Obama's approval percentage is 48% (50% if we exclude the Rasmussen tracking), with recent individual surveys ranging from a low of 44% to a high of 54%. A "bump" above 60% percent would be extraordinary.
It may not be obvious in Charles Franklin's graphic above, but presidents rarely see sustained approval rating spikes that big (i.e. changes that are not statistical noise or persist for more than a week or so). If you play with the interactive chart of Gallup approval ratings published by USA Today, you will see that virtually all of the real bumps of this magnitude over the last 30 years occurred as Americans "rallied around the flag" in the aftermath of crises like the 9/11 2001 or the taking of American hostages in Iran in 1979 or in the run-up to the wars in Iraq. Charles points out that when presidents recover from dips in approval, as Reagan and Clinton did, their recoveries are usually slow, steady and extended. They have not been about quick gains produced by some event that persist.
The other problem is that perceptions of the "health care legislation now being considered by Congress," or "Barack Obama's health care plan" -- as measured by the various polls included in our chart -- are typically more negative than positive. Sure, an important chunk of that opposition comes from the left (a result reconfirmed in this week's CNN poll), but opposition is still opposition. Some liberal voters unhappy with the current legislation may rally to the president once a reform bill passes, but how many is anyone's guess.
The bigger problem -- especially if you are hoping for a health reform bounce -- is that so few Americans believe they will personally benefit, and majorities worry about the impact on their own costs and the government's budget. Specifically,
- Only 35% of the adults surveyed by the Kaiser Family Foundation believe the reforms will make "you and your family better off." Most of the rest think it will make no difference (32%) or leave them worse off (27%).
- 53% of adults on the recent ABC/Washington Post poll said their own health care "will cost more" if the legislation is passed, as opposed to 33% who believe their costs will go up "if the current system is left as it is now."
- 73% of registered voters surveyed by Quinnipiac University this week agree that "any health care plan that Congress passes and President Obama signs will add to the federal budget deficit," only 18% believe Obama will "keep his promise" that reform "will not add to our federal budget deficit over the next decade."
These perceptions bring me to the second question: Will passage of health care reform affect opinions about the legislation itself? Ben Smith's item yesterday also passed along an argument from Democratic Senator Chuck Schumer that polling on the reform bill will turn around "soon":
"When people see what is in this bill and when people see what it does, they will come around," Schumer said. "The reason people are negative is not the substance of the bill, but the fears that the opponents have laid out. When those fears don't materialize, and people see the good in the bill, the numbers are going to go up."
That assertion is the basis of a memo from Democratic pollster Mark Mellman that Politico and others reported today. Mellman concedes that voters express opposition to a "content-less" plan -- in other words, a plan they know little about:
Focus group research makes clear that voters know little about the substance of the plan--all they know is that some on both the left and the right don't like it and that it is the subject of intense controversy. In essence then, these questions ask people whether they favor or oppose "a controversial plan that is in constant flux." Understood that way, it is surprising we find any support.
Mellman goes on to argue that "the individual elements of the legislation," once described, "are very popular, as is the bill in total, when it is explained."
But those findings lead us to the really critical question: Will Americans come to understand the reform bill as Mellman's questions explained it, especially since most of the key provisions will not take effect until 2014? As I explained in a column earlier this year, seniors soured on the prescription drug benefit in Medicare after it passed. Their assessments reversed, but not until almost two years later when they finally started experiencing the benefit.
If the process of debating and moving the bills through Congress has not brought much clarity, where will attitudes go during the next two election cycles? Here are two very different views of the future. The first comes from the Weekly Standard's Michael Goldfarb:
Liberals seem to be under the impression that health care reform will be like a new entititlement, and that Republicans will run against it at their own peril -- as was the case with Social Security reform in 2005. And they may be right, but not until this monstrous bill actually goes into effect some time in 2013. Which means that for the next four years, Republicans will be able to say whatever they want about the health care reforms that were passed but won't come into effect for years. Republicans will be able to come up with another "death panel" every week.
To put this in some perspective, the "death panel" story was named "lie of the year" by the non-partisan Politifact.com, and Goldfarb's comment was deemed "divisive and intemperate" enough to merit nomination for one of Andrew Sullivan's "Malkin Awards."
The second view of the future comes from the Washington Post's Ezra Klein:
There's a lot of talk over whether the health-care bill should begin before 2014, and whether the long delay will give the GOP sufficient time to foment a backlash.
[...]
A year after the president signs health-care reform, the country will have largely forgotten about it. That's not to say it won't be mentioned in the elections, or argued over in occasional op-eds. But what keeps it on the front page? It's easy enough to write about health-care reform when it's dominating the congressional agenda. When it's waiting to be implemented? Or when it's being implemented, and the main effect is that 16 million people without political power now have health-care coverage? [As for the backlash,] I don't buy it.
As different as they are, both visions have some merit. The Republicans have made it clear they will not relent in efforts to paint the reform as a budget-busting government takeover, and Klein is probably right that the news media will move on to other topics. As such, I would not advise supporters of reform to take much comfort in the health care debate falling off off the front pages. Americans may forget about the bill for awhile, but when Republicans candidates bring it up in 2010, voters will have memories of what they learned in 2009. People won't "see the good in the bill," unless someone makes a concerted effort to tell them. If you believe in reform, I'd recommend listening closely to my friends Sheri and Allan Rivlin:
The answer is that we need more message discipline. We need more voices of support for the underlying effort at health care reform. For every blog post about the public option that will effect fewer than 10% of the public, there needs to be 10 posts about the other provisions of the bill - no more pre-existing condition exclusions, no dropping coverage for people who get sick, insurance exchanges that offer choice and competition, etc. And if Democrats do not stop attacking other Democrats as being too much like Republicans there will soon be a lot more Republicans around to sharpen the comparison.
My last column for 2009 takes a look back at the strangest and most troubling polling story of the year: The allegations of fraud swirling around the firm Strategic Vision, LLC in the wake of a rare censure from the American Association for Public Opinion Research (AAPOR) for a failure to disclose basic information about its weighting procedures and response rates. Please click through for all the details.
The most troubling aspect of this story may be that so little has happened since early October. Most notably: Strategic Vision abruptly stopped releasing polls.
Back in October we decided that in the absence of significantly better disclosure from Strategic Vision, we will no longer include their releases in our charts or publish them among our "poll updates." We were ready to announce that policy when they released a new survey, but...they never did. Our general philosophy has been to include any survey that purports to provide a representative snapshot of voter or adult opinion, but given the troubling questions raised about the integrity of the numbers released by Strategic Vision, we really have no choice but to make an exception.
Not that it appears to bother anyone at Strategic Vision, LLC. One odd epilogue of this story is that, for whatever reason, we can no longer access www.strategicvision.biz from our offices of the National Journal Group and Atlantic Media. Attempts to load their site from any computer that accesses the internet through the Atlantic Media corporate IP address -- and that includes the computers used by The Hotline and anyone at The Atlantic -- produces a "403 - Forbidden" error. According to wikipedia, "The 403 Forbidden HTTP status code indicates that the client was able to communicate with the server, but the server won't let the client access what was requested."
What makes this especially odd is that I am able to connect to www.strategicvision.biz through every other internet connection I have tried. On Friday, I also posted queries on Twitter and in our 'outliers' item asking followers and readers to report if they had any trouble accessing the Strategic Vision website (David Johnson did not return my call requesting comment). No one reported receiving the 403-Forbidden error. I followed up with those who got error messages produced by the apparently over-capacity tr.im URL shortening service asking them to enter www.strategicvision.biz directly. Everyone was able to access the site. So far,no one outside of the Atlantic Media offices has reported receiving a 403-Forbidden error.
Now beyond this circumstantial pattern, I cannot prove that Strategic Vision is intentionally blocking access to their site by Atlantic Media and the National Journal Group. If anyone reading this post encounters the same error, please comment or email me and I'll gladly update this post. But like everything else involving the Strategic Vision story, it is very strange.
Paul Krugman shares "the right poll to look at" on Massachusetts' health reform (via Ezra Klein).
Greg Sargent reports on a health care reform poll question on an individual mandate commissioned by Howard Dean's Democracy for America, Nate Silver critiques its question wording; Brian Beutler has more.
John Sides reviews lessons learned from the health reform debate.
Jon McHenry argues the AARP doesn't speak for all seniors.
Politifact finds Karl Rove's characterization of Obama's approval trend "mostly true."
Lymari Morales blogs on the (behavioral) economics of Christmas (through Gallup data).
Gerald Seib weighs the damage to the American spirit evident in the NBC/WSJ poll.
Fred Yang and Bill McInturff look forward to 2010.
Tom Jensen considers two potential sources of damage to Democrats prospects in 2010.
Andrew Gelman points to a draft article on predicting elections using the "most important issue" question.
And could you help me with some research for a future post? Click this link (it leads to a "pollster" web site) and email me or leave a comment if you get any sort of error message (if you can, please include the text of the message)
The Kaiser Family Foundation has just released its latest tracking survey. Although a majority (54%) still say it "is more important than ever to take on health care reform now (41% choose the alternative, "we cannot afford to take on health care reform now), the December survey shows a "dip on several measures" of health care reform:
The number of Americans who say they personally will be better off if reform passes fell to 35 percent in December, down from 42 percent last month. Meanwhile, 27 percent say they will be worse off, and 32 percent said they don't expect to see much of a difference. Similarly, 45 percent say the country would be better off if health care reform passes down from 54 percent in November. This compares to 31 percent who say the country will be worse off and 17 percent who see no impact. Public opinion in December looks more like it did in August, the last time this debate became so contentious.
In addition to their usual tracking measures, this month's survey also includes open-ended questions asking respondents to offer the "main ways" that their families would be better or worse off if health reform passes (see the toplines for coded results). Their summary of key findings (which includes selected verbatim responses from those open-ends) is well worth the click).
Regular readers will recall that after ABC News polling director Gary Langer posted a new paper on a study conducted by a team of Stanford University researchers that found surveys conducted using op-in internet panels, we subsequently ran two guest posts by pollsters whose companies produce such surveys. One was from Douglas Rivers, the president and CEO of YouGov/Polimetrix (the company that is also the principal sponsor of Pollster.com).
A second came from Humphrey Taylor, chairman of the Harris Poll at Harris Interactive. Taylor's response argued that "social desirability bias" -- the notion respondents might not be comfortable accurately reporting on sensitive behaviors such as smoking and drinking -- might have caused errors in both the live interviewer telephone surveys and the in-person government surveys used as benchmarks in the paper authored by David Yeager, Jon Krosnick and their colleagues.
Today, Langer shares a response by Yeager and Krosnick to Taylor's critique that provides a long and detailed argument that the benchmark measures used in their analysis were not contaminated by social desirability bias. It is "perhaps not," as Langer notes, "the most casual reader's cup of Java," but is certainly worth your time if you read Taylor's guest post here in October.
As just about everyone involved seems to agree, these are serious issues for the survey research profession and worthy of this sort of high level argument. For those interested in a more basic review of the issues raised by the initial Yeager, et. a. paper, I devoted two columns to the subject in October.
Last night's news that Senate Democrats have agreed to abandon both a government run public option and a Medicare buy-in provoked a storm of criticism from the Netroots left. Markos Moutlisas (the Kos of DailyKos) tweeted a desire to "kill" the Senate bill and later explained that his main objection is the mandate on individuals to purchase health insurance: "My position on #HCR -- kill it if it includes mandate. Strip out the mandate, then what's left is inoffensive."
The irony in this latest twist in the health reform debate is that Obama's opposition to individual mandates was the key point of differentiation with Hillary Clinton on health care reform during campaign 2008. This new turn will inevitably draw more attention to an aspect of the reform bill that has received surprisingly little attention so far.
So it's probably worth reviewing what surveys can tell us about public opinion regarding an individual mandate.
Not surprisingly, the monthly tracking surveys done by the Kaiser Family Foundation have some of the most useful data. Their November tracking survey finds 72% of Americans in favor or "requiring all Americans to have health insurance, either from their employer or from another source, with financial help for those who can't afford it" (39% say they favor the idea strongly, 25% are opposed). Support on their tracking surveys has ranged between 66% and 72% over the last 12 months.
However, just last month the Kaiser pollsters demonstrated just how tenuous that positive reaction might be. When the informed those in favor that an individual mandate "could mean that some people would be required to buy health insurance that they find too expensive or did not want," support plummets to just 21%.
So this data takes us back to the main point explored in my column earlier this week. To those in the middle on health reform, affordability is the key word. Requiring that everyone buy affordable insurance sounds mostly inoffensive. Requiring people to buy coverage that is too expensive could prove to be a very unpopular proposition. The real impact that this proposal will have on the cost of insurance is what should be the focus. That's the subject of Nate Silver's analysis this morning and is a worthy subject for further discussion and debate.
We are excited to share some minor changes involving our RSS feeds, Twitter and email updates that we hope will make life a little easier for our regular readers.
RSS - As you may know, we have a variety of RSS feeds set up -- all available here -- that allow you to read Pollster.com blog entries using RSS readers like Google Reader or FeedDemon. We upgraded the process we use to produce these feeds in a way that is mostly invisible except that the feeds should update a little more quickly than before. The new feeds do use a new URL, so if you are reading this entry via one of our previously RSS feeds, it would be a good idea to resubscribe now using one of the links below, as we will shut off the old ones in a few weeks.
Twitter - Since many are starting to use Twitter in lieu of RSS, we have also set up some new automated Twitter feeds that feed a headline and a link for every new blog post on Pollster.com. Details on the specific options below, followed by links to each feed.
Email Updates - If neither Twitter nor RSS are your thing, starting today, you can also sign up for a daily email update that will deliver links to our most recent posts. You can subscribe using the links in the table below right now or via our RSS/Twitter page.
Feed Options - We have set up these RSS, Twitter and email feeds for a variety of different categories of blog content. "All content" gets you everything posted to Pollster.com. "Poll updates" includes only the brief posts on the latest polls. "Analysis" gets you everything but the poll updates from all of our contributors.
We also have automated RSS and email alerts specific to individual authors. The four Twitter accounts listed here for yours truly, Charles Franklin, Steve Lombardo and Kristen Soltis are personal accounts that can and will include commentary beyond what you see posted here.
Follow us on RSS, automated Twitter, or email alerts:
Blog Author RSS, Email alerts, and personal Twitter:
| Mark Blumenthal: |
RSS, Email alerts, Twitter |
| Charles Franklin: |
RSS, Email alerts, Twitter |
| Margie Omero: |
RSS, Email alerts |
| David Moore: |
RSS, Email alerts |
| Steve Lombardo: |
RSS, Email alerts, Twitter |
| Brian Schaffner: |
RSS, Email alerts |
| Kristen Soltis: |
RSS, Email alerts, Twitter |
| Brendan Nyhan: |
RSS, Email alerts |
| Robert Moran: |
RSS, Email alerts |
| Guest Pollster: |
RSS, Email alerts |
|
If you have any questions, comments or complaints about these feeds, please drop us an email or leave a comment below.
My column for the week picks up where this post from last week left off and attempts to answer the question, so what do Americans want with respect to health care reform?
One additional thought I didn't take the time to express in the column. For all its messiness, the legislative negotiation and debate has narrowed to the two priorities that Americans, and especially those most uncertain about the propose legislation, rank highest: making sure affordable
insurance is available and not adding to the budget deficit. From the
perspective of democratic representation, the system is working as it
should.
On their blog CenteredPolitics, my friends Sheri and Allan Rivlin (Allan is a Democratic pollster) focus on an issue related to what Americans want: Why do so few Americans -- only 22% on the most recent CNN/ORC poll -- believe they will benefit from health care reform?
The reason support for health care reform is declining is really very simple. The only messages the public is receiving is that health care reform is bad. Turn on Fox News any given night and the message is this or that health reform bill is bad. Turn on MSNBC any given night and the message is this or that health reform bill is bad. Fox News blames all Democrats and MSNBC blames some Democrats.
They continue with advice for supporters of health reform.
Ordinarily, I might save this clip for the "outliers" feature, but this one won't wait. Jon Stewart discovers some poll numbers mentioned on Fox and Friends that don't quite add up:
And if you think that clip is impressive, wait until a little after the 1:00 mark on this one from October (via Media Matters)
Update (12/11): Politico's Michael Calderone explains the error in the Fox graphic first flagged by Media Matters. The 59% labeled as "somewhat likely" was really the total of "very likely" and "somewhat likely." A correct labeling would have been either "total likely," "very + somewhat likely" or "at least somewhat likely."
Calderone includes this reaction from Fox News:
But Lauren Petterson, executive producer of Fox & Friends, told
POLITICO that she sees no error in the graphic. And for that reason,
there will be no reprimand of staff under the "zero tolerance" policy.
"We were just talking about three interesting pieces of information
from Rasmussen," Petterson said. "We didn't put on the screen that it
added up to 100 percent."
[...]
While Petterson maintains that Fox & Friends didn't err in
displaying the information from Rasmussen, she acknowledges that the
presentation wasn't perfect. "The mistake I do see is we could have
been a little clearer here," she said.
As you probably know, we run a chart on Pollster.com that aggregates the various questions asked on national surveys about general support or opposition to (variously) "President Obama's health care plan," "the health care reform proposals being discussed," and so on. Although these questions show a lot of variation from pollster-to-pollster, the overall patterns and trends have been reasonably consistent: Most surveys have shown more opposition than support, and support has declined modestly over the last month.
Support has fallen in recent weeks from the 43% and 45% range where our trend estimate had been from Labor Day through early November to 39% as of this writing. Meanwhile, opposition has risen from a range of 47% to 49% over much of the fall to 53% today. That trend is reasonably robust across pollsters, showing up on Rasmussen surveys, other polls that sample registered or likely voters, and traditional live interviewer telephone polls that sample adults. Again, the current level of opposition is slightly lower (47%) in the all adult samples than the likely/registered voter samples (53%), but that gap is roughly consistent with what we see for presidential approval.
A few weeks ago, New Republic columnist Jonathan Chait wrote that despite the apparent opposition, "health care reform actually remains quite popular." The problem with the general favor-or-oppose question about health care reform, he argued, "is that it lumps together Obama's critics from the right with those from the left." Other questions he cited appear to show that "majorities of the public either support Obama's approach or wish it went further."
At about the same time, the pollsters at CNN and the Ipsos-McClatchy poll were testing follow-up questions that provided an effective test of Chait's theory. I wrote about the CNN results in my own column, Ipsos pollsters Cliff Young and Aaron Amic wrote up their results in a guest post, and Nate Silver chimed in as well. The gist of both results was that a big chunk of opposition appeared to come from the left: 10% of the total CNN sample who said they oppose the reform bill because it is "not liberal enough," and 12% of the Ipsos sample who say they oppose reform because it "does not go far enough."
This week, the Democratic leaning pollster PPP released results of an automated survey showing more overall opposition and asking a different follow-up question showing only 3% opposing reform because "it doesn't involve government enough." So at the very least we have some inconsistency: The CNN and Ipsos surveys imply that a majority of Americans "favor either the House-passed version of health care reform or something further to the left," as Alan Reifman puts it, while in the PPP survey the combination of those who support or something further left (42%) is still less than the number who oppose reform because "it gets government too involved in health care" (47%).
Why the inconsistency? As usual, there are many possibilities: three different though similarly structured questions, two different populations (registered voters and all adults), two different modes (automated and live interviewer), different timing and ordinary random sampling error. Choose your own theory, I'm not going to begin to speculate about which is best.
I can shed some light on one issue raised earlier today by my Atlantic colleague Megan McArdle. As she points out, "going too far," the follow-up phrase from the Ipsos survey, has many potential interpretations:
I could go down to [the] Cato [Institute] right now and poll 65% support for the proposition that the health care reform doesn't go far enough--in the direction of taking away the employer health care tax exemption, means testing Medicare, and other ideas that no one would call "left". Republicans who want liability caps and bigger HSAs might have similar complaints.
As it happens, I asked the folks at IPSOS for some tabulations this week that help shed a little light on that question. The results are what we might expect in some ways and surprising in others. On the one hand, the 136 respondents who said they oppose the current proposals because they "don't go far enough to reform health care" lean Republican (54% to 38%) -- not surprising, given that most of the opposition is Republican or independent. Like those who think reform goes too far, they overwhelmingly oppose "a single payer system in which the government controls the entire healthcare insurance system" (85% to 15%).
On the other hand, the same group, those who opposes reform because "it does not go far enough", supports the public option concept (59% to 35% - described as "creation of a public entity to directly compete with existing health insurance companies"). In contrast, those who oppose reform because it "goes too far" also oppose the public option by a nearly two-to-one margin (62% to 32%).
The picture you should get from all of this is a lot of fuzziness of opinion, and I think that's the most important point.
Yes, the gradual increase in opposition to health care reform over the last month should concern reform supporters, and yes, it's important to take into account that some of the expressed "opposition" on these questions is coming from the left rather than the right. But my main advice in interpreting these results echoes something that Young and Amic argued here last week: We mislead ourselves by treating the general favor-or-oppose-reform question as analogous to a measure of candidate preference before an election
[P]olling on healthcare reform is quite different than polling on presidential elections because our "true value" is not fixed. This makes the construction of singlequestions impossible and misleading. Such issues are, well, fuzzy and, therefore, only a multiple indicators approach will tell the entire story-some generic, some specific questions.
I'll have more to say on that question in my next column on Monday.
White House press secretary Robert Gibbs offers the quote of the day: "I tell you, if I was a heart patient and Gallup was my EKG, I'd visit my doctor."
Gallup's Frank Newport defends daily tracking: "I think the doctor might ask him what's going on in his life that would cause his EKG to be fluctuating so much."
Obama job disapproval ticks ahead of approval on our chart; conservatives rejoice.
Nate Silver worries about the Daily Kos/Research2000 enthusiasm gap poll; Joshua Tucker suggests a reference point of comparison.
Chris Beam asks how well the Rasmussen "Tea Party" poll question measures actual support for a Tea Party.
Gary Langer warns against literalism in data analysis; explains why the margin of error grows (slightly) when pollsters interview via landline and cell phone.
Pew Research notes that while only 23% can identify cap and trade legislation, 50% support the idea.
John Sides reviews a not-so simple poll question.
Michael Meyers creates tag clouds to show that neither Bush nor Obama promised "victory" via surge.
Chris Bowers ponders the connection between unemployment and presidential popularity.
Jay Cost looks for the sources of opposition to health care reform.
Resurgent Republic compiles results showing independents favoring military tribunals for 9/11 terrorists.
MoveOn finds is members oppose the Afghanistan War, Steve Benen has more.
Five Texas newspapers team up to conduct election polling in 2010.
Northwestern University journalism student Justine Jablonska explains how to make sense of polls.
ISR has a facebook fan page.
