Alex Lundry is a political pollster, microtargeter, data-miner and data-visualizer. He spends most of his time searching for big ideas hidden inside of big data. He has visualized historical tax receipts,White House visitor logs, ideological estimates of Supreme Court justices (called a "very cool graphic" by the Washington Post), and hundreds of thousands of survey interviews. In 2009, Politics Magazine named him a "Rising Star."
President Obama's recent appointment of Yale professor Edward Tufte to the independent commission charged with tracking stimulus funds underscores the growing importance of data visualization in both public policy and political debate.
Tufte is inarguably the modern era's leading authority on data visualization, the transformation of raw data into graphical form. These visuals - graphs, charts and other types of information graphics - are frequently responsible for remarkably stunning revelations and deep insights that may otherwise have been obscured among large and cumbersome spreadsheets or databases.
The federal stimulus is just that - an incomprehensibly enormous $787 billion piece of legislation being distributed across 50 states, 435 congressional districts, 28 federal agencies and over 160,000 individual projects. President Obama's challenge is to convincingly show the American public that their money is being well-spent.
Thanks to a neurological phenomenon called the pictorial superiority effect, the human brain is hardwired to find visualizations more compelling than a spreadsheet, speech or memo. So it's no wonder that Obama has turned to a data visualization guru for the monitoring of his administration's largest legislative accomplishment to date. Meaningful visualizations of stimulus data can make the project more transparent, accountable, and could ultimately even impact the legislation's perceived success.
Transparency, allowing the public to see the who, what, when and where behind stimulus funding, will help alleviate any perceptions of waste, inefficiency, or unfairness. Indeed, the most common criticisms of government spending are that it is unequally or unfairly distributed across communities, that it goes to unworthy projects, or that it simply isn't doing those things it was meant to do: stimulate the economy and create jobs. But states like California have already engaged with design firms to visualize the disbursement of stimulus funds, mapping dollars to projects and locations, in turn increasing voters' investment in the bill as they see its direct benefits to their community.
Data visualization can also make the federal stimulus more accountable, revealing fraud, abuse or even honest mistakes. A case in point: the public outcry over the recent revelation that stimulus funding seemed to go to congressional districts that didn't exist. This seemingly innocuous data entry error quickly became an anti-stimulus talking point, whereas a simple visualization of the data could have revealed the problem well ahead of its entry into the news cycle.
Finally, there is also great political advantage to effective visualizations of the Stimulus Act. Convincing voters of its merit will take more than declarative speeches and number-drenched spreadsheets, and the Obama administration knows this. Their appreciation for the political power of data visualization was on display last month when it released a graph of weekly job losses since December 2007. The bars, color-coded by presidential administration, tell a distinct, if not debatable, story about the stimulus' impact. The visualization took the internet by storm as pro-stimulus voters shared, linked, blogged and tweeted the image, and anti-stimulus voters denounced it as infographic propaganda, all the while scrambling to create their own charts telling their side of the story.
These chart wars are only going to become more and more common in political discourse. President Obama understands this acutely - and this was certainly the subtext in appointing Edward Tufte to the stimulus board.
Robert S. Erikson is a professor of political science at Columbia University.
If the health care reform bill finally passes Congress and is signed into law, what will be the response of public opinion? Would it turn out that support goes up once the public learns the details of the law, as the Democrats claim? Would Obama's image improve following successful passage? Which party would receive the net benefit?
For clues, we can turn to public opinion polls from the 1960s both before and after passage of Medicare in June 1965. There was a far lesser density of public opinion polling in that era, but the small set of available polls from back then (retrieved via iPOLL) reveal the following.
During the 1965 health care debate, public opinion was ambivalent on how to deliver health care to seniors. Whether a plurality favored President Johnson's public plan or the Republican alternative designed to expand private coverage depended on the exact question wording. But there was considerable popular support for Medicare when presented to the public for an up-or-down vote. In a February 1965 Harris Poll, 62 percent answered affirmatively when asked "Do you favor or oppose President Johnson's program of medical care for the aged under Social Security?"
The lesson for today is that following passage in June 1965, support for Medicare increased further. By December of 1965, the percent who told Harris they "approved" of Medicare rose to a consensus of 82 percent. Ever since, the public's support for Medicare has never been in doubt.
Perhaps even more telling, support for Johnson's handling of health care rose even as his overall popularity began to plunge. In April 1965, when President Johnson was enjoying 67 percent approval in the Gallup Poll, a similar 65 percent told Harris they favored "what [Johnson] has been doing on Medicare under Social Security." After passage, in October 1965, 80 percent of Harris respondents rated Johnson's job as "excellent" or "very good" on "working for Medicare for the aged."
The year 1966 brought a fading of Johnson's political fortune, largely due to declining support for his handling of Vietnam. By August 1966, Johnson's overall approval in the Gallup Poll had sunk to 47 percent. But in the same month, the percent in the Harris Poll who rated Johnson's performance as "excellent" or "very good" on Medicare held firm at 84 percent.
The lesson of 1960s polling can provide some encouragement to today's Democrats. If the analogy holds for today's political scene, a Health Care Reform Law of 2010 will become popular and Obama will be credited with a success in the eyes of public opinion. But like all analogies when applied to today's politics, it must be interpreted with considerable caution. Medicare was considerably more popular at the time of passage than is the current health care bill on the eve of its final vote. And Medicare's opponents at the time of passage were weaker politically than today's Republican leadership, united in opposition.
Harry Joe Enten is a junior at Dartmouth College and will be interning with Pollster.com this spring and summer.
Yesterday, Nate Silver posted a well thought out post on why the 50% incumbent rule no longer applies. I think Nate's post is straight on, but I think that he misses a potentially larger point. In his chart, you'll notice something very interesting: no incumbent from 06, 08, or 09 won when trailing by more than 1.5 points in the January to June average of polls. I think that points to potentially very large problems for Democrats in the 2010 United States Senate Elections. Why? If current polling averages hold through June, the Democrats would be on the verge on losing the United States Senate, according to Silver's findings. What follows is a simple rundown of the top (and some not so top) United States Senate races involving seats held by Democrats. I apply Nate's rule of averaging all the polls available (including partisan ones). I supply a two month (starting in January as Nate did) and six month (using length of time of) Nate's average (when available) to try and catch short and long term trends. To be fair, I take only the highest polling Republican candidates. I don't intend this to be a be all end all, but the results are still amazingly scary for Democrats.
I find 6 Democratic incumbents who would most likely lose re-election, if the polling averages held through June. One Democratic incumbent does lead, but she is also going to have a difficult time in her fight for re-election.
1. Arkansas- Democratic Senator Lincoln trails by an average of 21.5 points since January to Rep. John Boozman and 10 points since January and 5.4 points since September to Gilbert Baker. Not only is Lincoln in trouble, but her trouble seems to be getting worse by the day. Unless a dramatic turn occurs in the polls (and considering Boozman is the likely Republican candidate), Lincoln is probably a goner.
2. Nevada- Senate Majority Leader Harry Reid is in major trouble. He had trailed potential Republican candidates Danny Tarkanian by an average of 7.8 points since January and 6.9 points since September and Sue Lowden by 7.8 points since January and 8.2 points since September. Such polling and past history would argue that Reid is dead in the water; however, the emergence of Tea Party candidate Jon Ashjian has thrown the race somewhat into doubt. Reid still trails both candidates, but, with Ashjian in the race, Lowden leads by only 5 and Tarkanian only leads by 1 point in the only poll including the Tea Party candidate. Still, Reid's position is precarious at best, and he would almost definitely lose to Lowden, if the averages held through June.
3. Colorado- Senator Michael Bennet is not an incumbent in the traditional sense (he was appointed to the post), and both appointed Senators Bob Menendez and Roger Wicker were among the incumbents who performed significantly better than the average of polls between January and June indicated. Bennet is also facing a primary challenge from Andrew Romanoff. If Bennet makes it out of the primary (an if, but the only poll conducted so far indicated Bennet leads), he trails by 9.5 since January and 9.3 points since September to Republican Jane Norton, 2.3 points since January and 2 since September to Tom Wiens, and 2 points since January and 0.8 since September to Ken Buck. If those leads hold (and they seem to slightly be expanding), Bennet is in major, major trouble especially against Norton. Ramonoff does not do much better; he trails Norton by 6.8 points since January and 7.7 points since September, 1.7 points since January and 1.5 points since September to Tom Wiens, and 2 points in both the January and September averages to Ken Buck. Romanoff seems to be a slight underdog, especially against Norton.
4. Pennsylvania- Republican turn Democrat Arlen Specter is in as in much troubled as the 2010 New York Mets. He trails Republican challenger Fmr. Rep. Pat Toomey by 8.8 points in an average of the polls since January and 4 points since September. Both of these averages would render him on life support applying Silver's standard come June. Specter is being challenged in the Democratic primary by Congressman Joe Sestak. Specter currently leads Sestak in that primary by 20 points (a lead, which is growing). If Sestak somehow won the primary, he trails Toomey by an average of 12 points and 6.9 points in polls conducted since January and September respectively. In a Republican year, it would be very difficult for Sestak to make a comeback from being this far back.
5. New York- Senator Kirsten Gillibrand, like Senator Bennet, is an appointed Senator in trouble. Her negative net approvals indicate a good challenger would have a fair shot. The mostly unheard of Bruce Blakeman trails Gillibrand by 22 points and 24.7 points in the polling average since January and September respectively. Potential candidate Fmr. Governor George Pataki would make it a race. He leads Gillibrand by 5.5 points in the average since January and 1.4 points in the average since September. Pataki leads the other potential Democratic candidate Harold Ford (who has trailed by 14 points or greater in every primary poll against Gillibrand) by an even larger 14.8 points since January. If Pataki does get into the race, he would be a very formidable challenger. Of course, even if Pataki does not enter the race, Gillibrand's approvals leave her in a vulnerable position.
6. Washington- Senator Patty Murray is not the first Senator you think of as in danger. The Cook Political Report, Rothenberg Political Report, and Larry Sabato's Crystal Ball all have this race rated as safely Democratic, but one Republican challenger could make it a race. Republican Dino Rossi leads by 2 points over Murray in two recent polls. Rossi nearly won the Governor's mansion in 2004, losing in a recount, and he only lost by 6.5 points in 2008 when President Obama carried the state by 17 points illustrating his appeal as a statewide candidate. If the recent polls hold, Rossi could give Murray one heck of a fight.
7. California- Democrat Barbara Boxer leads her strongest challenger Republican Tom Campbell by on an average of 5.5 points in polls taken since January. Boxer has the edge in this matchup, but the Cook Political Report and Larry Sabato's Crystal Ball have the race at only Lean Democratic. Keep in mind, Republicans Carly Fiorina and Chuck DeVore poll considerably weaker. As such, Republicans should hope that Campbell wins the nomination, if they are looking for the candidate with the best shot at winning. He leads in recent primary polls.
The two and six month polling averages indicate that the Republicans are in a position to defeat six Democratic incumbents. This position seems to have strengthened over the last two months. In two states, New York and Washington, they need to hope they can recruit two candidates. If they do and the averages hold, Republicans could easily be up to 47 seats in the United States Senate.
When you combine these races with open Democratic seats, the Democratic majority looks like it could fall.
1. North Dakota- Republican Governor John Hoeven leads all opponents by at least 21 points, and he is over 50% in all polls conducted since January. He'll win unless a divine miracle happens for the Democrats.
2. Delaware- Congressman At-Large (meaning he represents the entire state) Mike Castle leads Democrat Chris Coons by an 22.7 and 20 points in polls conducted since January and September respectively, and he has been over 50% in every poll ever conducted in this race. Coons is not as dead as the Democrats in North Dakota, but he has a very high hill to climb.
3. Indiana- Republicans Fmr. Senator Dan Coats and Fmr. Representative John Hostettler lead both Democratic Congressmen Brad Ellsworth and Baron Hill by at least 14 points in the only poll conducted since Democratic incumbent Evan Bayh announced he was not running for re-election. We'll have to see if this poll is an aberration, but the Cook Political Report already has this seat leaning Republican.
4. Illinois- Republican Congressman Mark Kirk trails Democrat Alexi Giannoulias by 0.2 points in the polling average since January, but Kirk leads Giannoulias by 0.4 points in the average since September. It could go either way.
In conclusion, I am by no means saying that the Republicans will take back the Senate; however, the polling in conjunction with past results indicate that it not that long of a shot that they do. Democratic candidates seem to be consistently weak over the last six months, and the Republicans seem to be moving into a stronger position in the last two months. Keep in mind that in 06 and 08, Democrats pretty much swept all the hotly contested races (save Tennessee in 06 and Georgia in 08. In these years as well as 1994, the party who lost seats (Democrats in 1994 and Republicans in 2006 and 2008) did not win a single seat belonging to the other party.
If the national environment for Democrats does not improve, these polling averages probably will not get that much better for Democrats. And if the averages do not get better, Silver's findings show the Republicans are at least in a position to win 10 seats and take back the United States Senate.
A survey The Washington Post conducted in Massachusetts last week in the aftermath of the U.S. Senate special election shocker was unusual for a political poll in that it interviewed non-voters as well as voters. (See Post story and full release with topline and other links.) I consulted for the Post on this project – which was fielded in conjunction with the Kaiser Family Foundation and the Harvard School of Public Health – and with permission I’ll take a look here at Massachusetts adults who sat out the special election. I’ll particularly focus on those who said they did vote for president in 2008 – to try to assess evidence of an "enthusiasm gap" in Republican Scott Brown's victory over Democrat Martha Coakley for the seat long held by the late Democrat Edward M. Kennedy, and possible implications for the state's elections in November.
The sample of special election non-voters was small - 242 adults (sampling error plus or minus 8 points) - but it's safe to say they generally differed from voters little if at all on many questions such as the direction of the country and whether Brown should work with or mainly try to block Democrats when he gets to Washington.
Non-voters also were no different from voters on overall support for proposed health care reform, though they were more likely than voters to think those changes would be good for themselves and their family and for Massachusetts. Those who didn’t vote last week also may have been slightly more likely to favor a bigger role for government; only 37% of them, compared to 47% of voters, said government is doing too many things best left to businesses and individuals.
Of those 242 who did not vote in the special Senate election, 104 said they did vote for president in November 2008. We start getting into pretty big sampling errors with that subgroup but I feel comfortable concluding that what for shorthand I'll call these "occasional" voters were predominantly Democratic in their outlook:
- Seven in 10 occasional voters said they voted for Obama in 2008 and about as many approve of how he's handling his job now (Obama got 62 percent of the Massachusetts vote in 2008, and 61 percent job approval in the Post poll)
- Nearly half call themselves Democrats (this includes independents who lean Democratic), just one in 10 Republican
- Fewer than three in 10 said they feel "enthusiastic" or "satisfied" about policies offered by Republicans in Washington, while nearly six in 10 felt that way about the Obama administration's policies
- Only around two in 10 said that when Brown gets to Washington, he should mainly work to block the Democratic agenda and should stop Democrats on health care reform; nearly all the rest said he should work with Democrats.
On these and other measures, these occasional voters looked more like people who cast ballots for Coakley than Brown supporters. That suggests some people who on the whole might have been inclined to vote Democratic were not sufficiently motivated to turn out last week - evidence supporting the notion of an enthusiasm gap that the Democratic get-out-the-vote operation could not overcome.
The poll had little good news for Democratic Gov. Deval Patrick as he prepares to face the electorate himself. Just 40 percent of who voted in 2008 but not last week approved of how Patrick is handling his job – a number not significantly different than among those who did vote last week (36 percent). And should Democratic-oriented voters remain less inclined to turn out in November, obviously that could hurt other Democrats on the ballot, for U.S. House and other offices.
Mike Mokrzycki is an independent consultant who was the founding director of the Associated Press polling
unit. He may be reached at mike@mikemokr.com. His guest contribution is cross-posted from his blog, MJM Survey Musings.
One thing is certain about the polling in the last days before
Tuesday's special election in Massachusetts to fill the late Ted
Kennedy's U.S. Senate seat: Someone's going to end up being very, very
wrong.
Polls completed in the past week and recorded at Pollster.com
range from a 14-point lead for Democrat Martha Coakley - just weeks ago
considered a shoo-in in heavily Democratic Massachusetts - to a
15-point advantage for Republican Scott Brown, who has become a darling
and major fund-raising beneficiary of conservatives nationwide.
I'm not going to do a deep methodological dive into all these polls to try to explain the differences. Pollster.com and Fivethirtyeight.com
have done their usual stellar job with that already, including
analyzing the extraordinary uncertainty inherent in trying to determine
who really will vote in this mid-January special election.
I will try to provide a little perspective as someone on the ground in Massachusetts who also knows a thing or two about polls.
My hypothesis: While Brown supporters clearly are more enthusiastic
than Coakley backers, that may serve him relatively better in the
pre-election telephone polls than it will Tuesday.
I've lived in Massachusetts on and off since 1980 and I can't ever
recall Republicans here as energized as they are now. Sure, they had a
16-year run in the governor's office despite the state's overall
leftward tilt. But Bill Weld, elected in 1990, was fairly unusual -
socially liberal enough that "Weld Republican" became its own label.
Paul Cellucci sure didn't inspire a lot of passion and I can't say Mitt
Romney did either, with his eye on the White House all along. Brown
seems an agile campaigner but I don't think his personal charisma is
what's charging up Republicans here and elsewhere; rather, it's the
once almost-unthinkable notion that any Republican might
actually win the seat Ted Kennedy held for nearly half a century,
especially with such extremely high stakes for policy and politics
nationally.
This enthusiasm is abundantly evident in internal data from numerous
polls. I'd add a couple anecdotes: I don't put stock in lawn signs but
when you see a voter (like someone on the main street in my town)
posting a handmade Scott Brown placard, or an ice cream stand
using its roadside sign to advertise "VOTE FOR SCOTT BROWN," it may be
an indication something beyond rote partisanship is at work.
This race has been Coakley's to lose, and she's seemingly been doing
her best to do that. The most recent example was in a radio interview
the other day when she called Curt Schilling - famous for helping pitch
the Boston Red Sox to a long-awaited World Series championship in 2004
on an ankle stitched together and visibly bleeding through his sock - a
New York Yankees fan, of all things. In little more than the
time it used to take a Schilling fastball to reach the plate, his
recorded voice was on my phone telling me this faux pas was
proof Coakley was out of touch with Massachusetts voters. Silly,
weighed against the import of issues such as health care reform?
Perhaps. But - last baseball metaphor, I promise - Coakley served up a
big fat meatball and I sure don't blame the Brown campaign for hitting
it out of the park.
Schilling's was one of countless phone calls we've gotten on this
race since before the primaries last month. Many have been "robo-calls"
like his (as I write this paragraph I just got one from Brown's
daughter), though plenty feature live human beings (like someone from
Coakley's phone bank who called as I started writing this post).
At this point it's hard to blame people in Massachusetts for
screening incoming calls even more so than usual. For years there's
been plenty of screening, part of the reason why response rates for all
kinds of telephone polls have declined dramatically. (An article in the Winter 2009 issue of the journal Public Opinion Quarterly
(subcription required) gives response rates for numerous respected
telephone polls it cites and many of them barely crack 10 percent. A
response rate greater than 20 percent now is extraordinarily good.)
Response rates are even lower for automated polls, which use a recorded
voice for interviews and require respondents to punch in answers on the
touchtone keypad.
But - and this is an important "but" - a growing body of research
indicates decreasing response rates have not hurt the accuracy of
survey estimates. That happens when there's no systematic difference
between those who cooperate and take the survey and those who decline.
I'm thinking the Massachusetts Senate race may be a case where we do
see non-response bias in surveys. It comes down to relative enthusiasm
for the candidates. It's tough to prove, but I'd venture a guess the
dynamic works like this:
- Republicans are excited Brown might win and thus more likely to
answer their phone and listen to political messages -possibly be
invited to take a survey - when the phone is practically ringing off
the hook with such calls. I suspect they'd be particularly enthused to
participate in a poll and tell the world they're voting for Brown, to
help build the sense he has unstoppable momentum. These folks certainly
will vote but there's no upside in Brown's election-day numbers
compared to the pre-election poll estimates.
- Democrats may be demoralized and scared after several weeks of
Coakley campaign missteps and bad headlines. They may not be all that
eager to pick up the phone for political calls. They also might be more
skeptical of or angry about polls since they've been such downers for
Coakley and President Obama lately, and thus, I would speculate, more
likely to take a pass if invited to participate in one. None of that
means these folks are less likely to vote, though - by now any sentient
Democratic-leaning voter will know Coakley needs all their votes, and
what's at stake. They might not be happy about how Coakley has run her
campaign but they'll still be motivated to vote by a desire to deny a
Republican the chance to do serious harm to Obama's agenda from Ted
Kennedy's old seat. Obama is in Massachusetts today to remind them of
exactly that (not that they're necessarily all that enthused about him
at this point, either).
Of course, truly independent or "swing" voters are another vital
factor, and if Brown wins enough of them he could overcome the inherent
Democratic advantage in Massachusetts. But I'd think enthusiasm, or
lack of it, would be more of an issue among stronger partisans.
In pollster speak, what this boils down to is "differential
non-response," where one candidate's supporters are more likely than
the other's to take a survey. It's suspected to be a big reason why
exit polls in recent years have tended to overstate support for
Democratic candidates. In the Massachusetts special Senate election I
suspect it's inflating the Republican's poll numbers. Coakley has room
to outperform the polls Tuesday even if her natural base is motivated
by nothing more than fear of what would happen if her opponent pulls
off an historic upset.
Humphrey Taylor is chairman of the Harris Poll at Harris Interactive, which conducts surveys on the internet.
I have read Yeager and Krosnick's recent, well researched essay on this subject with great interest. It was written in response to my comments (of October 26) on their paper comparing the accuracy of RDD telephone surveys and Internet surveys conducted with probability and non-probability samples posted in August 2009.
In their new essay Yeager and Krosnick provide evidence to refute my two criticisms of their original paper.
"Consistency"
My first criticism was the data they presented, even if completely accurate, did not show that the "RDD telephone data was consistently more accurate than the non-probability surveys." Yeager and Krosnick agree with me that the Harris Interactive's data points are closer to the benchmarks on two of the six items they used by 2.64 and 0.56 percentage points. They argue that the word "consistently" was justified because these differences are small (and they are). So this is really a question about semantics. The Oxford English Dictionary defines consistently as "uniformly, with persistent uniformity." IF the RDD sample produced more accurate data on six out of the six variables, that would be consistently more accurate but four out of six is not.
Social Desirability Bias
Yeager and Krosnick agree with me that "Internet surveys are less subject to social desirability bias than are surveys involving live interviewers," and provide some useful references to support this conclusion. However, they argue that "the measures of smoking and drinking we examined were not contaminated by social desirability bias."
Smoking and Drinking
The authors provide several hypotheses, other than social desirability bias, that might explain why Harris Interactive's online surveys found more drinkers and smokers than the benchmark survey and the RDD survey, both involving live interviewers. For example they suggest that "perhaps the people who agreed to participate in the opt-in Harris Interactive Internet surveys generally possessed the studied undesirable attributes at higher rates than did respondents to the RDD sample." This is possible, of course, just as it is possible that Harris Interactive's online respondents are much more likely to be gay or lesbian, and less likely to give money to charity, to clean their teeth, believe in God, go to religious services, exercise regularly, abstain from alcohol and drive under the speed limit. However this hypothesis sounds very like the argument used by the tobacco industry for thirty years or more that the correlation between smoking and lung cancer could be because those prone to this disease were more likely to smoke .
Yeager and Krosnick also address the evidence I quoted from the Federal government's NHANES survey which found that based on blood samples more people had apparently smoked than admitted to smoking cigarettes when they were interviewed.The authors present several hypotheses to explain this difference, all of which may be true but none of which are proven. It is surely true, as they suggest, that part of the increase is due to people using tobacco in ways other than smoking cigarettes. But they also argue that the data from the blood samples cannot be usedas a check on respondent's answers because for most respondents there was a gap of "between two and nine weeks" between the interview and the drawing of the blood sample, and that smoking behavior may have changed during this time. If so this would be a big increase in the number of smokers over a short time, and this trend if it continued would rapidly increase the number of adult smokers, which has not happened.
As I suggested at the beginning, I am impressed by Yeager and Krosnick's research on the literature on this topic. Furthermore, I concede that I have not proved that social desirability bias is the only possible explanation for the differences between our online survey data and the live interviewer surveys on smoking and drinking (including our own). However, Yeager and Krosnick have not proved my hypothesis is wrong and their explanations for these differences are also hypothetical and, I submit, less plausible.
The 7 "secondary demographics"
This was not part of my argument about "were the benchmarks wrong?" but was in the original paper by Yeager and Krosnick and was referenced again in the authors' reply, so a few comments may be useful. The seven variables were picked by the authors from a long list that they might have used. Had they chosen other variables the results might have told a different story, but we do not have those data. The average errors involved were modest ( 3.0 and 1.7 respectively) and the differences between the two samples was small. One of the seven variable was the number of adults in the household, a variable for which Harris normally weights; I am not sure why it was not weighted in this survey. By far the biggest error in the Harris survey was for people in households with incomes of between $50,000 and $60,000 (why that particular bracket and not others?) Replies to questions about incomes are notoriously unreliable and here again social desirability bias may well be at work.
One other thing
At the risk of extending this dialogue, there is one other important point that should be made about the research on which Yeager and Krosnick have based their paper and their conclusions.
They reported that the RDD telephone survey used in these comparisons was very different from the typical telephone surveys used by any of the published polls. It was in the field for six months, non-respondents were offered a $10 incentive to participate, and it achieved a 35.6% response rate. In other words, the sample was presumably much better than the samples used in all the published telephone polls, which do not pay incentives, are usually in the field for only a few days, and achieve much lower response rates. Even if the RDD survey used by the authors had been more accurate than our online poll (which, of course, I dispute) it would say nothing about the accuracy of the RDD telephone polls published in the media.
Doug Usher is Senior Vice President and Research Director at Widmeyer Communications.
Margie Omero is President and founder of Momentum Analysis LLC and a frequent Pollster.com contributor.
Back when the two of us collaborated on polling presentations (in the mid-to-late 1990s), PowerPoint had more competitors and transparencies were as common as LCD projectors. Even today, it can sometimes take more technical savvy than it should to create a slide that's both legible and informative. But presenting data in an understandable visual way is one of the most important things pollsters do for their clients. While pollster.com usually chooses to simply lead by example on this topic, we thought we'd have some back-to-work fun.
Inspired by this comment from a couple of months past, we've selected a few examples of (frankly) awful charts. Certainly we're not perfect. But it's time to take a stand! Clients pay us to help use data to build effective strategies - and part of our job is to present graphics that illuminate, not confuse and distract.To paraphrase a famous political philosopher, pollsters of the world, unite - the only thing you have to lose is your outdated and clunky templates.
To this end, below are a few examples of subpar graphics from mainstream polling firms - to give a sense of just how far we have to go as a profession.To protect the guilty, we've obscured references to the pollsters, but have kept everything else intact.
This is just a start - do you have some better examples of graphic crimes by pollsters?
EXAMPLE A: Getting too much interpretive analysis out of very little data:
EXAMPLE B: Color schemes that add no insight.
EXAMPLE C: Do big numbers convey your point more effectively?
EXAMPLE D: Are two graphics always better than one?
EXAMPLE E: Really?
Here are a few "action items" for pollsters to think about as they put together charts for presentations.
- What is the point of your chart - and what data are critical to making that point? Try to have the exact amount of information to make your point, not too little and not too much.
- Is it accessible to a non-pollster? The goal of an effective presentation is for it to be passed along to (and understood by) many.
- Are there extraneous slides/data that can be effectively summarized in a few words? Yes, a picture is worth a thousand words, but in too many presentations hundreds of numbers can be replaced by a few summary points about significant subgroups.
- Is every additional color, font, chart type and piece of clip-art required to make an additional insight? If not, then refrain. What once was seen as "plain" is now more likely to be viewed as "crisp" and "concise."
Let's make it a New Years' resolution: simpler, clearer data slides!
Mike Mokrzycki is an independent consultant who has studied implications of the growing cell-phone-only population for survey research. He was the founding director of the Associated Press polling unit. He may be reached at mike@mikemokr.com.
Sometimes a study is more intriguing not for what it finds but for what it doesn't. That's the case with the latest federal estimates, released this morning, of how many Americans can no longer be reached by landline telephones.
First, what the semiannual update from the Centers for Disease Control did find: Steadily worsening news for surveys that exclude cell phones. Americans keep abandoning landline phones at about the same pace as in the last couple years - in the first half of 2009, 21.1 percent of adults live in households with no landline, up from 18.4 percent in the second half of 2008. By a slightly different measure - particularly relevant to random digit dial surveys using households as a sampling frame - 22.7 percent of households now have only wireless phones, up 2.5 percentage points from six months earlier. (With sample sizes of 12,447 households and 23,632 adults, sampling error for overall results is generally around plus or minus 1 percentage point.)
Somewhat surprisingly, though, the CDC's National Health Interview Survey (NHIS) - the ongoing in-person study that is the benchmark for telephone status estimates in the United States - did not find a disproportionate increase in cord-cutting overall or among the poor or unemployed, despite the deep and sustained economic downturn. In the latest NHIS, 33 percent of those households falling below the U.S. Census poverty threshold were cell-only, up 2.1 percentage points from six months earlier; 14 percent of those who were unemployed or gave "something else" as their employment status (but weren't students) were cell-only, up 3 percentage points from the previous report.
"We would have expected that the recession would have led to outsized increases, both in overall rate of wireless substitution and also perhaps among the poor relative to those with higher incomes. We did not see that effect," Stephen J. Blumberg, co-author of the CDC study with Julian Luke, told me this morning.
"It appears that lifestyle issues, such as where you live, who you live with and age are still bigger predictors of cord-cutting," Blumberg said. "Ever since we've been tracking these data, income has not been a strong predictor of being wireless-only. Yes, the poor are more likely to be wireless-only than those with higher income, but that has largely reflected the fact that people who have substituted wireless for landlines are younger, more likely to still be in school, and more likely to be renters than homeowners."
The NHIS not only measures the cell-only population but attempts to gauge what proportion of Americans still have landlines but can't really be reached on them, contributing to non-coverage for survey researchers. The NHIS began tracking cellular telephone trends in 2003 to understand the implications for landline-only federal health surveys and in 2007 also started asking respondents whether in their households "all or almost all calls are received on cell phones, some are received on cell phones and some on regular phones, or very few or none are received on cell phones." Some of these results are eye-opening:
About one in seven U.S. households (14.7 percent) are "cell-mostly." Add that to the cell-only figures and at least 37 percent of households definitely or probably cannot be reached by landline. (The cell-mostly group has been growing at a slower rate than cell-only.)
Landline abandonment is most prevalent among people age 25-29, 63.5 percent of whom live in cell-only (45.8 percent) or cell-mostly (17.7 percent) households. (Not as many people age 18-24 can't be reached by landline as they're less likely than those 25-29 to live in wireless-only households, probably because some younger people still live with parents who haven't cut the cord.)
Blumberg observed: "Interestingly, we see an increase in cell-phone usage among people living with relatives, people living with children, and older adults. More people in these groups tell us they receive all or most calls on their cell phones, but they haven't given up their landlines in disproportionate numbers."
What does it all mean for surveys that only sample landline phones? Clearly, sample non-coverage is a growing problem, at least as a perception - it's easy to wonder about survey validity if more than a third of the population of interest has little to no chance of being included. True, excluding cell phones didn't appreciably harm presidential vote preference in 2008 pre-election polls. But a deep dive into a phone-status question on the 2008 national exit poll yields cause for concern for anyone interested in not just the overall horserace but understanding why different subgroups behave and think as they do - more on this in an article I wrote with two co-authors for a soon-to-be-published issue of Public Opinion Quarterly (an earlier draft, presented in May at the annual conference of the American Association for Public Opinion Research, is available here). See also extensive Pollster.com coverage of who is abandoning landlines and what it means for the survey profession.
David Winston is President of the Winston Group, a strategic planning, communications, and survey research firm. He was formerly Director of Planning for Speaker Newt Gingrich and is presently an election analyst for CBS.
This week, President Obama finds himself facing his first public opinion crisis as several different national surveys showed his job approval below 50% over the past 10 days. The Marist and Quinnipiac surveys both put his job approval at 46%. CNN had it at 48% while Ipsos/McClatchy had it at 49%. But it was the Gallup daily tracking, which finally dipped below 50%, that pushed Press Secretary Robert Gibbs to an uncalled for denigration of the respected polling organization, comparing their results to a "six-year-old with a crayon."
Why is this important? Gibbs testy response isn't. But the president's downward spiral certainly has serious implications for both his ability to govern and to enact his policy agenda. Simply put, if a Presidential job approval is below 50%, a governing majority coalition does not exist and without a governing majority, controversial policies like health care and cap and trade are relegated to the uphill climb of minority status.
But presidential polling numbers can be worse than simply slipping below that 50% mark. When a President's job approval is under water, meaning more people disapprove than approve of the job he is doing, that's when every alarm bell in the West Wing ought to go off. President Obama is dangerously close to needing a life jacket.
In the CNN survey, 48% of those surveyed approved of his job performance, while 50% disapproved. Ipsos/McClatchy had it at 49-49, and both Marist and Quinnipiac had it at 46-44.
If Obama's numbers continue to slide, his policy agenda is at serious risk. Don't think for one moment that members of the House and Senate don't pay attention to these national polls. They do, especially those who find themselves in competitive races. Equally important, their own internal state or district polls will likely also have a presidential job approval question. Whether Obama is under 50% or under water back home could and, in many cases, will impact their voting behavior in D.C.
It's premature to suggest that it's time the Obama team break out the life boats, but contrary to Mr. Gibbs assertions, numbers do matter. They will determine, in part, whether his legislative agenda succeeds this year and survives the elections next year.
Harry Enten is a student at Dartmouth College
The past two years have shown us that predicting voter support for same-sex marriage ballot measures is no easy task. Pollster.com's aggregate trend estimates, reflecting pre-election polling, incorrectly projected that voters in California and Maine would vote against measures to ban same-sex marriage. Nate Silver, using a regression model that included a state's religiosity, year of the measure, and whether the measure included a ban on civil unions, also incorrectly predicted that Maine's amendment to ban same-sex marriage would fail.
In a post this past Friday, Silver offered a possible explanation: "It's not clear that the results in Maine are comparable to those in other states. Question 1 was the only gay marriage ballot initiative that did not seek to rewrite its state's constitution... there was no particularly good way to model the uncertainty."
While Question 1 was rare in that it did not amend the state constitution, it is not the only anti-same-sex marriage ballot measure to do so. In 2000, California voters passed Proposition 22 (the California Defense of Marriage Act), an ordinary statute, by a margin of 61%-39%. I was interested to see if including California's 2000 vote and a variable signifying that it was not a constitutional amendment would have improved Silver's model. To do so, I simply added a dummy variable controlling for whether the measure in question amended the state's constitution or merely altered state law.
The result is a model that would have actually done worse in Maine with a predicted yes vote for Question 1 of only 33.4% (vs. 43.5% for Silver's initial model), when the actual yes vote was 52.9%. If one were to add a dummy variable for an off-year election to this model as Silver did "ad-hoc" to his, the yes vote would still only get 37.9%.
Still, I was inspired by Silver's 2008 presidential regression models that combined polling and states' demographic data to find out if combining polling data with other variables could create a more accurate prediction of same-sex marriage ballot measures.
I have built a linear regression model based on 25 state gay marriage referenda from 1998 to 2009. The model attempts to predict support for banning same-sex marriage using five variables: projected support for the measure from pre-election polls, a state's religiosity, year of the measure (where 1 is 1998, 2 is 1999, and so on), a dummy variable controlling for whether the measure in question amended the state's constitution or merely altered state law, and a dummy variable controlling for whether the election was off-year.
The results for this model are very encouraging for those of us hoping to add value to polling data and predict future results of same-sex marriage ballot measures. I found that 92.1% of the variation between the different same-sex marriage elections was explained by the model compared with 80.7% for Silver's unaltered model. The average difference between the model's predicted support for an amendment in an election and the actual support for the amendment was 2.69% (compared with Silver's 4.46%). Importantly, this difference was greater than 2.00% in only 4 instances (Michigan 2004, Montana 2004, North Dakota 2004, and South Dakota 2006) and greater than 4.00% in only two (Michigan 2004 and North Dakota 2004).
The polling data is the best predictor for support for same-sex marriage amendments. Indeed, a simple regression in which the poll variable alone predicts the final result explains 86.4% of the variation in support for same-sex referenda across elections.
Despite the polling variable's dominance, the year variable is statistically significant with 95% confidence in the model. That is, we can be 95% sure the effect this variable has on the model did not occur by simple chance. The year variable has a negative coefficient, suggesting that in more recent years polling is less likely to underestimate support for the propositions. This finding supports a study by NYU's Patrick Egan that concluded that any possible "gay Bradley Effect," the theory that some respondents were uncomfortable sharing their opposition to gay marriage with a stranger on the telephone, has subsided in recent years.
The reason for this abatement is unclear, but it may have to do with the fact that the issue of same-sex marriage is no longer heavily used as a wedge issue nationally. Senator McCain mentioned the issue fewer times in 2008 than President Bush did in 2004, and Congress has not voted upon the Federal Marriage Amendment since 2006. This explanation would be consistent with Georgetown's Daniel Hopkins finding that the Bradley Effect for black candidates began to disappear in the mid 90's once issues (such as welfare reform and crime) with a racial undertone began to recede from the national debate.
The off-year and religiosity variable are statistically significant with 90% confidence in the model. The coefficient for the off-year variable is positive implying that polling underestimates support for the "yes" vote in off-year elections. This is not surprising considering these elections tend to have lower turnout (and are thus more difficult to poll) and are dominated by older voters who are more likely to be opposed to same-sex marriage.
The coefficient for the religiosity variable is positive meaning that, when controlling for the other variables, polls tend to underestimate support for the measures in more religious states. Last year, Mark DiCamillo, director of The Field Poll in California, argued that polling errors for same-sex marriage referenda resulted from late shifts and a boost in turnout among Catholics and regular churchgoers. He speculated that these shifts resulted from "last minute appeals" from religious figures. If DiCamillo is correct, and if gay marriage opponents have used similar tactics elsewhere, we would expect this effect, and thus the polling error, to be larger in more religious states.
The variable controlling for whether the measure in question amended the state's constitution or merely altered state law is not statistically significant. That is, there is a relatively high probability that any effect this variable had on the predictive value of this model occurred only by chance. It is important to point out that the results from this variable should be viewed with caution because we only have two observations.
Of course, I was also interested in testing if my model can work proactively and not merely explain past results. I wanted to investigate if, unlike the Pollster.com aggregate, it would have accurately predicted the results for California and Maine. To estimate the result for California as I would have prior to the 2008 election, I eliminated all the observations from the 2008 and 2009 elections from my dataset: California 2008, Florida 2008, and Maine 2009. This altered model called for the "yes" side to win in California with 51.9% of the vote, an error of 0.3%. To estimate the result for Maine, I simply eliminated the Maine 2009 observation. This modified model called for the same-sex marriage ban to pass in Maine with 50.6% of the vote, an error of 2.3%.
All of these findings support the argument that we can add value to polling data on same-sex marriage amendments when we control for them with variables such as religiosity of a state and year of the measure. We should recognize that polling ballot measures is always very difficult due to their confusing language. Polling same-sex marriage measures is especially problematic because of added factors such as a possible same-sex marriage Bradley Effect. My model helps to eliminate some, but no means all, the possible errors that result from these problems.
Notes on Data
1. For my model, off-year is defined as any election that did not place during a presidential election (primary or general) or a midyear general. This includes Missouri 2004, Kansas 2005, Texas 2005, and Maine 2009. Silver's model only counts Kansas 2005, Texas 2005, and Maine 2009 as off-year elections. I used my measure because non-presidential primaries, like traditional off-year elections are often plagued by low turnout.
2 For Silver's and my model, religiosity is measured by the percentage of adults in a state who considered religion an important part of their daily lives in a 2008 Gallup study.
3. Because prior studies have found that due to the confusing nature of ballot questions voters become increasingly aware of the meaning of a "yes" and "no" vote for same-sex marriage ballot measures closer to the election (most likely relying on advertisements), my polling variable only uses data taken within three weeks of the election. In the case that more than one firm conducted a poll within three weeks of the election and less than a week separated the polls, I used an average of the firms' final polls. For Maine, this rule means I included an average of the final Public Policy and Research 2000 polls in my dataset, but not the Pan Atlantic poll because it was taken more than a week before the Public Policy's final poll was conducted.
While most of the data in my model is easily available, prior polling for same-sex marriage referenda is surprisingly difficult to find. I managed to locate and verify 25 elections with a measure to ban (or allow the state legislature to ban as is the case with Hawaii) same-sex marriage and a poll within three weeks of the election. I simply allotted undecideds to how already decided voters were planning on voting: projected vote in favor of the amendment by polls = those planning on voting yes / (those planning on voting yes + those planning on voting no).
Complete dataset is available here.
David C. Wilson is an assistant professor of political science and international relations at the University of Delaware, who previously served as a Senior Statistical Consultant for The Gallup Organization in Washington, D.C.
In this edition of "Toplines and Headlines," (previous notes can be found at the CPC blog) I examine headlines and data from a recent poll about the economy. The poll was sponsored by CNN, and conducted by the Opinion Research Corporation (ORC). The headline from the story on CNN's website read, "CNN Poll: Optimism on economy fading." The headline implies that positive beliefs about the economy are actually on the decline, and that readers should be concerned. Yet, after reading the Topline results provided by CNN, a sophisticated reader of polls might (and probably should) come to the exact opposite conclusion as CNN.
CNN's polling director, Keating Holland, supports the headline citing data that suggest "Americans don't see economic conditions getting better any time soon," and notes that 34% of respondents say that things are "going well in the country today," a 14% increase from a year ago, BUT a 3% point decrease since November. Holland also cites a 6% increase, from 33% to 39%, in the percentage of people who say the country is still in a downturn. These Toplines form the initial thrust of the support for the "negative" headline.
Yet, this narrative should be questioned just on a couple of simple survey methodological factors. The margin of error (MOE) for the poll is plus or minus 3%, which means the aforementioned percentage decline in things "going well in the country today" is within the margin of error; thus, since last year, while more people see things going well, statistically, those numbers have not changed since last month. This counters CNN's headline.
More questions about the negative headline are raised when one examines the entire trend (p. 7 of the Topline release) since November of 2008; at that time, the percent saying things in the country are going well was 16%. In every poll since that date, except last month, the trend increased. Thus, it's very possible, and quite likely, that the results from November were a random (larger than expected) bump in the trend. In reality, the percentage thinking things are going well is actually continuing to increase rather than decline; another counter to the headline.
Turning to another question ostensibly supporting the headline, the trend showing an increasing percentage of those who say "the country is still in a downturn" is important, but the results from that particular question do not necessarily describe the fading optimism cited in the headline (see p. 7 of the Topline release). In fact, since June, 60% or more of Americans believe the economy is either "recovering" or "stabilized and is not getting any worse." This trend may have gone DOWN 6% since October, but has gone virtually unchanged since June.
In the CNN story, Holland also notes that 43% say the "chances of the recession turning into another Great Depression" are either somewhat or very likely (see p. 7 of the release). Yet, strong majorities in 2009--58% in the Dec. poll, 58% in a July-Aug poll, and 54% in a March poll--believe this is less likely to happen. Moreover, it's true the trend is up 5% from a year ago, but if one examines the entire trend, the Dec. 2008 poll cited by Holland appears to be another blip in the trend (see the Topline data for yourself).
On another question, 84% say the economy is still in a recession, but since May of this year that trend has decreased by 6%, while those believing the country is "not" in a recession has increased 6%, from 10% to 16%, over the same period. Thus, while there's broad agreement the American economy is in a recession, that agreement is actually decreasing, rather than increasing.
Lastly, Holland connects his interpretation of the pessimism about the economy to President Obama. He says, "it's clear why Obama is again addressing the economy," noting that "most Americans (40%) continue to say that the economy is the most important issue for them. Yet, one need only examine the trend in this question (p. 2 of the Topline release) to become skeptical of the narrative.
Since March of this year, the percentage saying the economy is the "most important issue facing the country today" has DECREASED by a whopping 23%, while the percentage saying "the wars in Iraq and Afghanistan" has INCREASED by 10%. Even Health care has a significant increase since March (the 3% decline since Aug. is within the margin of error). Thus, while the economy is the most important problem, it's been losing steam since March of this year.
When reading the headline and the story in concert with the data, it becomes clear that there is a narrative that CNN, Holland, or both CNN and Holland are trying to promote. At some points the story ignores the overall trend, and at others it mentions only the snapshot point estimates, dismissing the trend completely. In other words, the "trends" (i.e., "fading") that the story and headline emphasize are selective, not comprehensive, and thus in many ways the story is an overly biased take on things.
The point to remember here is that the reader of polls, and "headline," that emphasize trends must be considerate of the starting date of the trend, as well as the other responses not mentioned in a story/narrative. Bottom line, while the CNN headline reads that hopes are fading, a sophisticated poll watcher might easily disagree.
Alan I. Abramowitz is the Alben W. Barkley Professor of Political Science at Emory University in Atlanta, Georgia. He is also a frequent contributer to Larry Sabato's Crystal Ball.
In his recent post, Mark Blumenthal provides an excellent discussion of some of the possible explanations for the differences between the results of Rasmussen polls and the results of other national polls regarding President Obama's approval rating. What needs to be emphasized, however, is that regardless of the explanation for these differences, whether they stem from Rasmussen's use of a likely voter sample, their use of four response options instead of the usual two, or their IVR methodology, the frequency of their polling on this question means that Rasmussen's results have a very disproportionate impact on the overall polling average on the presidential approval question. As of this writing (December 4th), the overall average for net presidential approval (approval - disapproval) on pollster.com is +0.7%. The average without Rasmussen is +7.1%. No other polling organization has nearly this large an impact on the overall average.
A similar impact is seen on the generic ballot question reflecting, again, both the divergence between Rasmussen's results and those of other polls and the frequency of Rasmussen's polling on this question. The overall average Democratic lead on pollster.com is 0.7%. However, with Rasmussen removed that lead jumps to 6.7%. Again, no other polling organization has this large an impact on the overall average.
According to Rasmussen, Republicans currently enjoy a 7 point lead on the generic ballot question among likely voters. Democracy Corps, the only other polling organization currently using a likely voter sample, gives Democrats a 2 point lead on this question. To underscore the significance of this difference, an analysis of the relationship between popular vote share and seat share in the House of Representatives indicates that a 7 point Republican margin of victory in the national popular vote next November would result in a GOP pickup of 62 seats in the House, giving them a majority of 239 to 196 over the Democrats in the new Congress. This would represent an even more dramatic shift in power than the 1994 midterm election that brought Republicans back to power in Congress. In contrast, a 2 point Democratic margin in the national popular vote would be expected to produce a GOP pickup of only 24 seats, leaving Democrats with a comfortable 234 to 201 seat majority.
One of the biggest problems in trying to compare Rasmussen's results with those of most other polls is that Rasmussen is almost alone in using a likely voter sample to measure both presidential approval and the generic ballot. Moreover, Rasmussen has been less than totally open about their method of identifying likely voters at this early stage of the 2010 campaign, making any evaluation of their results even more difficult. However, there is one question on which a more direct comparison of Rasmussen's results with those of other national polls is possible--party identification. Although the way Rasmussen asks the party identification question is somewhat different, reflecting its IVR methodology, Rasmussen's party identification results, like almost all other national polls, are based on a sample of adult citizens. Despite this fact, in recent months Rasmussen's results have diverged rather dramatically from those of most other national polls by showing a substantially smaller Democratic advantage in party identification. For example, for the month of November, Rasmussen reported a Democratic advantage of only 3 percentage points compared with an average for all other national polls of almost 11 percentage points.
Rasmussen's party identification results have only a small impact on the overall average on this question because they only report party identification once a month. However,
Rasmussen's disproportionately Republican adult sample does raise questions about many of their other results, including those using likely voter samples, because the likely voters are a subsample of the initial adult sample. If Rasmussen is starting off with a disproportionately Republican sample of adult citizens, then their likely voter sample is almost certain to also include a disproportionate share of Republican identifiers. Of course, there is no way of knowing for certain whether Rasmussen's results are more or less accurate than those of other polling organizations. All we can say with some confidence is that their results are different and that this difference is not just attributable to their use of a likely voter sample.
Cliff Young is Senior Vice President at Ipsos Public Affairs. Cliff is head of the Public Sector practice and responsible for the Ipsos McClatchy poll. Aaron Amic is Vice President at Ipsos Public Affairs and is responsible for analytics for the Ipsos McClatchy Poll.
When the definitive history of the 2009 healthcare reform
debate is written, one footnote will read how varied, even contradictory, the
polls had been. We see this now. Indeed, on any given day, different people can
cite different polls and come to very different conclusions. "Americans are in
favor of healthcare reform-no, wait, they are against it!"
It goes without saying that given this uncertainty, cherry
picking of polls has been rife on both the right and the left. Democrats prefer
to cite polls on the "public option" which has consistently shown strong
majority support. Republicans, on the other hand, point to polls on general support
for healthcare reform-most showing only a plurality.
At a methodological level, pollsters have been grappling
with this dilemma as well. The original debate centered around the variability
of question wording and its effect on levels of support. The overriding
question was-what is the ideal healthcare question, if even such a thing
exists?
More recently, the debate has shifted to explaining the differences
between generic healthcare questions and more specific ones referring to the "public
option". The controversy lies in the
differential levels of support-generic questions have shown only plurality
support, while specific questions referring to the "public option", show
majority support. The consensus
explanation is that the healthcare debate is quite distant from people's
day-to-day lives and so their answers are "uninformed"-in methodological speak,
a classic case of "non-attitudes."
Both lines of reasoning have their merit. However, we
believe that they miss the mark because they assume that polling on healthcare
reform is analogous to polling on presidential elections. In our opinion, it isn't.
Indeed, in presidential elections, our job as pollsters is
made easy with ballot questions being basically fixed after the primaries. Simply
put, we know which candidates will be running. This, in turn, all but defines our
ballot question for us.
In contrast, issues like healthcare reform are quite fuzzy
as no bill typically exists at the beginning of the process. This makes the
construction of a single question impossible if not simply disingenuous.
Put another way, we have no "true value" to measure against-
no concrete bill exists (or at least did not exist until recently). You
can't measure what doesn't exist!
The problem is most apparent when looking at generic questions
on healthcare[1]. Such
questions are broadly worded and lack any concrete anchor. People, consequently, can (and do) read into
them what they want, making their meaning variable. To illustrate our point
let's look at table 1 below.
The above question shows that only a plurality (34%) of
Americans support healthcare reform (or at least the proposals in Congress). Simple
Conclusion: Americans do not support healthcare reform.
However, a simple follow up question shows that about a
quarter (25%) of those that oppose the reform bills actually think the
proposals "do not go far enough" (see table 2 above)! This same 25% actually is much more likely to
be Democrat and more likely to support the public option. People, once again,
read into the question what they want.
In contrast, questions which refer to "the public option"
and other specific policy measures can introduce greater certainty into the
ballot question, helping to establish a clear reference point for people (See
table 3 below). However, once again, such questions are nothing more than
hypothetical as we do not know a priori
which items will (and will not) be included.
So what are our takeaways here? What does polling on American
healthcare reform teach us about polling on non-electoral policy issues
involving the legislative process?
First, polling on healthcare reform is quite different than
polling on presidential elections because our "true value" is not fixed. This
makes the construction of singlequestions impossible and misleading. Such
issues are, well, fuzzy and, therefore, only a multiple indicators approach
will tell the entire story-some generic, some specific questions. Here
triangulation is key.
Second, genericquestions should be used with caution. At the least, they should include a follow up
question in order to determine why people favor or oppose healthcare reform. We
only included such a follow up after struggling with interpreting the results.
Are such generic questions valid at all? We think they are but with caveats.
Indeed, before the final bill, such questions seem to be
nothing more than a measure of optimism about the reform process, much like "right
track, wrong track" questions. Looking
forward to a final bill, we do expect that such generic questions will become
relevant. Only then will they have a "true value" to be measured against.
Third, questions which reference specifics like the "public
option" are hypothetical and have to be understood as such. Indeed, without a
final bill, they should be used more for sensitivity analysis than anything
predictive-which policy measures garner more support, which ones less so. While such questions say nothing about "general
support for healthcare reform," they do help us understand which measures are
more (and less) likely to be in the final bill as politicians read polls too.
To this end, we have tracked specific items for most of the
healthcare debate. Here we understood that healthcare reform would be fundamentally
a debate about the role of government (or lack thereof). All of our items fall
along a government intervention continuum. In our experience, polling on
"fuzzy" issues places a premium on understanding the underlying value cleavages
related to the policy debate at hand. At its essence, healthcare reform is a
debate about the proper role of government.
Fourth, from an analytical perspective, the combination of
generic and specific (hypothetical) questions makes total sense. They allow us
to be both predictive as well as diagnostic with our clients but only make
sense when used together.
Fifth, from a media polling perspective, the combination of
general and specific ballot questions is much less tidy than a single "up or
down" measure and, thus, more complicated to explain. Looking forward to future
non-electoral legislative reform debates, we, as an industry, need to do better
in explaining these complexities.
[1] Examples
of some questions recently fielded:
Ipsos wording:
As of right now, do you favor or oppose the healthcare reform proposals
presently being discussed?
ABC wording:
Overall, given what you know about them, would you say you support or oppose
the proposed changes to the health care system being developed by Congress and
the Obama administration?
AP-GFK wording:
In general, do you support, oppose or neither support nor oppose the health
care reform plans being discussed in Congress?
PEW wording:
As of right now, do you generally favor or generally oppose the health care
proposals being discussed in Congress?
CBS wording:
Do you mostly support or mostly oppose the changes to the health care system
proposed by Barack Obama, or don't you know enough about them yet to say?
Adam Geller is the CEO of National Research, Inc. and conducted polling for Chris Christie's campaign in New Jersey this year.
I'd like to contribute a few thoughts on the performance of the public polls during the recently concluded New Jersey Gubernatorial race. On this topic, I bring a unique perspective, as the pollster for the Christie campaign, and I'd like to offer my thoughts not as any type of authority, but rather to contribute to an important professional discussion.
I should mention that, for what it's worth, some observers may have been surprised by the results on November 3rd, but neither Governor Elect Christie nor his advisers were surprised.
Before the cement hardens and ink dries on the post election wrap up, let me offer the following five thoughts:
- The automated polls were more accurate than the live interview public polls, due in part to the methodology of the live interview polls.
From polls that were in the field for an entire week (Quinnipiac) or even longer (FDU), to polls that oversampled Democrats (Democracy Corps, among several others) to polls that asked every single name in the ballot (Suffolk), an essential reason for the poor performance of the live interview polls had less to do with the fact that a live person was administering the poll and more to do with methodological issues.
- The partisan spread in the polls ought to be reported up front.
Some public pollsters make it difficult to determine how many Republicans, Democrats and unaffiliated voters they interviewed. Why not just put it into the toplines? Reporters and bloggers should demand this before they report on the results. Not to pick on Quinnipiac, but they had Corzine and Christie winning about the same amount of their own partisans, and they had Christie winning Independents by 15 percentage points, and yet they STILL had Christie trailing overall by 5 points. Quinnipiac did not publish their partisan spread, but then an astute blogger was able to ascertain the fact that there were, in fact, too many Democrats in the sample. Other polls, notably Democracy Corps, regularly produced samples with too many Democrats (though, in their parlance, some of these were "Independent - Lean Democrat"). That their sample was loaded up with Democrats had the obvious effect on their results. Whether this was intentional or not, I would leave to others to speculate.
- In general, RDD methodology is a bad choice in New Jersey, if the goal is predictive accuracy.
In New Jersey, there are many undeclared voters (commonly but mistakenly referred to as Independents). These undeclared voters identify themselves as Republicans or Democrats - even though they are not registered that way. In our polls, we frequently showed a Democrat registration advantage that matched their actual registration advantage - but when it came to partisan ID, the spread was more like a six point Democrat advantage. By using a voter list, we knew how a respondent was registered - and by seeing how they ID'ed themselves, we gained insight into the relative behavioral trends of undeclared voters and even registered Democrats who were self identifying as Independents. Public pollsters who dialed RDD missed this. Partisan identification in New Jersey is not enough, if the goal is to "get it right."
- The public polls oversampled NON voters.
Again, this is a function of RDD versus voter list dialing. It is easy for someone to tell a pollster they are "very likely" to vote. With no vote history and no other nuanced questions, the poll taker has little choice but to trust the respondent. Pollsters who use voter lists have the benefit on knowing exactly how many general elections a respondent may have voted in over the past five years, or when they registered. By asking several types of motivation questions, the pollster can construct turnout models that will have a better predictive capacity. The public polls did not seem to do this.
To this end, we had heard all about the "surge strategy" that the Corzine campaign was going to employ. This refers to targeting "one time Obama voters" and driving them out in force on election day. With voter lists, we were easily able to incorporate some "surge targets" into our sample. After running our turnout models, we saw no evidence that the surge voters would be game changers.
- The Daggett effect was overstated in the public polls.
Conventional wisdom holds that Independent candidates underperform on election day. But the reality is, many analysts could have easily predicted Daggett's collapse, based not on history, but on simple a simple derivative crosstab: for example, voters who were certain to vote for Daggett AND had a very favorable opinion of him. They could have asked a "blind ballot" where none of the candidate choices were read. We did these things - and we estimated Daggett's true level of support to be around 6%.
None of this is meant to pick on the "live interview" public pollsters. For the most part, these polls are conducted and analyzed by seasoned research professionals. But in non-Presidential years, RDD methodology can lead to inaccurate results, which can then lead to inaccurate analysis. It is tough to conclude that the automated polls are somehow superior to live interview polls, given the methodological issues I've outlined.
What does it mean for next year? At the very least, journalists, bloggers and reporters need to ask more questions about the methodology and construction of the poll sample. They need to understand the partisan spread, and the extent to which it conforms to reality. They need to know how long the survey was in the field. They also need to beware of polls being released that are designed to manipulate opinion rather than manage it. They need to ask if certain polls are being constructed to reflect what is happening, or if they are being constructed to reflect what the poll sponsor would LIKE to happen. The public polls add to the dialogue, and given their ever increasing contributing role, we all ought to be more demanding when reporting their results.
Humphrey Taylor is chairman of the Harris Poll at Harris Interactive, which conducts surveys on the internet.
These comments are prompted by the paper Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Non-Probability Samples by Yeager, Krosnick, et al, and by Mark Blumenthal's two excellent articles in the National Journal reviewing their paper.
The paper's conclusions were based on a comparison between six "benchmarks" and the findings of the various polls they examined. They assumed that the benchmarks were perfectly accurate, and that any differences between the polls and the benchmarks were "errors." I believe that this is not the case and that some of the benchmarks were inaccurate because of the social desirability bias that is often found in surveys where respondents are interviewed, by telephone or in-person, by live interviewers.
Social desirability bias occurs where respondents are not comfortable telling interviewers the truth because they are embarrassed to do so, or where their behavior or attitudes may be seen as unethical, immoral, anti-social or illegal.
Our online surveys have always found substantially more people than our telephone surveys who tell us they are gay, lesbian or bisexual (by a 3-to-1 margin). Our online surveys also find fewer people who claim to give money to charity, clean their teeth, believe in God, go to religious services, exercise regularly, abstain from alcohol, or drive under the speed limit.
Furthermore, in-person surveys by the Census Bureau report substantially more people claiming to have voted in elections than actually voted. If there is a better explanation than social desirability bias, I haven't heard it.
This conclusion - that surveys with live interviewers underreport "socially undesirable" behavior is supported by the data used by Yeager et al.
Our online survey, used by Yeager, found more smokers and more people having had 12 drinks in a life time than either the benchmark surveys conducted by government agencies or the RDD sample (and our own telephone surveys). Our online survey found that (to the nearest whole number) 28 percent were smokers compared to 26 percent in the RDD sample and 22 percent in the benchmark survey. Our online survey found only eight percent who had not had 12 drinks in their lifetime compared to 15 percent in the RDD sample and 23 percent in the benchmark survey.
Another government study, the NHANES study reported that 24.9 percent of adults said they were smokers but that blood tests showed that an additional 4.5 percent had smoked in the previous 24 hours but had not reported it when asked by an interviewer. The resulting NHANES estimate of 29 percent is closer to our estimate of 28 percent than to Knowledge Network's 26 percent or the RDD sample's 24 percent.
Two of the six benchmarks used by Yeager et al come from government sources where one would not expect to find any social desirability bias. In both cases, the Harris Interactive data were slightly closer to the benchmark data than were the findings of the RDD telephone survey. Our surveys found 28 percent of adults with passports compared to 30% for the RDD sample and the 23 percent in benchmark. Our survey found 92 percent having a driver's license compared to 93 percent in the RDD sample and the 89 percent benchmark.
In addition to the presence or absence of live interviewers there is one other reason why our online polls may have less social desirability bias than most telephone and in-person surveys. Our panel members have agreed in advance to be surveyed, which suggests that they trust us with confidential information, and are therefore more likely to tell the truth.
All this evidence suggests that the Harris Interactive data used by Yeager et al is generally more accurate than the RDD sample and that some of the so-called benchmarks probably overstate socially desirable behaviors because they were obtained in surveys with interviewers.
This guest contribution comes from Michael McDonald, an Associate Professor of Government and Politics in the Department of Public and International Affairs at George Mason University and a Non-Resident Senior Fellow at the Brookings Institution.
Saturday Night Live's sketch mocking Obama prompted CNN to run a story stating that the 'SNL' Obama sketch marks end of [Obama's] honeymoon. Actually, SNL is not leading public opinion here. Polling suggests that Obama's honeymoon ended in early August. Since then, Obama's job approval rating has remained essentially flat.
If you are an Obama supporter, you might ask how this is possible, since an Oct. 1-5 AP-GfK survey shows a resurging six percentage point increase in support for Obama since their Sept. 3-8 survey. Or, if you oppose Obama, you might point to the slight downward trend in Obama's job approval among all polling firms from early September clearly evident on Pollster.com.
What is going on here is that Pollster.com's trend line behaves fine when there are lots of polls to average together, but it does not work as well when two daily tracking polls are averaged together with more sporatic national polling. The two daily tracking polls - Gallup and Rasmussen - consistently find lower Obama job approval ratings than other polling firms. In addition to these two daily tracking polls, there are approximately bi-monthly internet polls from YouGov/Polimetrix and Zogby that also consistently show lower Obama job approval numbers compared to other polls.
These so-called "house effects" whereby different pollsters consistently report different numbers is well-known. I do not want to get sidetracked into speculation about why these polls have lower numbers, since we really cannot know what the true population value is for Obama's job approval rating.
What is interesting is what happens when these polls are disaggregated into two types (1) the tracking and internet polls and (2) all other polls.
To examine the first type of polls, let's use Pollster.com's filter tool to include all internet polls and the two daily tracking polls.
According to this trend estimate, Obama's job approval rating leveled out in early August at about 50 percent, and may be slightly increasing since.
To examine the second type of polls, let's use Pollster.com's filter tool to exclude all internet polls and the two daily tracking polls.
According to this trend estimate, Obama's job approval rating leveled out in early August at about 53 percent.
Seen in this light, Obama's job approval rating has remained steady since early August, and it is here that Obama's honeymoon likely came to an end. Most pollsters took a vacation during August, except those conducting the first type of polls, which show lower Obama job approval than the second type. The bump up in Obama's job approval at the beginning of September is an artifact of the increased number of the second type of polls conducted when Obama delivered his health care speech to Congress. Subsequently, the absence of the second type of polls allows the first type of polls to again dominate the trend line, thereby giving the appearence that Obama's approval is now decreasing from the (non-existent) short-term early-September rally. The different mixes of the first and second types of polls are confounding the trend line and incorrectly coloring perceptions of the direction of Obama's job approval rating. Indeed, if you squint closely at Pollster.com's trend line for all pollsters, you'll see a long-term periodicty that apparently fluctuates along with the mix of the first and second types of polls.
[Editor's Note: So that Professor McDonald's commentary will always match the graphics, we replaced the embedded, interactive version of charts with screenshots, although you can click the link above each chart to see the most recently updated version with the filtered polls he selected].
Robert Y. Shapiro is a professor of political science at Columbia University who specializes in public opinion, policymaking, political leadership, and mass media. He is a member of the board of directors of the Roper Center for Public Opinion Research.
The polling and pundit world is now looking to see if President Obama's
speech will rally public support for his health care reform plan. In
addition to looking at the stream of polls that will now follow, I direct your attention, hot off the presses, to the latest issue of the journal Political Communication. A timely article by Brandon Rottinghaus provides a broader political science view on presidential efforts to influence public opinion. What we know from George Edwards' book, On Deaf Ears: The Limits of the Bully Pulpit (Yale, 2003), is that it is difficult for presidents to succeed at influencing public opinion. However, Rottinghaus's article provides evidence for why Obama correctly chose to take his best shot in a nationally televised speech.
The article
uses "a comprehensive data set spanning 1953 to 2001," to examine
"several strategic communications tactics through which the presidents
might influence temporary opinion movements." Specifically, it finds
that "presidential use of nationally televised addresses is the most
consistently effective strategy to enhance presidential leadership, but
the effect is lessened for later serving presidents." In contrast,
other strategies such as those involving domestic travel do not have
positive effects and "televised interactions"--press conferences and the like - tend to have negative effects. While some may not be surprised with these findings, it is good to have empirical evidence to wrestle with.
But getting to the point, how will this now play out for Obama? My sense is that Obama's speech will come out on or above average in impact, though there is a question of what its half-life will be. What I see as most important, however, is not the new polls that we will soon see (if they are not out already). Putting Rottinghaus' article aside, what will count most is not what the public thinks at this moment, but rather the extent to which Democratic leaders unite around Obama's plan (which may well be close to Baucus'?); it is this elite consensus that will enable any positive effect of the speech to last or even widen. This assumes that the consensus will be more salient and striking than any continued Republican opposition.
Echoing the famous political scientist, V.O. Key, what matters more than the immediate polls is political leadership more broadly. The speech itself is the start of what could be a stronger consensual message than we have seen to date from Democratic and potentially other political leaders. The relevant public opinion research comes from Richard Brody's book on presidential leadership, (Assessing the President: The Media, Elite Opinion, and Public Support. Stanford, 1991), John Zaller's seminal book on public opinion (The Nature and Origins of Mass Opinion, Cambridge, 1992), and what Ben Page and I examined (The Rational Public. Chicago, 1992).
Larry Jacobs and I (Politicians Don't Pander, Chicago, 2000) looked at the President Clinton's 1993-94 health care reform effort from this perspective. What happened there was the Democratic leaders never supported any Clinton plan, and this, along with the strong Republican leadership opposition caused the public to become apprehensive and turn against health care reform. This happened much earlier in the legislative process than what occurring now, as the Clinton plan got to Congress later in Clinton's first term. In contrast, we are at that same juncture now --- however, earlier in Obama's first term but later in the legislative process, as there are now actual bills that have made it through congressional committees. Clinton never made it that far. The Democrats now have a better chance than Clinton did, since at this moment they are poised to unite around a president's plan. But if they don't do that quickly, then it's 1994 all over again. If by all appearances they come together, they can prevent public support from tapering off and very likely increase it.
In the end, Obama may have timed his entry into the fight just right--it's earlier than when Clinton entered the actual legislative fray in 1994--and this may have been the only way he could have gotten a major health care reform bill through. Given the financial crisis, the stimulus bill, and the two wars, he may well have been stopped in his tracks earlier on--without the health care reform bills making it through multiple committees as they have. He needed to enter the fight when he could rally congressional support in both houses, with drafted legislation in hand and already substantially debated. Of course we will never know since as we can't replay history. For now, the main point is don't just watch the polls-watch the leaders. The public will not just be responding to Obama but to the extent to which he has liberal, blue dog, and any (albeit unlikely) Republican leadership support.
Today's Guest Pollster article comes from Thomas Riehle, a Partner of RT Strategies.
Technological capabilities can become temptations to conduct research studies that add nothing to our knowledge of public opinion, just because we can. Get thee behind me, Satan!
For example, it would be no problem, technologically, to display squiggly lines with the moment-by-moment reactions of a panel of viewers to the blathering of the talking heads on news show panels. The Onion demonstrates what a mess that would be, in a parody entitled "New Live Poll Allows Pundits to Pander to Viewers in Real Time."
What would happen if we let the talking heads see whether viewers at home agreed or disagreed with what they were saying, "using the Insta-Poll Tracker on our web site"? The talking heads would become self-conscious about the direction of their own squiggly line and start tailoring their statements...word by word...to make the squiggly line go up.
Insta-polls like September 9th's CNN/Opinion Research Corporation poll of adults who watched President Barack Obama's address to Congress may have a similar effect on poll respondents. Mark Blumenthal correctly points out the age-old problem of such polls--the partisan make-up. Last night, the audience for this address was heavily weighted with Obama supporters rallying to watch their leader, supplemented with a few civic-minded Americans who would watch any Presidential address, regardless of their own partisanship. Of the 427 adults in this study, all of them interviewed September 5-8 in advance of the speech, and all of whom indicated both an intention to watch the speech and a willingness to be re-interviewed after the speech, 18% were Republicans, 45% Democrats. These kinds of post-speech poll samples always skew heavily in favor of the speaker. Pollster.com's report on this poll last night squeezes out what knowledge can be gleaned by comparing the "bump" among this group of speech watchers to the bump registered among similarly situated groups of speech watchers in the past.
The problem with this kind of insta-poll may be exacerbated when the study is designed, as this one was, to compare the pre-speech responses of speech watchers to opinions after the speech. In the pre-speech survey, I would guess that respondents would strive to express their opinions as forthrightly as possible, as most survey respondents do. In the follow-up poll after the speech, however, I am afraid respondents would be like the Onion's self-conscious pundits. They'd be aware that they are about to become as much a part of the story as South Carolina Republican Rep. Joe Wilson who heckled the President. They'd tailor their answers to make their leader look good. Drawing much of a conclusion from their answers would not be any fairer than judging the entire Republican caucus by the boorishness of a few Members.
Douglas Rivers is president and CEO of YouGov/Polimetrix and a professor of political science and senior fellow at Stanford University's Hoover Institution. Full disclosure: YouGov/Polimetrix is the owner and principal sponsor of Pollster.com.
I woke up on Tuesday morning to find several emails pointing me to Gary Langer's blog posting, which quoted extensively from a supposedly new
paper by Jon Krosnick. These data and results appeared previously in a paper, "Web Survey Methodologies: A Comparison of Survey Accuracy," Krosnick coauthored with me and presented at AAPOR in 2005. The "new" paper has added some standard error calculations, some late arriving data, and a new set of weights, but the biggest changes in this version are a different list of authors and conclusions.
The 2005 study compared estimates from identical questionnaires fielded to a random digit dial (RDD) sample by telephone, an Internet-based probability sample, and a set of opt-in panels. Of these, Internet probability sample had the smallest average absolute error, followed closely by the RDD telephone survey, and the opt-in Internet panels were around 2% worse. In his presentation of our paper at AAPOR in 2005, Krosnick described the results of all the surveys, both probability and non-probability, as being "broadly similar." My own interpretation of the 2004 data, similar to James Murphy's comment on AAPORnet, was that although the opt-in samples were worse than the two probability samples, the differences were small enough--and the cost advantage large enough--to merit further investigation. Even if it were impossible to eliminate the extra 2% of error from opt-in samples, they could still be a better choice for many purposes than an RDD sample that cost several times as much.
Krosnick now concludes that "Non-probability sample surveys done via the Internet were always less accurate, on average, than probability sample surveys" and, tendentiously, criticizes "some firms that sell such data" who "sometimes say they have developed effective, proprietary methods" to correct selection bias in opt-in panels.
In fact, the data provide little support for Krosnick's argument. The samples from the opt-in panels were, as we noted in 2005, unrepresentative on basic demographics such as race and education because the vendors failed to balance their samples on these variables, while the two probability samples were balanced on race, education, and other demographics. This is not a result of probability sampling, but of non-probabilistic response adjustments. It is too late to re-collect the data, but the solution (invite more minorities and lower educated respondents) doesn't involve rocket science.
Instead, Krosnick tries to fix the problem by weighting, and concludes that weighting doesn't work. A more careful analysis indicates, however, that despite the large sample imbalances in the opt-in samples, weighting appears to remove most or all selection bias in these samples. Because the samples were poorly selected, heavy weighting is needed and this results in estimates with large variances, but no apparent bias. In fact, if we combine the opt-in samples, we can obtain an estimate with equal accuracy to the two probability samples.
First, consider the RDD telephone sample. The data were collected by SRBI, which used advance letters, up to 12 call attempts, $10 incentives for non-respondents, and a field period of almost five months. Nonetheless, the unweighted sample was significantly different from the population on ten of the 19 benchmarks. RDD samples, like this one, consistently underrepresent male, minority, young, and low-education respondents. These biases are reasonably well understood and, for the most part, can be removed by weighting the sample to match Census demographics.
Next, consider the Probability Sample Internet Survey, conducted by Knowledge Networks (KN). The unweighted sample does not exhibit the skews typical of RDD. How is this possible, since the KN panel is also recruited using RDD? Buried in a footnote is an explanation of how KN managed to hit the primary demographic targets more closely than SRBI (which had a much better response rate). The answer is that "The probability of selection was also adjusted to eliminate discrepancies between the full panel and the population in terms of sex, race, age, education, and Census region (as gauged by comparison with the Current Population Survey). Therefore, no additional weighting was needed to correct for unequal probabilities of selection during the recruitment phase of
building the panel." That is, the selection probabilities that are
supposedly so important to probability sampling were not used because they would have generated an unrepresentative sample!
The opt-in panels, for the most part, were not balanced on race and education. Only one of the opt-in samples, Non-Probability Sample Internet Survey #6 actually used a race quota. Another, the odd Non-Probability Internet Sample #7, claims to have sent invitations proportionally by race and ended up with 46% of the sample white, despite a 51% response rate. (This survey will be excluded from subsequent comparisons.) Non-probability Sample Internet Survey #1 involved large over-samples of African Americans and Hispanics. I could find no explanation of how Krosnick dealt with the oversamples in the 2009 paper, but it should either match exactly (if the conventional stratified estimator is used) or be far off (if the data are not weighted). In fact, the proportion of whites and Hispanics is off by 1% to 2%.
The selection of a subsample of panelists for a study is critical to the accuracy of opt-in samples. Regardless of how the panel was recruited, the combination of nonresponse or self-selection at the initial stage along with subsequent panel attrition, will tend to make the panel unrepresentative. In 2004, we instructed the panel vendors to use their normal procedures to produce a sample representative of U.S. adults. The practice then (and perhaps now for some vendors) was to use a limited set of quotas. If you didn't ask most opt-in panels to use race or education quotas, they wouldn't use them.
Even without correcting these obvious imbalances, the opt-in samples
provided what most people would consider usable estimates for most of
the measures. For example, the percentage married (unweighted) was between 53.7% and 61.5% vs. a benchmark of 56.5%). The percentage who worked last week (unweighted) was between 53.6% and 63.1% (vs. a benchmark of 60.8%). The percentage with 3 bedrooms (unweighted) was between 41.2% and 46.1% (vs. a
benchmark of 43.4%). The percentage with two vehicles (unweighted) was
between 40.1% and 46.9% (vs. a benchmark of 41.5%). Home ownership
(unweighted) was between 64.8% and 72.8% (vs. a benchmark of 72.5%).
Has one drink on average (unweighted) was between 33.8% and 40.2% (vs.
a benchmark of 37.7%). The KN sample and phone samples were better,
but the difference was much less than I expected. (Before doing this
study, I thought the opt-in samples would all look like Non-
probability Sample Internet Survey #7.)
The 2009 paper attempts to correct these imbalances by weighting, but the weighted results do not show what Krosnick claims. He uses raking (also called "rim weighting") to compute a set of weights that range from .03 to 70, which he then trims at 5. The fact that the raking model wants to weight a cell at 70 is a sign that something has gone wrong and can't be cured by arbitrarily trimming the weight. If there really are cells underrepresented by a factor of 70, then trimming causes severe bias for variables correlated with the weight and not trimming causes the estimates to have large variances. In either case, the effect is to increase the mean absolute error of estimates.
The fact that the trimmed and untrimmed weights have about the same average absolute error does not mean that weighting is unable to remove self-selection bias from the sample. The mean absolute error is a measure of accuracy. It is driven by two factors: bias (the difference between the expected value of the estimate and what it is trying to estimate) and variance (the variation in an estimate around its expected value from sample to sample). The usual complaint about self-selected samples is that you can never know whether they will be biased or the size of the bias. Inaccuracy due to sampling variation can be reduced by just taking a larger sample. Bias, on the other hand, doesn't decrease when the sample size is increased.
Obviously, uneweighted estimates from these opt-in samples will be biased because the vendors ignored race and education when selecting respondents. This wouldn't have been difficult to fix, but it wasn't done. Apparently very large weights are needed to correct demographic imbalances in these samples, but the large weights give estimates with large variances and, hence, a high level of inaccuracy. If one tries to control the variance, as Krosnick does, by trimming the weights, then the variance is reduced at the expense of increased bias. The result, again, is inaccuracy. We are asking the weighting to do too much.
A simple calculation shows that all of Krosnick's results are consistent with the weighting removing all of the bias from the opt-in samples. One way to combat increased variability is to combine the six opt-in samples. Without returning to the original data, a simple expedient is to just average the estimates. Since the samples are independent and of the same size, the average of 6 means or proportions should have a variance about 1/6 as large as the single sample variances. The variance is approximately equal to the square of the mean absolute error which, after weighting, was about 5 for the opt-in samples, implying a variance of about 25. If there is no bias after weighting, then the variance of the average of the estimates should be 25/6 or approximately 4, implying a mean absolute error of about 2%.
How does this prediction pan out? If we average each of the weighted estimates and compute the error for each item using the difference between the average estimate and the benchmark, the mean absolute error for the opt-in samples is 1.4% -- almost identical to the mean absolute error for each of the weighted probability samples. That is, the amount of error reduction that comes from averaging the estimates is about what would be predicted if the all bias could have been removed by weighting. Thus, the combination of these six opt-in samples gives an estimate with about the same accuracy as a fairly expensive probability sample (which also required weighting, though not as much).
There is no reason, however, why you should need six opt-in samples to achieve the same accuracy as a single probability sample of the same size. If the samples were selected appropriately, then we could avoid the need for massive weighting. It is still an open question what variables should be used to select samples from opt-in panels or what the method of selection should be. In the past few years, we have accumulated quite a bit of data on the effectiveness of these methods, so there is no need to focus on a set of poorly selected samples from 2004.
Probability sampling is a great invention, but rhetoric has overtaken reality here. Both of the probability samples in this study had large amounts of nonresponse, so that the real selection probability--i.e., the probability of being selected by the surveyor and the respondent choosing to participate--is not known. Usually a fairly simple nonresponse model is adequate, but the accuracy of the estimates depends on the validity of the model, as it does for non-probability samples. Nonresponse is a form of self-selection. All of us who work with non-probability samples should spend our efforts trying to improve the modeling and methods for dealing with the problem, instead of pretending it doesn't exist.
Prof. Alan Reifman teaches social science research methodology at Texas Tech University, and is compiling the results of public opinion polls on the specifics of health care reform at his blog, Health Care Polls.
There's been a lot of discussion of how seniors, who already are on Medicare, appear to be the least supportive age group of President Obama and the Democrats' plans for enacting health care reform. Seemingly at the center of seniors' concerns is the idea of cutting federal support for a program called Medicare Advantage. According to a Los Angeles Times article:
Although scaling back payments would have no effect on a sizable majority of Medicare users, it would create an opening for opponents to make the blanket allegation that the president wants to cut back on Medicare benefits -- as some Republicans are already starting to say.
Also, of course, seniors were more likely to vote for John McCain in last year's presidential election than were younger voters, who went overwhelmingly for Obama.
The diagram below (which you may click on to enlarge) compares different age groups' attitudes toward health care reform in four recent polls. Compiling these percentages was not as easy as I thought it might be, for a variety of reasons. First, only some pollsters make a public release of cross-tabulations between demographic characteristics and health care-related attitudes (other pollsters reserve such cross-tabs for paid subscribers). Second, age cross-tabs on a common attitude item were not always available. My plan was to use general favor/oppose items toward Obama and the Democrats' reform plan, but such an item was not always available so I had to substitute other types of items, as described below. Third, different pollsters use different cut-points to create their age groups. There's always a youngest age group, for example, but some pollsters bracket it from 18-29 whereas others use 18-34; similar discrepancies exist for other age groups, as well.
Having said all this, the pattern of seniors showing the least support for Obama/Democratic reform plans is clear and well replicated. For any given color of bar (purple, light blue, green, or orange; each representing a different pollster and question), the shortest height is with the seniors.
One other thing to notice is that two polls, ABC/Washington Post and The Economist/YouGuv, only reported on a 30-64 broad middle-age group rather than having two groups like other pollsters; whether groups in the lower and upper halves of the 30-64 age range were combined because they did not differ much in their responses, or the pollsters never broke 30-64 year-olds into smaller subsets, I don't know. For these two polls, I have taken the percentage on the respective attitude measures attributed to 30-64 year-olds and plotted them twice (linked by a light-blue or green horizontal line), where a 30s-40s group and a 50s-60s group would ordinarily go. Now that these "housekeeping" matters are out of the way, here are the question wordings used:
Survey USA (Aug. 19): “Now I am going to tell you more about the health care plan that President Obama supports and please tell me whether you would favor or oppose it. The plan requires that health insurance companies cover people with pre-existing medical conditions. It also requires all but the smallest employers to provide health coverage for their employees, or pay a percentage of their payroll to help fund coverage for the uninsured. Families and individuals with lower- and middle-incomes would receive tax credits to help them afford insurance coverage. Some of the funding for this plan would come from raising taxes on wealthier Americans. Do you favor or oppose this plan?”
ABC/Washington Post (Aug. 13-17): “Reform’s supported by 58 percent of adults under age 30, but 44 percent of 30- to 64-year-olds and just 34 percent of seniors, apparently concerned about its potential impact on Medicare” (this quote comes from an article and does not depict the actual survey item).
Economist-You Gov (Aug. 16-18): “If President Obama and Congress pass a health care reform plan, do you think you personally would receive better or worse care than you receive now?" (% Saying Better).
Kaiser Family Foundation (Aug. 4-11): “Do you think you and your family would be better off or worse off if the president and Congress passed health care reform, or don’t you think it would make much difference?” (% Saying Better).
The four polls above were not the only ones that made some type of age-related comparison. Others did, as well, but their age groupings and/or survey items appeared non-comparable in some way to the four polls whose results I plotted. Two additional polls are as follows:
A Harris Interactive poll used what I think are the most interesting age-group descriptors (shown in Table 2 of the linked document): "Echo Boomers (18-32), Gen. X (33-44), Baby Boomers (45-63), Matures (64+)." Harris plotted the percentage of respondents in each age group who rated Obama's job performance in various issue domains as "fair" or "poor." On health care, higher percentages of Matures (71%) and Gen. X (69%) gave Obama these unflattering ratings than did Echo and Baby Boomers (each 62%). Along with some of the figures from other polls plotted above, this finding from Harris shows a non-linear trend (i.e., support does not decline in perfect progression from the youngest to the oldest voters).
Finally, a Penn, Schoen, & Berland poll released in conjunction with AARP reported only comparisons between respondents younger than 50 and 50-plus. A section of this poll's report entitled "Specific Policy Proposals" (on pages 6-7) is perhaps the most worthy of attention. On most of the items, the younger respondents are more favorably inclined, but on others, there is little or no difference.
(Cross-posted to Health Care Polls)
Prof. Alan Reifman teaches social science research methodology at Texas Tech University, and has begun compiling the results of public opinion polls on the specifics of health care reform at his new blog, Health Care Polls.
Perhaps the most contentious issue among congressional negotiators and interest groups in Washington, DC (and elsewhere) is the so-called public option. The idea is that the government would create a new health-insurance program (modeled to one degree or another on Medicare, the government insurance program for seniors) that people could join. Proponents argue that, by having it compete with private insurers, the public option would help control costs. Opponents, on the other hand, see the public option as yet another government intrusion into an area they feel should be left to the private market.
Where does the public seem to stand? Not surprisingly, the public option has been widely polled, and we shall focus exclusively on it today. As seen in the diagram below (which you can click on to enlarge), levels of support for the public option vary widely according to different polls, despite the relative consistency of question wording (all the survey items refer in some fashion to the public option being a government health-insurance program that would compete with private insurance companies). The predominant trend, I would say, is that a majority of respondents supports a public option, with five of the eight polls showing between 52-66 percent in favor.
Still, though, two other polls show support in the mid-40s and one poll (Rasmussen) has support way down at 35%. What to make of this? Let's start with Rasmussen. Whereas Rasmussen's presidential-election polling has tended to be highly accurate (relative to the actual results), other types of polls from this outfit appear to have had a Republican slant. Here are some examples:
*Whereas most polls tended to have George W. Bush's job-approval ratings during the waning months of his administration in the low-30s or even the 20s, Rasmussen consistently had it around 35%.
*Whereas virtually every pollster other than Rasmussen has shown a majority of voters to prefer the Democrats (at this early point) in next year's U.S. House elections, Rasmussen has been showing the Republicans in the lead (albeit with large percentages undecided).
Polling analysts refer to systematic differences in the results (on the same basic issue) between different survey firms (or survey "houses") as house effects. These may stem from different firms' practices regarding question-wording, sample weighting, etc. On health care reform and other issues, it looks to me as though Rasmussen has a substantial house effect.
There's one other aspect of the public-option polling I'd like to point out. As can be seen in the diagram above, I have highlighted in red the words "option" and "offering" in the wording of some of the survey items. It appears that wordings stressing the voluntariness of the public option (i.e., that it is an "option," or something "offered" to the consumer) tend to elicit higher support than wordings that don't highlight voluntariness as much. This is just a hunch. If anyone has other explanations for the large variation in support between the polls, please share them in the comments section.
(Cross-posted to Health Care Polls)
Today's guest pollster contribution comes from Brendan Nyhan, a political scientist and Robert Wood Johnson Foundation Scholar in Health Policy Research at the University of Michigan. This entry is cross-posted at his blog, Brendan-Nyhan.com.
Matthew Yglesias calls the public "ill-informed and hypocritical" based on a New York Times poll that found "Most Americans continue to want the federal government to focus on reducing the budget deficit rather than spending money to stimulate the national economy... [y]et at the same time, most oppose some proposed solution for decreasing it."
The problem, however, is that the available evidence doesn't support Yglesias's conclusion (which is encouraged by the way the poll is framed in the Times). When you look at the raw poll results (PDF), you'll see that the public prefers reducing the deficit to stimulating the economy 58%-35%, but 53% oppose cuts in public services and 56% oppose higher taxes. Those numbers may seem "ill-informed and hypocritical," but the problem is that we're dealing with aggregate data (this is what is known as an ecological inference problem). We can't draw any strong conclusions about the proportion of individual members of the public who have incoherent preferences about deficit reduction without access to the raw data. Ideally, we would break out the members of the public who advocate deficit reduction over stimulus and see how many of them oppose both higher taxes and reduced services. That's the quantity of interest, but it's unfortunately not available to us at this point.
Update 7/30 12:12 PM: Yglesias has generously updated his post to note that you "can't infer very much about individual preferences from this aggregate data."
Today's guest pollster contribution comes from Brendan Nyhan, a political scientist and Robert Wood Johnson Foundation Scholar in Health Policy Research at the University of Michigan. This entry is cross-posted at his blog, Brendan-Nyhan.com.
Just to briefly elaborate on the point I made last week, here are comparable plots of President Obama's overall job approval and approval of his handling of health care:
As you can see, what's happening on health care is a leading indicator of the end of Obama's honeymoon period. As we return to our normal, highly polarized political climate, most Republicans and Republican-leaning independents will disapprove of a Democratic president's performance in office and his handling of high-salience issues, especially in a bad economy. As a result, Obama's numbers will inevitably decline across the board -- this reality shouldn't be surprising to anyone who works in or reports on politics.
Going forward, we should focus on more important questions. First, how much will Obama's approval numbers decline? Given the state of the economy, it wouldn't be surprising to see him in the low- to mid-40s by the end of the year. Second, what is the distribution of opinion on Obama's handling of health care? Aggregate public opinion on the issue is less relevant than how it's playing in the states of key senators whose votes will determine the fate of the legislation in Congress.
Today's guest pollster contribution comes from Brendan Nyhan, a political scientist and Robert Wood Johnson Foundation Scholar in Health Policy Research at the University of Michigan. This entry is cross-posted at his blog, Brendan-Nyhan.com.
The Washington Post is reporting that a new ABC/WP poll shows a major decline in Sarah Palin's favorability ratings. Her favorables have dropped from a peak of 58% after the GOP convention in September to 40% now, while her unfavorables have surged from a low of 28% to 53% now. Her 40/53 favorable-unfavorable ratio puts her into Hillary/Bush/Cheney territory as one of the most polarizing figures in American politics -- quite an achievement for someone who was a completely unknown less than a year ago.
It's almost impossible to imagine Palin getting the GOP nomination in 2012 at this point (though Intrade still puts the probability at 16%). With numbers like that, her general election prospects are dim, and the Post poll shows growing doubts about her among Republicans as well:
Republicans and GOP-leaning independents continue to rank Palin among the top three contenders in the run-up to 2012, however, with 70 percent of Republicans viewing her in a positive light in the new poll. But her support within the GOP has deteriorated from its pre-election levels, including a sharp drop in the number holding "strongly favorable" impressions of her.
And while Palin's most avid following is still among white evangelical Protestants, a core GOP constituency, and conservatives, far fewer in these groups have "strongly favorable" opinions of her than did so last fall.
...Perhaps more vexing for Palin's national political aspirations, however, is that 57 percent of Americans say she does not understand complex issues, while 37 percent think she does, a nine-percentage-point drop from a poll conducted in September just before her debate with now-Vice President Biden. The biggest decline on the question came among Republicans, nearly four in 10 of whom now say she does not understand complex issues. That figure is 70 percent among Democrats and 58 percent among independents.
Her favorability numbers also stack up extremely poorly against the rest of the expected 2012 field, as this graph illustrates:
The candidates are ordered left to right by their favorable-unfavorable ratio in the most recent poll on Pollingreport.com. As you can see, Palin's numbers are even worse than Newt Gingrich (!) -- the other highly polarizing candidate -- and she has less room to change her image because so many Americans already have an impression of her. By contrast, Romney, Huckabee, Jindal, and Pawlenty start the race without that sort of baggage and are therefore much more likely to make a serious run for the nomination.
To be sure, it's not impossible to come back from numbers like Palin's. Hillary Clinton overcame numbers that were nearly as bad and almost won the Democratic presidential nomination, but she did so with a great deal of hard work and discipline -- qualities that Palin appears to lack. Runner's World photo spreads, feuds with David Letterman, and useless policy op-eds are not going to turn her image around anytime soon.
Patrick Murray is the founding director of the Monmouth University Polling Institute and maintains a blog known as Real Numbers and Other Musings.
There are a couple of pieces of accepted wisdom when it comes to contested primary elections versus general elections: 1) turnout has a bigger impact on the ultimate margin of victory in primaries and 2) primaries are more difficult to poll (see point #1).
The voters who show up for primaries come disproportionately from either end of the ideological spectrum. Even in states with closed primaries (i.e. one has to pre-register with a party to vote in its primary), there is still a particular art for determining which groups of voters should be included in the likely voter sample.
Voters' likelihood to turnout generally correlates with their ideological inclination. Last year's Democratic presidential nomination provides a good illustration of this. Lower turnout caucus states saw a bigger proportion of higher educated liberal activists participate in the process. These same voters also showed up in the primary states, but they were joined by a good number of less educated, blue-collar Democrats. Result: Obama basically swept the caucus states, while Hillary Clinton held her own in the primaries. Texas, which held both a primary and a caucus that were won by different candidates, is a stark illustration of this turnout effect.
The same is true for Republican primaries. Lower turnout means a larger proportion of the electorate will be staunchly conservative in their views. As turnout increases, it's moderates who are joining the fray, thus diminishing the conservative voting bloc's overall power. And with the GOP being in its present ideologically-splintered state, small changes in turnout can have a real impact in primaries cast as battles between the party's ideological factions.
To some extent, we saw this play out in New Jersey's recent gubernatorial primary where the two leading candidates were seen as representing different wings of the Republican party. Former mayor Steve Lonegan cast himself as the keeper of the conservative flame, while former U.S. Attorney Chris Christie claimed to adhere to core conservative principles (e.g. anti-abortion), but presented himself as a more centrist option. New Jersey's Republican voters agreed - a plurality of 47% described Christie as politically moderate while a majority of 56% tagged Lonegan as a conservative.
The Monmouth University/Gannett New Jersey Poll released a poll nearly two weeks before the June 2 primary showing Christie with an 18 point lead over Lonegan - 50% to 32%. New Jersey has a semi-open primary - meaning both Republicans and "unaffiliated" voters are permitted to vote (although unaffiliateds have their registration changed to Republican if they do vote). So, technically about 3.5 million out of New Jersey's more than 5 million registered voters were eligible to vote in the recent GOP primary. But in the last two contested gubernatorial primaries only between 300,000 and 350,000 voters were actually cast.
So, how do you design a sampling frame for that? First, it's worth noting that state voter statistics show that extremely few unaffiliated voters ever show up for a primary - certainly not enough to impact a poll's estimates. So we are left with about one million registered Republicans, of whom still only one-third will vote. That is, of course, IF turnout is typical (more on that below).
Our poll for this primary used a listed sample of registered Republican voters who were known to have voted in recent primaries. It was further screened and weighted to determine the propensity of voting in this particular election (based on a combination of known past voting frequency and self-professed likelihood to vote this year). In the end, our model assumed a turnout of about 300,000 GOP voters, based on turnout in the past two gubernatorial primaries.
However, turnout in other recent GOP gubernatorial primaries in New Jersey have gone as low as 200,000 - that was in 1997 when incumbent Christie Whitman went unchallenged. Turnout in contested U.S. Senate primaries is also generally around the 200,000 level. On the other hand, turnout has been much higher than 300,000 as well. It even surpassed 400,000 as recently as 1981.
The GOP primary saw higher than average turnout in 1993 - another year when a trio of Republicans were vying to take on an unpopular Democratic incumbent. So, it was fair to speculate that Governor Jon Corzine's weak position in the polls would give GOP voters extra incentive to turn out in the expectation of scoring a rare general election win. On the other hand, perhaps the state's Republicans have become so demoralized by their poor standing nationally and 12-year statewide electoral drought that turnout could be lower than the 300,000 used for our poll estimate.
Because we had information on actual primary voting history for each voter in our sample - i.e. rather than needing to rely on notoriously unreliable self-reports - it was possible to re-model the data from two weeks ago with alternative turnout estimates. If the GOP primary turnout model was set well above 430,000 - a 40-year record turnout for a non-presidential race - the Christie margin in our poll grew to 23 points. Alternatively, if the turnout model was pushed down to about 200,000 - a typical U.S. Senate race level - the gap shrank to 13 points. In other words, adjusting the primary poll's turnout estimate from 5% to 12% of eligible voters could swing the results by 10 points!
Why? The analysis showed that "strong" conservatives comprise about half of New Jersey's 200,000 "core" GOP turnout - and this group was largely for Lonegan. But when we widened the turnout estimate, more and more moderates entered the mix. As a result, Chris Christie gained one point on the margin for approximately every 25,000 extra voters who "turned out."
On primary day, Christie ended up beating Lonegan by a respectable 13 point margin - 55% to 42% - on a 330,000 voter turnout. Based on the model above, if Republicans had been a lot less enthusiastic, Lonegan may have been able to narrow this gap to 8 points. On the other hand, record level turnout would have given Christie a 16 or 17 point win.
Alan I. Abramowitz is the Alben W. Barkley Professor of Political Science at Emory University in Atlanta, Georgia. He is also a frequent contributer to Larry Sabato's Crystal Ball.
On May 15th, the Gallup Poll reported what they described as a significant shift in Americans' attitudes on the issue of abortion. For the first time since Gallup began asking the question in 1995, more respondents described themselves as "pro-life" than "pro-choice" on the issue of abortion. The proportion of Americans describing themselves as "pro-choice" fell from 50% in May of 2008 to 42% in May of 2009 while the proportion describing themselves as "pro-life" increased from 44% to 51%. To back up this conclusion, Gallup cited a recent Pew Poll that showed a decline from 54% to 46% in the proportion of Americans who wanted abortion legal in all or most cases and an increase from 41% to 44% in the proportion who wanted abortion legal in only a few or no cases.
While the results of these two polls appear to show a shift in public opinion on abortion, Gallup neglected to report an important fact about the Pew results that might have undercut this claim. Pew has asked the same question on at least seven occasions since early 2007 with results ranging from a 45-50 split in February/March of 2007 to a 57-37 split in June of 2008. Taken together, these results show no clear trend. The 2009 results could reflect a real change, or they could just be random noise.
Gallup also made no mention of a CNN poll in late April of this year that showed a 49-44 advantage for the "pro-choice" label over the "pro-life" label. CNN has asked the "pro-life" vs. "pro-choice" question three times since 2007 with results ranging from a 45-50 split in June of 2007 to a 53-44 split in August of 2008 to the recent 49-44 split. Again, no clear trend is evident in these results.
And now a new AP poll appears to show continued stability in public attitudes on the issue of abortion. This poll, conducted between May 28 and June 1, found that 51% of Americans want abortion legal in all or most cases vs. 45% who want abortion illegal in all or most cases. These results can be compared with two polls conducted last year. An NBC/Wall Street Journal Poll in early September found 49% of Americans wanted abortion legal always or most of the time while 49% wanted it illegal with no exceptions or only a few exceptions. And a Washington Post/ABC Poll in August found that 54% of Americans wanted abortion legal in all or most cases while 44% wanted it illegal in all or most cases.
The Washington Post/ABC Poll has actually asked this question 23 times between June of 1996 and August of 2008. In these 23 polls, support for keeping abortion legal in all or most cases has ranged from 49% to 59%. Interestingly, the highest and lowest levels of support for legal abortion were found in two polls conducted only a few months apart in 2001.
The safest conclusion one can draw from these results is that at this point the evidence for a significant shift in public attitudes toward abortion is far from conclusive.
[J. Ann Selzer is the president of Selzer & Company and conducts the Des Moines Register's Iowa Poll.]
Can you trust your data when response rates are low? And, in this age of the ubiquitous internet, do we make too much out of its inability to employ random sampling? We asked and answered those questions in a study we conducted a few years ago, commissioned by the Newspaper Association of America. Given recent online discussions of data quality, I revisited this study.
In April and May of 2002, five surveys-asking the same questions-were conducted in the same market. The only difference was the data collection method used to contact and gather responses from participants. This rare look at what role data collection methodology plays in the quality of data yields some fascinating results. Our goal for each study was to draw a sample that matched the market, to complete interviews with at least 800 respondents for each separate study, and to gather demographics to gauge against the Census.
Method of contact. Our five methods of contact were:
- Traditional random digit dial (RDD) phone (landline sample);
- Traditional mail;
- Mail panel, contracting with a leading vendor to send questionnaires to a sample of their database of previously screened individuals who agree to participate in regular surveys, with a small incentive;
- Internet panel, contracting with a leading vendor to send an e-mail invitation to a web survey to a sample of online users who agree to participate in regular surveys, with a small incentive; and
- In-paper clip-out survey, with postage paid.
The market. We selected Columbus, Ohio as our market. It was sufficiently large that the panel providers could assure us we would end up with 800 completed surveys, yet it is perceived to be small enough that mid-sized markets would feel the findings would fit their situation.
Analysis. To compare datasets, we devised an intuitive method of analysis. For each of six demographic variables-age, sex, race, children in the household, income, and education-we compared the distribution to the 2000 Census, taking the absolute value of the difference between the data set and the Census. For example, our phone study yielded 39% males and the Census documents 48%, so, the absolute value of the difference is nine points. We calculated this score for each segment within each demographic, added the scores, then divided by the number of segments to control for the fact that some demographics have more segments than others (for example, age has six segments, education has three). We then summed the standardized scores for each method and those raw scores give us a comparison allowing us to judge the rank order of methods according to how well each fits the market. Warren Mitofsky improved our approach for this analysis.
Problem with the internet panel. I'll just note that both panel vendors were told the nature of the project-that we were doing the same study using different data collection methods to assess the quality of the data. I said we wanted a final respondent pool that matched the market. They would send reminders after two days. Participants would get points toward rewards, including a monthly sweepstakes. The internet panel firm e-mailed 7,291 questionnaires; after 850 completed responses were obtained, they made the survey unavailable to others who had been invited. Because the responses to the first 850 completed surveys were so far out of alignment with the Census, we opted to implement age quotas post-hoc, to systematically substitute some in the 45-54 age group (which were too plentiful) with respondents in other age groups (which were underrepresented) with additional invitations to the survey. We reported out both findings-those before and after the adjustment.
Results. Unweighted, the RDD phone contact method was best; the in-paper clip-out survey was worst.
Weighting just for age and sex improved all data collection methods. Most notable is traditional mail, which comes close to competing with traditional phone contact after weighting for age and sex. The in-paper survey showed the greatest improvement because the respondent pool was strongly skewed by older women. One in four respondents to that survey were women age 65 and older (26%). The median age was 61 (meaning, just to be clear, half were older).
Other data. This study was commissioned by the newspaper industry, so it was natural to look at readership data. Scarborough is to newspapers what Nielsen is to television, and we had their data from the market for comparison. Partly because of the skew toward higher income and especially in higher educational attainment in the internet panel, that method produced stronger readership numbers-higher than the Scarborough numbers and higher than any other data collection method. This was one more check on whether a panel can replicate a random sample, and casts suspicion on whether a panel can ever sufficiently control for all relevant factors to deliver a picture of the actual marketplace.
Concluding thoughts. I have to wonder how this study might change if replicated today. The rapid growth in cell-phone only households probably changes the game somewhat. Panel providers probably do more sophisticated sampling and weighting than was done in these studies. Our mail panel vendor indicated they typically balance their sample draw, though their database in Columbus, Ohio, was just on the low end of being viable for this study, so we're confident less rather than more pre-screening was done. We did not talk with the online vendor about how they would draw a sample from their database, though we repeatedly said we wanted the final respondent pool to reflect the market. It is our sense little was done to pre-screen the panel or to send out invitations using replicates to try to keep the sample balanced. Nor did they appear to have judged the dataset against the criteria we requested before forwarding it to us; it did not look like the Columbus market. We specified we did not want weighting on the back end because we were wanted to compare the raw data to the Census. Had they weighted across a number of demographics, they certainly could have better matched the Census. And, maybe that is their routine now. But, I wonder how the readership questions might have turned out, for example. The Census provides excellent benchmarks for some variables, but not all. Without probability sampling, I always wonder if the attitudes gathered in from panels do, in fact, represent the full marketplace.
Epilogue. Of course it would be a good idea to replicate this study given recent changes in cell phone use. The non-profit group that commissioned this study just announced it is laying off half its staff, so they are unlikely to lead this quest.
Sheri Rivlin and Allan Rivlin are the Co-Editors of CenteredPolitics.com. Allan Rivlin is a Partner at Hart Research Associates. In 1993 Allan Rivlin was a Special Assistant in the U.S. Department of Health and Human Services.
Remember 1993? Snoop Dogg was on the radio. Grunge ruled the world of fashion, and one of the top movies was "Groundhog Day" where Bill Murray had to relive the same day over again until he figured out just what he had to offer the world and finally got it right.
A charismatic young Democrat had just been elected President promising, among other things, to reform a broken health care system. Public opinion seemed to be behind him but the effort ultimately failed and a more careful reading of public opinion in those early months of the Clinton Administration reveal some of the fault lines that eventually sank the effort. Not only did reform fail to make it out of either house of Congress, but in the 1994 election voters ratified the decision and punished Democrats who supported reform rather than the Republicans who had defeated the plan.
Now a new Democrat has taken office promising healthcare reform. The question becomes; has enough changed in public opinion to offer hope that the outcome will be different this time around? A thorough review of the available polling then and now is less than encouraging for supporters of comprehensive health care reform (a category that includes the authors who should be understood to be supporters of comprehensive reform albeit sobering ones.)
Where common questions can be found in polls leading up to health reform 1993 and 2009, the public is currently less attuned to the issue, expresses less dissatisfaction with the status quo, and offers lower levels of support for the general prospect of reform. But an even greater challenge for reformers is the fact that the basic contours of public opinion that undercut the previous effort continue to be true today - perhaps even more so.
Just as in 1993, it would be easy to read current polls as highly encouraging. Many of these measures appear quite strong, it is just that they are not as strong as comparable numbers in surveys taken before the start of the 1993 effort when many pollsters, including those advising the White House were fooled into believing they had a clear mandate for major change.
Now: A 2008 Harris Interactive survey finds 29% saying so much is wrong with the current health care system that it needs to be completely rebuilt, and an additional 53% says that while there are some good aspects the system needs fundamental changes. That adds up to 82% calling for fundamental change. Just 13% say the system works pretty well and only needs minor changes.
Then: The problem is, these results were typical, though a little stronger in the period before the failed effort. As early as 1991, the same pollsters (then Lou Harris and Associates, the word "Interactive" as we know it today had not yet been coined) using the same question recorded 42% saying so much is wrong with the current health care system that it needs to be completely rebuilt, and an additional 50% said that while there are some good aspects the system needs fundamental changes - for a total of 92% calling for fundamental change and just 6% said the system worked well and only needed minor changes.
Now: A 2008 Harvard School of Public Health survey found a 55% majority in support of "national health insurance" with 35% opposed. While this is unlikely to be a phrase that this round of reformers will find useful or descriptive of their proposals, the term that was in common use in 1993 does allow for an apples to apples comparison.
Then: The same researchers using the same phrase in 1993 found 63% supporting "national health insurance" and just 26% were opposed.
Then as now the real problems facing health care reformers were structural and clearly visible in the polls. As the nation reached near consensus that there was a problem, there was never any such agreement on the specific solution. While many people agreed then as they do now that it is wrong that so many Americans are either uninsured or underinsured, the priority then, as now, for most people was on finding ways to lower their own health insurance cost. Then as now most people had health insurance that they judged to be pretty good.
Then: In 1993 a 77% majority told Martilla and Kiley that they were at least somewhat satisfied with their own health care coverage.
Now: For comparison, 82% expressed a similar level of satisfaction with their own insurance in a 2007 Greenberg Quinlan Rosner Poll.
Then: A 1993 Gallup Poll asked people about their priorities for reform and 38% said they wanted health insurance that included all Americans. The bare majority, 51% wanted to control costs, and 10% volunteered that they want reform that did both.
Now: The comparison here is a little less direct, but in 2008 the Harvard School of Public and the Kaiser Family Foundation found similar results with 45% saying they want to make health care insurance more affordable and 22% saying their goal for reform would be to expand insurance to the uninsured.
Then: An NBC News Wall Street Journal Poll in March 1993 found 66% agreeing with the statement "I would be willing to pay higher taxes so that everyone can have health insurance." Just 30% were opposed. A Martilla and Kiley poll found a similar result but in a clear sign of the problems that would emerge, among their 65% willing to pay higher taxes, just 25% said in a follow up question that they would be willing to pay as much as $50 more a month, 40% said they would pay $30, and the majority 62% were only willing got go as high as $10 per month more in order to give coverage to everyone.
Now: In the most recent NBC News Wall Street Journal Poll conducted February 26 to March 1, 2009 the public is now split with just 49% agreeing with the statement "I would be willing to pay higher taxes so that everyone can have health insurance" and nearly as many 45% do not agree.
Does all of this mean that the Obama plan is doomed before it has even begun? Of course not, but putting the apparently positive number from many of today's poll questions in the context of even more positive numbers from polls taken before the previous failed effort should serve to underscore the difficulty of the challenge ahead.
It is clear that the new team will benefit from lessons learned in the earlier health care reform effort. Reflecting a hard won understanding that most Americans are fairly satisfied with their current coverage, the first words out of any Administration spokesperson, including President Obama, on the subject of health care reform is that if you like what you have now you will be able to keep it. Also reflecting the priorities expressed in public opinion polls today (and back then), far greater emphasis is now being placed on cost containment than on extending coverage.
The real question will of course come in the details of the proposal. If Obama can come up with a plan that extends coverage to more Americans without a major increase in the burdens it places on the individuals and businesses who pay for it, then it will be difficult for those who want to see this effort fail to generate much public opposition. Naturally this is a tall order, but we would not want to be among the legions of commentators who have had to swallow their doubts that Barack Obama can achieve the difficult.
The only thing we will predict is that there will be a lot of articles written looking at statistics like some of the ones mentioned here (in fact they are likely to grow stronger as the heat is turned up on the issue) to make the case that this time around the public strongly supports reform. We hope this little bit of context will help keep these articles in perspective.
The authors wish to thank Julia Kurnik for Research Assistance and Robert Blendon of the Harvard School of Public Health for invaluable assistance. Would anyone try to write this article without first calling Bob Blendon?
[This Guest Pollster contribution comes from Philip Gould, who served as a polling and strategy adviser to the British Labour Party for general elections held from 1987 until 2005.
Editor's note: Gould was a central figure in the dispute between pollsters Stan Greenberg and Mark Penn that we have covered this week, as he was responsible for managing the services that each provided to the Labour Party. He submitted his comments to Pollster.com in an effort to help clarify and resolve some of the issues raised here this week.
Since I emphasized the question of whether Penn delivered complete marginals and cross-tabulations, I want to promote the following paragraphs that come toward then end of Gould's memo:
After a poll Stan normally presented a filled in questionnaire, a full banner book containing complete cross tabs.
Mark had a different approach. Following a poll he quickly made available a full and extensive polling report. This went immediate to the whole campaign. This was not an inconsiderable document. I have one in front of me now: it is 18 pages long; it contains historic voting and favourability data; it closely examines 12 targeting groups ranging from rural lower class Conservatives to union households; it uses seven different batteries to examine campaign issues. It analyses responses to the news and key policy areas. And of course it contains numerous message batteries: in all well over 100 questions were asked and recorded. All of these were analysed by voting preferences, and sometimes by demographic categories.
These reports were extensive and useful documents, far in excess of a normal filled in campaign questionnaire. They did not constitute a full banner book and did not contain 'full marginal's' in the manner favoured by Stan Greenberg, but what Penn did supply was both exhaustive and useful, and certainly met the regular needs of the campaign. As one senior campaign official with responsibility for polling in 2005 has said: 'Mark Penn 'could quite fairly argue that the memos were intended for an audience that had no time or interest in delving into every corner of the data. I don't think that in any way illegitimises the findings or his advice'. On a personal note Mark Penn invariably supplied any additional cross tab or targeting data that I required, and I presume the same is true of others. Two pollsters, two approaches.
Gould's piece covers far more ground than this narrow excerpt. It is well worth reading in full.
-- Mark Blumenthal]
I am aware that intercession in the Greenberg/Penn polling war can precipitate what has probably never happened before: uniting Stan and Mark in the face of a common enemy (i.e. me). But with all the risks it entails I will press on. From the start I must declare an interest- I suspect I am one of the very few people around who can claim that they like and respect both Greenberg and Penn (I can already feel them starting to unite against me!). I worked with Stan for well over ten years and believe him to be an outstanding pollster and strategist. I worked with Mark for a much shorter time, and came to greatly appreciate his skills too, different from Stan's certainly, but considerable for all that. It is in that spirit that I write this piece.
There are so many issues here, of methodology, strategy, personality and of course memory that getting to the truth of what actually happened in the UK election campaign of 2005 is probably impossible, but I will try at least to clear away some of the fog. Not by focusing on the smaller, although I accept crucial disagreements between the two pollsters, but by trying to paint a bigger picture, and using where possible contemporary sources, notes written at the time, my rather sketchy diary, and in particular a lecture I made to the LSE on the campaign in 2006 which pretty accurately sums up what I believe about the campaign.
[Continue reading after the jump]
For me the starting point was a letter faxed to me by Stan in 1992 asking me to fly to Little Rock to observe the Clinton campaign, and to debrief on the negative campaigning that the Conservatives had used to win the 1992 election in the UK.
'I left immediately and in a way it saved my political life, taking me from a failing and dismal Labour project, to a world of political confidence and optimism. I wrote about this later: 'I still vividly recall arriving in Little Rock in 1992 still stunned by Labour's awful defeat in that same year, feeling the late summer heat as I left the airport, and arriving at the campaign headquarters and seeing a whole new world of possibility emerge. I remember the kindness I received; embarrassed by our failure in Britain but being told that defeat was a step on the road to victory, a badge of honour not of blame. Above all I remember the incredible energy and pulsating life of the campaign, its extraordinary confidence, and the way it had simply revolutionised the way campaigns had been run.'
I learned much in that campaign, and bought most of it back to the UK, including Stan Greenberg who became our pollster in 1994 when Tony Blair became leader of the Labour Party
For ten years Stan was part of the team, but by 2004 there were signs of dissatisfaction on both sides. You can clearly sense in Stan's book a growing sense of disenchantment with the New Labour project, provoked in part by the Iraq war but deeper than that. Equally there was in Downing Street a sense that it was time for new voices, and new ideas, in the face of mounting political difficulty. This was not my view but it was the view of some, and in the summer of 2004 Mark Penn was commissioned to conduct a series of polls, to see if new insights could be gained. I was not told of this principally I am sure because of my closeness to Stan (we used to be business partners as well as colleagues). Finally in mid-Sept I was told of Penn's involvement, and informed that Downing Street wanted to use Mark but keep Stan involved. I was mandated to manage both relationships, which was not to prove an easy task.
In December I told Stan of Mark's involvement, and it was a pretty grim meeting which caused me some distress. In his book Stan claims that I was afraid to meet with him alone to inform him which was quite untrue as I did tell him, was the first to do so, and did so with just the two of us present. By then I had started to have meetings with Mark Penn and so began the period of dual pollsters. This situation continued through 2005 until the election in May. Stan hated it and was clearly resentful and unhappy - but he kept going, and made the best of it. In the campaign itself Stan felt more unhappy still, with Mark Penn the dominant pollster. I certainly do not claim that I handled Stan perfectly in this period, and was too irascible and sometimes angry. I regret that, especially when it sometimes rubbed off on Sam Weston his excellent assistant on the campaign. It was tough dealing with two formidable pollsters, but I do not agree with Stan that it would have been better if he had left the campaign in 2004, as he suggests in his recent posting. In the first place there was the issue of loyalty to Stan, who had helped us so well over so many years. Secondly I felt that in this campaign, fighting as it was the headwinds of public hostility on Iraq and other issues, that more voices were better than less. Greenberg and Penn are very different pollsters, with very different approaches, and crucially with very different value sets, but both have significant contributions to make. Stan puts methodological exactitude first: he is the Volvo of pollsters highly engineered and meticulously thorough. He is strategically astute, but follows the data carefully because he has so much respect for it. His politics are modernising but rooted hard in fairness based populism: his favourite dividing line will always be based on a contrast between the many not the few, his emotional heartland will always remain hardworking families. He is a natural iconoclast, always challenging, often doubting, and leading him sometimes to put flexibility ahead of consistency. Mark also uses and understands data well but leans to strategy ahead of data, and he can be strategically brilliant. He prefers consistency to flexibility, believing that a strategic position once adopted should be held unless there is compelling evidence to the contrary. His instinct is to stick rather than to shift. Mark's politics are far less populist than Stan's, favouring aspiration to fairness as a guiding concept.
This then was the background to Stan's book, and to Mark response to it. If that is the context, what then of the issues - I will take them in turn.
1 Mark is wrong to say that I knew of his work from the outset, I did not. I first discovered Penn's involvement in the Labour Party election campaign on September 13th.
I repeat verbatim my diary entry for that day:
'Sally (Morgan) said to me: look we have been using Mark Penn for a polling project with Tony and we want you manage that relationship, and they were worried about Stan. I was very shocked by this, that behind my back I think they had conducted at least two and possibly three polls, and had a complete operation going since the July period. They did not want to involve me because they did not want to hurt Stan'.
This account will be supported by many who worked at Downing Street and on the campaign, and is unarguably the truth of what happened.
2 I disagree with Stan's characterisation of the campaign as over-rigid, and too inflexible. In my 2006 LSE lecture I explain why I believed that in this campaign consistency was at a premium, not least because of our determination to avoid the fate of the Kerry campaign. I wrote then:
'In truth Senator Kerry was trapped by ambivalence, not certain about the war, and he found it hard to appear certain about anything else. He had very talented advisers but he did not have a political project. The flexibility of the war room, essential twelve years ago was inadequate as a compass in the rough and treacherous waters of a nation at war. We needed battle ships not light cruisers. In Britain we watched and we learned. This time the view of the campaign at almost every level, and certainly my absolute conviction, was that in a time of uncertainty, turbulence, and electoral sullenness the first imperative was absolute strategic clarity, robustness, and constancy. That is why we were so determined to show courage and consistency under fire. To hold our nerve and certainty despite all the usual noises off. Above all to be strategic not tactical. That is why we were so determined to make the economy not just the campaign message but the campaign anchor, repeated endlessly until it broke through. Why Tony Blair and Gordon Brown campaigned together to hammer home the economic message. Why our message was simple and potent -Forward not back- and why it was repeated endlessly from the start of the campaign until the finish. The vicissitudes of terror, war and insecurity made robust confident clarity essential but the need to engage had not disappeared nor had the need to listen and respond as the public vented anger and concern. In our campaign strength and connection had to learn to co-exist. This was the paradox of the 2005 campaign. We had to be confident and strong in our message and leadership, but sensitive and responsive in our relations with the electorate. And these two apparently conflicting imperatives had to be implemented simultaneously, strong and responsive at one and the same time.'
This was the campaign we hoped to build and I believe we did: consistent, but also flexible, constant but also responsive. In my judgment it was this balance of strength and responsiveness that took us through to victory. And our polling played a big part in this.
3 I do not agree with Mark Penn that the distribution of polling information was restricted, and nor do I agree with Stan that the polling was rigged. The very nature of the campaign, the need for a plurality of polling sources, the immediate and rapid distribution of polling information make both positions impossible. Once again I wrote in 2006:
'The third element of connection was a new approach to polling which I called 'wrap around polling'. Which is diagnostic; intuitive; responsive; multi-faceted and pluralist, but also systematic and rigorous. We wanted polling not just to tell us what had happened, but to alert us to what might happen. To be a kind of early warning system to anticipate where an uneasy and dissatisfied public might flare up in protest or anger. Effectively research became radar for the election. To do this we used multitude of polling instruments: strategic message polling; standard daily tracking; internet panels of various sizes; marginal polling; daily focus groups. And this polling was not kept tight to a small group of insiders but distributed widely and openly throughout the campaign. Basically if anyone wanted to see the polling they could. The old days when a pollster is effectively Doctor dispensing tough medicine to uninformed politicians are gone. We are all polling experts now, or at least equal partners in the polling process. This approach was effective. For example the Prime Minister had always said that the issue of immigration should be left until the public turned against Michael Howard the Conservative leader when he went too far on the issue which he inevitably did. Night after night we tracked the public response on the issue seeing it move from whole hearted support of Howard on the issue, to a gradual bemusement that it was all they appeared to talk about, to a kind of contempt that this was the only issue they had, and they seemed to exploiting race for reasons of political advantage. At this point we pounced and the Prime Minister made a powerful speech arguing for a balanced approach to immigration shredding Tory polices and assumptions. That was the end of immigration as an issue in that campaign'
This was the culture of polling in the campaign, as pluralist as possible, and as open as possible. Every poll that Mark did, Stan got and so did everyone else. In this context I simply cannot agree with Stan that the polling was rigged, because there was so much of it, and it was so varied in methodology and type. Mark Penn's polls had so many questions, that were so varied, and which had so much outside input into them, that 'rigging' them seems impossible to me, and too strong a word to use. Equally it is not true to suggest, as Mark claims, that 'Stan was out of the loop', and to suggest that he could not be trusted with 'highly sensitive questions' is completely false. Everything the campaign got Stan got, and the campaign got everything, in its distribution of polling information this was probably the most open campaign the Labour Party had ever conducted.
4 I am confident that our strategy was right, and feel that Stan is unfair to it. At the core was a relentless focus on the economy, which dominated all else. That was the absolute bedrock of our campaign, everything else secondary to it. We did focus on women voters, and in particular younger female voters, and this focus worked, as the evidence shows. In the Ipsos-Mori exit poll we were shown to be level with men, but led by 6 points with women. Our focus on women gave us consistency, and gained us the vital votes we needed to win, as well as being in my judgement, the right and progressive thing to have done. As for older voters it was not those that deserted us but younger voters, as Stan acknowledges in his book.
5 Mark is wrong to say that Stan worked for Gordon Brown, and Mark for the Prime Minister. Both pollsters worked for the Labour Party, and in the campaign all information was shared. We are a party system, not a presidential one.
6 As for forecasting landslides: both pollsters got close to doing so, but at different times in the campaign. At the end Stan was probably more optimistic than Mark; at the campaign start the reverse was true. Stan's marginal polling was exemplary and accurate, but it was Mark's last major poll that was accurate to within one point.
7 On Iraq I did not 'delete' messages on Iraq as the book claims, certainly not as a consequence of polling by Mark Penn of which I had no knowledge. To give you an example, in one mid-summer polls that Stan mentions there are several Iraq batteries, and many references to Iraq as an issue. In any event it was not within my power to 'delete' anything, I was just one member of a team who collectively supervised questionnaires.
8 Finally one of the most contentious points of all: Did Mark Penn make available the 'agenda's, marginal and cross-tabs as requested and without reservation'. This is a grey area, but I will try and clear it up.
After a poll Stan normally presented a filled in questionnaire, a full banner book containing complete cross tabs.
Mark had a different approach. Following a poll he quickly made available a full and extensive polling report. This went immediate to the whole campaign. This was not an inconsiderable document. I have one in front of me now: it is 18 pages long; it contains historic voting and favourability data; it closely examines 12 targeting groups ranging from rural lower class Conservatives to union households; it uses seven different batteries to examine campaign issues. It analyses responses to the news and key policy areas. And of course it contains numerous message batteries: in all well over 100 questions were asked and recorded. All of these were analysed by voting preferences, and sometimes by demographic categories.
These reports were extensive and useful documents, far in excess of a normal filled in campaign questionnaire. They did not constitute a full banner book and did not contain 'full marginal's' in the manner favoured by Stan Greenberg, but what Penn did supply was both exhaustive and useful, and certainly met the regular needs of the campaign. As one senior campaign official with responsibility for polling in 2005 has said: 'Mark Penn 'could quite fairly argue that the memos were intended for an audience that had no time or interest in delving into every corner of the data. I don't think that in any way illegitimises the findings or his advice'. On a personal note Mark Penn invariably supplied any additional cross tab or targeting data that I required, and I presume the same is true of others. Two pollsters, two approaches.
These are the big points, there are many small ones, but this is not the time or the place for that.
My overall view is clear: the strategy was right, the balance between consistency and flexibility was right, the polling open and freely available, the campaign a success, conducted in the war-room spirit but in 2004 not 1992. Penn made a major contribution to that success, and deserves credit for that. Mark came into a new and difficult situation and helped give the campaign the consistency and strategic clarity. Whatever people say about Mark in other campaigns, and at other times, in this campaign he got most things right and did so with grace and good humour. I very much enjoyed working with him. Stan of course hated the whole process and I understand why. But he showed great courage in keeping going and got much right, particularly in his marginal polling. He played a part not just in one election but three, and he should be proud of that.
In all this lies the truth, but it is not clear-cut, nor certain. Finding the truth is never easy, and in any event it is always multi-faceted and complex, especially in a tense hard fought election campaign as this was.
This has been a long piece but unless you understand the circumstances of that campaign you have no chance at all of understanding why Stan wrote that book, and why Mark responded as he did.
In the end there is another, deeper truth. The battle between these pollsters may be intriguing to us, but in the great scheme of things it is politicians and leaders who decide, who make successful campaigns, and build great political projects. One of the most powerful recurring themes of this book is how Stan laments the fact Tony Blair so often ignored his advice and just went his own way. That was true of Stan, but was also true of me and Mark Penn as well. Tony Blair listened to pollsters and advisers but in the end went his own way on his own terms. He marched to his own drum. That may have led to bumps along the way, but that is why he was then, and still is, a great political leader. He listened, but he led. When the final histories are written, it is not Stan's or Mark's or my view that will matter, but the actions and decisions of politicians entrusted with the responsibility of leadership. We should all have the humility to recognise who are the real hero's in the world of politics that we all love so much.
This guest pollster contribution from Stan Greenberg is part of Pollster.com's week-long series on his new book, Dispatches from the War Room and responds to comments from Mark Penn in Mark Blumenthal's post earlier today. Greenberg is chairman and CEO of Greenberg Quinlan Rosner.
To avoid this discussion descending into an ugly mud-wrestling match between two squabbling pollsters, I will only take up issues where the "facts" are indisputable and where we learn something about Tony Blair and political leadership and about differing approaches to polling and strategy.
What this exchange reveals even more clearly than the book itself are the limits of building a strategy from a coterie of target groups, rather than from the leader's vision or party's mission for the times. It underscores the need for frankness about what is holding voters back and the need to challenge leaders with blunt truths. It underscores the need for transparency and methodological rigor.
Penn's basic argument is straightforward. He took over the campaign's polling in July 2004 about nine months before the election when Blair was at a low point, working under Philip Gould, Blair's long time advisor for research and media. Greenberg was pushed out and was in no position to judge the character of Penn's work, as he was "not in the loop." Seems straightforward enough.
When I first learned in December of Penn's involvement and in January of our dividing the polling, I was convinced that Gould had played just such a role and I wrote about it. I was wrong. Philip was hurt by the accusation that he had concealed Penn's involvement and wrote me with detailed diary entrees that show he only learned of it in September and resisted Penn's involvement until the end of the year, when he decided to "make the best of it."
Penn's premature rush to anoint himself as Blair's pollster obscures Blair's effort to examine competing solutions to the problems he faced. In May, Blair had reached a low point in the polls, dragged down by Iraq, the "hyping" of pre-war intelligence and Abu Ghraib. He was very despondent, seriously considering not running again and consulted widely, including with President Clinton and Senator Clinton who urged him to run and to use Penn.
Penn offered his own path back for Blair, aided by huge surveys and "clustering work" that coughed up "school gate mums" as a key target. Because Labour got its highest marks on the economy, his message started there, but Penn's emphasis was on policies that appeal to the groups that can grow Blair's coalition. Penn's imprint was immediately evident in Blair's September conference speech when he spoke of the stresses of the need for "more choice for mums at home and at work." Blair's policy offer was grounded in this clustering and coalition building.
At the very same time, we were commissioned by Phillip to do a special research project and I reported in July with a very different approach to the problem - centered on New Labour's central mission. For the first time in a long time, respondents shifted to Labour on hearing of Blair's commitment to "a better life for hardworking families," though only when Blair expressed his own frustration with the state of public service reform and offered some learning by showing independence from Bush on climate change. Iraq was the elephant in the room. Finding a way to acknowledge it, even indirectly, allowed people to come back to Blair's project.
In the September party conference speech, Blair was eloquent about "hardworking families," but just could not get himself to be reflective on Iraq - perhaps with Penn's support. That was the learning voters needed if they were to come back.
I respect Blair for rejecting my advice and deciding to go with Penn who did not push him to address the Iraq question and who offered a way to make electoral gains. The mistake was not firing me and leaving both of us in the campaign.
In fact, I have all of Penn's memos - about a two-inch pile on my desk at the moment, available for inspection by Mr. Blumenthal. Philip's note to me confirms he shared all of them during the course of the campaign, as did many of my friends "in the loop."
The whole concept of "in the loop" betrays a lack of transparency and openness in Penn's approach to campaigns - painfully evident in the Blair campaign, perhaps a precursor to Hillary Clinton's presidential run two years later.
Pollsters as a rule share the results for all their questions and hypotheses, even the ones that didn't pan out. In the Blair campaign, Penn provided a memo with large tables including only the questions he wanted to report; he did not provide a standard book of demographic cross-tabulations. Read Penn's words carefully, "The campaign received all of the agendas, marginals, as requested without reservation." In short, he provided breakouts only when asked, in effect keeping his own client and campaign team "out of the loop."
The surveys were methodologically sloppy and included biased tests, though it is important to underscore here that Philip Gould came to value Penn's research and rejects my characterization of it in the book.
Specifically:
1) Penn failed to incorporate professional learning from Britain. Penn national polling - not some errant tracking program - showed Labour with landslide leads of 8 or 9 points for the entire six weeks prior to the election being called. Penn discovered just 27 days before the election what every pollster in Britain has knows: you have to weight to offset the "shy Tories" - Conservatives reluctant to be interviewed. In an instant, the Tories gained 6 points in Penn's polls.
2) Penn's fixed targeting let real targets slip away. With Penn focused on "mums," the campaign regularly rolled out initiatives on breast cancer screening and childhood obesity. But voters in the key marginal seats were older and among those most likely to return to Labour, two-thirds had no children at home and found this campaign irrelevant.
3) Penn exaggerated the reliability of findings. Penn conducted a valuable weekly open-ended Internet panel of undecided voters. When the sample dropped to 100, so did the reporting of sample size that produced a testy email exchange that restored it. Still, Penn reported this as a "Survey of Undecided Swing Voters" and reported the full percentage results over 18 pages, including results for men and women, with about 50 cases each.
4) Penn created biased tests. Two weeks before the election, Penn declared that "our policy approach remains stronger than the Tories," but the Labour statement was more than twice as long, with more rhetorical flourishes and covering a much broader range of policies with greater specificity (which I'm happy to share). Even with this biased test, the Conservative's statement ran 6 points ahead of its vote. An unbiased test might have revealed potential Tory gains.
To inform the decision of whether to close positively or negatively, Penn constructed a sensible experiment where half the respondents were read positive statements about Labour's progress and half read attacks on the Conservatives' record and plans, and then respondents were asked to vote again. But this was not meant to be a fair test. The negative statements were 50 percent longer by word count and helped foreclose an uplifting close.
Penn describes the 2005 third-term as "historic" but in the campaign everyone was disappointed with the result, what the media called Labour's "drastically reduced majority," produced by a disengaged electorate and historically low turnout. Many factors contributed to the result, but among them were Penn's research, not to mention having two polling teams with different theories on how to win.
This guest pollster contribution from Stan Greenberg is part of Pollster.com's week-long series on his new book, Dispatches from the War Room. Greenberg is chairman and CEO of Greenberg Quinlan Rosner.
Brian Schaffner focuses on the role of pollsters in identifying groups and thus empowering them -- making their opinions relevant to political leaders. I am very conscious of the role and as you correctly point out, I put the spotlight on Macomb County's "Reagan Democrats" and after this election, moved the spotlight next door to upscale suburban Oakland County.
What groups "matter" in my work is not some blind search of the data to find interesting and distinctive groups.
In the period when the wall between my academic and political lives was starting to crumble, I was very taken by E. E. Schattshneider's argument that whomever decides what the fights about likely wins. Successful political leaders and campaigns control the subject, define the choice and choose the fight. Drawing that line decides what issues are important and critically, who gets engaged and who loses interest. In 1992 Clinton made the election about change and the economy stupid and President Bush failed to make it about trust and experience. This year, Obama made it about change and Hillary Clinton tried unsuccessfully to make it about experience, but when she shifted to the economy and the middle class, she put the spotlight on white working class voters who rallied to her.
"Reagan Democrats" derived from the political project that tried to put the middle class back at the center of a renewed Democratic Party -- but the groups emerged from the project. In the book, I argue for the strength of these five leaders because they made politics purposeful.
Related to this point is David Moore's important discussion of "intensity" of beliefs and and the ability of leaders to get people to change their views on an issue and follow them. Whether a leader touches people, understands the times and poses a choice that impacts their lives impacts both which issues get highlighted and how intense are reactions.
I fully agree that mapping intensity will give you a much better view of public thinking and how issues are likely to break. But what is interesting about my Jerusalem example is that people held intense views (which I measured and monitored closely) when they rejected the idea of dividing Jerusalem, but shifted their views nonetheless once the public debate forced them to think about all the possibilities. This is a life and death and emotional issue and voters followed it very closely but Ehud Barak, like earlier Israeli leaders, was able to move the deliberation to a longer-term framework for preserving a Jewish state.
Focusing on intensity will help pollsters know which opinions really matter and difficult to move, and I did a lot of simulation in my polls to see how dynamic are opinions. But I'm still in awe of how much opinions shifted on such a central issue in such a short period and still learning from the fact.
This guest pollster contribution from Stan Greenberg is part of Pollster.com's week-long series on his new book, Dispatches from the War Room. Greenberg is chairman and CEO of Greenberg Quinlan Rosner.
Charles Franklin rightly begins his comments by putting up my quote on page 58 that "the endgame in presidential campaigns brings out all sorts of irrationalities, starting with the media polls. Many are criminally bad." One of the problems in writing a book and a memoir is living with your words and thoughts, particularly when as unnuanced as those.
In retrospect, I might have been more nuanced. First, I made the comment in the context of the Clinton presidential campaign when the statement was clearly true, as described in the book. Second, it reflects my experience during the final weeks in campaigns in Britain and Israel and in Latin America, even very recently. But because of sites like Pollster.com, there is more transparency and exposure of shoddy methods, and despite strong budget pressures, the national media organizations in the US produced very credible polling programs in this last election. But as recently as 2004, there were stark examples of volatile polls without political weighting conducted by Gallup and aired on CNN, along with commentary on how fickle were the voters. The challenge will be what happens with media polls, as there is more upheaval in the industry and need for more costly multi-modal methodologies and greater use of IVR.
This is a very different matter when one goes down to the state and congressional level and when you are in lower turnout elections and primaries. The media polls, as well as polls conducted by universities and institutes, are often out of line with the campaign surveys, as they are less likely to screen or filter for likely voters, factor-in historic turnout patterns and consider use of exit polls, as well as CPS. That one in four state polls in 2008 were conducted one day suggests we are dealing with a genuine issue.
I Amen, Franklin's Amen. The biggest problem is the reporting, not the polls themselves. It is the "outlier" poll -- not the boring average that gets headlines. But it is even worse in the war rooms I'm writing about that are poised to explode in the closing week of the campaign. It is the errant poll, not the average, that sets off the sparks in the war room and gets the attention of the candidate.
This guest pollster contribution from Stan Greenberg is part of Pollster.com's week-long series on his new book, Dispatches from the War Room. Greenberg is chairman and CEO of Greenberg Quinlan Rosner.
I want to come back in my next post to the first half of Charles Franklin's piece where he raises legitimate issues about my characterization of media polls. But before that I want to develop a point he makes -- about order of questions -- because I actually have some new information on the subject and it is hard to find anyone interested in such issues.
When it comes to the vote, I have spent a lot of time assessing how to get people most comfortable with answering and to minimize the number of false undecided. In my experience, the closer the question to the start of the survey, the larger the undecided. There is a price in possible bias in introducing prior questions but if those questions reflect the broad political context in which the vote choice is being made, you can risk that. So, we will usually have a right direction/wrong track question, most important problem (either open-end or closed), a favorability or thermometer battery on political leaders, parties and organizations -- broadly distributed and balanced. Each election is a test of whether that structure produces unbiased estimates.
Our national presidential results for Democracy Corps or by Greenberg Quinlan Rosner closely resemble the national media polls, with a somewhat smaller undecided (though I haven't done the comparison on this issue).
Presidential approval is different. People are comfortable answering the question and at presidential level, not a lot of undecided once in office for a period. When I polled in the White House for President Clinton, we started by asking the job approval after the thermometer/favorability battery and right before the vote. With President Clinton, I discovered that asked at that point, his approval rating was higher than reported by other organizations. So, we conducted an experiment where half the respondents heard the job approval at the front of the survey and half heard it right after the favorability battery about Clinton and other political leaders. The implicit comparison led people to rate Clinton higher. Since job approval was the indicator most important to us, we moved up to the front in the survey.
We continued that practice during Bush's term and Democracy Corps' approval for Bush was higher than the norm -- and often cited by the White House. Bush's job approval, unlike Clinton's, likely fell if considered alongside other leaders or if the survey dealt with Iraq, the economy, health care or any other topic where is approval was probably even lower. I know this is obvious, but order really matters.
I agree that openness is the best route to sorting these issues -- including publishing the full surveys wherever possible.
This guest pollster contribution from Stan Greenberg is part of Pollster.com's week-long series on his new book, Dispatches from the War Room. Greenberg is chairman and CEO of Greenberg Quinlan Rosner.
Kristen starts with a key piece of the role of pollster -- keeping elites and elected officials "in-touch" with people; a reality check. Lost in all the talk about politicians with their fingers to the wind is how hard it is for the voter to get heard amidst the lobbyists, experts, bureaucrats, donors and more. If there is a problem to solve, that's not a bad one to address.
I really share Kristen's frustration with the search for the "silver bullet" -- particularly a word or phrase -- when the "silver bullet" is really have a theory of the race, knowing why you are running, defining the choice in a way that really impacts people's lives. Look what happened when Obama's "change" encountered Clinton's "experience" and coalition of small groups. Clinton lost ground because Obama had the force of what was happening in society and the economy with him. Understanding your times and having a mission puts you in a more powerful electoral position.
I agree that voters not voters not grounded in the current ideological polarization. That's why it hasn't worked to label Clinton a "liberal" in 1996 or Obama in 2008. But that has led to some to say that voters aren't moved by big ideas and political political forces and alignments and they have rushed to advance a bunch of small policies. I think 2008 showed that America is a country moved by big currents and open to big ideas about how to address our problems. I think Obama and McCain debated big philosophical issues and those mattered in how people became engaged and voted.
This guest pollster contribution from Stan Greenberg is part of Pollster.com's week-long series on his new book, Dispatches from the War Room. Greenberg is chairman and CEO of Greenberg Quinlan Rosner.
I
think Steve Lombardo makes an important point. Some elections and some
periods invite big choices and political projects, while many do not.
When I raised the issue, I was not so much suggesting what is the best
choice in a particular campaign. Obviously, you will do what works and
use the opportunity of the moment to win. But I was making a
self-reflective judgment about my work over a number of decades, and
the parallel emergence of polling that was self-consciously tactical,
some ideologically centered and some post-ideological. There is
evidence cited in the book that more recent generations of political
consultants (p. 423) give greater weight to the thrill of the contest
rather than partisan or ideological goals, compared to earlier
generations. I don't think that was just a consequence of diminishing
issues in the late 1990s -- as many of them advanced this approach
earlier -- e.g., the 1996 Clinton campaign -- and later.
I do think any campaign -- even ones in less tumultuous times and a
lower place in the ticket -- will seek to pose a choice that draws on
the issue and partisan environment, the candidate's goals and project,
and that poses a defining choice with the opponent. That is strategic
and possibly purposeful, regardless of how small bore. But
collectively, pollsters and political consultants elevate the political
discourse in the country as they are part of that process. But where
they are opportunistically jumping on tactics or "swing groups"
detached from that kind of process, they may well be diminishing the
quality of political discourse.
Today's guest pollster contribution comes from Stan Greenberg, chairman and CEO of Greenberg Quinlan Rosner. He will be discussing his new book, Dispatches from the War Room all week on Pollster.com.
I very much appreciate Pollster.com hosting this discussion of the book and Mark's introductory note that raises central issues.
Let me underline one point that is at the heart of the book and then address the two key questions raised by Mark.
I started the book with columnist Joe Klein's assertion that the polling-media industrial complex diminishes politics, leaders; it makes them less bold and more risk averse. I wasn't sure he wasn't right, if you can excuse the double negative.
And as you can see, I'm pretty critical of trends in polling and critical of some of my own choices, which we can discuss.
But, I come out of this believing that strong political leaders build a special bond with people, rather than flying in the face of it. Strong leadership is not defying the public, but engaging with it -- using support to get things done; mobilizing the public, educating the public on challenges and goals and working to shift opinion. I look at the example of Lincoln and Franklin Roosevelt who were both intensely solicitous of public opinion. Engaging with the public was a precondition for boldness. That contrasts with Bush and Cheney who thought they were strong because they pursued bold policies, never guided by polls and focus groups, but I think we can look now at the consequences. President Obama's special bond with people is part of his leadership but he will struggle like these leaders to keep people with him and enhance his chances of success. That makes for stronger and more democratic leadership and produce greater civic engagement. I'm not making a partisan point -- only the case for strong leadership that is solicitous of public opinion.
Mark raised two issues.
First was tactics. Yes, all campaigns are tactical but divorced from a political project, it becomes just a game and our techniques risk diminishing politics, as Klein suggested.
Example. Reassurance. Bill Clinton reassured voters by his commitment to "end welfare as we know it," support for death penalty, and commitment to cut middle class taxes. That is tactical. But with voters more comfortable about Clinton's values and how he would use government, they now were much more supportive of his bigger agenda for investing to create sustainable growth, investing in people and education, and allowing all to have health insurance. The reassurance built support for the main project, thus a strategy.
Tony Blair reassured by promising not to raise taxes and to not increase the budget over the next two years -- and that allowed voters to support him so that would invest in public services, particularly health care and education.
But when Dick Morris advised President Clinton, he relished "stealing" the Republicans' issues -- like welfare reform -- to make the Republicans irrelevant, not to advance Clinton's larger vision. Here it becomes a game and diminishes politics.
And then there is Jerusalem. My conclusion from that is that pollsters need to be unbelievably careful about assuming current attitudes are static or can't be changed. Political pollsters should not be focused on depicting current thinking, but instead, on searching for the underlying dynamism. Even strongly, deeply emotional positions can give way -- if leaders with authority in certain areas are committed their educative roles.
Remember, we also said in Bolivia that proceeding with the export of natural gas would lead to violent opposition and that voter opinions could not be moved on this deeply emotional issue. The president there was determined to make this bold move and ultimately was forced from office, as violence grew in the country.
So, Jerusalem is a lesson but so is La Paz.
[Follow the complete series here].
Today's guest pollster contribution comes from Humphrey Taylor, who has served as chairman of The Harris Poll, a service of Harris Interactive, since 1994.
In the late 1970s, Louis Harris & Associates (one of the two firms that merged to form Harris Interactive in 1996) was commissioned to conduct a survey of American attitudes to allowing oil companies to drill for oil in wilderness areas that were off-limits. Unfortunately neither Lou Harris nor any of his senior colleagues knew anything about this survey until we read the results in the media -which showed strong public support for opening up wilderness areas for drilling, quite contrary to the findings of other polls by Harris and other firms.
When we looked at the questionnaire it was immediately obvious that this was a particularly egregious example of a "hired gun poll" designed to get the answers the client wanted, a survey designed to mislead rather than to inform policy makers about public opinion.
Lou Harris, to his great credit, publicly disowned the survey and said that the findings did not reflect public opinion. Soon afterwards, he received a phone call from an irate Senator Stevens of Alaska, who was apparently close to our clients, trying unsuccessfully to persuade Lou to back down. And the clients never paid the bill.
Were I a lobbyist with a strong point of view, I (or any competent pollster) could without much difficulty write a series of questions which might be technically acceptable, but which - taken together or separately - would mislead the media and many legislators into believing that public support for my client's position was much stronger than it really is. And here I'm not talking about bad samples or manipulating data to get the answers I want, only about the design of the questionnaire itself.
Over the years I have read many published surveys that fitted the description "hired gun polls." They are often easy for pollsters to spot because the results are surprising and are strikingly different from the results from other polls. Furthermore, they provide powerful arguments that can be used by those who paid for the survey to lobby government, to influence elected officials and to generate favorable publicity. However, it is much harder for most people, including policy makers and influentials, who are not themselves pollsters to recognize, and discount, those "hired gun polls." And this can be made more difficult when the funding source - which may be a company, a trade association, a public relations firm, an NGO or an advocacy group - is hidden behind a supposedly independent third party.
This would not matter if polls had no influence, but I strongly believe that polls sometimes influence the political agenda, policy makers, regulators and the way the media cover issues. At the risk of sounding pompous, I believe all pollsters whose polls are released to the media and the public have a moral obligation that they inform and do not mislead. Unfortunately, hired gun polls are designed to mislead.
In the polling community, some polling firms are seen as particularly bad practitioners of hired gun polling. Some will defend their right to ask whatever questions they and their clients want. Most major polling firms probably try to avoid hired gun work, but in an imperfect world it is not always easy to spot them early enough to prevent them.
Those who are not pollsters may be puzzled by this whole issue. They may believe that the public opinion is what it is and that as long as a representative sample is surveyed the replies will reflect public opinion. Pollsters know better. Different questions on the same issue can produce very different and apparently contradictory results, and questions asked earlier in a survey can have a big impact on questions that are asked later. It is not difficult to write questionnaires that greatly increase the number of people who give a particular response. Fortunately the reputation of our organizations can be hurt if we are seen as to be doing this, and our business may suffer. But, sadly, some firms that have a track record of doing hired gun polls are still in business.
Some pollsters' clients may be puzzled. Once in Latin America, I asked a presidential candidate why he had published polls showing him in the lead, when no other polls did so (he was soundly defeated). His immediate and refreshingly honest reply, "If I pay for a poll, I should be able to get the answers I want and am paying for." He is not unique.
To address this problem Harris Interactive has a set of guidelines and procedures that I summarize below. I would be delighted to hear that other survey firms have similar rules. And I would be thrilled if the leading polling firms could agree on a code of ethics based on similar principles.
We pollsters need to put our house in order so that we do not have to defend polling that is morally and ethically indefensible.
Hired gun polls damage the credibility and reputation of polling and pollsters and, I am glad to say, can - when spotted and criticized - be damaging to the firms that do them and to their clients. A much more serious effect is that they can mislead and misinform policy makers, opinion leaders and the media.
The leading polling organizations have a code but it is a code of disclosure, not a code of ethics. It is necessary but not sufficient. Many years ago they came together to found the National Council on Public Polls (NCPP) and agreed to a code of disclosure that describes what they must publish whenever they release a new poll (they must describe the universe, the methodology, the fieldwork dates and the relevant questions, and who commissioned the research). But complying with the NCPP code of disclosure does not do much to help the readers to recognize a hired gun poll, that is intended to mislead rather than to inform.
Maybe the time has come for pollsters to agree on a code of ethics that would inhibit them from conducting surveys that they know are designed to get the answers the clients want. This is not an argument against conducting and publishing any polls for interest groups and advocacy organizations. They fund many valuable and useful surveys. Rather it is a plea that pollsters do everything they can to ensure that such polls are sufficiently comprehensive and are fair and balanced (although those words seem to have taken on a new meaning) that they do not mislead.
Note: This document provides a summary if the ground rules that bind us and our clients who commission surveys for public release.
Today's Guest Pollster contribution comes from Peter Holm, a Ph.D candidate in Political Science at the
University of Wisconsin Madison. His research focuses on the military
and political attitudes.
Last month, the Military Times newspapers released their quadrennial election survey showing that 68% of currently serving military respondents favored John McCain for president, as compared to only 23% for Barack Obama. Examining the results, Duke professor of political science Peter Feaver noted that "A lot of people thought that eight years of frustration with the Bush administration was going to undermine [the conservative Republican leanings of the military as an institution]. This evidence suggests that it hasn't undermined it as much as they thought, at least not yet."
In one respect, Feaver is clearly right: the military remains significantly more conservative and more Republican than the public generally. But let's take a closer look at the evidence to see whether there has actually been some moderation in military Republicanism over the course of the Bush administration.
Both Times reporter Brendan McGarry, in his piece reporting the poll results, and West Point professor Jason Dempsey, here on Pollster, explained that the Military Times survey cannot be regarded as representative of the military population as a whole. The papers' readers are whiter, older, more likely to be male, more senior in rank, and more highly educated, for example, than are the armed forces at large. Further, the survey (like all the Times' annual surveys) used non-random sampling of this already unrepresentative group, simply allowing any subscriber or former subscriber to respond to an emailed questionnaire. Among active-duty personnel, for example, junior enlisted soldiers (E-1 through E-4) comprise about 22% of the force, but only 6% of the Times 2008 election survey respondents came from the junior enlisted ranks. So how can we use these data to make inferences about political attitudes among the military generally?
One good way to do this is to look at trends over time, as Dempsey demonstrated. A second approach, which I use here, is to weight the military survey data to bring the sample into line with the demographic characteristics of the force as a whole. This is as simple as the weighting we see in most national opinion surveys every day. When the sample doesn't conform to the expected distribution of sexes, races, and ages among registered (or likely) voters, for example, most organizations use post-stratification or raking procedures to "weight up" the responses from underrepresented groups.
I constructed weights for the Times annual surveys of active-duty servicemembers going back to 2003 using race, sex, rank, age, education, and branch as raking variables. I used Department of Defense personnel data from 2005 to construct these weights, as full data on all the demographic variables I included are not available for each year individually. In any case, the demographic profile of the armed forces changes quite slowly; the biggest change over the past eight years has probably been the aging of the active force as recruitment and retention pressures have pushed the services to raise the age limits for entering and leaving the force. Unfortunately, the 2008 survey focused almost exclusively on McCain-Obama comparisons and did not ask respondents to identify with a party or place themselves on an ideological spectrum, so the data from that survey cannot be included here. (The 2008 annual survey, distinct from the election survey, will ask these questions, as annual surveys in previous years have.)
Presidential Approval and Party Identification
Let's look first at President Bush's approval ratings among active-duty servicemembers. Figure 1 shows that, in fact, Bush's support inside the military has declined significantly, while his disapproval rate has increased. In 2003, 62% approved of his job performance and 17% disapproved. In 2007, those numbers were 44% and 36%, respectively. The largest shift came during the year 2006, when civil violence in Iraq reached its peak and disapproval shot up from 22% to 39%.
Figure 1. Source: Military Times annual surveys, 2003-2007;
weighting done by the author using Department of Defense personnel data.
Has this dissatisfaction with the president translated into declining Republican identification among military personnel? Yes, but less strongly. As Figure 2 shows, the gap between Republican and Democratic identifiers closed significantly during the first two years of the second term - in fact, the Democratic deficit was more than cut in half from 37 to 16 points between 2004 and 2006. In 2007, though, this trend moderated. Republican identification rebounded and Democratic identification receded as violence in Iraq abated, the president appeared to have developed a more coherent and successful strategy for managing the conflict, and Republicans attacked congressional Democrats for wanting to "pull the rug out" from under the troops in their attempts to force a timetable for withdrawal into war funding bills. At the end of 2007, the weighted results found Republicans making up 44% of the military population, as compared to 17% calling themselves Democrats.
Figure 2. Source: Military Times annual surveys, 2003-2007;
weighting done by the author using Department of Defense personnel data.
The Military Vote in 2008
How did these trends extend to military voting behavior in 2008? Let's consider two potential data sources. First, we can look at the Military Times 2008 election survey, weighted to correct for the demographic distortions in the subscriber base. Among active-duty respondents, the weighted results show a slightly closer contest than was originally reported: 60% favored McCain, 29% favored Obama, and 11% were undecided or favored someone else. Obama still trailed by a 2:1 ratio, but these same respondents reported that in 2004, they voted nearly 4:1 in favor of George Bush over John Kerry. Although the military vote almost certainly still favors Republicans for president, the gap has clearly narrowed.
It is important to note that the demographic weighting done here does not render the Military Times poll data fully representative of the active-duty population as a whole. It does make the sample demographically representative, at least along the dimensions I have included, but there is no way to tell with the data available what other factors may exist that are a) correlated with political attitudes; b) correlated with a person's propensity to subscribe to the Times newspapers or to respond to the survey; and c) that are not accounted for by the demographic characteristics included. It is the case, for example, that military donations reported to the FEC ran just about even between Obama and McCain, a result that suggests these survey findings, even after being weighted, may still underestimate Obama's support within the armed services. Of course, it could also indicate that Obama's supporters are simply more willing to give money but still constitute a distinct minority. We will have to await the results of better-designed surveys of the military population to find out.
In the meantime, one place to gauge how Obama actually did in relation to previous Democrats among military voters is in the election returns from military bastions around the country. I looked at returns in 29 of the 30 counties with the highest proportion of residents serving in the military as measured in the 2000 census. (Alaska does not report its returns by county, so one county is excluded from the analysis.) Figure 3 shows how Obama did in relation to Kerry's 2004 performance in these 29 counties, where the military percentage of the adult population ranges from 9.8% (in Hardin County, Kentucky) to 63.3% (in Chattahoochee County, Georgia).
Figure 3. Source: state election agencies and cnn.com.
The 45-degree line represents the divide between counties where Obama outperformed Kerry (in the upper left) and where he underperformed against Kerry (in the lower right). In all but one of the top 29 military counties, Obama's share of the two-party vote was higher than Kerry's, and by an average of over 6 points. Nationwide, Obama outperformed Kerry by 4.4 percentage points (Kerry received 48.8% of the two-party vote; Obama got 53.2%). This means that in the military bastions of the country, the counties most dominated by military personnel, Obama not only gained over Kerry's 2004 totals, but he improved on them in these areas even more than he did in the country as a whole. Although Obama still fell well shy of a majority in these counties (achieving 42.6% of the two-party vote, on average), the strong performance of a relatively young and inexperienced Democrat running against a decorated Republican war hero is highly suggestive that military allegiance to the Grand Old Party is on the wane.
Today's Guest Pollster contribution comes from Mark DiCamillo, director of The Field Poll in California.
Having put to rest the so-called Bradley effect in this year's presidential election, we are now seeing numerous references to a so-called "Bradley effect" regarding the California vote on Proposition 8, the same sex marriage ban. The Bradley effect in the California gubernatorial election, even back in 1982 was minimal (at most 2 pts out of the 8 point error in the pre-election polls). It was a convenient theory for people to use when describing the fallibility of the pre-election polls conducted in California in that year, but a closer examination would find most of the polling errors were not due to factors relating to racial bias.
While the notion that social desirability effects could have played a role on a controversial social issue like same-sex marriage, tit s theory without any real evidence, whereas an alternate explanation of the deviation between the pre-election polls and the election outcome is far more compelling and is supported by the data.
First, a quick review of the pre-election polling done by the two leading public opinion polls in California, The Field Poll and the Public Policy Institute of California. They show the following trend:
- July Field: No = +9
- August PPIC: No = +14
- Early Sept Field: No = +14 or +17 (depending on wording)
- Mid-Sept PPIC: No = +14
- Mid October PPIC: NO = +8
- Late October Field: No= +5
- Election outcome: Yes = +4
These data show the No side ahead by double- digit margins throughout most of the pre-television campaign stages. However, as the TV advertising hit in mid to late September, the Yes campaign ads proved to be more effective, and the polls showing the No side advantage slipping.
The movement continued into and through the final weekend of the election when the churches and various religious groups made a concerted effort to win over the support of their congregations. The evidence shows that they were successful.
When comparing the findings from The Field Poll's final pre-election survey of likely voters (n-966) to the Edison Media Research exit poll in California, the biggest differences relate to the turnout and preferences of frequent church-goers and Catholics. The Field Poll, completed one week before the election, had Catholics voting at about their registered voter population size (24% of the electorate) with voting preferences similar to those of the overall electorate, with 44% on the Yes side. However the network exit poll shows that they accounted for 30% of the CA electorate and had 64% of them voting Yes. Regular churchgoers showed a similar movement toward the Yes side. The pre-election Field Poll showed 72% of these voters voting Yes, while the exit poll showed that 84% of them voted Yes.
The same kind of phenomenon occurred when the first same-sex marriage ban was voted in California in the March 2000 election (Prop. 22), although because of the size of its victory( 61% Yes vs. 39% No) it didn't matter much back then. In that year The Field Poll's final pre-election poll, also completed about one week prior to the election, had 50% of Catholics on the Yes side, and accounting for 24% of the vote. Yet, the network exit poll conducted that year by Voter News Service showed them to account for 26% of the electorate with 62% voting Yes.
My take is that polling on issues like same-sex marriage that have a direct bearing on religious doctrine can be affected in a big way in the final weekend by last minute appeals by the clergy and religious organizations.
This Guest Pollster contribution comes from Samuel Popkin, Professor of Political Science at the University of California, San Diego, and Visiting Scholar at the Russell Sage Foundation, and Douglas Rivers, Professor of Political Science at Stanford University, a Senior Fellow at the Hoover Institution, and CEO of YouGov/Polimetrix.
John McCain was dealt a bad hand of cards and then played it poorly. His party has been in power eight years, and even in good times, third terms are never easy. On one hand, there is governing fatigue from a combination of satisfied, complacent partisans, and, on the other, sniping from embittered disappointed former supporters. If we add to the typical eighth year blues an unpopular war, soaring gas prices and the worst financial crisis in a generation, the odds would be against any candidate from the incumbent party.
It is nonetheless surprising just how Senator McCain managed to lose this election. Running against an unknown, inexperienced candidate with a foreign name and dark skin, the McCain strategy was obvious from day one: contrast the security and confidence Americans could have in a known, heroic and inspirational leader with the gamble of an inexperienced, risky choice. With a nation at war, as FDR said, you shouldn't change horses in the middle of the stream. Or as Gerald Ford said every day when making up nearly all of a thirty point deficit against Jimmy Carter in 1976, "Trust means saying what you mean and meaning what you say."
The odds were against the McCain campaign , but this is not how it should have played out. The campaign's primary strategy was to overcome the Bush-Cheney stigma by emphasizing McCain's leadership, heroic sacrifice for his country and willingness to take on both right and left to do what is right for America.
The big surprise of 2008 is that John McCain made voters nervous about himself and not about Barack Obama. Reporters and scholars will be studying the brilliance of Barack Obama and the genius of his campaign staff for many years. We focus on the declines in public regard for Senator McCain as an essential part of the Obama victory, not to denigrate Obama or his campaign. A better Republican campaign would not - in all probability - have succeeded in winning this year. A better campaign with a credible economic plan might well have saved many Republican Senators and Congressmen from defeat.
In the end, we show that voters became more comfortable with Obama and less comfortable with McCain. Suburban, middle class voters and older voters who should have belonged to McCain are voting in substantial numbers for Obama. Obama persuaded voters that his policies would be favorable to the middle class, and that he understands them. Senator McCain did far worse at selling middle class policies or generating strong excitement about his leadership. Maverick did not become synonymous with bold or strong. By the end, more people considered Obama inspiring, and bold, as well as intelligent.
This election was a rejection of McCain as well as his party. When he could talk about security and danger he held his own despite sagging support for his party and no support for the president. When he could argue that offshore drilling would hold down gas prices, he also stayed close. Once he chose Governor Palin and started to talk about the economy his ratings on traits related both to empathy and competence declined. He failed to demonstrate that he would be different from President Bush or that he understood ordinary people or that his policies would be good for the middle class.
For the past year, we have been conducting weekly surveys for The Economist using the YouGov/Polimetrix Internet panel. For this article, we have collected a time series of items that show how and where the McCain campaign failed to achieve its strategic objectives. The analyses presented here are preliminary, but, from the perspective of election day, we can start to understand how the campaign unfolded.
Download the complete report, with detailed charts and a description of the methodology.
Nick Panagakis is president of Market Shares Corporation, a marketing and public opinion research firm headquartered in Arlington Heights, Ill.
A few weeks ago you may have seen a debate regarding voter indecision on these pages. David Moore argued that in a poll, decided voters who said there was a chance they could still change their minds before election day should be counted as undecided voters
This post is an update based on new poll data. I argued then that this wasn't indecision. I said response to the hypothetical, point-in-time "if the election were held today" question will yield some voters willing to decide on a candidate who won't rule out the possibility that some incident or disclosure, however remote, could lead them to vote otherwise which is not indecision. In other words, this is not candidate induced indecision but calendar induced because the election is still weeks away.
ABC is the only poll that follows up its ""could change mind" question with another that asks chances of doing so. See current ABC Poll numbers here. Click PDF report:
The columns show: 1. likely or registered voters would definitely vote for a candidate, 2. any chance "could change mind", which breaks down to, 3. "good chance" could change, and 4. chances "pretty unlikely" which is thisclose to no chance in h*ll.
The table shows that over seven weeks, chance of mind-changing drops from 20% to 9%. Moreover, good chance of doing so drops even faster, from 8% to only 3% overall (2%-3% of Obama voters, 3%-4% of McCain voters, last three reports.). In 2004, ABC polls showed the same "good chance" trend up to a few weeks before the election.
This effect appears predictable, regardless of election, regardless of candidate or campaigns. Imagine if polls up until last week were showing undecideds 10 to 20 points higher - or still showing 9 points greater this week. Again, this not indecision between the candidates as I understand it but the understandable effect of asking how voters would vote today, the only way to characterize election contests over time. ABC tracking poll results in the days to come will continue to confirm this.
Nick Panagakis is president of Market Shares Corporation, a marketing and public opinion research firm headquartered in Arlington Heights, Ill.
This week our poll for the Chicago Tribune reported that Illinois Governor Rod Blagojevich had a job approval rating of 13% versus 71% disapproving of his performance - astonishingly low job approval. Some 16% had no opinion, which includes ambivalence. Our poll of 500 likely voters was fielded October 16-18.
But another Illinois poll found even lower numbers - an October 13th Rasmussen poll of 500 voters as reported by the Huffington Post.
Their question used a four-point scale asking respondents to "rate the way that Rod Blagojevich is performing his role as Governor". Results were: Excellent 0%; good 4%; fair 29%; poor 65%. Huffington reported individual ratings. But approval is often reported by combining the top two and bottom two scores. In this case, approval would be 4% and disapproval 94%, quite a difference from our 13%/71%. (Don't know response could be lower because of the 4-option scale or because this was an automated poll.)
Some background. In my earlier years, I conducted phone and exit polls for WBBM-TV, the owned and operated CBS station in Chicago. About a year after Jane Byrne was elected Mayor, we obtained her job approval using the four-point scale, combining the excellent to poor ratings to approve/disapprove. Byrne's late husband Jay McMullen, a former Sun-Times reporter, wrote to the station's general manager objecting to the use of that scale. So we did another poll asking both the excellent to poor rating and approve/disapprove questions.
McMullen was right. Not only was her approval score higher than excellent-good combined, but some of the "fair" raters also said "approve" when asked. Moreover, when asked reasons for rating Byrne the way they did, we got answers like "doing a pretty fair job" from those rating her both "fair" and "approve".
Turning to more current examples, two national pollsters use the four-point scale, Zogby and Harris. Harris asks "only fair" not fair. All others use the dichotomous approve/disapprove. George Bush approval ratings dating back to January 2001 appear on the Pollkatz site.
In the chart, Zogby polls are light gray diamonds and Harris polls dark gray diamonds. Note how often these symbols appear at the bottom of clusters of other scores, Zogby more often than Harris. The chart confirms my findings nearly three decades ago.
The four-point job rating has its supporters in the polling community. But it can be not compared with dichotomous approve/disapprove questions.
Guest Pollster Clark A. Miller is an Associate Professor at Arizona State University. His post expands on a comment left on Pollster.com on Friday.
As Mark Blumenthal and Nate Silver have both noted in detail of late, the design of likely voter models can significantly impact how pollsters interpret and transform the raw data of voter samples into the topline results we see at pollster.com, fivethirtyeight.com, and other sites covering election polling. In turn, Mark and Nate observe, likely voter model design depends significantly on judgments that pollsters make about how to model the likelihood that any voter sampled will actually turn out and vote in the election. As we have all seen in the last few days, differences in how such judgments get made by different pollsters, combined with differences in the samples of voters collected by each poll, can mean the difference between a 1-point and a 14-point spread between the respective candidates for President.
A key challenge for consumers of polls - whether citizens, journalists, or politicians - is sorting out to what extent the likely voter model or the underlying raw data sample is responsible for variations in poll outcome. In fact, this sorting out of how judgments made by modelers impact model design and outputs is a general challenge in the use of science to inform policy choices, which I have studied for much of the past two decades. Judgments like this are inevitable in any scientific work, which is why policy officials turn to experts to make judgments on the basis of the best available knowledge, evidence, and theories.
One case that I have looked at in detail is the use of computer models of the Earth's climate to make predictions about whether the planet is experiencing global warming. As I'm sure most of you know, models of climate change have been viewed skeptically by many people. I believe the trials and tribulations of climate modelers - and also their approaches to addressing skepticism about their judgments - offer three useful insights for pollsters working with likely voter models.
- Transparency - climate models are far more complex than most polls, but climate modelers have made significant efforts to make their models transparent, in a way that many pollsters haven't. (In much the same way, computer scientists have called for the code used in voting machines to be open source.) By making their models transparent, i.e., by telling everyone the judgments they use to design their model, pollsters would enhance the capacity of other pollsters and knowledgeable consumers of polls to analyze how the models used shape the final reported polling outcome. They would also do well to publish the internal cross-tabs for their data.
- Sensitivity - climate modelers have also put a lot of effort into publishing the results of sensitivity analyses that test their models to see how they are impacted by embedded judgments (or assumptions). This is precisely what Gallup has done in the past week or so, in a limited fashion, with its "traditional" and "extended" LV models and its RV reporting. By conducting and publishing sensitivity analyses, Gallup has helped enhance all of our capacity to properly understand how their model responds to different assumptions regarding who can be expected to vote.
- Comparison - climate modelers have also taken a third step of deliberate comparisons of their models using identical input data. The purpose of such comparison is to identify where scientific judgments were responsible for variations among models, and where those variations resulted from divergent input data. Since the purpose of polling is to figure out what the data are saying, it is essential to know how different models are interpreting that data, which can only be done if we know how different models respond to the same raw samples.
The reason climate modelers have carried out this activity is to help make sure that the use of climate model outputs in policy choices was as informed as possible. This can't prevent politicians, the media, or anyone else from inappropriately interpreting the outputs of their models, but it can enable a more informed debate about what models are actually saying and, therefore, how to make sense of the underlying data. As the importance of polling grows, to elections and therefore to how we implement democracy, pollsters should want their polls to be as informative as possible to journalists, politicians, and the public. Adopting model transparency, sensitivity analyses, and systematic model comparisons could go a long way toward creating such informed conversations.
Guest Pollster Nick Panagakis is president of Market Shares Corporation, a marketing and public opinion research firm headquartered in Arlington Heights, Ill.
The National Council on Public Polls analyzed national presidential poll accuracy In 2004. The found that eleven of fifteen pollster margins were off by 0% to only 2% from the winning margin., well within the margin of error.
This year polls are now showing Obama from +5 points to +13 ahead, more variability than four years ago this week. This year a demographic variable appears to be having an acute effect on voting estimates. That variable is race and may explain the difference in poll margins.
I did a "what if" spreadsheet analysis with hypothetically variable percentages of black composition of total voters and corresponding variable percentages of the white/other races of turnout. I set a constant result for the vote for president by African-Americans.
Typically, polls lack sufficiently large samples of black voters to reliably estimate their voting intentions. I derived my estimate from a tracking poll's pooled results: 95% of African-American would vote for Barack Obama, 3% for John McCain and 2% for other candidates. A one or two point disparity from 95% for Obama doesn't make much difference in this analysis. If anyone has better results, please respond to this column.
The key variables are the racial distribution of voters and how white/race voters will vote.
Exit polls in the last few elections have shown African-American varying from 10% to 12% (in 2004) of total voters. My analysis ranged from 10% to a hypothetical 15% to check incremental margin gain.
General voter interest in this election may be so high, black composition could remain at 12%. But if it increases to 13%, that adds a one-point margin gain for Obama. For each one percentage point increase in black composition, the overall margin for Obama increases by one percentage point.
The other variable is how whites and others would vote. The following hypothetical scenarios cover most of the overall margins current polls are showing. They range from +4.8 to +9.4 Obama winning margins.
Assuming McCain is ahead by 7 points among whites/others and with 12% African-Americans of all voters, overall results are a Obama win of +4.8%. At 13% black voter of total turnout this would result in a +6.0 point Obama win.
Assuming McCain ahead by only 5 points among whites/others and with 12% African-American sample composition. This would yield an Obama win of +6.6 points. At 13% black voter composition results are a +7.6 point Obama win.
Assuming McCain down to +3 points among whites/others and with 12% black voter sample composition. This would yield an Obama win of +8.4 points. At 13% black voter composition results in a +9.4 point win for Obama.
In closing this analysis has nothing to do with the Bradley Effect theory. And nothing to do with reverse-Bradley. (Since when can a theory have it both ways?) Pre-election poll versus exit poll or post election analysis examining such variables could have confirmed or denied the effect.
Alex Lundry is the research director at the Republican polling firm, TargetPoint Consulting.
With only two weeks remaining, pollsters and journalists alike have rightly been reexamining the legitimacy of their polling numbers. In particular, there have been caution flags regarding three phenomena that may be unaccounted for in many of the public polling numbers: 1) the difficult to poll cell-phone only voter, 2) a possible surge in youth and minority turnout missed by likely voter models, and 3) the Bradley-Wilder Effect causing artificially deflated numbers for McCain.
Unfortunately, poll-watchers today have another reason to be wary: a likely over-report of voter registration, especially among African American voters, possibly causing surveys of both registered voters and likely voters to overstate support for Barack Obama.
A 2007 article in Public Opinion Quarterly (link, gated) by Andrew Fullerton, Jeffrey Dixon, and Casey Borch, looked specifically at the problem of registration over-reporting - in which unregistered respondents inaccurately state they are registered voters. Their analysis relied upon National Election Study (NES) validation studies between 1976 and 1980 (the most recent year for which both registration and voter validation data are available - an analytical shortcoming the authors freely admit to). Seeking the drivers of this behavior, they found that blacks are more likely to overreport their registration (along with those that are better educated, live in the "Deep South", and have strong partisan beliefs). The implications of this are particularly relevant to this year's election polls, as the authors detail in this critical point:
If the level of registration overreporting is comparable today, as we believe it is, this subpopulation inflates the number of potential voters in pre-election surveys because they are typically based on samples of self-reported registrants. More importantly, if our finding that blacks are more likely to overreport registration than voting holds true today, as we think it may, this could skew the results of pre-election surveys, likely in favor of a Democratic candidate given blacks' historical affiliation with this party.
If ever there was an election in which black respondents felt a social desirability bias to over-report their registration, this would be it. Support for Barack Obama among African Americans is nearly monolithic, and we are treated to frequent numeric and anecdotal accounts of increased enthusiasm and engagement among the black community. A reasonable person would conclude that an unregistered African American, called to participate in a survey, would feel some sort of pressure (either known or unknown) to say that he or she is indeed registered, and continue with the survey.
How significant could the bias be? Word of increased registration and enthusiasm among African Americans makes it difficult to assign a precise number, but the data itself can at least provide us with a guidepost: between 1976 and 1980, 11% of NES respondents overreported their registration. Among this group, two-thirds (7% of all respondents) later claimed in a post-election study to have actually voted.
These findings should lead us to be especially wary of recent polling in Georgia and North Carolina showing Barack Obama within striking distance of John McCain, as Fullerton et al.'s analysis indicates that residence in the "Deep South" - states with the heaviest concentration of blacks - also makes a meaningful difference in registration over-reporting (though, to be fair, North Carolina is considered a "Peripheral South" state in their treatment). Still, one way to mitigate this problem - voter registration based sampling - is used by Insider Advantage, a frequent pollster in Georgia, as well as PPP, which has been active in North Carolina.
Still, as much as these findings intuitively "make sense" there are a number of reasons to be skeptical: first, the authors themselves point to a number of issues with their analysis (old data, problems with the validation of African Americans' registration records, etc.), and second, the very reasonable assumption that even if this effect did exist, it could be cancelled out if African Americans turnout at higher rates than pollsters predict they will.
Despite these limitations, Fullerton et al.'s analysis should give pollsters and poll consumers sufficient pause as they read the inevitable flood of horserace results over these remaining weeks.
Alan I. Abramowitz is the Alben W. Barkley Professor of Political Science at Emory University in Atlanta, Georgia. He is also a frequent contributer to Larry Sabato's Crystal Ball.
It may take some time for historians to decide whether the contest between Barack Obama and John McCain should be considered one of the most important presidential elections in American history. But we already know that it will be the most thoroughly polled presidential election in American history. Lately it seems that not a day goes by without dozens of new national and state polls being released. There are now no fewer than eight national tracking polls underway. These are polls that interview a random sample of voters every day and then combine the results over three or more days, adding one new day and dropping one earlier day, in order to measure changes in candidate preferences within the electorate.
In the past few days there has been a lot of speculation in the media about whether the presidential race has been tightening. Some pundits have argued that such tightening is inevitable in the final days of a presidential race. So what do the national tracking polls tell us about the state of the race between Obama and McCain with less than two weeks left until Election Day?
An analysis of the seven national tracking polls that have been up and running since at least October 12th (Rasmussen, Gallup, DailyKos/R2K, Diageo/Hotline, Battleground, IBD/TIPP, and Zogby) leads to several conclusions. Perhaps the most important one is that despite differences in sampling, interviewing, and weighting procedures, Barack Obama led John McCain on every day in every poll. Beyond that basic finding, however, there are some clear differences in the results of these seven tracking polls.
The results in Table 1 show that while all seven tracking polls have had Obama ahead over the past ten days, the size of that lead has varied considerably. During this time period Obama's average lead has ranged from a low of 4.4 points in the IBD/TIPP Poll to a high of 9.8 points in the DailyKos/R2K Poll. In addition, some of the tracking polls have shown much more volatility than others: the standard deviation of Obama's lead has ranged from a low of 0.8 points in the Rasmussen Poll to a high of 3.9 points in the Battleground Poll which has had both the smallest (1 point) and the largest (13 points) Obama lead in the past ten days. In contrast, Obama's lead in the Rasmussen Poll has varied only from a low of 4 points to a high of 6 points. In general, polls like Rasmussen that weight their results by party identification tend to produce more stable results than polls that do not weight by party identification.
There is no evidence in these data of any tightening of the presidential race over this time period.
Figure 1 shows the trend in Obama's average lead in six tracking polls that provided results every day between October 12 and October 20 (the Battleground Poll does not report results on the weekend). While Obama's lead increased in some polls and decreased in others during this period, the results in Figure 1 show that the overall average changed very little. Obama led by an average of 6.5 points on October 12thand he led by an average of 7.0 points on October 20st, the final date included in this analysis.
If there is no overall trend, then what explains the day to day movement in the tracking polls? One possibility is that most if not all of the day to day movement was due to sampling variation-that it was nothing more than random noise. In order to test this hypothesis, I calculated the correlations among the day-to-day results of the seven tracking polls over these ten days. The correlations between individual pairs of polls varied considerably. Some were strongly positive, some were very weak, and some were strongly negative. Nothing much should be made of this, however, because of the very limited number of days on which these correlations were based. What is significant, however, is that the average correlation among the seven tracking polls over this ten day period was -.06. This means that there was basically no relationship in the day-to-day movement of these polls during this time period. Whether Obama's support was going up or down in one poll was unrelated to whether his support was going up or down in the other six polls.
The lesson that should be drawn from these findings is not that there is any fundamental flaw in the tracking polls. Random variation is unavoidable in public opinion polling. What these findings do indicate, however, is that poll-watchers should not pay too much attention to the day to day movements in these polls unless they see all or most of them moving consistently in one direction over a period of time. Similarly, polling organizations should avoid overemphasizing the significance of the day to day movements in their own polls and pay more attention to whether their results are consistent with those of other polls.
Today's Guest Pollster contribution comes from Jason Dempsey, who is an infantry officer assigned to the Army's 10th Mountain Division and the author of the forthcoming book, Our Army: Soldiers, Politics, and American Civil-Military Relations. He also has an article on the political attitudes of military personnel in the most recent issue of the The New Republic. The views presented here are his own and do not reflect the views of the U.S. Army or the Department of Defense.
While the veteran vote is not attracting as much attention as it did in 2004 it is still a salient election issue, and we could use more discussion of available data. For the most part, the most current available data is provided by the Military Times family of newspapers. [Note--does anyone know if NAES is attempting to do a focused military survey again this year?] However, the Military Times surveys have to be used for what they are: Surveys of subscribers to the Military Times papers (Army Times, Air Force Times, etc.) As such they are not representative of the entire military population. And we should note that they don't claim to be, although that is often lost in interpretations and use of the data.
Briefly, I'd like to address the methodology of the surveys and the ways in which these surveys can be useful, some trends revealed in these surveys since 2004, and some thoughts on the results of their 2008 election survey.
Methodology
First off, I think the Military Times do a good job of explaining that these surveys are not representative (see here), even if the headlines and commentary resulting from these surveys often imply otherwise. As the crowd at Pollster.com is the type that likes the fine print I think this is a great venue for discussing the potential as well as the limitations of these surveys.
The survey in the news last week was this year's election-specific survey. In September the Military Times sent e-mails to about 69,000 subscribers. (They sent original messages to about 80,000, but many came back as undeliverable. Some of this should be expected, and appropriately discounted, given the high mobility of the active-duty military community, but it is not clear how many with invalid addresses were active as opposed to being retirees or in the Guard/Reserve). From this they collected responses from 4,515 retirees, 1,515 members of the National Guard and Reserves, and 2,982 active-duty members of the military (although of these 316 were left out of analysis because they were not registered or did not intend to vote). Of the active-duty respondents, 1,543 were in the Army. I limit analysis in my research, and here, to the active Army population within the surveys as I can appropriately compare this subgroup with the overall Army population. However, I think the discussion of the representativeness of the Army subsample probably applies equally to the other active-duty groups.
As with previous Military Times surveys the respondents in 2008 were disproportionately white, male and officers. The actual Army population is about 85% male, 14% regular commissioned officers (not including Warrant Officers), and 60% white. The active-duty members of the Army who responded to the Military Times poll were 90% male, 45% regular commissioned officers, and 71% white. Furthermore, the Army's junior enlisted ranks are dramatically underrepresented in the Military Times surveys. About 47% of the Army serves in the ranks of E-1 through E-4. These ranks comprise only 6% of the active Army population included in the 2008 Military Times survey. (The samples of each of the previous Military Times surveys are nearly identical in the degree to which they represent the active military population). Bottom Line: these surveys should in no way be used to assess aggregate attitudes across the force.
However, this does not mean that the Military Times surveys aren't valuable (that is far from the case). Rather, it highlights that interpretations of the Military Times survey results have often been inappropriately extrapolated to the entire military population.
These surveys can be useful in two ways. First, they can be useful as a gauge of opinion trends. While the results of these surveys might not present an accurate estimate of overall military attitudes in a given year, over time they reflect how the opinions of a portion of the military are shifting. By extension we might assume that the rest of the military is shifting to a similar degree, even if the starting point is not the same. (See the discussion of Robert Shapiro and Ben Page on 'parallel publics' in The Rational Public).
Secondly, if we limit analysis of the survey data to senior officers then the 'subscriber bias' is likely to be minimal, in that the attitudes of senior officers in the Military Times subscriber database are likely to be similar to the attitudes of senior officers generally. Whereas a junior soldier or officer who subscribes to the Army Times is likely to have a more careerist outlook than his or her peers, in that subscribing can be interpreted as an act of dedicated interest in the profession as a whole, the difference between subscribers and non-subscribers is likely to be more muted in the senior officer ranks.
Trends: 2003 to 2007
If one assumes these surveys can be useful as a gauge of opinion trends, if not a comprehensive view of aggregate attitudes, then the Military Times surveys do tell us something about military attitudes over time. I believe we can also view them as fairly accurate portrayals of the opinions of the subset of senior Army officers. Below are the results of what I found when parsing out the opinions of active-duty Army officers in the ranks of major and above (typically 10 to 20+ years of service) from the 2003 through 2007 Military Times annual surveys.
The single datapoints reflect the results from the Citizenship & Service: 2004 Survey of Military Personnel (Completed with Bob Shapiro and support from the Institute for Social and Economic Research and Policy at Columbia University. See here). The C&S Survey included a higher number of women and minorities and the resulting data was weighted to reflect the general Army population on the dimensions of rank, race/ethnicity, and gender. Notably, the results from comparing senior officer attitudes from the C&S Survey with the attitudes of senior officers in the Military Times survey show a pretty close match.
These Military Times survey results show that support for the Republican Party among senior members of the Army, the group most likely to identify as Republican, declined significantly between 2004 and 2006 before leveling off at about 49% in 2007. Also interesting is that the data show no corresponding change in support for the Democratic Party.
Because the Military Times did not conduct these surveys before 2003 we can't assess what this means historically, but we do have data from Ole Holsti's and James Rosenau's Foreign Policy Leadership Project surveys that were conducted every four years between 1976 and 1996. Looking at this data, the military is experiencing a shift comparable to what occurred between 1976 and 1980. During that period military leaders shifted decidedly toward the Republican Party. By the end of Carter's presidency the proportion of senior military leaders who identified with the Republican Party had increased by 13%. This data show a shift of comparable magnitude--only during this administration the military has begun to shift away from the Republican Party. Over the last three years the Military Times surveys have shown a decline in Republican Party identification of 14% among active-duty Army respondents and an overall decline of 13% among senior Army officers.
Notes on 2008
Unfortunately the 2008 Military Times Election survey did not ask party affiliation. They did, however, ask respondents both who they planned to vote for during this election and who they voted for in 2004. Not exactly panel data, but this again offers an opportunity to assess shifts in attitude among survey respondents.
The primary headline to come out of the Military Times surveys was that 68% of respondents backed the Republican presidential candidate, John McCain. However, lost in the analysis was a significant shift in support for the Democratic nominee. Looking at just the subset of active-duty members of the Army in the Military Times poll, 64% of these respondents reported voting for George Bush in 2004 and 15% reported voting for John Kerry. As for the 2008 election, 66% planned to vote for McCain while 25% reported planning to vote for Barack Obama.
There are two significant points to draw from these results. The first is the 10-point uptick in support for the Democratic candidate. While not indicative of a reversal of military preferences among officers, this increase in support for the Democratic candidate signals a significant shift in military opinion and indicates that military aversion to the Democratic Party may be on the wane.
A key discussion point from initial reports on the survey was that black members of the military overwhelmingly indicated support for Barack Obama, but looking at the demographics of those who reported shifting their preference to Barack Obama in 2008 reveals that this dynamic was not driven solely by minority respondents. Of those who shifted from Bush to Obama, 95% were male and 55% were white (n=79, again, I am only looking at active-duty Army respondents). Among those who voted for neither the Republican nor Democratic candidate in 2004 but were planning to vote for Obama in 2008, 77% were male and 39% were white (n=121). This indicates that the increased support for the Democratic presidential candidate among members of the Army is due to both a shift of the Army's traditional voting block away from the Republican Party as well as an infusion of new, predominantly minority voters into the Democratic column.
The second significant point to draw from these results is that McCain has been able to hold onto military votes at a time when the Republican brand is hurting nationally. He is holding a slightly higher portion of the senior officer vote than Bush did in 2004. This is probably not indicative of a shift back to 60% Republican Party identification among senior officers, but is probably due more to his own veteran status. Among active-duty Army respondents, 73% felt that the veteran status of the candidates was important in making the decision about who to vote for (32% felt it was 'very important'. Another 41% felt it was somewhat important.) I suspect that this explains a good portion of McCain's military support and that post-election assessments of the party identification of senior officers will be closer to the 2007 figures.
Finally, however, it must also be noted that comparisons with 2004 are problematic. As outlined by Jeremy Tiegen in his analysis of the veteran vote in 2004, the Swiftboat attack ads were successful in getting otherwise Democratic voters to vote for George Bush, so 2008 may be the 'norm' although, again, McCain's strong identification as a veteran further muddles analysis. Hopefully someone will attack the data and give us an answer after the election.
For those interested enough in military attitudes and polling to have read this far there will be more analysis of the attitudes of the active-duty Army (to include the enlisted ranks) in my forthcoming book Our Army: Soldiers, Politics, and American Civil-Military Relations due from Princeton University Press in 2009.
Nick Panagakis is president of Market Shares Corporation, a marketing and public opinion research firm headquartered in Arlington Heights, Ill.
Some polls are reporting the cross-over Clinton voters; i.e., the percentage of Clinton primary/caucus voters who tell pollsters they would vote for McCain in November. In the first half of September this was reported by CBS, ABC, Pew, and Quinnipiac polls. Results ranged from 12% to 25%with an average of 19%.
While the numbers are interesting, how will they affect the November election? The findings are in need of some context if you are willing to accept some assumptions
First consider how many votes Clinton won in state primaries and caucuses. According to realclearpolitics.com she won 18 million votes after including Michigan and state caucus estimates. Assuming they all vote in the general election, multiply 18 million by the 19% average above and you get 3.4 million November voters who say they will vote for McCain.
What percentage of the general election vote will they represent? Assuming turnout will be near the record 122 million votes cast in 2004 (source: FEC), those 3.4 million Clinton cross-over voters represent 2.8% of all voters. If similar magnitudes of Clinton voters make good on their intentions in states that prove to be battlegrounds, this could make the difference in a close election.
This year the Democratic primary remained a tight contest much longer keeping many Clinton voter hopes alive so similar cross-over data from past elections may not be useful.
Another benchmark would be cross-over Democratic voters voting for any Republican candidate in past exit polls. The New York Times "Super Table" of past exit poll results provides answers. 1992 to 2004 exit polls show 10% or 11% of Democrats voting for the Republican candidate. (Earlier years are much higher due to Reagan Democrats, not likely based on pre-election polls this year.)
In three of those last four elections Democrats won the popular vote overcoming the 10%-11% Democratic cross-overs who voted for the Republican. So the question becomes: will that 2.8% of Clinton cross-over voters add to those numbers or be mostly included in those numbers?
John
Coleman is a the Chair of the Department Political Science at the University of Wisconsin-Madison
Joe Biden has supported President Bush 70% of the time. You
may not have heard this mentioned at the Democratic National Convention or in
Barack Obama's acceptance speech.
The Obama team--and Obama himself--has been working hard to
link John McCain to George W. Bush by noting that he "votes with Bush 90% of
the time." And if 90% isn't enough Bush for you, Democrats note, McCain
supported the president 95% of the time in 2007. One Obama ad
even lists this voting record as the first plank in McCain's economic program.
The figures being used by Democrats are presidential support
scores computed by CQ Weekly, a leading weekly magazine
monitoring events in Washington.
The score is based entirely on recorded roll-call votes in Congress. CQ identifies those votes where
the president has taken a clear stand and then records whether a senator or
representative voted in the president's preferred direction. The votes need not
be key on the president's agenda or be anything the president encouraged
Congress to do--they are simply cases where CQ has determined a clear
presidential position. In the Senate, the president's nominations, which are
usually noncontroversial, are a sizable portion of the votes used by CQ to
compile its support score. In 2007, nominations were 30% of the votes used by
CQ to calculate presidential support in the Senate.
As the chart below shows, John McCain has indeed voted
consistent with the preferences of President Bush about 90% of the time on
these presidential support roll-calls. This has been roughly the same level of
support as the average Republican senator.
McCain's presidential support level was 95% in 2007, but
this is somewhat misleading. Because he was running for president, McCain was
present for only 38 of the 97 roll calls CQ used to calculate the presidential
support score. There were 442 roll-call votes in total in the Senate in 2007. Looking
at only those votes for which both McCain and Obama were present that year--33
votes--McCain's support score was 94% while Obama's was 48%. CQ also noted
in a recent post that McCain, Obama, and Biden voted on less than half the
presidential support votes from January through August 2008.
Using the same figures the Obama campaign has used to tie
John McCain to President Bush, Biden was a 77% supporter of President Bush's
positions in 2002, 70% in 2004, and over a 50% supporter of Bush in 4 of the
president's 7 full years in office. Up through the August 2008 congressional
recess, Biden
had supported Bush's positions 52% of the time since January 2001. Obama
himself supported the president's positions just under 50% in 2006 and 40%
since he joined the Senate in January 2005.
It is doubtful that many Americans hearing the Obama team's
90% charge against McCain realize that Obama and Biden themselves have
supported the president anywhere from 33 to 77% of the time during his term.
In addition to linking McCain to Bush, another goal of the
Obama campaign in using the 90% support figure is to blunt McCain's claim to be
a maverick who shows independence from his party. Establishing McCain's
independent credentials was a major theme at the Republican National Convention
on Tuesday night.
Given that even Obama and Biden sometimes had relatively
high levels of support for Bush, a better measure of independence than the
presidential support score would be to look at the party support score, also
calculated by CQ Weekly. Looking at
"party votes"--those roll-call votes on which a majority of Republicans oppose a
majority of Democrats--CQ calculates whether a senator voted with his party's
majority or against it. The party support score is the percentage of times a
senator voted with his party majority on party votes. There were 266 party
votes in the Senate in 2007, or 60% of all Senate roll-call votes.
Looking at his party support scores during the Bush
presidency, the chart below shows that McCain regularly was less supportive of
his party than the average Republican senator. His voting in 2007, when McCain
was frequently out of Washington
and missing more roll-call votes than usual (he voted on 48% of the 266 party
votes), is an exception.
McCain's professed independent streak is supported by these
data. About 75 to 85% of the time, McCain voted with his party's majority. More
frequently than the average Republican, however, McCain voted with the
Democratic majority rather than the Republican majority on votes that put the
two parties on opposite sides.
Obama and Biden, on the other hand, have both been more
likely than the typical Democratic senator to vote with the Democratic party
position. In each of his three full years, Obama voted over 95% of the time
with the Democratic majority on party votes. McCain reached 90% only once, in
2007.Biden's party support level has hovered between about 90 to 95%. From
these data, McCain can more credibly make the claim that he is willing to buck
his party. He has voted against his party majority about 15 to 25% of the time
across the Bush years, compared to about 3% for Obama and 5 to 10% for Biden.
I've plotted these data in a different format in the chart
below. Here, zero on the left axis indicates the baseline party support level
of the average senator for each party. I then plot the difference between the
average Republican senator's party support and McCain's, and the average Democratic
senator's support and Obama's and Biden's. During the Bush years, McCain was
usually about 5 to 10 percentage points less likely to vote with his party than
the average Republican senator. Obama's party support level was about 10 points
higher than the average Democratic senator, while Biden was usually between
about 5 to 12 points more likely to vote with the party majority than the
average Democrat.
These numbers burnish McCain's independent credentials, at
least compared to his two senatorial rivals. But they also point to one of the
key dilemmas of the McCain candidacy. To weaken McCain's maverick image,
Democrats can tie McCain to Bush by emphasizing McCain's presidential support
percentage, while not mentioning the sometimes high Bush support level of his Democratic
opponents themselves. McCain can respond by noting that, compared to his
rivals, his party support percentage shows he is less likely to vote along
party lines and has more of an independent streak. Emphasizing that streak may
endear him to independents and some Democrats, but it is of course one of the
chief aspects of McCain's legislative life that has historically created
problems for him within his own party and among party activists. It is one of
the tasks of the Republican convention to convince Republicans of the virtue of
that independent streak as a matter of character, even if they disagree with
McCain on policy particulars.
Today's guest pollster contribution comes from Michael P. McDonald, an Associate Professor of Government and Politics in the Department of Public and International Affairs at George Mason University and a Non-Resident Senior Fellow at the Brookings Institution.
A media storyline surrounding the Democratic convention is how a sizable number of Hillary Clinton supporters are backing John McCain over Barack Obama. A recent CNN/ORC poll provides grist for the mill. Twenty-seven percent of self-identified Clinton supporters are reported backing John McCain, an increase from 16% in a similar June survey.
Yet, there are indications that something is amiss in this survey. CNN reports they interviewed 1,023 adults. The organization does not report the sub-sample size of Democrats who support Clinton, but they do provide a margin of error of this sub-sample from which we can infer the number of Clinton supporters. The reported margin of error for Democrats who support Clinton is 7.5 percentage points, which is equivalent to 171 persons assuming a simple random sample. That is 16.7% of all adults in the survey, which when applied to my 2006 voting-age population estimate of 227 million persons means that there are 38 million self-identified Clinton supporters among Democrats in the CNN/ORC poll (with a 95% confidence interval between 20.9 and 54.9 million persons).
As one might recall, Clinton received 18 million votes in the primaries. If she had received 38 million votes, she would be accepting the Democratic Party's nomination on Thursday.
The question arises, who are these 20 million or so self-identified Democrats who support Clinton who did not participate in the primaries? It is difficult to tell without analyzing the survey in depth. While there are many reasonable explanations for the discrepancy between the election and survey results, a plausible explanation consistent with the large percentage of self-identified Clinton supporters who report supporting McCain in a two-way contest against Obama is that the CNN/ORC questionnaire is worded in such a manner that elicits persons who self-report supporting McCain to report that they are a Democrat who supports Clinton for the party's nomination.
The implication is obvious: if these surveys that purport to measure Clinton supporters who will vote for McCain actually measure McCain supporters who would like to see Clinton as the Democratic nominee, the media storyline of Democratic dissention quickly unravels.
