This week my firm, along with Republican pollsters Neil Newhouse & Alex Bratty from Public Opinion Strategies, released a survey of women overall, and Walmart moms. The survey, commissioned by Walmart, was conducted online May 20-27 using the EMIl online panel. We surveyed 1250 women, and 380 Walmart moms, defined as women with children under 18, who have also shopped at the store in the last month. (The full presentation, along with more methodological info, can be found here. Some coverage of the results can be found here and here.)
Both motherhood itself and the unique economic pressures mothers face are also at work in these results. Of the women we surveyed, 35% were moms, and 86% of those moms shop at Walmart at least once a month. Below are some key findings about these Walmart moms:
They are younger than women overall, but with similar incomes and education. These women are younger, naturally, because they have young children. And they are less white (67%) than women overall (75%). But they are very similar along economic and educational lines.
They are true swing voters--they support Obama and an involved government, but lean Republican in the Congressional ballot. Like women overall, they lean Democratic (+4 Dem advantage), they voted for Obama (+7), and they are currently favorable toward Obama (+6). However, unlike women overall, they lean slightly toward voting Republican in the upcoming Congressional election (-3). These advantages are small--this group could go either way in November.
They see an active role for government. Six in ten (60%) agree with the statement "government should do more to solve problems and help meet the needs of people." While they oppose the recent health care reform (-14), it's unlikely to be a vote driver this November when compared to the economy. Two-thirds (63%) say the economy and jobs are their first or second priority, compared to just a third (35%) for health care. In fact, Walmart moms who support health care reform are actually more interested in health care as an issue (40%) than are Walmart moms who oppose health care (31%).
They are middle of the road on social issues, although they support gay rights more than the Tea Party. Nearly half of Walmart moms are moderates (46%). As such, they are in the middle of the road on social issues. They are more likely to support "the Gay Rights movement" (51%) than the "Tea Party movement" (46%). They are also nearly evenly divided on whether they support "conservative religious groups" (51%).
These moms are also feeling a real personal squeeze across the board. To be sure, given that large numbers of the moms surveyed also shop at Walmart, motherhood itself seems to covary with both more swing political views and personal economic pressures. In this recent Gallup poll (and in the slightly older studies in this Pew report), mothers report more stress, less time, and less rest than fathers. Our findings below about personal economic insecurity and household task division are consistent with this.
While these moms may be similar to women we surveyed in income and education, yet they feel economic pressures more acutely. Unlike women overall, they are both affected by the current economic downturn, and dissatisfied with their personal economic situation. They are also more likely to identify as working class (or lower), and are more likely to feel anxious about slipping out of that class. In fact, middle- or upper-class Walmart moms are even more likely to feel that anxiety, a pattern not found among women overall. In this swing group, anxiety extends across social strata.
We dug a little deeper to examine what specifically were women's top economic concerns. Daily expenses top the list for these moms, with three-fourths rating them an 8,9, or 10 on a 0-10 scale. But concerns about the future also loomed large, with majorities concerned about their retirement or future job loss. Nearly half expressed strong concern about their credit card debt, and a third say they use their credit cards to get by during this crisis. Taking all concerns together, about half (47%) of Walmart moms rated most of these items as top concerns, compared to only a third of women overall. And candidates should take note that those Walmart moms with more interest in the election are even more likely to have multiple strong concerns.
Economic challenges beget personal and family challenges. This economic insecurity leads to personal and household strain. Over four in ten Walmart moms (42%) said the economic crisis has put strain on their relationship with their spouse or partner, and younger women even more so. It's not surprising given how many cost-cutting actions these women are taking. When read a list of activities one might do to deal with the economic crisis, two-thirds of Walmart moms (66%) said they had done most or all of them. A majority even said they have put off getting health care.
Along with these daily economic challenges, Walmart moms feel even more of a burden when it comes to household tasks. Nearly four in ten Walmart moms say they alone are responsible for nearly every item from a series of tasks like "doing laundry" or "cooking at home." Even married Walmart moms were more likely to do most of these tasks (31%) than married women overall (26%).
Swing moms view the political climate through a personal, economic lens.
What we began to do in this survey is something we don't see much of--an exploration of both women's personal and family concerns along with their political views. Personal financial insecurity, broader economic concerns, and swing political views are all related. In order to better talk to women, swing moms, Walmart moms, or however we define them, it's crucial to understand the link between the personal and the political. Other advice includes the following:
Acknowledge the continued tough economic climate, and that more work needs to be done.
Personalize a candidate's own narrative and personal journey, to demonstrate relatability, and an understanding of hard times and sacrifices.
Remember this group is likely not following the daily squabbles of Washington, as they are preoccupied with concerns about daily expenses and family life.
Focus more on the economy and jobs, less on divisive social issues.
Put policy positions in the context of how they would affect women and families. Something I've written about here, and elsewhere.
Was it appropriate for that commander, General Stanley McChrystal, to criticize the president and other top U.S. officials in an interview with Rolling Stone magazine?
32% Yes, 50% No
Following release of that interview, was it appropriate for the president to fire General McChrystal?
47% Yes, 36% No
Is President Obama's decision to name General David Petraeus as the new top commander in Afghanistan good for the U.S. war effort there, bad for the war effort, or will it have no impact?
47% Good, 9% Bad, 30% No impact
If there is only one person in the world for whom the oil spill disaster in the Gulf is a blessing in disguise, that man is Charlie Crist.
From photos of the Governor surveying the spill to soundbites of him demanding full compensation for Florida's spill related damages, Crist's handling of the spill has offered him the chance to look like a leader, above politics, fighting for Florida.
But his favorables, according to Quinnipiac's June 9th survey, haven't changed dramatically from the more difficult days of early 2010 and late 2009. His current job approval, at 57%, is lower than it was in October 2009 when Rubio's insurgency was underway. His favorables today are lower than the October poll as well, currently at 52%.
Meanwhile, Marco Rubio has struggled to pivot out of primary mode and into a general. The shift from running as "the true conservative" to a general election candidate will not be an easy one, and it becomes more and more critical with each tough poll that the Rubio campaign make that transition and begin to build his case to an audience beyond Tea Parties and local GOP groups.
There are a number of things going in Crist's favor - but don't count Rubio out. Five months is an eternity in politics. Looking at the recent polls and exit polling data going back to 1994, there are a variety of factors that will keep this race interesting through November.
1)Florida's unemployment rate is the fifth highest in the nation at 11.7%. There's an anecdote my colleagues and I have been using recently to describe the current political environment. Imagine a run-down house on that is on fire. Sure, the windows need repair, the house could use a coat of paint, the lawn needs to be cut. But until you put out the fire, the rest of that is irrelevant. The fire in politics today is the unemployment rate; until jobs come back to Florida, everything else is a distraction. When you can't drive down a suburban street without seeing foreclosure signs, voters have bigger issues they are voting on than whether or not former party chair Jim Greer had an illegal consulting arrangement with the Florida GOP. The temptation will be high for candidates to get into discussions about party credit card statements and backroom deals but things in Florida are very serious, and voters will respond to the candidates that take the economic crisis seriously.
2) Around one out of four voters in 2010 in Florida is likely to be independent. In the 2006 election, 24% of voters in the Governor's race were independent - a number that jumped to 29% in the Presidential race in 2008, in congruence with the nationwide trend of a small bump in independents. Capturing these voters is key. Currently, Crist is winning 51% of independent voters according to the June 9 Quinnipiac poll. This is not particularly surprising - both Meek and Rubio have been fighting for their partisan supporters - but if Crist continues to sustain a majority of the independent vote, he will be incredibly formidable heading into November.
3) As a result, Rubio must improve his brand with independents. Republicans know Marco Rubio. They love Marco Rubio. Only a quarter haven't formed an opinion about him, and only 11% don't like him. When it comes to locking down his side, he's good. His bigger problem comes from independents, where his fav/unfav is roughly even at 31-30. He absolutely needs to have favorables that are over 50% among independents in order to be competitive with Crist.
4) Kendrick Meek still doesn't have a statewide brand, and if he develops one, he will slightly erode Crist's share of the vote. Crist currently pulls in a whopping 37% of Democratic voters. I believe this has a lot to do with the fact that 69% of voters, including 59% of Democrats, say they haven't heard enough about Meek to form an opinion. As the election proceeds and all candidates hit the airwaves one can expect Crist's advantage to erode. These days, a candidate can build a brand almost overnight - consider that Rick Scott came out of nowhere and now boasts 53% of Florida voters who have an opinion about him. Meek may not be armed with the same kind of war chest, but by election day it is highly unlikely that Meek will still be an unknown to 7 out of 10 voters.
5) Painting Crist as an opportunist is not enough - people think everyone does what's popular. The conventional wisdom is that if Rubio pulls down Crist's favorables and brands Crist as a political opportunist, he can gain ground. The Quinnipiac poll showed that almost half of Florida voters (48%) think Crist makes decisions based on "what's popular" - a charge they also believe about Marco Rubio (42%). When the question is asked generally about "most public officials", 74% say they usually do what is popular. Fighting the battle over whether or not Crist is "principled" isn't fighting a battle on which Rubio has some major advantage in the general electorate. Furthermore, it's not as if Florida voters didn't associate Crist's defection from the GOP with ulterior motives - 60% said he left the Republican party because he couldn't win the primary, including 57% of independents. Voters aren't naïve on this point. If Rubio spends five months beating up on Crist as an opportunist and neglecting to build his own favorables among independents, it's not likely to be as productive as he'd like.
Most folks I talk to say that in order for Rubio to have a fighting chance against Crist, he needs to bring down Crist's favorables. Of course, that strategy might yield a slight bump in standing, but I don't believe it is nearly enough to win. Voters already assume politicians do what they need to do to get elected. They already assume Crist has made politically motivated moves in this race. And they vote for him anyways. The problem isn't Crist's favorables, the problem is Rubio's neutral brand image among independents. And the way for Rubio, Crist, or Meek (or any candidate in any race, for that matter) to build that brand is to become the leader on the issue of the economy and jobs.
Crist may be getting a break in the press with his handling of the oil spill. But the ultimate impact of the oil spill is more than environmental, it is economic. If tourism dollars start leaving the state and the economic situation grows more dire, the primacy of the economy in this and all races will become even greater. In January 2007 when Crist was sworn into office, Florida's unemployment rate was 3.5%. Besides March 2007, every month that Charlie Crist has been Governor, Florida's unemployment rate has gotten worse. Even the national unemployment rate doesn't have a trend as dramatically consistent as that, and even though the national rate has levelled off, Florida's keeps getting worse.
If Rubio wants to take Crist head on, he should - but with economic policy contrasts that demonstrate both how Crist failed to ameliorate the jobs situation and with how Rubio would propose to fix the problem. Rubio rose to fame as the "ideas" man in Tallahassee, and it is that same focus on "ideas" that can be his ticket to Washington in November.
The narrative that President Obama's approval ratings are being heavily damaged by the oil spill in the Gulf of Mexico is quickly overwhelming the critical faculties of the media.
Last week, I laid down a marker on this point, noting the potential appeal of the spill as a journalistic narrative to dramatize Obama's political difficulties. It appears that process is already well underway.
For instance, a story by Mark Murray, the deputy political director of NBC News, wrote an article on a new NBC/WSJ poll for MSNBC's website headlined "Poll: Spill drags the president's rating down." Murray's lede states that "Two months of oil continuing to gush from a well off the Gulf Coast, as well as an unemployment rate still near 10 percent, have taken a toll on President Barack Obama and his standing with the American public." However, given the relative stability of Obama's numbers, Murray resorts to hyping small changes:
In the poll, Obama's job-approval rating stands at 45 percent, which is down five points from early last month and down three points from late May...
What's more, Obama's favorable/unfavorable rating is now at 47 percent to 40 percent, down from 49 percent to 38 percent in early May and 52 percent to 35 percent in January.
What's left unsaid is that the changes in favorability and approval from the most recent NBC surveys asking those questions are within the poll's margin of error.
Similarly, Chris Matthews claimed on Hardball Tuesday that Obama's approval ratings "have been falling steadily since that oil rig blew up in the Gulf of Mexico two months ago" and that "his disapproval ratings [are] well above his approval ratings":
That was President Obama just last week calling upon Americans to seize the moment in that Oval Office address. The president's poll numbers, however, have been falling steadily since that oil rig blew up in the Gulf of Mexico two months ago. Is this just a blip on the screen for him, or has the president lost his political touch?...
Let's take a look at the president's poll numbers in the Pollster.com trend line. These are a combination of all the polls. They show his disapproval ratings well above his approval ratings.
There may well have been a small downturn in Obama's numbers in recent weeks. But as Media Matters points out, Matthews' claim that Obama's poll numbers have been "falling steadily" since the spill is overstated. Excluding Rasmussen polls, which often dominate the Pollster.com ratings due to their frequency and have a pronounced pro-GOP house effect, the shift in Obama's ratings since the spil on April 22 is on the order of 2-3 points, which is only clearly visible if we zoom in closely on the data:
In the broader view, however, the change (if it is real) is very small:
In addition, any change in Obama's approval ratings is not necessarily the result of the oil spill -- the economy (for instance) or many other factors could also affect his ratings.
In short, given what we know at this point, a more appropriate headline for reports on Obama's approval numbers is the one used by Pew: "Obama's Ratings Little Affected by Recent Turmoil." That may not be "news," but it's what the evidence tells us.
Update 6/25 11:47 AM: CJR's Greg Marx independently wrote an excellent post on the same theme, which includes a very nice useful interview with Pollster.com's Charles Franklin (a University of Wisconsin political scientist):
The point is not that Pew is right and the NBC poll is wrong, but that both data sets are legitimate--so journalists should include both, and be circumspect about sweeping conclusions. Any given poll contains uncertainty, so "until we see several of them moving in the same direction, it’s pretty hard to be sure that you’re picking up true change," said Charles Franklin, a professor of political science at the University of Wisconsin and co-founder of the polling aggregation site Pollster.com.
Media institutions have an obvious incentive to play up the polls they pay for. But "a story written entirely from the point of view of either of those two polls would be misleading to readers," Franklin said. A more accurate story would present the fuller range of data--which remains, at the moment, ambiguous.
To be fair to NBC’s journalists, the Pew poll hadn't yet come out when they started reporting on their numbers. But they didn't need it to offer more perspective. Franklin passed on via e-mail the chart below, which shows the trend for every pollster who has conducted surveys before and after the oil spill (click the image for a larger version as a PDF):
There's a considerable amount of variation there. Taking a comprehensive view, "the trend lines do show some modest long-term decline," according to Franklin. But while supposition that the spill might become a drag on the president is reasonable, "statistical tests show little evidence that the decline is specifically after the oil spill--rather, [we see] a continuation of a very slow decline since the first of the year." What you'd need to see to make claims about the spill's impact is not a downward trend after that point, but a worsening trend--and while it might show up eventually, "I just don't think the evidence is there yet."
On Monday of this week I reviewed these data and concluded: "No Sign That Obama's Overall Job Approval Rating Has Been Significantly Affected."
Based on weekly Gallup averages, here's what I said:
There is less evidence that the oil spill has affected Obama's standing in the public's eye from a comparison of his weekly overall job approval average before the BP spill on April 20 with his average after the spill. Obama's ratings have been slightly lower in the last four weeks than they were in the four weeks prior to that, but his average in either time period is not much different from his 48% average in the four weeks immediately prior to the spill . . . However, weekly trends in Obama's overall job approval rating show no significant impact from the oil spill; his weekly average now is little different from what it was in the weeks prior to the spill.
Now. That was through last week; Obama's weekly job approval average for June 15-21 was 47%. This week, so far, Monday through Thursday, Obama has been averaging 45%. If this lower trend continues through the weekend (by no means guaranteed), then we will report a dip by next week.
But, by this point in time, it's getting more and more difficult (and it never was easy) to establish causality between changes in Obama's approval rating and the oil spill. There are other variables coming into play as each day goes on. In particular, this past week we have the firing of Gen. Stanley McChrystal (and we'll have some specific data on the public's reaction to that on Monday), plus the tentative passage of a new financial reform bill much pushed by the Obama administration. Either could be affecting Obama's job approval rating -- along with economic news, the Dow, and a host of other variables not as dramatically obvious as the oil spill.
As noted, the further away we get from the April 20 oil spill, the lower the certainty with which we can attribute changes in Obama's job approval rating to it -- or any specific event.
But the data to date do show little evidence of a dramatic drop in Obama's job approval rating immediately after -- or two months after -- the BP oil platform explosion and the beginning of the oil leak/spill.
Pew Research Center
6/16-20/10; 1,802 adults, 3% margin of error
528 Republicans, 5.5% margin of error
581 Democrats, 5% margin of error
596 independents, 5% margin of error
Mode: Live telephone interviews
As you may know, the state of Arizona recently passed a law that requires police to verify the legal status of someone they have already stopped or arrested if they suspect that the person is in the country illegally. Do you approve or disapprove of Arizona's new immigration law?
64% Approve 32% Disapprove
Would you favor changing the Constitution so that the parents must be legal residents of the U.S. in order for their newborn child to be a citizen, or should the Constitution be left as it is?
41% Change constitution, 56% Leave as is
How well is the U.S. military effort in Afghanistan going?
9% Very well, 40% Fairly well, 32% Not too well, 13% Not at all well
34% Democrat, 27% Republican, 34% independent (chart)
Well, here's a story you don't read every day: A pollster for Florida Democratic Congressman Alan Grayson is running as a Florida Tea Party candidate for the Florida legislature. Roll Call's Nathan Gonzales has the truth-is-stranger than fiction full story:
On Friday, Victoria Torres, 44, of Orlando qualified to run as a Tea Party candidate in state House district 51 in the last hours of the qualifying period.
A call to Torres was returned by Nick Egoroff, communications director for the Florida Tea Party, who described Torres as a "quasi-paralegal assistant who works in a law office." But apparently, Torres is also a pollster.
According to records from the Florida Department of State office, Torres incorporated Public Opinion Strategies Inc. in December 2008. In the first quarter of this year, Grayson's campaign made two payments to her firm, totaling $11,000, for polling and survey expenses.
The whole story is a bit murky but the gist of the conspiracy theory is that Democrats like Grayson want to assist the organization known as the Florida Tea Party field candidates in the fall elections, rather than in Republican primaries, in order to divide the Republican vote. You can judge for yourself by reading the full piece, but Gonzales has gathered evidence of a lot of odd coincidences.
As for candidate Torres' work as a pollster, Gonzales got a Grayson spokesman to confirm that the House candidate is the same person who did polling for Grayson as a "side business," but could find no further evidence that her company had done polling for anyone else. He also reports that Torres was one of three Grayson pollster, adding, "the use of multiple pollsters simultaneously in the same cycle is highly uncommon for a Congressional candidate."
Yes it is.
Also, as Gonzales explains, "Public Opinion Strategies" is also the name of one of the "largest and best known" DC-based Republican polling firms, led by Glen Bolger, Bill McInturff, Neil Newhouse and a long list of partners. His article includes what for us is easily the quote of the day:
"We definitely do not poll for Democrats, nor do we have an office in Orlando," said Glen Bolger of the Virginia-based POS. "However, we do wish Congressman Grayson the worst of luck in November."
Last week, I criticized Matt Bai's claim that it was an "ominous sign" for Democrats that President Obama's approval rating is under 50%.
Writing in The Hill, top Democratic pollster Mark Mellman goes even further, calling 50% approval a "magic number" for midterm elections:
[P]erhaps there are some magic numbers after all.
Take the effect of presidential approval on midterms. We graph the relationship between a president's approval rating and his party's gains and losses in midterm elections, thinking of the result as a smooth relationship. The lower the president's approval rating, the more seats his party loses.
But the pattern is not really so linear after all. There is a sharp discontinuity at 50 percent. Presidents whose approval rating is at 50 percent or above have lost, on average, just 11 seats in the House, while presidents under the 50 percent mark have lost an average of 33 seats.
Averages can obscure as much as they reveal, so pick apart the numbers. No president with an approval rating under 50 percent has lost fewer than 15 seats. The next smallest number is 26. Even a president just below 50 percent can lose a lot. When Democrats were punished with the loss of 52 House seats in 1994, President Clinton's approval rating rested just under the 50 percent threshold, at 48...
So where does President Obama stand? Last week Gallup put his approval at 45 percent, this week at 49. The Pollster.com weighted average of all polls says 46 percent. In short, for now, the president is hovering just below what may prove to be a magic number for Democrats in 2010.
Unlike Bai, Mellman brings historical evidence to the table, so let's consider his argument. He acknowledges the seemingly linear relationship between presidential approval and changes in House seats for the president's party in midterm elections. However, he argues that there is a "sharp discontinuity at 50 percent" in which "[p]residents whose approval rating is at 50 percent or above have lost, on average, just 11 seats in the House, while presidents under the 50 percent mark have lost an average of 33 seats."
First, let's replicate Mellman's numbers. (He appears to be excluding the replacement presidents -- Truman in 1946 and Ford in 1974 -- so I do the same here.) Using a cutpoint of 50 percent, I find that presidents with greater than 50 percent approval in the most recent Gallup poll before Election Day lose an average of ten seats and those below 50 percent lose an average of 33.5 seats (these slight discrepancies are likely the result of how different sources calculate seat change).
The problem is that the choice of 50 percent is arbitrary. For instance, presidents with approval ratings above 45 percent lost an average of 15 seats, while those below 45 percent lost an average of 33 seats -- results that aren't that different from Mellman's original numbers. Going the other direction, any cutpoint from 51 percent to 56 percent will yield the same results as 50 percent because there are no presidents who had approval ratings in that range in the data.
If you prefer graphical evidence, here is the data with a standard linear fit:
If we instead use a more flexible local polynomial fit to allow for nonlinearity, the predicted values show an inflection point around 50 percent approval, but the 95% confidence intervals reveal a great deal of uncertainty in that estimate -- hardly enough to justify a claim of a "sharp discontinuity":
With so few data points, it's very difficult to demonstrate a non-linear relationship. Absent further evidence, we can't be confident that a discontinuity exists at 50 percent.
It's also hard to believe the claim of a discontinuity in the context of the upcoming midterm elections. Given the state of the economy and the generic ballot, it's clear that Democrats are likely to lose a substantial number of seats regardless of whether Obama's approval rating is 49 percent or 51 percent on Election Day. Does Mellman believe otherwise?
Update 6/24 4:16 PM: To illustrate the point a bit further, I created a simple simulation of a linear relationship between approval and seat change that produces data approximately similar to what we observe above:
Over 1000 iterations of the simulation, the average outcome for presidents below 50 percent approval was a 12 seat loss while the average outcome above 50 percent approval was a 29 seat loss. In other words, it is very easy for a linear relationship to produce the sort of outcomes that Mellman describes.
"If the choice in your district had the following, would you be more likely to vote for a --: Republican candidate for Congress, a Democratic candidate for Congress -- or an independent or third party candidate for Congress?"among registered voters
34% Democratic candidate, 31% Republican candidate, 25% independent/third party candidate (chart)
Preference for Congress after 2010 electionsamong registered voters
Republican Control: 45%, Democratic Control: 43%
Will your vote for Congress this November be a vote to send a signal of support for President Obama, a signal of opposition to President Obama, or not a signal either way about President Obama?among registered voters
27% send support, 37% send opposition, 39% not a signal either way
If there were a place on your ballot that allowed you to vote to defeat and replace every single member of Congress, including your own representative, would you do this, or not?among registered voters
47% yes, 50% no
Recently Passed Health Care Plan
40% Good idea, 44% Bad idea (chart)
"How responsible is __________ and his administration's policies for the country's current economic conditions?"
Pres. Obama: 7% solely responsible, 20% mainly responsible, 51% only somewhat responsible, 21% not really responsible
Former Pres. W. Bush: 12% solely responsible, 28% mainly responsible, 46% only somewhat responsible, 14% not really responsible
"When it comes to oil drilling off U.S. coasts, which of the following statements comes closer to your point of view?"
48% Harm to environment outweighs economy, 46% Benefits to the economy outweighs environment
State of the Country
29% Right Direction, 62% Wrong Track (chart)
33% Democrat, 26% Republican, 36% independent (chart)
The U.S. Senate has the constitutional authority to confirm all Supreme Court nominees. Based upon what you know at this time, should the U.S. Senate confirm Elena Kagan as a Supreme Court Justice?
35% Yes, 42% No
USA Today / Gallup
6/11-13/10; 1,014 adults, 4% margin of error
Mode: Live telephone interviews
As you may know, earlier this year Congress passed legislation that restructures the nation's healthcare system. All in all, do you think it is a good thing or a bad thing that Congress passed this legislation?
49% Good thing, 46% Bad thing (chart)
How would you rate the job ________ has been doing responding to the oil leak in the Gulf of Mexico?
The Obama administration: 8% Excellent, 25% Good, 29% Only fair, 35% Poor
BP: 2% Excellent, 16% Good, 28% Only fair, 49% Poor
Thinking about the economic stimulus passed by Congress last year do you think the stimulus has helped the job situation, or has it NOT helped the job situation?
33% Helped, 60% Not helped
From what you've seen and heard so far, do you think the Senate should or should not confirm Elena Kagan to the Supreme Court?
33% Should, 25% Should not
29% Democrat, 27% Republican, 34% independent (chart)
My column for this week follows up on last week's topic from a different angle: Nate Silver's intriguing finding that as a group, pollsters that are members of the National Council of Public Polls (NCPP) or that endorsed the worthy Transparency Initiative of the American Association for Public Opinion Research (AAPOR) appear to be more accurate in forecasting election outcomes than other pollsters. While I'd like to see more evidence on this issue, it is definitely a topic worth further exploration. I hope you click through and read it all.
And yes, this has been the fourth or fifth item from me on Silver's ratings in a week, with two more from guest contributors, so it's time for me to move on to other subjects. However, since Nate responded yesterday, I want to clarify two things about my post on Friday:
First, he quarrels with my characterization of his effort to rate polls "from so many different types of elections spanning so many years into a single scoring and ranking system" as an "Everest-like challenge:"
Building the pollster ratings was not an Everest-like challenge. It was more like scaling some minor peak in the Adirondacks: certainly a hike intended for experienced climbers, but nothing all that prohibitive. I'd guess that, from start to finish, the pollster ratings required something like 100 hours of work. That's a pretty major project, but it's not as massive as our Presidential forecasting engine, or PECOTA, or the Soccer Power Index, all of which took literally months to develop, or even something like the neighborhoods project I worked on with New York magazine, which took a team of about ten of us several weeks to put together. Nor is anything about the pollster ratings especially proprietary. For the most part, we're using data sources that are publicly available to anyone with an Internet connection, and which are either free or cheap. And then we're applying some relatively basic, B.A.-level regression analysis. Every step is explained very thoroughly.
The point of my admittedly imperfect Everest metaphor was not that Silver has attempted something that requires a massive investment of time, money or physical endurance, but rather that the underlying idea is ambitious: Using a series of regression models to combine polls from 10 years and a wide variety of elections, from local primaries to national presidential general elections, fielded as far back as three weeks before each election, with controls to level the playing field statistically that all pollsters are treated fairly.
I am not an expert in statistical modeling, but when I ask those that are, they keep telling me the same things: Nate's scoring system is based on about four different regression models (only one of which he has shared), and he does not provide either standard errors of the scores (so we can better understand what the level of precision is) or the results of sensitivity testing (to test what happens when he varies the assumptions slightly -- do the results change a little or a lot). If there is "nothing especially proprietary" about the models, then I don't understand the reluctance to share these details.
Second, I will concede that my headline on Friday's post -- "Rating Pollster Accuracy: How Useful?" -- was an attempt to be both pithy and polite that may have implied too broad a dismissal of the notion of rating pollster accuracy. I do see value in such efforts, as I tried to explain up front, especially as a means of assessing polling methods generally and new technologies in particular. SurveyUSA, for example, has invested much effort into their own pollster scorecards over the years as a means of demonstrating the accuracy of their automated survey methodology in forecasting election outcomes. That sort of analysis is highly "useful."
And I also agree, as Berwood Yost and Chris Borick wrote in their guest contribution last week, that individual pollster ratings offer the promise of "helping the public determine the relative effectiveness of polls in predicting election outcomes [that] can be compared to Consumer Reports." The reason why I have found past efforts to score individual pollsters not very useful toward that end is that it's difficult to disentangle pollster-specific accuracy from the loud noise of random sampling error, especially when we have only a handful of polls to score. And as I wrote in December 2008, very small changes in the assumptions made for scoring accuracy in 2008 produced big differences in the resulting rankings. Except for identifying the occasional clunker, efforts to rate the most prolific pollsters usually produce little or no statistically meaningful differentiation. So in that sense, they have not proven very "useful."
That said, as Nate argues, the challenges may be surmountable. I'm confident that other smart statisticians will produce competing ways of assessing pollster performance, and when they do, we will link to and discuss them. David Shor's effort, posted just last night, is an example with great promise.