The campaigns do the best they can under the circumstances fate has dealt them. Their campaigns may do better or worse than expected, but the basic trends provide the context of the election contest.

Congress is the favorite whipping boy, even for members of Congress. So low approval isn't shocking. But approval at 15% is. Now as low as ever since 1992.

An opportunity to throw all the bums out. But the implications for the parties have not equally matched the views on the institution as a whole. Despite being out of power for two years, Republicans in Congress remain more negatively viewed than do Democrats, despite the latter's difficulties in passing a legislative program.

The campaigns do the best they can under the circumstances fate has dealt them. Their campaigns may do better or worse than expected, but the basic trends provide the context of the election contest.

At the time of the 2006 election the war in Iraq was the most important problem facing the country, topping 30% and by far the dominant issue. The economy, by contrast, was under 10%. Let me repeat that: the economy was under 10%.

Improving conditions in Iraq (see previous post) started a decline in the MIP for Iraq, before views of the economy started to change. But then at the end of 2007 and start of 2008 the economy exploded as the most important problem by far.  It rapidly reached the 30%+ level the war had previously held, and has now nearly doubled that to over 50%. That dominance changed the gound on which the campaign has been fought.

The war? Now under 10%.

What about perceptions of economic conditions? The Michigan Consumer Sentiment measure is the longest running estimate of how the public views the economy, and is included in the Fed's index of leading economic indicators:

The campaigns do the best they can under the circumstances fate has dealt them. Their campaigns may do better or worse than expected, but the basic trends provide the context of the election contest.

The public has been amazingly negative about the direction of the country for four years, but more amazing is how the trend has moved to ever more negative views without periods of increased optimism. No wonder "Change" is a winning theme this year, and one both candidates have tried to adopt.

The campaigns do the best they can under the circumstances fate has dealt them. Their campaigns may do better or worse than expected, but the basic trends provide the context of the election contest.

In 2006 the war in Iraq was the dominant issue. Public opinion had  moved from support of the war in 2003-2004 to a substantial opposition in 2006. More than 60% judged the war not worth the cost, while only 35% thought it was worth it.  Those facts have not changed over the past two years. They provide the basis for Obama's initial run, and the challenge for McCain as a supporter of the war and the surge.

But in a crucial sense opinion of the war has fundamentally changed. The public no longer thinks the war is going badly. In fact the contrary is now true.

The campaigns do the best they can under the circumstances fate has dealt them. Their campaigns may do better or worse than expected, but the basic trends provide the context of the election contest.

The current administration is one part of those basic trends. And in this case the familiar trend for approval of President Bush dramatically demonstrates the burden faced by McCain and the opportunity provided to Obama.

Adding to the long term impact is the recent drop in approval from already poor low 30s to the current 24.2. The financial crisis (and campaign rhetoric criticizing him) has driven his approval to new lows.

How will undecided voters break, and will racial attitudes color their votes?

We've seen an enormous amount of speculation but little evidence based on data, so let's try to tip the balance back to empirical evidence.

Thanks to the Diageo/Hotline tracking poll data, we can model individual vote choice and see what we would expect of undecided voters.

During October 3-11 our colleagues at the Diageo/Hotline poll included a racial attitude question we had previously used in the Big Ten Battleground survey in September and which NBC/Wall Street Journal used in January. That question was shown in both the earlier polls to have a statistically significant effect on vote choice, even after controlling for other political attitudes and demographics.

The question text is: "I'd like you to tell me whether you strongly agree, somewhat agree, somewhat disagree or strongly disagree with the following statement. ... African Americans often use race as an excuse to justify wrongdoing"

58% of the total sample, and 61% of whites agree either strongly or somewhat. (For comparison, 56% of hispanics and 40% of African Americans agree strongly or somewhat.)

I model the vote choice for those who expressed a preference with a model using a variety of attitudes and demographics, including favorability to Obama and McCain, party id, marital status, kids at home, education, race, age, sex, church attendance, region and urban, suburban or rural residence. Then I added the racial attitude responses from the "black excuse" question. To check against people hiding their feelings by refusing to answer the "black excuse" question I also included a variable to capture the effect of refusal to answer.

This model produces a predicted probability of voting for McCain or Obama, including predicted probabilities for those who had said they were undecided or who refused to respond to the vote question. From this we can estimate the likely vote of undecided, and compare the estimates to the responses of those who gave a vote preference in the survey.

Bottom line: Undecided and refuse to say voters are estimated to break 50% for McCain and 50% for Obama. As even as it gets. There is no evidence here of a large bias towards McCain that is hidden within the undecided respondents.

Nor is there evidence of a pronounced racial bias among these undecided voters as compared to the public at large. Among the undecided 27% strongly agree and 32% somewhat agree on the "black excuse" item. For the public as a whole 26% and 32% give the corresponding responses.

The model does a good job predicting survey response as well. 97% of both Democratic and Republican voters are predicted by the model to vote that way. For those who say they only "lean" towards one party or the other, 77% of Democratic leaners and 80% of Republican leaners are predicted to vote as they lean. The symmetry of results here suggests that there is not a visible bias in the model estimates for either party or for intensity of preference.

Finally, what happens if we ignore racial attitude and predict vote among the undecided without it? The split is 52% Obama to 48% McCain. So at most the impact of incorporating racial attitude in the model is a rise of 2% for McCain among undecided. Given the sample sizes involved, that is well within the margin of error. And if we take out candidate favorability from the model we get estimates of 52-48 without racial attitude and 53-47 with racial attitude.

So what can we conclude? There is no evidence of a hidden support for McCain among undecided voters. They split more evenly than does the "decided" pool of respondents, who split 54-46 in this sample (Oct 3-11) but that's well within normal expectations and is a modest difference in any case.

Second, the role of racial attitude is important at the individual level, but the aggregate consequence is extremely modest. Some are moved away from Obama yet others are moved towards him. And among the undecided, the distribution of opinion on this measure of racial attitude is virtually identical to that in the population.

In a year of endless discussion about racial effects there has been far more speculation and far less data analysis than is good for us. Let's put our data on the table before continuing to opine about this subject.

Obama has now recovered his lead in national polling, rising at or above his post-DNC, pre-RNC level. This sharp turnaround erases the very sharp pro-McCain/Palin convention bounce we saw in early September.

But the Obama recovery is not concentrated in Democratic states. As with the McCain advantage, this turn is visible in all three types of states-- Red, Yellow and Blue.

The biggest recovery is actually in the Red states, where McCain enjoyed his biggest bounce. Those states are not back where they started, or even a shade less pro-McCain, but they are a far ways from being "close".

The states we have classified as tossup or leans (what I'm calling yellow here, though on the map leans are light red or blue) have also seen a significant Obama recovery. The range of movement is rather modest, but the roughly three point McCain gain has now been balanced by a 3+ point Obama recovery in these most contested states, putting Obama up by just over a point.

Dem states showed a small move to McCain and have now seen an Obama recovery, though with a one point fall off most recently.

    "I think one of the things driving the national polls is that the red states are redder," said David Axelrod, one of Obama's closest advisers. "In the battleground states, the race has held pretty firm." 

An interesting claim. Let's take a look at the data based on state polls, rather than national.

Among the strong Republican states, McCain has gained more than 8 points over Obama since shortly before the conventions, turning a 14 point lead into a 22.5 point margin, a huge gain.

Among the strong Democratic states, the effect of the conventions is a tiny 2 point move in McCain's direction, from an Obama lead of 12 points before to 10 points now.

But the rest of the states, rated lean or toss up, have also shown movement. These swing states had a 1.5 point Obama lead before the conventions, and that has now turned into a 3 point McCain lead, a 4.5 point shift.

So Axelrod is right that the biggest gains for McCain have come in the reddest of states, and those may influence national polling. But the evidence doesn't back his second claim, that the battleground has held firm, unless of course you mean they are still battleground states. But now battlegrounds that on balance favor McCain rather than favor Obama as they did before the conventions.

One caution: the lean and tossup states are themselves heterogeneous, so a single trend estimate such as the 4.5 McCain lead here is itself a simplification. If you wanted to focus on the six or eight states that probably hold the key to the electoral vote, you could slice this more finely.

We currently rate eight states as pure tossup: NH, VA, OH, MI, CO, NM, NV and MT. (Note the last has few polls and the latest 9/8 shows a 53-42 McCain lead. But it does fit our statistical criteria, and Montana was listed by the Obama campaign as a target state.)

When we fit the data to just these eight tossup states, we see a 3.5 point move in McCain's direction, from a 2 point Obama lead to a 1.5 point McCain lead. Only a point less shift than among all lean and tossup states.

The bottom line is neither campaign should be complacent or despondent. There is a lot of time left and recent history shows that both up and down swings of 6-9 points are entirely plausible.

The focus on polling starts at 11:00 minutes into the show.

Cross posted at Political Arithmetik.

The McCain campaign has gained significant support in national polling since the end of the Republican Convention, but what about the state polls? Has the shift also been reflected there?

State pollsters appeared to go on vacation for the conventions, with very little new polls during the two weeks of conventions (and the week before). Now the pollsters are back, tanned and rested and ready to go. We've added 17 new state polls since the RNC ended, and while we'd love to see more, it is enough to get started with some analysis.

The chart above shows the national trend in blue and the trend based on those states with post-convention polls in purple. Over the course of the year, the two trends have followed each other rather well with some small differences in details but qualitatively similar patterns of up and down movement.

Now in the post-RNC period, the states with new polls match the national polls quite closely, both giving estimates of about a one point McCain lead, with the states maybe a shade less than that.

This post-convention bounce may or may not last, but at the moment the evidence is that it is moving across the states (and these are mostly competitive states) at about the same rate as it is for the national polls.

States for which we have new polls are Colorado, Florida, Maryland, Michigan, Montana, New Jersey, North Carolina, Ohio, Oklahoma, Pennsylvania, Virginia, Washington and Wisconsin.

Cross-posted at Political Arithmetik.

The post-convention bounce is now moving in the Republican direction, but with an enormous spread in estimates. A Gallup/USA Today (9/5-7) has an enormous 10 point McCain lead over Obama, 54%-44%. In contrast, Gallup's tracker over the same days shows a 5 point McCain lead, 49%-44%. Now would be a good time to note that the tracker is a registered voter (RV) sample, while the Gallup/USAT is a likely voter (LV) sample. LV samples typically are more favorable for Republican candidates, so at least some of this difference is probably due to these different sampling frames. We'll no doubt be talking a lot about this issue in days ahead.

But other polls on the same days show a tied race. Diageo/Hotline has the race 44%-44% and CNN has it 48%-48%. And Zogby's Internet poll done 9/5-6 puts the race at 50%-46%.

All of these are much better for McCain than the 5-9 point Obama leads we saw in the immediate aftermath of the Democratic convention.

So it looks like both parties got nice convention bounces.

Our trend estimate is still hungry for more data. The standard, blue line, estimate is now less persuaded that Obama had a convention bounce OR that McCain is getting one either. That is standard behavior of our estimator which is designed to be a bit conservative when faced with conflicting polls and short term changes of trend.

But that is why we have our "sensitive" estimator for comparison. The red line is a trend estimate that is about twice as sensitive a the blue line. It is considerably quicker to respond to short term changes and to fewer polls. The down side is it will often chase random noise.

Since there is good reason to believe convention bounces are real, it is reasonable to think that the red line's indication that the race has indeed tightened is probably a real signal in the data, and not just noise. On the other hand, the Gallup 10 point McCain lead is out of the range of any other current polling data. So "red" may be chasing that outlier just a bit more than is good for him. As the figure makes clear, red and blue usually agree quite closely after enough data are in hand, but can diverge especially when data are sparce.

A prudent approach is to wait for a few more post-convention and post-weekend interviewing polls to see just how big and how sustained the RNC bounce is. But both estimators agree we have ourselves a real horserace now.

Cross-posted at Political Arithmetik.

A little interesting movement in views of the candidates has taken place since the end of the primaries in June. All three candidates, McCain, Obama and Clinton, have seen rises in their favorable ratings and an initial decline in unfavorable views though with a slight upturn recently. McCain and Obama are enjoying essentially identical ratings, with 60% favorable and only 35% unfavorable. Even after a significant amount of negative portrayals of him in RNC and McCain ads, Obama's rating has risen over the summer, and so has McCain's. (According to the Wisconsin Advertising Project, which monitored and coded all 100,000 ad airings in June and July, one third of McCain's ads contained negative information about Obama and 100% of RNC ads were negative. In the same two months, 10% of Obama's ads mentioned McCain.)

Whatever happens after the conventions, both candidates enjoy an enviable standing with voters as attractive figures instead of a pair of lesser evils. The fall capaign may alter this, but even after a hard fought primary season the nominees remain attractive figures.

Meanwhile, Senator Clinton has also enjoyed an upturn in favorable ratings and a decline in unfavorable ratings since the end of the primary season. While improved, Clinton remains a more polarizing figure than either McCain or Obama, with slightly lower favorable but noticeably higher negative ratings.

Senator Clinton is far more popular among Democrats than among either Independents or (especially) Republicans. In that sense, her speech to the Democratic Convention last night was an example of speaking primarily to the party and her supporters, rather than to the broader public. The contast between former Virginia governor and now Senate candidate Mark Warner's speech and Clinton's is a good example of this difference. Warner stressed unifying themes and appeals across political groups, which was greated warmly but which fell short of electrifying the Democratic delegates. In contrast, Clinton played to the party and produced a predictably enthusiastic response within the DNC convention hall. Conventions contain both elements. Monday, the party celebrated Sen. Kennedy's life and family legacy, primarily an inside the family affair, perhaps touching some independents but not likely to attract Republicans. In contrast Michelle Obama's speech could have easily been given at the Republican convention, with its themes of family, hard work, pulling oneself up from working class circumstances. Hers was a speech designed to reach out beyond the party.

The one remaining question from the Clinton speech is whether her supporters also resepect her enough to follow her lead. For Clinton to be a power in the party includes the requirement that she be able to deliver her supporters for Obama. If any significant number of her supporters refuse to be delivered, they reduce her status as a result. This is hard to judge from the cable news coverage, who can easily find individual delegates willing to say they are unpersuaded. But what effect the Clinton speech has with her supporters outside the convention hall will be critical.

"What happens if you leave out 'x'?" is probably the single most asked question at Pollster.com. Everyone has their favorite pollster to hate, and wonders if only that one were removed would the results be closer to the truth. It is a really good question because it goes to the heart of the robustness of our trend estimates and the role of one (or a couple) of pollsters in shaping the conventional wisdom of what "the polls show". The former issue is statistical, the later goes to how shared understandings are constructed. If our estimators are highly sensitive to any one pollster then we have a statistical problem. If one pollster unduly influences shared perceptions, then we better hope they are "right".

Today's question from Mark (and many readers) is what role the tracking polls play in our estimates. This is an issue Mark and I debated quite a lot during the winter when Gallup and Rasmussen began their daily tracking polls. Because they produce so many numbers, including all their data runs the risk that these two dominate our trend estimate to an unacceptable degree. But do they exert that much influence-- there is the question.

And just to be contrarian, take note of the opposite problem: data are valuable. You should never want to ignore information. In that sense excluding data from prolific sources is a mistake unless the data are biased in some uncorrectable way.

The first decision we reached in January was that we would only include each INDEPENDENT sample from tracking polls. This was an easy call. Rolling samples are great for daily updates but Thursday's poll isn't independent of Wednesday's because they both contain Tuesday's and Wednesday's results, if it is a three-day track. In that sense, there isn't as much new information as it seems. So we take only the independent results: Mon-Tues-Wed, Thur-Fri-Sat, Sun-Mon-Tues and so on for a three day tracker. This means we are only including independent data collections, and cuts down on the number of entries in our data that come from any single tracking poll.

Despite this, we get a lot of data in the national track from two primary sources: Rasmussen accounts for 63 of 286 data points in our national trend data. Gallup's tracker provides 41 more. (We keep Gallup's USAToday polls separate from the tracker.) And a third source, The Economist/YouGov's internet poll accounts for 24 data points. (Full Disclosure: YouGov/Polimetrix Pollster.com and supports our work here.) The next most common pollster is Zogby with only 12. So let's take a look at the influence of these top-three pollsters in terms of data. Together they account for 128 of 286 data points, or 45% of our national data.

Let's begin with recognizing that every data point MUST have some influence on our trend estimator. If it didn't then the trend would not be responding to the data! So in that simple sense, the Rasmussen, Gallup and YouGov data must play some role in determining the value of our trend estimate. That really isn't the issue that concerns people. The question is whether these three pollsters DISTORT the trends we would otherwise estimate from all other sources. It would be fine if Rasmussen or Gallup or YouGov had a huge influence on our estimate so long as their trends were exactly in line with everyone else's trends. The concern arises when there is the possibility that one of these is both influential AND out of line with the rest of the world.

We need to look at three things: the overall trend with all pollsters included, the trend only for a single pollster, and finally the trend we'd estimate if we excluded this pollster. If a pollster is different from others, that's a concern. But if they don't substantially change the trend estimate, then we aren't that worried. But if they are different AND shift the trend, then we have to worry.

So let's look at the data. The chart below plots the overall trend (the blue line), the trend for each of the three most prolific pollsters (solid red), and the trend estimate if we exclude that pollster (dashed red line). A fourth plot shows what happens if we exclude all three prolific pollsters and rely only on the 28 different pollsters who've done 12 or fewer polls each (dashed blue).

Over all our polls, we estimate an Obama advantage over McCain of 3.4 points (as of early morning on 8/14). If we exclude Gallup, the trend estimate is 3.2. If we exclude Rasmussen, the estimate is 4.5. If we exclude YouGov the estimate is 3.3. And if we omit all three (and 45% of our data) the trend estimate is 5.1. So it DOES matter which of these we include. By as little as 0.1 points or as much as 1.7 points.

The most striking thing to me about these figures is that all three tracking polls trend a bit below the overall trend, which is why omitting them all produces the biggest change in the current trend estimate. Gallup is only a bit below trend, YouGov a bit more in May but less recently. Rasmussen stands out as the most consistently below trend, with convergence only in June for a while.

At first glance, the worst thing about Rasmussen is that his trend seems much more sharply downward since late June than either other frequent pollster (both Gallup and YouGov see flat or rising Obama margins in that time.) The dashed line without Rasmussen looks flat or possibly rising slighting, while including Rasmussen with all others produces a modest downward slope recently. So is Rasmussen determining our current trend's tendency to be moving down? This is especially relevant given the upward moves by Gallup and YouGov.

The bottom right panel of the figure offers some reassurance. While Rasmussen does look different from Gallup or YouGov, when we take all three tracking polls out, the dashed blue line in the bottom right figure trends slightly down, approximately in parallel with the overall trend estimate using all the polls. To be sure, omitting the tracking polls does produce a higher current trend estimate: 5.1 vs 3.4 for all polls. Clearly the tracking polls are showing a lower margin and that is reflected here. But from my point of view, the happy news is that the trend with or without the three trackers moves in pretty much the same way over the year. Granted some minor differences, both curves move up and down at about the same time and the gap between the solid and dashed blue lines is roughly equal over time. This suggests that the effects of the three trackers may be to lower the estimated Obama margin over McCain, but they don't distort the dynamics of the race. When trends are up or trends are down, they are reflected in both the with and without tracker estimates.

It is reassuring that both the Gallup and YouGov trackers have very little influence on the overall trend estimate. Including or excluding either of these polls has very little effect on the trend estimate.

A final point is what this says about the validity of the polls. If Gallup and YouGov are flat or slightly up, and Rasmussen is sharply down, how are we to know which is "right"? The data here say a bit of both are right. Gallup and YouGov do somewhat better jobs tracking the overall trend than does Rasmussen. But the recent decline in Obama support, even though modest, is not captured by Gallup or YouGov. Rasmussen clearly overstates the decline (compared to other polling) but the consensus of the 158 polls NOT from these three sources is that there has been a little downturn in Obama's lead since late June.

It is easy to exaggerate how large these differences are, especially in light of the intrinsically hard problem of knowing what "the truth" is at any moment. The chart below compares the trend estimates we would get from dropping each of the 31 different pollsters in our our data. Two things stand out. Dropping any single pollster has very little effect on the trend estimate, with one exception. Omitting Rasmussen, who is both the most prolific pollster and the one with considerably more variation than others, does make a noticeable difference in the trend estimate. But the reassuring element of this graph is that even the line omitting Rasmussen still falls within the 95% confidence interval around our overall trend estimate. While there was a time in