September 17, 2006 - September 23, 2006


Happy New Year

Apologies for lack of posts today.  The entire staff here at Pollster.com world headquarters will be leaving early to celebrate the New Year 5767, and we have been preoccupied with backroom tasks all day. 

We will be back on Monday, and this coming weekend will likely be the last without posts until the Election.  So please stay tuned.

Meanwhile, our Slate Election Scorecard will update again later tonight.  Here's a hint: The big numbers on the Scoreboard will be changing.

L'Shana Tova.

Slate Scorecard: New Polls in PA & MD

Our daily update to the Slate "Election Scorecard" is now up and focuses on new polls in Pennsylvania and Maryland.  Read it all. 

Another Day, Two New Polls

Topics: 2006 , Measurement , The 2006 Race

The morning brings two new surveys: One from CBS and the New York Times (article, results, CBS analysis, results on Iraq/terror, elections) and one from Bloomberg and the LA Times (article, results). Both surveys are lengthy and the accompanying analysis of each goes into great depth on attitudes toward Congress, the upcoming elections, Iraq, and Terrorism among others. I want to take a quick look at the two measures we have been considering here for the last few days: the Bush job approval rating and the generic Congressional ballot.

First, let's add the new survey results to the table I posted last night showing how results from all national pollsters changed between August and September:


Both surveys indicate an increase in the Bush job approval rating, although the change is bigger in the LA Times/Bloomberg poll (+5, from 40% to 45%) than in the CBS/NYT poll (+1, from 36% to 37%). Given the obvious random variation across the various pollsters in the table above, the difference is not terribly striking. While the CBS/NYT approval rating is certainly a lot lower than the LA Times rating (see Charles Franklin on "house effects"), the change since August on both polls seems within the sampling error of the average change (+2.6) seen across all eleven pollsters.

Not surprisingly, the relatively small difference in the two trends makes a big difference in the coverage. The gain in the Bush approval rating is the lead in Ron Brownstein's article:

President Bush's approval rating has reached its highest level since January, helping to boost the Republican Party's image across a range of domestic and national security issues just seven weeks before this year's midterm election, a new Times/Bloomberg poll has found.

The first reference to the Bush job rating comes in the fifth paragraph of the New York Times article by Adam Nagourney and Janet Elder and reaches a very different conclusion:

The poll also found that President Bush had not improved his own or his party's standing through his intense campaign of speeches and events surrounding the fifth anniversary of the 9/11 attacks. The speeches were at the heart of a Republican strategy to thrust national security to the forefront in the fall elections. Mr. Bush's job approval rating was 37 percent in the poll, virtually unchanged from the last Times/CBS News poll, in August.

Later the Times article offers an explanation for the contrast between this result and the apparent upswing in Bush approval reported by the recent USA Today/Gallup Poll:

The New York Times/CBS News poll began last Friday, four days after the commemoration of the fifth anniversary of the 9/11 attacks, and two weeks after the White House began its offensive on security issues. A USA Today-Gallup Poll published Tuesday reported that Mr. Bush's job approval rating had jumped to 44 percent from 39 percent. The questioning in that poll went through Sunday; The Times and CBS completed questioning Tuesday night. Presidential addresses often produce shifts in public opinion that tend to be transitory.

The obvious problem with that theory is that the LA Times/Bloomberg poll was conducted over essentially the same period (both ended on Tuesday, but the LA Times poll started a day later). A more likely explanation is simply that the real trend was slightly less than that measured by LA Times/Bloomberg and Gallup, and slightly more than that measured by CBS/NY Times. But I can also suggest one more theory. Like the Gallup poll, the CBS/New York Times poll changed the question order, adding new questions that appeared just before the job approval item at the beginning of the questionnaire:

1. I'd like you to compare the way things are going in the United States to the way they were going five years ago. Generally, would you say things are going better today, worse today, or about the same today as they were going five years ago?
     18% Better, 60% worse, 19% same, 3% DK/NA

2. And what is your best guess about the United States five years from now? Generally, if things go pretty much as you now expect, do you think things will be better, worse, or about the same as they are today?
     31% Better, 37% worse, 27% same, 5% DK/NA

These questions have appeared on CBS/New York Times surveys before, but not since January 2005. In the previous instances (January 2004 and January 2005), the questions about the direction of the country preceded the job approval question. I exchanged emails with Kathy Frankovic, director of surveys for CBS News, and she explained that "we have usually put the [state of the country questions] first (so people won't automatically connect the time frame to the President)." Most pollsters (including yours truly) use the same rationale for asking the "right direction, wrong track" question up front, just before the president's job approval rating.

Of course, it is also possible that the ordering of the most recent survey primed negative attitudes about the direction of the country and knocked down the Bush approval rating a point or two. Similar to the issue raised yesterday, without a controlled experiment that systematically compares both orderings on a survey with an enormous sample size, it would be impossible to know for certain.

Meanwhile, both surveys show Democrats holding a wide lead on the generic Congressional ballot among registered voters: The advantage is 15 points on the CBS/New York Times poll (50% to 35%), and 10 points on the LA Times/Bloomberg poll (49% to 39%). CBS also reported "similar" results among the smaller subgroup of those who say they will "definitely vote" - Democrats lead 50% to 37%. The "definite" voter subgroup is *not* equivalent to the more rigorous CBS likely voter model, which they typically use when reporting surveys conducted closer to Election Day.

Keep in mind that the result among registered voters on yesterday's USA Today/Gallup poll had the Democrats ahead by nine points (51% to 42%) among all registered voters. It was only the results obtained when applying their likely voter model that showed a significantly closer race.

More on that USA Today/Gallup Poll

Topics: 2006 , George Bush , Likely Voters , Measurement , Slate Scorecard , The 2006 Race

Our update to the Slate Election Scorecard yesterday tries to put the results for the generic congressional vote from the USA Today/Gallup survey into some perspective. It also reintroduces the controversy over likely voter models in general with specfic attention to the Gallup likely voter model. More on that below. Given the obviously high interest in this particular survey, as reflected in the sometimes heated debate in the comments section yesterday, I want to first share some of my own reactions.

First, remember it's just one poll. One of the inherent weaknesses in political polls is that they come with a lot of built in variation. Some comes from interviewing a sample rather than the whole population. Some comes from other methodological differences across surveys. As such, it is always better to look at more surveys than few. We like to average results across polls, despite some theoretical shortcomings, for just that reason.

In hindsight, the single discordant poll my be just a random statistical outlier, but not always. Sometimes it can be the proverbial "canary in the coal mine" that warns us of some new and emerging trend. So we pay attention to polls like yesterday's Gallup Poll, even if we typically recommend caution in interpreting them. 

Second, let's put aside the likely voter conundrum and focus on the larger sample of adults and compare the Bush job approval rating among all adults to trends on other surveys. I have updated the table from Monday's post below, and I averaged the two Gallup polls conducted in September to try to make the data as comparable across pollsters as possible.

The pattern is now strong and obvious: While the precise level of approval shows the usual variation across pollsters, eight of the nine pollsters show some small increase in the Bush job rating between August and September. That is a highly improbable result by chance alone, analogous to flipping a coin and having it come up heads eight of nine times (roughly 2% according to my favorite binomial calculator).

Third, consider one issue that everyone overlooked except one very alert MP reader: On previous Gallup polls, the Bush job rating came first on the questionnaire, or at least before questions about congressional vote preference. This is the first pre-election poll in which Gallup switched the order, asking the congressional ballot question first and then the Bush job rating.

This practice is not unusual. Media pollsters frequently juggle the order of questions with events, especially those that conduct surveys on a wide variety of topics year-round. They will generally try to position the most important (or newsworthy) questions first to reduce the chance of bias. The problem is that in moving questions around, they sometimes introduce some unforeseen new bias that unintentionally skews a time series trend.

We have no way to know whether that happened on the latest Gallup poll (absent a controlled experiment**), but it is certainly possible the change in question increased the Bush approval rating by a few points.

Fourth, as many comments on yesterday's post have noted, the 48% to 48% tie in the generic Congressional ballot question was based on the sometimes controversial Gallup likely voter model. Our Slate update yesterday provided a quick and dirty summary:

Ideally, pollsters and pundits prefer to watch likely voters because, well, they're more likely to vote than those who are simply registered. But identifying the likely electorate is much more difficult when an election is still months away, because respondents are less able to honestly assess whether they're really going to vote. (Getting a large enough sample of likely voters also costs more money, so media pollsters usually wait until closer to the election.)

The problem is that once pollsters start screening for likely voters, their methodologies vary widely. This produces the scattershot results we've seen recently. An AP-IPSOS poll conducted last week showed likely voters preferring the Democrats by a 14-point margin (53 percent to 39 percent). Other surveys conducted over the last two weeks by Zogby, Harris, and Fox show results that were more encouraging for Republicans but not the even split that Gallup shows. Further, the Gallup poll's likely voter model has been criticized for producing volatile results, especially when used a month or more before the election.

I have written extensively about the way pollsters choose likely voters. For those without time to read it all, the key point is that while evidence shows the Gallup likely voter model typically provides a better estimate of the vote on the final poll than looking at all registed voters, it can produce a lot of volatility before October.   As Mickey Kaus put it yesterday, the model may tell us more about:

'who would vote if the election were held today' as opposed to what we really want to know, which is 'whom would the people who are going to vote on November 7 vote for if the election were held today.'

Finally, consider that we are obsessing over measures with very limited utility to predict election outcomes -- the generic congressional ballot and presidential approval rating. In the races for Senate, on the other hand, we have direct measures of the actual contests and far more surveys to compare.  Our Slate Senate Scoreboard has logged 36 new state-level Senate polls released in September alone. And despite the modest increase in the Bush job rating, as of today, these continue to indicate momentum toward the Democrats.  Of course, the Scorecard also shows 49 seats currently held or at least leaning Republican, with 46 held or leaning Democrat.

With seven weeks remaining until the election, the data above should leave no one feeling too confident about the outcome. All of these trends can and probably will change. We'll be watching, so stay tuned...

About that ARG CT Poll...

Alert reader AL noticed what appeared to be an error in our "Most Recent Polls" update on the latest Connecticut Senate poll from American Research Group (ARG) poll.  We reported 3% voting for Republican Alan Schlesinger, but the ARG release indicates 3% for "other."  Their question asked about five candidates:  "Ralph Ferrucci, of the Green party, Timothy Knibbs, of the Concerned Citizens party, Ned Lamont, the Democrat, Joe Lieberman, of the Connecticut for Lieberman party, and Alan Schlesinger, the Republican."   

Actually, both numbers are correct.  We had already called ARG to check on the percentage voting for Schlesinger (since we plot no trend line for "other"), and they informed us that 3% supported Schlesinger.  The other category also displays as 3% in their release, since the preference for the two other candidates is so small that the combined result for Schlesinger, Farrucci and Knibbs still rounds to just 3%. 

And as long as we are on this subject, a note to all the other alert readers who have emailed:  Yes, we know that Joe Lieberman is running under the "Connecticut for Lieberman party" and not as an "independent."    Our use of "(i)" and independent labels was a programming compromise we adopted in putting this site together.   Labeling third parties correctly is on our short list of features to update soon. 

Also, we know that there are many other third party candidates not included in our charts.  In most cases, this is because pollsters are not asking or reporting results separately for such candidates.  We are hoping to expand our listing to include third party candidates where data is available. 

Gallup: Confirms Rise in Bush Job Rating

The latest USAToday/Gallup poll out this morning (USAToday story, results, Gallup summary) confirms the modest increase in the Bush job rating in evidence in some (but not all) recent polls conducted in September.  Among all adults, Bush's job rating approval rating has increased to 44%, which Gallup characterizes as "an improvement compared with the public's assessment of his performance in recent months" and "his highest rating so far this year." 

The new survey also provides results to the generic Congressional ballot question among likely voters.  While the Democrats lead among all registered voters (51% to 42%), the result is dead even (48%44% each) among those classified by Gallup as likely voters.

This is one poll, of course, subject to the same random variation as any poll. The Gallup summary notes how "fluid" the Bush job rating has been ("measuring as high as 42% in mid-August, but dropping back to 39% earlier this month").  As has been Gallup's recent practice, the summary also includes a three-poll rolling average and a "smoothed estimate" based on the Samplemiser program developed by Yale Professors Donald Green and Alan Gerber.  Both estimates suggest that Bush's current job rating has increased but to a slightly lower level (42%) than the result of this most recent poll.

Clearly anticipating the first question many will ask, however, the Gallup summary also includes this important point about party identification:

The improvement in Bush's ratings appears to result from a more positive evaluation of him from all party groups, rather than a short-term shift in more basic party loyalties. In the current poll, 34% of Americans identified as Democrats and 31% as Republicans. In the prior September poll, when Bush had a 39% approval rating, 35% identified as Democrats and 30% as Republicans. 

Both the new Gallup summary and a companion analysis that suggest the "Republican strategy on terror . . . may be having an effect," are free to non-subscribers for today only. 

IA Governor: A Toss-up

Topics: 2006 , Slate Scorecard , The 2006 Race

Our Slate update last night focuses on changes to the Election Scorecard for Governor's races over the last few weeks.  The most notable difference is a shift of Iowa from lean Democrat to toss-up.  Read it all. 

Momentum Shift?

Topics: 2006 , George Bush , Slate Scorecard , The 2006 Race

Our last Slate Election Scorecard update on Friday night shows a shift in national momentum for the first time, based on recently improving Democratic fortunes in states like Tennessee, Virginia, Missouri and Washington. These gains have occurred despite the small upward trend in the Bush job approval rating as seen on recent national surveys, as noted in Charles Franklin's post on Friday. How could these trends be moving in opposite directions?

Our Slate national momentum meter is an average of the averages of recent horserace results in each of thirteen competitive contests we have included on the Slate Scorecard. Much of the shift appears to come from states where Democratic candidates have benefited from issues or tactical advantages specific to those states. These include George Allen's "macaca" gaffe in Virginia, Harold Ford's post-primary spending advantage in Tennessee, the DUI revelations regarding Republican Mike McGavick in Washington.

At the same time, the more recent improvements in the Bush job rating are slight and may have less immediate impact on the attitudes of likely voters in the key races states.

It is worth considering the question raised by several astute readers in comments over the weekend: Could the improvement result from a recent shift by some pollsters from reporting results among all adults to likely or registered voters?

I did a quick "apples-to-apples" comparison table to show August to September comparisons among individual pollsters reporting comparable numbers. The results in the chart below, tend to rule out that explanation:

Only Fox News reported results that are not comparable (likely voters in September but only registered voters in prior months). The other organizations all reported comparable numbers (at least on the PDF releases with complete results). As the table above shows, most of the August to September change was small - too small in most cases to be statistically significant for any one survey. However, five of the seven organizations show at least some improvement and the overall average indicates a two point increase in the Bush job rating.

Incidentally, while the Rasmussen automated survey showed an upward spike in the Bush job rating that peaked in the three days following the September 11 anniversary (47% approve, 50% disapprove), it has since subsided to roughly the same level (41%-58%) as measured in August (40%-58%). Consider that the two September surveys showing the biggest changes (Zogby & AP-IPSOS) were also conducted immediately after the 9/11 anniversary.