Pollster.com

May 6, 2007 - May 12, 2007

 

Bush Approval: AP at 35%, Trend at 34.8%

Topics: George Bush

1BushApproval2ndTerm20070509.png

A new AP/Ipsos poll taken 5/7-9/07 is out today with approval at 35% and disapproval at 61%. With this addition my approval trend estimate stands at 34.8%.

There is not much to discuss here. The approval trend estimate has remained in the 33-35% range since January, and is still there. Once or twice we've bumped into those limits, only to promptly return to the range. At the moment we see a slight upward trend, but earlier this week saw a slight DOWNward trend. So I'd say the evidence so far gives no reason to think approval is doing anything other than remaining flat, with poll to poll variation bumping the trend up or down briefly.

Everything looks pretty normal in the diagnostics below, so I'll not add any more commentary. See the earlier posts on approval this week for more analysis, all of which holds true in light of today's AP poll.

2LastSixPolls20070509.png

3BushResiduals20070509.png

4BootApproval20070509.png

5Sentitivity2ndTerm20070509.png


POLL: InsiderAdvantage SC Primary


A new InsiderAdvantage/Majority Opinion automated survey of registered voters in South Carolina (conducted 5/8 through 5/9) finds:

  • Among 500 Democrats, Sen. Barack Obama runs at 31%, Sen. Hillary Clinton runs at 27%, and former Sen. John Edwards runs at 16% in a statewide primary.

  • Among 500 Republicans, former Mayor Rudy Giuliani leads former Speaker Newt Gingrich (22% to 15%) in a statewide primary; former Gov. Mitt Romney trails at 10%, Sen. John McCain at 9%, and former Sen. Fred Thompson at 8%.


Iraq War: Losing or Lost?

Topics: Iraq

Thanks to Washington Post blogger Dan Froomkin for tipping me off to this exchange from yesterday's press briefing by presidential press secretary Tony Snow that leads indirectly to some fundamental questions regarding public opinion and the Iraq War:

Q Tony, you mentioned the polls, and talked about the Republican support. All the polls also show that big majorities of the American public do not support the war. Have you heard the President talk about how difficult it is to fight a war or prosecute a war without the public's support?

MR. SNOW: The President understands the importance of public support. What's also interesting is that you see numbers coming up again on, do you think we're winning or do you think -- for instance, a pretty strong majority now, when asked, do you think we're losing, say no. That's an important data point. When it talks about, would you like the Americans to succeed, the answer is yes. So you always have mixed feelings.

[Emphasis added]

Is Snow right? Have the numbers really "come up" that much? Snow is presumably thinking about two nearly identical results asked on two recently released national polls:

  • CNN/ORC (5/4-6, n=1,208 adults): "Do you think that the U.S. war in Iraq is lost, or don't you think so?"

41% lost
55% don't think so
4% don't know

  • Quinnipiac (4/25-5/1, n=1,166 registered voters): "Do you think that the U.S. war in Iraq is lost, or don't you think so?"

41% lost
49 don't think so
11% unsure

Putting aside a quibble about whether those results amount to a "strong majority," they certainly show more Americans rejecting than accepting the notion that the war "is lost." The pollsters asked these questions after Democratic Majority Leader Harry Reid told reporters on April 19, "I believe ... that this war is lost, and this surge is not accomplishing anything."

As first glance, these new results certainly seem like a big change from a question traced by the ABC News/Washington Post poll: "All told, do you think the United States is winning or losing the war in Iraq?" As recently as January, a nearly two-to-one margin (57% to 29%) believed the U.S was "losing," a five point increase in the "losing" percentage since December:

05-11%20Post%20ABC%20Jan.png

So have opinions on the war changed really that much in just a few months? I doubt it. The more likely explanation is that, in the context of the Iraq War, some Americans are interpreting the "losing" and "is lost" very differently.

Consider the most recent results of questions tracked by the Pew Research Center and CBS News:

  • Pew Research Center (4/18-22, n=1,508 adults): "How is the U.S. Military effort going in Iraq?"

7% very well
31% fairly well
34% not too well
25% not well at all
3% don't know

  • CBS News (4/9-12, n=994 adults): "How would you say things are going for the U.S. in its efforts to bring stability and order to Iraq? Would you say things are going very well, somewhat well, somewhat badly, or very badly?"

2% very well
29% somewhat well
30% somewhat badly
36% very badly
3% unsure

So both organizations, in surveys conducted just before and just after Reid's remarks, show the same general pessimism reflected in the January ABC/Washington Post "is losing" results. By roughly two-to-one margins, Americans say the War effort is going "badly" or "not well."

More important, while both CBS and Pew show a slight up-tick (roughly six percentage points) in March and April as compared to December through February, the long term trend has been to greater pessimism. And the current overall results essentially match the reading of the January ABC/Post poll about the direction of the war. The following chart shows the results for the Pew survey. The CBS trend data (available via Polling Report) shows the same pattern.

05-11%20Pew%20Iraq%20Well_sml.png

Again, my hunch is that Americans interpret the words "is lost" very differently from "is losing," though the bigger issue is whether this semantic distinction has political consequences. A lot of conservative commentators seem to think so, given their reaction to Reid's remarks. Tony Snow is likely right that most Americans want our military to succeed in Iraq. However, if the seemingly contradictory results cited above are about a subset of American who believe that the war is not winnable yet perhaps not yet completely "lost," then we have a distinction without much difference.

Consider, for example the recent NBC/Wall Street Journal Poll (4/20-23, n=1,004 adults) that reported 55% saying it is "not possible" to achieve the U.S. goal of "achieving victory in Iraq." Now recall that academics that have studied public support for wars past and present - including one that has been advising the Bush White House - believe that "prospective evaluations of mission success" are the key to continuing support for the Iraq War.

All of this suggests some things my media pollster colleagues might want to test on future surveys. First, the issue of whether my hunch is right regarding the meaning of "losing" vs. "is lost." A survey could confirm this easily enough with an experiment that divides the sample randomly for a side-by-side test of two questions:

  • Do you think that the U.S. war in Iraq is lost, or don't you think so?
  • Do you think the United States is winning or losing the war in Iraq?

It might also be useful to try to get individual respondents to decide whether "lost" or "is losing" better describes their opinion. In other words, ask if they believe the U.S. "has won," "is winning," "is losing" or "has lost" the Iraq war. Do those who believe the U.S. "is losing" in Iraq but has not yet "lost" see any hope of winning in the future?

UPDATE: Professor Franklin sends along a slightly improved version of my "how are things going" chart of Pew Research Center data that adds the very similar results from CBS News and plots a smoothed regression line through the points. Click the image below for the easier-to-read full size version.

05-11%20Franklin%20-%20HowWellIsWarGoing_small.png


POLL: AP-Ipsos Approvals


A new AP-Ipsos national survey (story, results) of 1,000 adults (conducted 5/7 through 5/9) finds:

  • 35% approve of the job President Bush is doing; 62% disapprove.

  • 35% approve of the job Congress is doing; 60% disapprove.

  • 45% approve of the job Speaker Nancy Pelosi is going; 42% disapprove.


Even Polls About Baseball...

Topics: Sampling

Let's take a break from political polling controversies and focus on something of hopefully broader interest. A controversy involving a poll on baseball.

Larry Brown, who blogs MLB Clubhouse on AOL, posted scathing criticism a few days ago of a poll on Barry Bonds released over the weekend by ESPN and ABC News (hat tip to alert reader David Pinto of Baseball Musings). Brown sees evidence of a cooked poll:

If anyone actually bothered to read the poll, it says an oversample of 203 African-Americans were questioned out of 799 baseball fans. Considering only 12.8% of the population is African-American (according to the U.S. Census Bureau), and potentially a far lower percentage of baseball fans are African-American, I would say that the 25% mark of African-Americans used to conduct the study is designed to mislead the public and generate racially charged results.

Not exactly.

Pollsters sometimes "oversample" a survey sub-population in order to increase the reliability of the results for that group. More interviews means less potential random sampling error. Before tabulating the data for the full sample, however, they "weight" back the oversample its correct proportion with the larger sample.

I checked with Gary Langer, the director of polling at ABC News, and he provided a few additional details. The ABC Polling Unit started with a nationally representative sample of 1,803 randomly selected adults interviewed between March 29 and April 4. Of these, 660 described themselves as baseball fans (on the survey's first question). Of these, 64 were African-American.

The pollsters wanted a bigger and more reliable sampling of African-Americans. So they continued calling from April 5 to April 22 and interviewed another 476 randomly sampled African Americans, of whom 139 were self-described baseball fans.

Thus (adding everything up), the ESPN/ABC survey interviewed 799 baseball fans, including 203 among African Americans. Before tabulating the data, however, they weighted the combined sample of 2,279 (the original 1,803 plus the oversample of 476 blacks) in a way that reduced the proportion of African-Americans to its correct value as determined by the U.S. Census.**

This practice is not at all unusual. The intent is to generate more statistically reliable results by race, not -- as Brown puts it -- to "generate racially charged results."

One thing I will say in Larry Brown's defense. The two-sentence methodology blurb at the end of the ESPN was not entirely clear. For one thing, it said that the total sample of 799 baseball fans included "an oversample of 203 African-Americans." Technically, it included an oversample that increased the sample of African American to 203 interviews.

More important - and here is a message for all who write poll releases - it included the term "oversample" without any description of the weighting procedure. A sentence like this one (quoting from Gary Langer's email reply to me) would help reduce the confusion:

The combined sample (1,803 gen pop, oversample of 476 blacks) was weighted to Census norms, reducing the proportion of African-Americans to its correct population value.

**The US Census reports African-Americans as 11% of all adults. Browns 12.% statistic is the percentage African Americans among the full population, including children.

P.S.: And speaking of ABC Polling Director Gary Langer, the site Freakonomics posted some intriguing comments from Langer yesterday on the subject of whether polls have historically overstated support for minority candidates. It includes this accurate and well-deserved compliment from Freakonomics author Steven Dubner:

Gary is a force of nature. He not only runs ABC's polling but has become the network's top cop for keeping bad data off the air, vetting many of the surveys, studies, and polls that producers and reporters plan to use in their stories. I don't know of any other news organization that has such a resource. I am sure he is occasionally a thorn in the side of a reporter who's dying to cite some sensationalistic study from some biased organization ... but as consumers of news, we are all the better for it.


Kaiser's In-Person Katrina Study


As Today's Washington Post reports on a new in-person survey of current residents of the New Orleans area conducted by the Kaiser Family Foundation. The survey, according to the release on the Kaiser site,

documents the devastating impact that Hurricane Katrina and the failure to respond quickly and effectively to it has had on the economic well-being, physical and mental health, and personal lives of the people of the New Orleans area. The survey also found a sharp divide in the way that African Americans and whites in the New Orleans area experienced the storm and perceive the recovery efforts, especially in hard-hit Orleans Parish. Future Kaiser surveys are planned in 18 months and 36 months to monitor progress and changes.

The methodology of this survey is as interesting as the findings, at least to polling geeks like me. Telephone surveys present some obvious challenges in the areas affected by Katrina, particularly in reaching those still living in trailers or who otherwise lack landline telephone service. As such, the the Kaiser researchers went back to basics and conducted in-person interviews using a "two-stage, stratified area probability sample" and a team of 41 interviewers. Their interviewers visited 456 randomly selected census areas and documented the physical condition of nearly 17,000 housing locations, then attempted to conduct interviews in randomly selected households. The methology is all explained here; the complete 100-page study is available for download here.


POLL: Quinnipiac Connecticut Primary/GE


A new Quinnipiac University statewide survey (08 results, Issues results) of 1,427 registered voters in Connecticut (conducted 5/2 through 5/7) finds:

  • Among 543 Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (28% to 20%) in a statewide primary; former V.P. Al Gore trails at 13%, former Sen. John Edwards at 8%.

  • Among 375 Republicans, former Mayor Rudy Giuliani leads Sen. John McCain (36% to 15%) in a statewide primary former Gov. Mitt Romney trails at 9%, former Sen. Fred Thompson at 7%, former Speaker Newt Gingrich at 5%.

  • General election match-ups:

    Giuliani 48%, Clinton 42%
    Giuliani 44%, Obama 42%
    Giuliani 45%, Gore 42%
    McCain 40%, Clinton 47%
    McCain 38%, Obama 46%
    McCain 41%, Gore 45%


POLL: Gallup's Clinton/Obama Support


New analysis from the recent USA Today/Gallup national survey of 491 Democrats and Democratic-leaners (conducted 5/4 through 5/6) looks at support for Sen. Hillary Clinton and Sen. Barack Obama and finds:

  • Among 265 Clinton supporters, 35% are more likely to support her over Obama because "Clinton is more experienced;" 23% "like Clinton's views on issues/agenda."

  • Among 194 Obama supporters, 19% are more likely to support him over Clinton because "Obama is a fresh face/has new ideas;" 18% "agree with Obama's issue politions/agenda."


POLL: Branding Republican Candidates in South Carolina


Additional results from the recent Chernoff Newman/MarketSearch statewide survey of likely Republican primary voters in South Carolina tests "branding" and vote preference of the Republican candidates. Mark Blumenthal's analysis of the release of the Democratic candidates here.


POLL: SurveyUSA Fred Thompson in 16 states


New SurveyUSA automated surveys of 500 registered voters in 16 states testing general election match-ups that pit former Sen. Fred Thompson against Sen. Hillary Clinton, Sen. Barack Obama, and former Sen. John Edwards (conducted 4/13 through 4/15) finds:

KentuckyThompson40Clinton53
KentuckyThompson34Edwards56
KentuckyThompson48Obama42
TexasThompson43Clinton49
TexasThompson38Edwards49
TexasThompson46Obama42
OhioThompson38Clinton53
OhioThompson33Edwards57
OhioThompson43Obama43
MissouriThompson41Clinton49
MissouriThompson32Edwards53
MissouriThompson41Obama47
IowaThompson44Clinton46
IowaThompson35Edwards58
IowaThompson41Obama51
New MexicoThompson41Clinton51
New MexicoThompson34Edwards52
New MexicoThompson40Obama47
KansasThompson49Clinton42
KansasThompson37Edwards50
KansasThompson42Obama46
VirginiaThompson47Clinton43
VirginiaThompson38Edwards48
VirginiaThompson46Obama40
AlabamaThompson49Clinton44
AlabamaThompson44Edwards42
AlabamaThompson53Obama37
New YorkThompson30Clinton64
New YorkThompson29Edwards59
New YorkThompson38Obama50
WisconsinThompson43Clinton46
WisconsinThompson37Edwards50
WisconsinThompson42Obama45
MinnesotaThompson37Clinton53
MinnesotaThompson32Edwards56
MinnesotaThompson40Obama48
MassachusettsThompson31Clinton60
MassachusettsThompson25Edwards61
MassachusettsThompson37Obama48
CaliforniaThompson36Clinton57
CaliforniaThompson31Edwards54
CaliforniaThompson36Obama53
WashingtonThompson37Clinton54
WashingtonThompson32Edwards57
WashingtonThompson37Obama53
OregonThompson41Clinton48
OregonThompson34Edwards49
OregonThompson36Obama50


POLL: Critical Insights Maine Senate


A new Critical Insights statewide survey of 600 adults in Maine (conducted 4/20 through 4/27) finds Sen. Susan Collins (R) leading Rep. Tom Allen (D-01) (57% to 30%) in a senatorial general election match-up.

UPDATE: Full results.


POLL: American Research Group CA/FL/MI


Three new American Research Group statewide surveys likely voters in California, Florida, and Michigan (conducted 5/4 through 5/8) finds:

  • Sen. Hillary Clinton leads Sen. Barack Obama in California (37% to 28%), Florida (45% to 17%), and Michigan (38% to 25%); Sen. John Edwards trails in each state with 15% or less.

  • Former Mayor Rudy Giuliani edges out Sen. John McCain (27% to 24%) in California and leads him (31% to 18%) in Florida. In Michigan, former Gov. Mitt Romney runs at 24%, McCain at 22%, and Giuliani at 19%

.


Coulter's Pollster Rant

Topics: Pollsters

Although we try hard to promptly report polling related new items, we missed a good one yesterday. Conservative commentator Ann Coulter dropped another one of her bon mots during a Fox News broadcast, this one aimed at the pollsters at Newsweek. On his show At Large on Monday, host Geraldo Rivera asked Coulter for a reaction to Newsweek poll results showing Barack Obama leading the various Republican contenders. As reported by News Hounds (via colleagues on the AAPOR list serv):

"I think this is Newsweek doing more push polling for Al Qaeda," Coulter told Geraldo Rivera. Rivera responded "You don't think they're making it up?" to which Coulter answered "Yes."

She went on to say that in polls "where Republicans are actually allowed to vote," they "do a lot better." thus suggesting that Newsweek did not call any Republicans.

News Hounds also linked to a YouTube video clip:

Not surprisingly, reaction from the pollsters was swift. Rob Daves, president of the American Association for Public Opinion Research (AAPOR), released the following statement yesterday:

"Ann Coulter's kill-the-messenger assessment of the Newsweek poll exceeds the bounds of professional reporting and commentary," said AAPOR president Robert P. Daves, director of polling and strategic research at the Star Tribune. "It's pure-and-simple mudslinging on her part to question the ethics of a reputable polling firm and news organization that abides by a longstanding Code of Professional Ethics and Practices. Anyone who values solid public opinion research and intelligent public discourse should be outraged and ignore Coulter's irresponsible and groundless assertion that the results were fabricated."

[Interests disclosed: I serve as a member of AAPOR's executive council].

And finally, Evans Witt, the chief executive officer of Princeton Survey Research, the survey firm that conducts the Newsweek poll, had this comment for reported by the Associated Press [that was originally posted to the AAPOR member-only listserv):

"As the 2008 election campaign continues to heat up, I am sure that there will be informed and incisive criticisms of polls from many observers," he said. Coulter's comments "do not fit into this category," he added.

When it comes to Ms. Coulter's various lunatic comments, I tend to agree with those say that "she, like any bully, will go away if you ignore her." On the other hand, read over some of the comments left on this site over the last few days and you will see that some people will believe anything. So for those tempted to confuse the valid skepticism expressed by reputable voices about the reporting of Newsweek poll with Coulter's rant, let's restate what should be obvious: Coulter's remarks about the Newsweek poll are unfounded and preposterous.


POLL: Public Mind Bush, Iraq


Additional results from the recent Fairleigh Dickinson Public Mind statewide survey of 648 registered voters in New Jersey (conducted 5/1 through 5/6) finds:

  • 28% approve of the way President Bush is handling his job; 65% disapprove.

  • 29% say the military effort in Iraq is going well; 67% say not well.


POLL: Strategic Vision (R) Wisconsin Primary


A new Strategic Vision (R) statewide survey of 800 likely voters in Wisconsin (conducted 5/4 through 5/6) finds:

  • Among Republicans, former Mayor Rudy Giuliani runs at 22%, Sen. John McCain and former Wisconsin Gov. Tommy Thompson both at 16%, and former Sen. Fred Thompson at 10% in a statewide primary.

  • Among Democrats, Sen. Hillary Clinton leads Sen. Barack Obama 38% to 25% in a statewide primary; former Sen. John Edwards trails at 17%.

  • More Democrats express satisfication with the current field of Democratic candidates (59%) than Republicans with theirs (21%).


POLL: CNN Bush's Veto


Additional results to the new CNN/Opinion Research Corporation national survey (story, results) of 1,028 adults (conducted 5/4 through 5/6) finds:

  • 54% disapprove President Bush's veto of a bill providing additional funds for Iraq with a specific date for complete troop withdrawal; 44% approve.

  • When asked to choose among three bills, 40% prefer "a bill that would provide additional funds for Iraq that would set benchmarks for the Iraqi government but not set a specific timetable for troop withdrawal;" 33% prefer "a bill that would provide additional funds for Iraq that would set a specific date to start withdrawing troops;" 24% prefer "a bill that would not provide additional funds for Iraq and would require all troops to be withdrawn by March."


POLL: InsiderAdvantage Bush/Cheney Impeachment


A new InsiderAdvantage/Majority Opinion national survey of 621 registered voters (conducted 5/4 through 5/6) finds 39% of Americans favor the impeachment of President Bush and Vice President Cheney; 54% do not.


POLL: WNBC/Marist 2008 Survey


A new WNBC/Marist national survey of 823 registered voters (conducted 4/26 through 5/1) finds:

  • 33% approve of the job George Bush is doing as president; 61% disapprove.

  • Among 279 Democrats, Sen. Hillary Clinton (at 35%) leads Sen. Barack Obama (17%), former Sen. John Edwards (15%), and former V.P. Al Gore (9%) in a national primary.

  • Among 227 Republicans, former Mayor Rudy Giuliani (at 28%) leads Sen. John McCain (18%), former Sen. Fred Thompson (13%), former Speaker Newt Gingrich (8%), and former Gov. Mitt Romney (7%) in a national primary.

  • General election match-ups show:

    Giuliani 43%, Clinton 48%
    Giuliani 43%, Obama 41%
    Giuliani 43%, Edwards 49%
    McCain 42%, Clinton 47%
    McCain 46%, Obama 39%
    McCain 39%, Edwards 49%


The Rudy Discrepancy?

Topics: 2008 , Divergent Polls , The 2008 Race

The mysterious result de jour is a bit of a conflict between the most recent USA Today/Gallup and CNN/ORC polls on the national Republican primary horserace result. The Gallup survey has shown Rudy Giuliani leading John McCain by 14 and 13 points on their last two surveys (34% to 20% on the survey released today). Meanwhile, CNN/ORC shows a much closer race, with Giuliani leading by only 3 points last month and 2 points (25% to 23%) now. Both organizations poll over exactly the same period this week, and nearly overlapping periods a month ago. So why the difference?

05-08%20gallup%20and%20cnn.png

Why the difference? I have been looking at the numbers and see no obvious explanation. The sample sizes are relatively small (with a reported margin of sampling error of roughly +/- 5%), but differences in the Giuliani vote are statistically significant, especially given the April-to-May consistency for both polls.

The most important difference between the two polls - as Hotline OnCall points out - is that the Gallup polls asks its Republican primary vote question among all adults that identify or lean Republican, while CNN asks it only of self-described registered voters that identify or lean Republican. I might expect such a difference to work in Giuliani's favor, on the assumption that he might be less popular among the more hard core Republican primary voters. But there are two problems with that theory. The first is that despite the labeling, the percentage of adult respondents that were asked the Republican vote question is only slightly smaller on the last two CNN surveys (40% and 39%) than on the last two Gallup surveys (42% and 43%).

05-08%20sample%20size.png

That result is a little odd, since other pollsters that screen for registered voters who identify or lean Republican tend to ask their Republican primary vote question of a smaller percentage of their samples. Looking at recent surveys that provided comparable details, I get 35% (of the sample asked the Republican primary vote question) for the Pew Research Center, 32% for LA Times/Bloomberg, 31% for Time/SRBI and 30% for the Cook/RT Strategies.

Moreover, if we look at the other results from comparable questions over the last month, the other pollsters that screen for registered voters (including those with effectively tighter screens than CNN) have mostly not shown the race as close as CNN/ORC.

05-08%20recent%20gop.png

Which makes these results a puzzle, except for this general rule of thumb: When very small (even invisible) differences in methodology make big differences in the results, it usually means that the underlying attitudes are not strongly held.

In other words, most Republicans nationally have only begun to consider the candidates, much less who they will support in 2008. So their answers to pollsters are showing great variation. When voters start to make up their minds -- next year -- these results will show more consistency.


Bush Approval: NPR at 37%, Trend at 34.7%

Topics: George Bush

A1BushApproval2ndTerm20070506.png

The polls are really rolling in yesterday and today and they provide a striking contrast and hence a good lesson.

The latest to arrive is actually a bit old but only just released. The NPR poll conducted 4/26-29/07 is actually the sixth oldest of the recent polls. It found approval at 37% and disapproval at 59%. The NPR poll is the product of a bi-partisan team of Greenberg Quinlan Rosner Research (D) and Public Opinion Strategies (R). They sample likely voters, rather than adults or registered voters. This sample population mixes with house effects to generally puts the NPR poll about 1.9 percentage points above the trend estimate. In fact, the trend estimate is now 34.7 and NPR is 37%, a 2.3 point discrepancy which is reasonably close to what the house effect would predict.

The interesting comparison and lesson is that the Newsweek story over the weekend stressed a new all time low in approval of President Bush while other polling has failed to find any strong downward trend (except for a Harris poll taken 4/20-23 that also had approval at 28%.) In my trend estimator, when Newsweek was the endpoint, the trend estimator was pulled down to 33.2%. I warned then that Newsweek was an outlier and that an estimate of 34 or so was more likely. The flurry of new polls since Newsweek help reinforce that conclusion.

Here are the last twelve approval polls in order:

38 33 35 35 28 32 37 35 35 28 34 38

The mean is 34% and the median is 35%. The two polls at 28% stand out pretty strongly from the rest.

The lesson for interpreting polls is that a single poll does not a trend make. The strong interpretation Newsweek gave its poll, and the extra hype it got on Sunday talk shows, was not supported by other evidence. I stress that I'm complaining here about interpretation of poll results, and not about the poll itself. The Newsweek poll was an outlier, but outliers happen and that should not lead to a conclusion of anything other than a bad random outcome about the poll. The news story that was based on the poll, on the other hand, took the findings and made a very strong claim about approval at a new low when other evidence is clear that approval has not fallen into the 20s. One simply cannot ignore all other polls when making claims about a trend.

The ultimate irony of this is that Newsweek is right: President Bush IS at an all time low for his presidency, and has been since January! His stable approval rate of 33%-35% in my trend estimator is indeed the lowest of his presidency, and he has shown no signs of recovery (though no sign of further decline either.) But that all time low for the trend is 33-35%, not 28%. The desire to pick each poll number that happens to be well below the trend estimate and emphasize how low it is misleads us from attending to the central trend estimate which has far more to say about the actual level of presidential support. Each poll is a random draw around the trend. Some will be lower, some higher. But that poll to poll variation is largely random. It is the trend that captures the systematic path of approval.

And so I conclude as I did originally about the Newsweek poll. The best estimate of approval remains in the 33-35 percent range. At the moment the trend is at 34.7, but it could easily drop a bit with new polls, or possibly rise a little. My money is on an average of 34, for the moment.

Below are the usual poll diagnostics. I commented on these more extensively in the last two approval posts, so I'll not say more about them here.

A2LastSixPolls20070506.png

A3BushResiduals20070506.png

A4BootApproval20070506.png

A5Sentitivity2ndTerm20070506.png

Cross-posted at Political Arithmetik.


POLL: NPR Bush, Primary Voters (GQR/POS)


A new National Public Radio national survey (story, results, highlights) of 1,000 likely voters (conducted 4/26 through 4/29 by Greenberg Quinlan Rosner Research (D) and Public Opinion Strategies (R) ) finds:

  • 37% approve of the way President Bush is handling his job; 59% disapprove.

  • 56% of Democratic primary voters say Congress and President Bush should pay the most attention to the "war in Iraq," while 30% of Republicans agree.

  • A 37% plurality of Republican primary voters say Congress and Bush should pay the most attention to "terrorism and national security," while 11% of Democrats agree.


POLL: PublicMind Corzine


A new Fairleigh Dickinson University Public Mind statewide survey of 648 registered voters in New Jersey (conducted 5/1 through 5/6) finds:

  • 58% have a favorable opinion of Gov. Jon Corzine (53% in March).

  • 45% think Acting Gov. Richard Codey should do "as he thinks Gov. Corzine wants" on budget and policy questions; 34% think Codey should do "as he thinks best."


POLL: Rasmussen 5/8 Round-Up


New Rasmussen Reports automated surveys find:

  • Among 760 likely Democratic primary voters, Sen. Hillary Clinton leads Sen. Barack Obama (34% to 26%) in a national primary; former Sen. John Edwards trails at 16%. Last week, Obama edged out Clinton 32% to 30% (conducted 4/30 through 5/3).

  • Among 620 likely Republican primary voters, former Mayor Rudy Giuliani (at 25%) leads Sen. John McCain (17%), former Sen. Fred Thompson (16%), and former Gov. Mitt Romney (12%) in a national primary (4/30 through 5/3).

  • Among 800 likely voters, Clinton and Giuliani both run at 45% in a nationwide general election match-up; Clinton edges out McCain 48% to 44% (4/30 through 5/1).

  • Among 800 likely voters, 47% would vote for the Democrat in their congressional district if the election were today; 36% would vote Republican (5/2 through 5/3).

  • Among 1,000 adults, 32% say they have been following the DC Madam story closely; 63% are not paying much attention to it (5/2 through 5/3).

  • Among 1,000 adults, 58% think a "truly competetive third party" would be good for the United States (5/4 through 5/5).


POLL: USA Today/Gallup National Survey


A new USA Today/Gallup national survey (USAT Bush story; '08 story, results; Gallup 2008 analysis, video, Iraq analysis, video, Bush) of 1,010 adults (conducted 5/4 through 5/6) finds:

  • 34% of Americans approve of the way President Bush is handling his job; 63% disapprove.

  • Among 427 Republicans and leaners, former Mayor Rudy Giuliani leads Sen. John McCain (34% to 20%) in a national primary; former Sen. Fred Thompson trails at 13%, former Speaker Newt Gingrich at 8%, former Gov. Mitt Romney at 7%.

  • Among 491 Democrats and leaners, Sen. Hillary Clinton leads Sen. Barack Obama (38% to 23%) in a national primary; former V.P. Al Gore trails at 14%, former Sen. John Edwards at 12%. Without Gore, Clinton leads Obama 45% to 27%.


Bush Approval: 3 New Polls, Trend at 34.3%

Topics: George Bush

1BushApproval2ndTerm20070506.png

Three new polls put President Bush's approval at 34% or above, and bring my trend estimate to 34.3%. The Hotline poll, taken 4/26-30/07 but only released yesterday, has approval at 35%, disapproval at 62%. USAToday/Gallup, 5/4-6/07 has approval at 34%, disapproval at 63%. CNN/Opinion Research Corp., 5/4-6/07 has approval at 38% and disapproval at 61%.

These results contrast with the weekend release of a Newsweek poll at 28%. Newsweek's story on the poll stressed the new low rating for President Bush. A Harris poll some 10 days earlier had also pegged approval at 28%. However, other recent polls at 32-35% failed to offer evidence for a sharp decline in Bush's approval. With the addition of these new polls at 35%, 34% and 38%, the trend estimate has returned to "flat", continuing a long period of stable approval in the 33-35% range which has persisted since December.

The recent polls are plotted below. The CNN poll at 38% is a good bit above the trend estimate, while Gallup and Hotline are quite close. The CNN result is also a bit high compared to previous CNN polling which generally tracks the trend quite closely.

2LastSixPolls20070506.png

The addition of these new polls brings the trend estimate up, making the previous Harris and Newsweek surveys look like larger outliers than before. The upward shift also keeps the rather high CNN estimate within the 95% confidence interval, meaning that while it is high it is not a statistical outlier.

3BushResiduals20070506.png

The current trend remains within the uncertainty range of the trend estimate. The current uncertainty, with polls ranging from 28-38 percent, is rather large but not suspiciously so.

4BootApproval20070506.png

Finally, the various estimates of trend for the last 20 polls shows a strong clustering in the 34-35 percent range, so the evidence remains good for approval in this narrow range.

5Sentitivity2ndTerm20070506.png

Cross-posted at Political Arithmetik.


POLL: SurveyUSA California Primary


A new SurveyUSA automated survey of likely primary voters in California (conducted 5/4 through 5/6) finds:

  • Among 456 Republicans, former Mayor Rudy Giuliani leads Sen. John McCain (34% to 21%) in a statewide primary; former Gov. Mitt Romney trails at 12%, former Sen. Fred Thompson at 11%, and former Speaker Newt Gingrich at 9%.

  • Among 667 Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (48% to 27%) in a statewide primary; former Sen. John Edwards trails at 15%.


Gallup: Bush Job Approval at 34%

Topics: George Bush

Frank Newport is breaking news on his Gallup Guru blog that the latest USA Today/Gallup job approval rating for President Bush is 34%. Earlier, Newport voiced some caution about the "all time low" in Bush approval in this weekend's Newsweek poll, including a link to similar advice from Charles Franklin. USA Today and Gallup should have more details tomorrow, and we will post all the links.


POLL: CNN Bush Approval, 2008


More results from a new CNN/Opinion Research Corporation national survey (story, results) of registered voters (conducted 5/4 through 5/6) find:

  • 38% approve of the way President Bush is handling his job as president; 61% disapprove.

  • Among 414 Republicans, former Mayor Rudy Giuliani edges out Sen. John McCain (25% to 23%) in a national primary. Among 454 Democrats, Sen. Hillary Clinton leads Sen. Barack Obama 38% to 24%.

  • 70% of Republicans are satisfied with the field of Republican candidates; 26% are dissatisfied.


Did Nixon Win With Radio Listeners?

Topics: Debates

As long as we are on the subject of debates, I want to pick a nit with something Joe Klein noted in his assessment of the Republican debate last Friday:

Here's my gimmick: I listened to the debate on radio. Not that I wanted to...but I was on a forced march, driving north. I did this fully aware that Nixon "won" the famous debate with JFK among radio listeners as he was losing the presidency on TV.

My issue here that while entirely plausible, the evidence that Nixon "won" among radio listeners is about as sketchy as the evidence that Giuliani "won" among Californians last week. I don't mean to pick on Joe Klein. Two Colorado State University academics who reviewed the literature 20 years ago found that assertions Richard Nixon winning among radio listeners "prevail in nearly all accounts of the first Kennedy-Nixon debate" (Vancil and Pendell, 1987).

It turns out that aside from a few anecdotal accounts from journalists, only one true survey attempted to gauge reactions to the debate among both radio and television audiences. It involved a few questions asked on a larger omnibus telephone survey conducted by Sindlinger and Company, and the cross-tabulation by audience (television and radio) was published only once, in the November 7, 1960 issue of Broadcasting (from Kraus, 1996, p. 80):

Kennedy supporters may be grateful that television was invented before the "Great Debates" took place. The Sindlinger research showed that Mr. Kennedy was routed by Mr. Nixon on radio.

In answer to the question who won the debates, 48.7% of the radio audience named Mr. Nixon and only 21% picked Mr. Kennedy. Among those who watched the debates on tv, 30.2% named Mr. Kennedy the winner and 28.6% picked Mr. Nixon.

According to the Sindlinger projections, the total television audience was about 4 - ½ times the radio audience - 270 millions viewers of 5v to 61.4 million listeners to radio.

That result, however, has a few problems, the most important of which is the relatively small size and unrepresentative nature of the radio audience. It amounted to must 282 responses from Sindlinger's sample of 2138 respondents, but the pollster apparently misplaced the original data, because no information survived regarding the partisanship or vote preference of the radio or television subgroups. That omission is critical because, as Steven Chafee, a professor of communications at the University of California Santa Barbara has observed (2000, p. 334):

By 1960, those who could listen to debates only on radio were far from a random lot. Situated for the most part in remote rural areas, they were overwhelmingly Protestants and skeptical of Kennedy as a Roman Catholic candidate.

University of Minnesota Political Scientist James N. Druckman, sums up (p. 563):

Put another way, relative to television viewers, radio listeners may have been predisposed to favor Nixon over Kennedy. This lack of reliable causal evidence means that a prime example of the power of television images may be nothing more than "telemythology" (Schudson, 1995, 116).

Or is it? After all, much of the power of this debate anecdote is that most observers and commentators thought Nixon lost the debate to Kennedy on the basis of his appearance. It certainly seems plausible that had the same debate occurred on radio, Nixon may have fared better.

Druckman took the matter one step further. In the article quoted above, he describes an intriguing experiment conducted about five years ago. He recruited 171 respondents - mostly students - who demonstrated little or no knowledge of the Kennedy-Nixon debates. He then randomly divided the subjects into two groups. Half watched a video tape of the first Kennedy-Nixon debate and the other half listed to just the audio. All later answered questions about Nixon and Kennedy, yielding the results Druckman had expected (pp. 569-570):

Television images matter - they prime people to rely more on personality perceptions when evaluating candidates, which in turn, can affect overall evaluations. Images also enhance political learning, at least among nonsophisticates. The experiment provides evidence that Kennedy may have done better on television because of his superior image.

So there we have it. A great example of how not to interpret a post-debate survey, and a clever experiment by a social scientist that provides a bit of evidence to support the underlying truth beneath the well established "telemyth."

References used after the jump (links are to subscriber only databases).

Continue reading "Did Nixon Win With Radio Listeners? "


POLL: PPP (D) NC Presidential Primary


A new Public Policy Polling (D) automated survey of likely primary voters in North Carolina (conducted 5/1 through 5/3) finds:

  • Among 621 Republicans, former Mayor Rudy Giuliani leads former Sen. Fred Thompson (32% to 25%) in a statewide primary; Sen. John McCain and former Gov. Mitt Romney trail at 16% and 13% respectively.

  • Among 572 Democrats, former Sen. John Edwards edges out Sen. Hillary Clinton (33% to 27%) in a statewide primary; Sen. Barack Obama trails with 20%.


Partisan variation in 2007 polling

Topics: Measurement

1PIDBoxPlots.png

The Newsweek poll released this weekend was quickly criticized for under-representing Republican partisans. For the adult sample, Newsweek had 22% Republicans and 35% Democrats, while for registered voters their numbers were 24% and 36% for Reps and Dems. Independents were 36% of adults and 37% of registered, with 7% of adults and 3% of registered refusing or unable to answer the question.

How out of line are these results? The graph above shows the distribution of party id for all polls taken since January 1, 2007. The groups are for three populations (Adults, Registered Voters and Likely Voters) and for whether the party ID question is "leaned", that is does it include those who say they "lean" to one party or the other, after first saying they are independent or non-partisan. In the legend for the graph A.N is adults, not leaned, RV.L is Registered Voters, Leaned, and so on.

It is clear that the distribution of Republicans, Democrats and independents depends quite a bit on the population and the question wording. For the adult, not leaned, sample that Newsweek used for presidential approval, the 22% Republican they found is at the low end of samples in 2007, where the median is 25%. Among registered voters, not leaned, the median in 2007 is 29%, compared to Newsweek's 24%. For Democrats, the Newsweek sample at 35% for adults compared to a median of 35%, while among registered voters their 36% compares to a median of 35%.

So the Newsweek results are a bit low (-3%) for adult Republicans, and a bit lower (-5%) among registered voters, but quite close to the median for Democrats. While the adult Republican number is the lowest of the year in comparable samples and questions, it is only 3 points from the median. That is indicative of the relatively low spread of estimates of Republicans in the adult population this year, where half of all polls fall between 24% and 28% and only 3 of 24 polls fall above 30%. Among registered voters with unleaned questions, half of all polls fall between 27% and 32%, with four polls above 32% and six polls below 27%. Newsweek's registered voter sample is the second lowest reading for Republicans of this group.

The relationship between percent Republican and percent Democrat is not terribly strong within categories of population and question wording, as shown below.

2PIDRepByDemByPopByQ.png

If anything, there is a positive correlation of the two partisan groups, ranging from .44 to .81 for adults and registered voters. (The four cases of likely voters have little variation and produce negative correlations, and I discount this.) The positive correlation is a good indication that what is at work with partisanship is that some questions or survey organizations produce more or fewer independents, driving down or up both Reps and Dems simultaneously.

The variation we see above is a reminder that party identification is a central concept, but subject to quite a lot of variation in measurement depending on population and question wording (and probably "house effects" due to survey organization, a topic ignored in this post.)

Given the sample size of each partisan group, the range of variation we see across polls is not surprising, and that raises an important point. One of the first things we ask about a poll is what its partisan split is. That is an important diagnostic to have available, and Newsweek deserves to be praised for openly disclosing this bit of data. All pollsters SHOULD release this information but a number do not. The current criticism of Newsweek is exactly the disincentive these organizations have for such release. If your party id is even a bit off, it becomes an immediate target for criticism and partisan cries of "fraud". Such is an uninformed view. Normal sampling variation will produce a range of party id estimates, and the data above show that in general this variation is well behaved, doing about what we'd expect given the sample sizes involved. But it does provide a tempting target for anyone unhappy with a particular poll.

This is not to say we should not scrutinize pollsters based on their data. That is entirely what we do here, of course! And I think I was the first to point out that the latest Newsweek poll appears to be an outlier on presidential approval. But that conclusion was based on a great deal of evidence from over 1400 polls and a consistent methodology I apply to every new poll regardless of source and regardless of results.

Finally, it is easy to overstate the impact of the distribution of party id on other survey estimates, such as presidential approval or horserace results. While party id is a powerful individual level predictor of those things, a discrepancy of 3-5 percent in partisanship, as in this case, does not translate directly into a similar discrepancy in every question with partisan overtones. Partisans are not completely homogeneous in their views, so the impact of variation in party id is mitigated by the variation within partisan category in approval or candidate preference. If we want to analyze a poll's results, it is better to focus directly on the question of interest, rather than on another variable like party id which may influence but not entirely determine the result we care about.

Cross-posted at Political Arithmetik.


POLL: CNN Gas Prices, Queen


A new CNN/Opinion Research Corporation national survey of 1,028 adults (conducted 5/4 through 5/6) finds:

  • 45% think Britian would be "worse off" without a royal family; 41% say "better off."

  • 79% think it is likely they "will have to pay four dollars per gallon for gasoline" sometime this year; 21% say not likely.


POLL: SurveyUSA New Hampshire Primary


A new WBZ-TV/SurveyUSA automated survey (story, results) of likely primary voters in New Hampshire (conducted 5/4 through 5/6) finds:

  • Among 551 Republicans, former Gov. Mitt Romney (32%) leads former Mayor Rudy Giuliani (23%), Sen. John McCain (22%) and former Sen. Fred Thompson (11%) in a statewide primary. In January Giuliani ran at 33%, McCain at 32%, and Romney at 21%.

  • Among 589 Democrats, Sen. Hillary Clinton (at 40%) leads Sen. Barack Obama (24%) and former Sen. John Edwards (22%) in a statewide primary.


Asking About Obama's Race

Topics: 2008 , The 2008 Race

The 2,500-word examination of Barack Obama and race in today's Washington Post touches briefly on the issue of whether polling, "once" considered "wildly unreliable" in races involving black and white candidates might be more trustworthy than in past elections. It quotes Scott Keeter of the Pew Research Center, who co-authored a report on the issue back in February.

But what caught my eye is a bit of overlooked survey data relevant to the larger story. Consider the tongue-in-cheek reaction to the Post story this morning from Time's Ana Marie Cox:

Barack Obama is black. No, seriously, he is.

Washington Post is ALL OVER this story.

But believe it or not, not all Americans know that Obama is black. Here is the result released just last Friday from a national sample of adults surveyed by Quinnipiac University (Q33):

What race do you consider Barack Obama?

2% White
45% Black
0% Asian
7% Mixed/other
9% Mixed Black and White
37% Don't know

The results to this question show virtually no variation by party. Among Democrats, 48% consider Obama black, 7% mixed/other, 7% mixed black and white.

One thing unclear as of this writing: Did interviewers read the two "mixed" categories to respondents or did respondents volunteer those answers? I will check on that, but if "mixed" was not presented as an option, some confusion is to be expected given Obama's famously "biracial and multinational background" (to quote the Post article). [UPDATE: Doug Schwartz at Quinnipiac emails that this questions was entirely open-ended and did not prompt respondents with any categories].

Meanwhile, there is a very intriguing cross-tabulation I would like to see: How much different are the results to the primary horserace question and the Obama favorable rating among the 63% of Democrats that answered the question correctly as compared to the 37% that did not. The Democratic sample is relatively small (n=499) but still large enough for such a comparison. Unfortunately, the survey has nowhere near enough interviews to allow the really interesting test: How African-American Democrats answered this question.


POLL: Quinnipiac Thermometer 18 Countries


Additional results from the most recent Quinnipiac University national survey of 1,166 registered voters (conducted 4/25 through 5/1) tests thermometer ratings among 18 different countries among men, women, republicans, democrats, independents, and white evangelical/born again Christians.


POLL: Mason-Dixon Nevada Primary


A new Review-Journal statewide survey of likely caucus-goers in Nevada (conducted 4/30 through 5/2 by Mason-Dixon Polling & Research) finds:

  • Among 400 300 Democrats, Sen. Hillary Clinton (at 37%) leads Sen. John Edwards (13%), Sen. Barack Obama (12%), and former V.P. Al Gore (9%) in a statewide primary.

  • Among 400 300 Republicans, Sen. John McCain edges out former Gov. Mitt Romney (19% to 15%) in a statewide primary; former Sen. Fred Thompson runs at 13%, former Mayor Rudy Giuliani at 12%, and former Speaker Newt Gingrich at 7%.


POLL: Newsweek National Survey


A new Newsweek national survey (story, results, Charles Franklin's analysis) of 1,001 adults (conducted 5/3 through 5/3 by Princeton Survey Research Associates) finds:

  • 28% approve of way Bush is handling his job as president, "the lowest presidential approval rating in a generation;" 64% disapprove.

  • More Americans say Sen. Hillary Clinton, Sen. Barack Obama, and former Sen. John Edwards have "shown political courage in their current campaign" than have former Mayor Rudy Giuliani, Sen. John McCain, and former Gov. Mitt Romney; 48% say Giuliani has shown it "in the past," 43% say Clinton has.

  • General election match-ups among registered voters are as follows:

    Giuliani 46%, Clinton 49%
    Giuliani 43%, Obama 50%
    Giuliani 44%, Edwards 50%

    McCain 44%, Clinton 50%
    McCain 39%, Obama 52%
    McCain 42%, Edwards 52%

    Romney 35%, Clinton 57%
    Romney 29%, Obama 58%
    Romney 27%, Edwards 64%


Bush Approval: Newsweek at 28%, Trend at 33.2%

Topics: George Bush

1BushApproval2ndTerm20070503.png

A Newsweek poll taken 5/2-3/07 finds President Bush's approval rating fallen to 28%, with disapproval at 64%. This is a new all time low for Newsweek polling on Bush and ties the lowest readings of Harris polls (taken 4/20-23/07) and an older CBS poll taken 1/18-21/07. No poll has found a lower approval reading for the President.

With the addition of this Newsweek poll, my trend estimate stands at 33.2%, also an all time low.

Before concluding that Bush's approval has crashed, however, the Newsweek poll should be compared to the recent trend and to other recent polls. The last six polls have registered approval at 35, 28, 35, 32, 35 and 28. So the last six readings are three at 35, one at 32 and two at 28, a seven point range.

In the figure below, the Newsweek poll is clearly well below the current trend estimate, and more importantly, is below the typical Newsweek poll, which is normally about 2-3 points below the trend. The current poll is 5.2 points below trend.

2LastSixPolls20070503.png

This means that Newsweek is a bit atypically low, compared to both other polls and to its own normal trend.

A check of the outlier analysis shows that Newsweek, like Harris 10 days earlier, is in fact an outlier when compared to the variation around the trend estimate for all polls.

3BushResiduals20070503.png

The Newsweek and Harris polls are exerting a significant pull on the trend estimate, even when balanced against the three recent polls at 35%. The current small downward slope in the trend estimate is due to the addition of Newsweek. It was flat prior to adding the new poll.

The uncertainty of the trend estimate can be seen in the gray area in the plot below. While the trend is moving a bit down since early 2007, it has remained well within the gray region, suggesting we have no reliable evidence that approval is moving either way. Since January the approval trend has sometimes moved up slightly and sometimes down slightly, but has never sustained either trend long enough to move out of the uncertainty band. The current downward slope may again be reversed with new polling (or not-- let the data speak.)

4BootApproval20070503.png

The sensitivity analysis below shows how the last 20 trend estimates have moved around as each new poll has been added. the substantial effect of the new Newsweek poll is apparent in the gap between the current estimate and the previous cluster of estimates between 34% and 35%.

5Sentitivity2ndTerm20070503.png

The bottom line is that the evidence for a new decline in approval remains quite mixed, with two polls claiming a rather dramatic drop in approval, while several others suggest continued stability around 32%-35%. Given the outlier analysis and the fact that the Newsweek poll is clearly below that poll's own trend, we must still wait before concluding that approval has taken a sharp turn down. The evidence remains in favor of rough stability around 34%, though the trend estimate now stands a little below that, in deference to Newsweek's evidence.

Cross-posted at Political Arithmetik.


 

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR