Pollster.com

July 22, 2007 - July 28, 2007

 

National GOP Contest: Why are ABC/Post & Rasmussen So Different?

Topics: 2008 , IVR , IVR Polls , Likely Voters , The 2008 Race

A suggestion from alert reader and frequent commenter Andrew:

I write to suggest that you analyze the huge discrepancy between the latest Rasmussen and Washington Post/ABC polls. I'm talking about the Republican nomination. Rasmussen says Thompson is up by 4 over RG, while WP/ABC says Rudy is up by 20 pts over FT, who isn't even in second place here (36 RG to 14 FT). One of these pollsters is obviously very wrong. Two polls cannot both be accurate, if their margin of victory do not approximate each other. This is a humongous 24 point discrepancy.

Here, with a little assist from Professor Franklin, is a chart showing the discrepancy that Andrew noticed. The two surveys do seem to show a consistent difference that is clearly about more than random sampling error. The ABC News/Washington Post survey shows Giuliani doing consistently better, and Thompson doing consistently worse, than the automated surveys conducted by Rasmussen Reports, although the discrepancy has been largest in terms of how the most recent ABC/Post poll compares to Rasmussen surveys conducted over the last month or so.

ARasVsWaPo400.png

To try to answer Andrew's question, it makes sense to take two issues separately. First, why are the surveys producing different results for the Republican primary?

At the most basic level, these surveys seem to be measuring the same thing: Where does the Republican nomination contest stand nationally? And both surveys begin with a national sample of working telephone numbers drawn using a random digit dial (RDD) methodology. Take a closer look, however, and you will see some pretty significant difference in methodology:

  • The ABC/Post survey uses live interviewers. Rasmussen uses an automated recorded voice that asks respondents to enter their answers by pushing buttons on a touch tone keypad. This method is known as Interactive Voice Response (IVR). The response rates -- and more importantly, the kinds of people that respond -- are likely different, although neither pollster has released specific response rates for any of the results plotted above.
  • The ABC/Post survey attempts to select a random member of each household to be interviewed by asking "to speak to the household member age 18 or over at home who's had the last birthday" (more details here). Rasmussen interviews whatever adult member of the household answers the telephone. Both organizations weight the final data to reflect the demographics of the population.
  • Rasmussen Reports weights each survey by party identification, using a rolling average of recent survey results as a target (although their party weighting should have little effect on a sub-group of Republican primary voters). The ABC/Post survey does not weight national surveys at this stage in the campaign by party ID.
  • [Update -- one I overlooked: The ABC/Post survey includes Newt Gingrich on their list of choices. Gingrich receives 7% on their most recent survey. If the Rasmussen survey prompts Gingrich as a choice, they do not report it. It is also possible that Rasmussen omits other candidates as well, as t Their report provides results for just Giuliani, Thompson, Romney and McCain. Update II -- Scott Rasmussen informs via email: "We include all announced candidates plus Fred Thompson"].
  • And perhaps most important for Andrew's question: The ABC/Post survey asks the presidential primary question of all adults that identify with or "lean" to the Republicans. The Rasmussen survey screens to a narrower slice of the population: Those they select as "likely Republican primary voters."

Unfortunately, neither pollster tells us the percentage of adults that answered their Republican primary question, but we can take a reasonably educated guess: "Leaned Republicans" have been somewhere between 35% and 42% of the adult population on surveys conducted in recent months by Gallup and the Pew Research Center. If Rasmussen's likely voter selection model for Republican is analogous to their model for Democrats, their "likely Republican primary" subgroup probably represents 20% to 25% of all adults.

Consider also that, even before screening for "likely voters" and regardless of the response rate, those willing to complete an IVR study may well represent a population that is better informed or more politically interested than those who complete a survey with an interviewer.

Put this all together, and it is clear that the Rasmussen survey is reaching a very different population, something I would wager explains much of the difference in the results charted above.

Now, the second question, which result is more "accurate?" It is tempting to say that this question is impossible to answer, since we will never have a national primary election to check it against. But a better answer may be that "accuracy" in this case depends on what we want to use the data for.

If we were trying to predict the outcome of a national primary, and if all other aspects of methodology were equal (which they're not), I would want to look at the narrower slice of "likely voters" rather than all adult "leaned Republicans." Since the nomination process involves series of primaries and caucuses starting with Iowa and New Hampshire, and since the results from those early contests typically influence preferences in the states that vote later, we really need to focus on early states for a more "accurate" assessment of where things stand now. While interesting and fun to follow, these national measurements provide only indirect indicators of the current status of the race for the White House.

Why would the ABC/Post survey want to look at all Republicans, rather than likely voters? Here is the way ABC polling director Gary Langer explained it in his online column this week:

I like to think there are two things we cover in an election campaign. One is the election; the other is the campaign.

The campaign is about who wins. It's about tactics and strategy, fundraising and ad buys, endorsements and get-out-the-vote drives. It's about the score of the game - the horse race, contest-by-contest, and nothing else. We cover it, as we should.

The election is the bigger picture: It's about Americans coming together in their quadrennial exercise of democracy - sizing up where we're at as a country, where we want to be and what kind of person we'd like to lead us there. It's a different story than the horse race, with more texture to it, and plenty of meaning. We cover it, too.

We ask the horse race question in our national polls for context - not to predict the winner of a made-up national primary, but to see how views on issues, candidate attributes and the public's personal characteristics inform their preferences.

Questions like Andrew's are more consequential in the statewide surveys we are tracking here at Pollster.com, and those surveys have been producing some discrepancies even bigger than the one charted above. We will all be in a better to make sense of those differences if we know more about the methodologies pollsters use. I'll be turning to that issue in far more detail next week.


POLL: Mason-Dixon Florida


Via First Read, a new Mason-Dixon statewide survey of likely voters in Florida finds:

  • Among Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (31% to 17%); former Sen. John Edwards trails at 12%.
  • Among Republicans, former Mayor Rudy Giuliani edges out former Sen. Fred Thompson (21% to 18%) in a statewide primary; Sen. John McCain trails at 11%, former Gov. Mitt Romney at 7%.

We will update this post with additional links as they become available.

View all Florida Primary poll data at Pollster.com:


AFLTopDems190.png AFLTopReps190.png
Democrats Republicans


POLL: ARG Arizona Primary


A new American Research Group statewide survey of likely voters in Arizona (conducted 7/23 through 7/26) finds:

  • Among 400 Republicans, Sen. John McCain leads former Mayor Rudy Giuliani (32% to 23%) in a statewide primary; former Sen. Fred Thompson trails at 15%, former Speaker Newt Gingrich and former Gov. Mitt Romney both at 7%. All other candidates receive less than 5% each.
  • Among 400 Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (39% to 25%) in a statewide primary; Gov. Bill Richardson trails at 9%, former Sen. John Edwards at 8%. All other candidates receive 5% or less.


POLL: SurveyUSA Missouri Gov


A new SurveyUSA automated survey of 514 registered voters in Missouri (conducted 7/24 through 7/25) finds State Attorney General Jay Nixon leading Gov. Matt Blunt (57% to 38%) in a statewide gubernatorial general election match-up.


POLL: KCCI/KCRG Iowa Caucus


A new KCCI/KCRG statewide survey (via MyDD) of likely caucus voters in Iowa (conducted 7/23 through 7/25 by Research 2000) finds:

  • Among 400 Democrats, former Sen. John Edwards edges out Sen. Hillary Clinton (27% to 22%) in a statewide caucus; Sen. Barack Obama trails at 16%, Gov. Bill Richardson at 11%. All other candidates receive less that 5% each.
  • Among 400 Republicans, former Gov. Mitt Romney leads former Sen. Fred Thompson (25% to 14%) in a statewide caucus; former Mayor Rudy Giuliani trails at 13%, Sen. John McCain at 10%, former Speaker Newt Gingrich at 6%. All other candidates receive less than 5% each.
  • The survey also includes general election match-ups, job approvals, and questions regarding Iraq policy.

View all Iowa Caucus poll data at Pollster.com:

AIATopDems195.png AIATopReps195.png
Democrats Republicans


Unspinning Howard Wolfson*

Topics: 2008 , Iraq , The 2008 Race

Earlier today on Hardball, MSNBC's Chris Matthews had the following exchange with Howard Wolfson, communications director for the Clinton campaign (my transcript below the video):

Matthews: How would you describe [Sen. Clinton's] position in voting to authorize the war in Iraq believing we weren't going to war, that Bush really didn't intend to go to war. Was that naive?

Wolfson: Look, she's taken responsibility for the vote. She's been asked about this...

Mathews: Wouldn't you call that naive to believe...

Wolfson: No...

Matthews: ...that we're not going to war when everybody thought we were going to war? I thought we were going to war.

Wolfson: I guess 80 percent of the country was naive then.

Matthews: They didn't think Bush would take us to war?

Wolfson: I think people were, believed George Bush was going to do what he said he was going to do, which was to try diplomacy. And he didn't.

Matthews: Anybody who didn't think we were going to war, in the months leading up to the war in Iraq, wasn't paying attention.

Did 80% of Americans believe that President Bush would "try diplomacy" in an effort to avoid a war with Iraq?

Not exactly.

I checked the Iraq archives at the Polling Report. Most of the questions asked in late 2002 focused on whether and under what conditions Americans would support going to war. However, a CNN/USA Today/Gallup survey of 1,017 adults conducted November 22-24, 2002 (a month after the vote to authorize the use of military force in Iraq), showed that a majority of Americans believed President Bush had "already decided" to invade Iraq:

Which comes closer to your view about President Bush? [Rotate:] Bush has already decided to invade Iraq and has agreed to UN inspections mainly to gain international support for that action. OR, Bush has not yet decided whether to invade Iraq and has agreed to UN inspections mainly to determine if an invasion of Iraq were necessary.

58% - Bush has already decided to invade
38% - Bush has not yet decided whether to invade
4% - No opinion

A bit of context on the timing: The U.S. Senate passed the resolution authorizing the use of military force against Iraq on October 11, 2002 (with Senator Clinton voting in favor). President Bush signed it into law on October 16. Three weeks later, on November 8, the United Nations Security Council unanimously passed Resolution 1441 urging Iraq to "comply with its disarmament obligations" or face "serious consequences." Iraq agreed to the resolution on November 13, and under its terms, U.N. weapons inspectors were set to return to Iraq on November 27 after a four year absence to conduct onsite inspections in search of weapons of mass destruction.

Gallup fielded its survey on November 22-24, just days before the return of the U.N. inspectors, a time when one might expect optimism regarding the use of diplomacy to resolve the conflict. Yet even then, 58% of Americans believed the President had "already decided to invade Iraq."

*With apologizes to Frank Newport, who has made "unspinning" questionable assertions about public opinion on Sunday morning talk shows a regular staple of his Gallup Guru blog.


Is Bloomberg Polling Outside NYC?

Topics: 2008 , The 2008 Race

A few weeks ago, a reporter called to ask if I thought New York City Mayor Michael Bloomberg might be conducting polls to assess a potential independent candidacy for President. Needless to say, I have no way of knowing. Bloomberg is under no legal requirement to report on his personal expenditures, and as one of the wealthiest men in American, can certainly afford to spend his own money to commission polls.

I mentioned to the reporter that the only way we might discover that Bloomberg is polling is on the off chance that a respondent might report having participated in a Bloomberg-centric survey and post a comment on a blog somewhere (something that seems to happening more and more lately). As such, this comment from Open Left's Chris Bowers is very intriguing:

[T]hree weeks ago a friend of mine participated in a telephone poll conducted by Mountain West Research Strategies. Apparently, they asked many questions about his outlook on the two political parties, but the only politician they asked about was Michael Bloomberg. It should be noted that Mountain West Research Strategies was a polling firm used by Joe Lieberman in 2006.

Mountain West Research is a survey call center and, in this most recent case, was likely the vendor used by a political pollster to conduct interviewing. The link above involves a poll conducted in June 2006, when Lieberman's pollster of record was the Democratic firm Greenberg, Quinlan, Rosner Research. As I assume that Mountain West conducts surveys for many different political pollsters, the Lieberman association is probably not a good indicator of who sponsored this particular survey.

Update I - Stan Greenberg responds via email:

We have absolutely nothing to do with a recent survey for Bloomberg, though we use Mountain West and have surveys in the field there regularly - along with Sun Surveys, our own company.

Update II - There may be less to this story than meets the eye. Also via email, Chris Bowers adds more detail:

They only asked two questions about Bloomberg, but they came after a long battery opinion of the two parties. The rest of the questions, or about 30-40% of the call, were about education and No Child Left Behind. It is not clear to me that it definitely was a Bloomberg poll.

Given that description, the survey could have been conducted by any number of entities not affiliated with Bloomberg, including an education interest group or any group wondering about the potential impact of an independent presidential candidacy. It is also possible that this was an "omnibus" survey that included questions on different subjects paid for by several different clients.

Consider also that if Bloomberg had paid a campaign pollster to evaluate his potential as an independent candidate, the survey would have had many more questions about Bloomberg, including "message tests" involving his biography and positions on issues.

**Presumably, "Chris" is Open Left blogger Chris Bowers.


POLL: Diageo/Hotline National Survey


A new Diagio/Hotline national survey (story, results) of 801 registered voters (conducted 7/19 through 7/22) finds:

  • 33% approve of the job George Bush is doing as president; 63% disapprove.
  • 35% think that the next president will start withdrawing troops from Iraq "within three months, with all troops out within nine months;" 29% think that he or she will "leave a substantial number of troops in Iraq, but have them concentrate on training Iraqis and targeting Al Qaeda leaders in Iraq;" 14% say he or she will "begin an immediate withdrawal of all U.S. troops in Iraq;" 7% say he or she will "make no cutbacks in U.S. troops in Iraq."
  • Among 122 likely Republican primary voters, former Mayor Rudy Giuliani edges out former Sen. Fred Thompson 20% to 19%; Sen. John McCain runs at 17%, former Gov. Mitt Romney at 8%. All other candidates receive less than 5% each.
  • Among 393 likely Democratic primary voters, Sen. Hillary Clinton leads Sen. Barack Obama 39% to 30%; former Sen. John Edwards trails at 11%. All other candidates receive less than 5% each.


POLL: Battlegound Poll 2008


A new Tarrance Group (R)/Lake Research (D) Battleground Poll (results, graphics, Celinda Lake (D) analysis, Brian Tringali (R) analysis) of 1,000 likely voters (conducted 7/15 through 7/18) finds:

  • 35% approve of the job George Bush is doing as president; 61% disapprove.
  • The Democratic candidate leads the Republican candidate (44% to 37%) on a generic ballot for U.S. Congress.
  • General eletion match-ups:

    Clinton 44%, Giuliani 50%
    Clinton 47%, Thompson 45%
    Obama 52%, Giuliani 43%
    Obama 56%, Thompson 36%


POLL: Pew Campaign Media Coverage


A new Pew Research Center national survey (story, results) of 1,040 adults (conducted 7/20 through 7/23 by Opinion Research Corporation) finds:

  • In an open ended question, 42% say Sen. Hillary Clinton is the 2008 presidential candidate they've "heard the MOST about in the news lately;" 22% say Sen. Barack Obama, 3% say Pres. George W. Bush. 2% each say former Sen. John Edwards, Sen John McCain, former Mayor Rudy Giuliani, former Sen. Fred Thompson, and former Gov. Mitt Romney.
  • "Four-in-ten Republicans say news organizations are giving too much coverage to the campaign. Only 19% of Democrats feel the same way."


POLL: Gallup Church Attendance for 2008


New Gallup anaylses (and video) of recent national surveys of 3,006 adults (conducted in June and July) find:

  • Among 1,204 Republicans and those who lean Republican, former Mayor Rudy Giuliani leads former Sen. Fred Thompson (29% to 19%) in a national primary. Among the 480 who attend church every week, Giuliani edges out Thompson 24% to 20%. Among the 420 who seldom or never attend church, Giuliani leads 33% to 18%.
  • Among 1,515 Democrats and those who lean Democratic, Sen. Hillary Clinton leads Sen. Barack Obama (35% to 22%) in a national primary. Among the 364 who attend church every week, Clinton leads Obama 39% to 24%. Among the 794 who seldom or never attend church, Clinton leads Obama 32% to 20%.
  • In general election match-ups, Giuliani leads Clinton by more than ten points among Americans who attend church weekly and among those who attend nearly weekly or monthly. Among Americans who seldom or never attend church, Clinton leads Giuliani by more than ten points (54% to 43%).


Mid-Week Remainders


CBS's Kathy Frankovic provides some useful historical background on the application of random probability sampling to political polling and explains why pollsters "make such a big deal" about it.

ABC's Gary Langer explains "what we're after" in conducting national trail heat surveys on the presidential nomination contests.

"Gallup Guru" Frank Newport has an answer for the many Ron Paul supporters who want to know to ask why they have not been polled by Gallup.

The Washington Post's Chris Cillizza profiles the 29% of adults favor an immediate withdrawal of troops from Iraq on the most recent Post/ABC News poll.

The Post's Jennifer Agiesta blogs an extensive list of historical highs and lows of presidential job performance to compliment the Peter Baker analysis I blogged earlier today.

And speaking of the Bush job approval rating, TNR's Michael Crowley lists "some other things roughly one in four Americans believe."


POLL: InsiderAdvantage SC Dem Primary


A new InsiderAdvantage/Majority Opinion statewide survey of 536 registered voters would said they were likely to vote in the Democratic primary in South Carolina (conducted 7/24) finds:

  • Sen. Hillary Clinton leads Sen. Barack Obama (43% to 28%) in a statewide primary; former Sen. John Edwards trails at 13%, Gov. Bill Richardson at 5%. All other candidates receive less than 5% each.
  • Among African Americans, Clinton edges out Obama 44% to 41%.

Note: InsiderAdvantage/Majority Opinion reports that 193 out of 536 respondents were African American.


POLL: Quinnipiac NYC Survey


A new Quinnipiac University citywide survey of 1,194 registered voters in New York City (conducted 7/18 through 7/23) finds:

  • 73% approve of the way Mayor Michael Bloomberg is handling his job; 19% disapprove.
  • 56% think it is likely that Bloomberg will run for President; 41% think it is not likely.
  • If Bloomberg ran for President as an independent, 12% would definitely vote for him, 22% would probably vote for him, 26% would probably not vote for him, 31% would definitely not vote for him.

via USAToday On Politics


POLL: Lake Research (D) Oregon Senate


A new Lake Research (D) statewide survey of 500 likely voters in Oregon (conducted 7/7 through 7/11 for Steve Novick) finds:

  • Republican Sen. Gordon Smith leads Novick (50% to 27%) in a statewide general election match-up for U.S. Senate.
  • 59% have a favorable opinion of Smith; 25% have an unfavorable opinion.
  • 32% would vote to re-elect Smith, 34% would consider someone else, and 19% would vote to replace him.


POLL: CSRC Alabama Primary


A new Alabama Education Association statewide survey of likely voters in Alabama (conducted 7/11 through 7/13 and 7/16 through 7/19 by Capital Survey Research Center) finds:

  • Among 396 Republicans, former Sen. Fred Thompson leads former Mayor Rudy Giuliani (34% to 20%) in a statewide primary; Sen. John McCain trails at 11%, former Speaker Newt Gingrich at 7%, former Gov. Mitt Romney trails at 5%. All other candidates receive less than 5% each.
  • Among 371 Democrats, Sen. Hillary Clinton edges out Sen. Barack Obama (33% to 29%) in a statewide primary; former Sen. John Edwards trails at 9%, former V.P. trails at 6%. All other candidates receive less than 5% each.


Bush: Why So Unpopular?

Topics: George Bush

This morning, The Washington Post's Peter Baker takes a closer look at why President Bush, with a disapproval rating of 65% on the latest Post/ABC survey, "is in the running for most unpopular president in the history of modern polling." While poll junkies ought to read the whole thing, this bit of historical context from pollster Patrick Caddell is especially intriguing:

"It's astonishing," said Pat Caddell, who was President Jimmy Carter's pollster. "It's hard to look at the situation today and say the country is absolutely 15 miles down in the hole. The economy's not that bad -- for some people it is, but not overall. Iraq is terribly handled, but it's not Vietnam; we're not losing 250 people a week. . . . We don't have that immediate crisis, yet the anxiety about the future is palpable. And the feeling about him is he's irrelevant to that. I think they've basically given up on him."

Baker goes on to float the theory that "the changing nature of society," and in particular of the way Americans receive their news, is partly responsible for the difference:

"A lot of the commentary that comes out of the Internet world is very harsh," said Frank J. Donatelli, White House political director for Ronald Reagan. "That has a tendency to reinforce people's opinions and harden people's opinions."

Hmm. Thoughts anyone?

Update: TNR's Jon Chait has more thoughts, as does Open Left's Chris Bowers.


POLL: ABC/Post GOP Primary


Additional results from the recent ABC News/Washington Post national survey (ABC story, results; Post story, results) of 1,125 adults (conducted 7/18 through 7/21) finds:

  • Among Republicans and those who lean Republican, former Mayor Rudy Giuliani leads Sen. John McCain (34% to 16%) in a national primary; former Sen. Fred Thompson trails at 14%, former Gov. Mitt Romney at 8%, former Speaker Newt Gingrich at 7%. All other candidates received less than 5% each.
  • 13% of Republicans are very satisfied with the choice of candidates for the Republican nomination; 53% are somewhat satisfied, 26% are somewhat dissatisfied, and 6% are very dissatisfied.


Bush Approval: Three new polls, Trend at 29.6%

Topics: George Bush

1BushApproval2ndTermRough20070722.png

New polls have moved noticeably above President Bush's recent low point of approval, though the current trend estimate remains below 30%. New polls have generally fallen at 30% approval and above, though one is as low as 25%. This change of trend was first discussed about a week ago here. Since then, the evidence for a shift in trend has mounted.

The ARG poll, taken 7/18-21/07, sets the low point at 25% approval, 71% disapproval. Gallup, done 7/12-15/07 got approval at 31%, disapproval at 63%. CBS/New York Times has two recent results: 7/9-17/07 has approval at 29%, disapproval at 64%, while the newest CBS/NYT poll of 7/20-22/07 puts approval at 30%, disapproval at 62%. Fox, 7/17-18/07 has approval at 32%, disapproval at 61%, while ABC/Washington Post 7/18-21/07 estimates approval at 33% and disapproval at 65%. So that puts four of the last six polls above 30% with only one in the mid-20s, where polling in late June was falling.

The trend estimate now stands at 29.6%, and the slope of the trend has clearly begun to bend from a steep downward trend to a less negative one. This is classic behavior of my standard "old blue" estimator. It takes a while to change direction, and while it does it slowly bends until it shifts direction entirely. In contrast, the "ready red" estimator is more sensitive and picks up on direction changes more quickly. The red line in the figure above shows a sharp change at just under 29% to a current estimate of just under 31% approval. The red line is often too sensitive, mistaking short term random noise for real change. However, we've now accumulated enough supporting polls to make me more confident that the upturn in the red estimator is probably real. At the same time, this does not mean the trend will remain as sharply up as the red estimator seems to suggest. That won't be known for a while yet, as more polls help define what the path of the next month or so is.

Recent polls have fallen above and below the trend estimate, and at the moment none of the last 10 polls constitutes an outlier, though there are both high and low polls rather close to the confidence interval limits. This range of results is also reflected in the very wide gray area at the end of the trend in the Bootstrap plot below. The wide gray area reflects our uncertainty about where approval is right now in a time of change in direction.

Finally, the last plot below shows the most recent 20 estimates of approval. The red dots trace out the clear decline, low point, and move back upwards in the "old blue" estimator. So even that estimator is feeling a change.

2LastSixPolls20070722.png

3BushResiduals20070722.png

4BootApproval20070722.png

5Sentitivity2ndTerm20070722.png

Cross-posted at Political Arithmetik.


POLL: Zogby Bush Approval


A new Zogby telephone survey of 1,012 likely voters nationwide (conducted 7/12 through 7/14) finds that 34% of Americans approve of the job George Bush is doing as president; 66% disapprove.


Focus Grouping the YouTube Debate

Topics: Debates , Focus Groups

A few more thoughts about the various efforts to determine who "won" last night's CNN/YouTube debate. In addition to the SurveyUSA panel-back survey I blogged last night, there were (by my count) at least four different focus groups convened in various locations:

  • CNN reported last night on two "dial groups," one in Manchester, New Hampshire and another in Las Vegas, Nevada, in which selected voters constantly adjusted a dial that rates their reaction to whatever they are watching. (coverage here, here and here).
  • Dial group tester Rich Thau (Messagejury.com) did another dial group among 12 likely voters in Bedford New Hampshire. [Update: more on the Thau groups here, via First Read].
  • Fox News had Republican pollster Frank Luntz conduct what appeared to be a more traditional focus group (albeit with a larger than traditional group conducted on live television) in Charleston, South Carolina (transcript, video available from this page). [Update: A reader emails to say that the Luntz group also involved a "dial test"].

Because they are more qualitative, the "results" of these efforts can sometimes be in the eye of the beholder. Since they involve small, non-random samples of different kinds of voters, they frequently produce contradictory results. Some researchers like to use bigger groups (of 40-50 participants), but that approach increases the risk of creating a "group dynamic" that allows the most opinionated participants to sway others toward their point of view. We also know little from the various reports linked to above about the kinds of voters recruited for each group. How many were completely undecided before the groups? How many were political independents? The answers to those questions might tell us a lot about the reported outcome in each case.

Not surprisingly, last night's focus groups identified many different "winners." According to coverage from NBC's First Read, "Barack Obama got the most favorable [response] in terms of the best performance" in the CNN New Hampshire group, but their Nevada group showed "Bill Richardson and Hillary Clinton in a tie with six votes each." On the basis of favorable reaction from his focus group, Frank Luntz concluded that "Obama will be shown as the winner of tonight's debate."

Meanwhile, depending on which source we turn to, the Rich Thau group of 12 New Hampshire voters produced good news for John Edwards, Barack Obama and Joe Biden. Under the headline, "Very Much an Edwards Night," Thau reports that "a plurality of our group (four of 12) thought Sen. Edwards won the debate." However, NBC's First Read tells us, "his survey had Obama doing the best (in terms of improvement from pre-debate to post-debate; Obama and Edwards tied with the highest post-debate score)." And Thau's before-and-after "comfort scores" shows the biggest gains by Joe Biden.

If you're having trouble making sense of all this, you're not alone.

Also, I am guessing that the audience for the YouTube debate, while probably larger than usual for CNN, was still a tiny sliver of those who will ultimately vote in Democratic primaries or caucuses. Assuming the ratings followed typical patterns, we should assume that far more voters saw or read news accounts about the debate this morning than experienced it in full last night.

In that regard, consider this very sensible observation made last night by Chris Bowers, on the liberal site Open Left:

I noticed that both on CNN, and in the progressive blogosphere, several people started trying to determine who won, and even making declarations that Candidate X had, in fact, clearly won the debate. I was taken aback. There was a winner? There was even a competition? What did the winner, well, "win?" New hard-core voters in the Iowa and New Hampshire primaries? New donors? The deliverance of a biting comment that will psychological scar one of his or her opponents so badly that s/he cannot continue on the campaign trail with naught but a thoroughly shredded sense of self-esteem?

The truth is, there is only one objective way for any candidate to "win" the debate: if the debate, or the post-debate spin, cumulatively result in your campaign moving closer to taking the nomination than you were before the debate. That's it. From this "God's eye" perspective, it is doubtful that anyone won the debate. The factors that make the most difference in pushing someone closer to the nomination are, first, new Iowa supporters and, second, new supporters in New Hampshire. Everything else is pretty distant, and I seriously, seriously doubt anyone picked up much, if anything, in those two categories

That sounds about right.


POLL: Gallup Congress, 08


Additional results from a recent Gallup national survey of 1,001 adults (conducted 7/12 through 7/15) finds:

  • 27% approve of the job Congress is doing; 66% disapprove
  • General election match-ups:

    Giuliani 49%, Clinton 46%
    Giuliani 49%, Obama 45%
    Thompson 45%, Clinton 48%
    Thompson 40%, Obama 51%

* Typo corrected.


POLL: Rasmussen Florida, Ron Paul


Two new Rasmussen Reports automated surveys find:

  • Among 457 likely Democratic primary voters in Florida, Sen. Hillary Clinton (at 46%) leads Sen. Barack Obama (15%) and former Sen. John Edwards (13%) in a statewide primary (conducted 7/18 through 7/19).
  • Among 490 likely Republican primary voters in Florida, former Mayor Rudy Giuliani edges out former Sen. Fred Thompson (22% to 21%) in a statewide primary; Sen. John McCain and former Gov. Mitt Romney both trail at 13% (conducted 7/18 through 7/19).
  • Among 1,461 likely voters nationwide, Clinton leads Rep. Ron Paul (49% to 34%) in a general election match-up; Obama leads Paul 50% to 30% (conducted 7/20 through 7/22).

View all Florida Primary poll data at Pollster.com:

AFLTopDems190.png AFLTopReps190.png
Democrats Republicans


SurveyUSA's Post Debate "Panel Back"

Topics: 2008 , Debates , The 2008 Race

Tonight, partly in response to a critique offered here in May, SurveyUSA fielded an automated "panel-back" survey among 717 Americans who said they watched tonight's CNN/YouTube debate. The idea of a panel-back survey is to interview the same sample before and after some event, allowing for an individual level measurement of change. With a panel back survey, a pollster can determine how many respondents shifted their position

Here is the key question asked of these respondents before and after the debate:

Regardless of who you may vote for, and regardless of whether or not you would vote for a Democratic candidate for President, which Democratic candidate would make the best President of the United States?

The result:

Americans who watched the YouTube Democratic Debate 7/23/07 on CNN went into the debate thinking Hillary Clinton would make the best President of the 8 Democrats on stage, and came out of the debate even more convinced, according to an exclusive SurveyUSA poll. Before the debate, 40% saw Clinton as the most Presidential Democrat; after the debate, when the same respondents were re-interviewed, 43% saw her as the most Presidential Democrat

It is not obvious from the before and after snapshot, but it appears that most of those who responded to both calls were consistent in their answer to this question, before and after. To the extent that there was a shift, however, it worked slightly to the benefit of Clinton and Biden, slightly against Barack Obama.

They also asked if respondents "viewed each Democrat positively or negatively" before and after the debate. The results indicate that exposure to the debate improved perceptions of all the candidates except for Gravel. The summary reported by SurveyUSA (with bullets added):

  • Biden went up 38 points, from Minus 6 to Plus 32.
  • Obama went up 17 points, from Plus 24 to Plus 41.
  • Clinton went up 16 points, from Plus 34 to Plus 50.
  • Edwards went up 16, from Plus 22 to Plus 38.
  • Dodd went up 15, from Minus 21 to Minus 6.
  • Richardson went up 14, from Minus 1 to Plus 13.
  • Kucinich went up 7, from Minus 21 to Minus 14.
  • Gravel went down 3, from Minus 29 to Minus 32

The SurveyUSA summary does include some important caveats worth noting. The most important is that this sample of 717 is not a random sample of all Americans (or likely voters), or even a random sample of all Americans who happened to be at home this evening. They started with a first round of calls to a random sampling of households, but this final pool of 717 reflects only those who responded to the first survey, reported that they would watch the debate and take a second call and then actually completed a second call. The summary also notes that "panel back is not universally sanctioned" as a tool to measure debate reaction, pointing to the unsparing criticism by pollster David Hill. Writing for The Hill in 2004, he described the panel-back as "worthless:"

Considerable scholarly research demonstrates that simply being interviewed renders an otherwise normal voter abnormal. After being polled, voters are much more likely to seek out political information through the media, discuss politics with others and eventually to vote. The known effects are so great that in the earliest days of polling, voters would be screened at the outset of an interview to ascertain if they had ever been interviewed before.

I don't share Hill's blanket aversion to panel-back surveys, but he is right to warn that the first interview can add some artificiality to either the nature of the sample or the way those surveyed watch and react to the debate. For all its drawbacks, however, this approach is a great improvement to simply calling a fresh sample of those at home and asking "who won" without any data on which candidate they supported beforehand.

In the spirit of better understanding the data, I do have a few questions for the analysts at SurveyUSA:

First, what percentage of the 717 debate watchers provided the same answer to the first question before and after the debate?

Second, the summary reports:

SurveyUSA dialed at random into all 50 states. Respondents were asked if they planned on Monday night, 7/23/07, to watch the entire YouTube debate on CNN. Those respondents who said "yes" were then asked if SurveyUSA could call them back, immediately after the debate

What percentage of those adults interviewed on the first call agreed to watch the debate and be interviewed a second time?

Finally, did SurveyUSA have any way to confirm that they interviewed the same individuals within each household before and after the debate?


Flashback: The Polls in July 2003

Topics: 2008 , The 2008 Race

The blogs and mainstream media have featured much speculation, as they always do, on the trends in trial-heat vote questions on national and early state surveys and what they may foretell about the likely outcome next Spring. I thought a review of what these same surveys were saying four years ago about the Democratic nomination contest might serve as a valuable reality check.

07-23%20national.png

According to the Polling Report (my source for all the data that follows), seven organizations released polls in July 2003 on the Democratic race. The "undecided" option generally led on most surveys, and future nominee Sen. John Kerry received just 14% of the vote, typically slightly behind or tied with Rep. Dick Gephardt or Sen. Joe Lieberman. Howard Dean, who would emerge as a perceived "frontrunner" over the summer months was still averaging just 11%. Moreover, as the chart of the Annenberg National Election Survey data (described in more detail here several months ago) demonstrates vividly, Dean remained mired in the teens nationally until December 2003, but then plunged dramatically after his third place finish in Iowa.

03-27%20kenski%20chart_sml.png

Was it any different in the early states this time four years ago? Dean, whose campaign had started running television advertising in Iowa and New Hampshire in June, had increased his share of the vote as compared to polls conducted in both states earlier in the year. Two polls - one from the Des Moines Register and another internal poll later released by Dick Gephardt's campaign - both showed Dean and Gephardt leading with just over 20%. Meanwhile, John Kerry and John Edwards, the two candidates that would ultimately emerge to finish first and second in the caucuses with a combined 69% of the state delegates selected, receiving an average of 18% of the vote in the surveys take in July 2003.

07-23%20iowa.png

The New Hampshire polls of July 2003 probably came closest to the ultimate result in that state, although John Kerry's average in those polls (24%) fell far short of his ultimate share of the vote (38%). Moreover, John Edwards and Wesley Clark who each averaged just 2% in these early polls, ultimately received just over 12% of the vote.

07-23%20NH.png

Of course, 2008 is stacking up as a very different year than 2004. Hillary Clinton and Barack Obama are arguably better known now than any of the 2004 candidates were at this point four years ago. My point here is not to argue that 2008 will be like 2004, however, but to try to remind readers that there is still considerable room for change in the preferences voters express on these early surveys. We are just now coming to the close of the first phase of the campaign in the early states, where (except for Bill Richardson and Mitt Romney) none of the candidates have yet run significant amounts of television advertising. While the contest has started a bit earlier, we still have a long way to go.


POLL: ABC/Post Bush/Iraq


Additional results from the recent ABC News/Washington Post national survey (story, results) of 1,125 adults (conducted 7/18 through 7/21) finds:

  • 33% approve of the job George Bush is doing as president; 65% disapprove -- "a number surpassed only by Richard Nixon in the summer of 1974, albeit matched by Harry Truman and approached by Bush's father."
  • 37% approve of the job Congress is doing; 60% disapprove.
  • 55% trust Democrats in Congrss "to do a better job handling the situation in Iraq;" 32% trust Bush.


POLL: Democracy Corps (D) Health Care


A recent Democracy Corps (D) national survey (memo, results) of 1,000 likely voters (conducted 5/29 through 5/31; released 7/23) finds:

  • 35% approve of the job George Bush is doing as president; 60% disapprove.
  • 49% are satisfied with the quality of health care in this country; another 49% are dissatisfied.
  • 24% are satisfied with the cost of health care in this country; 74% are dissatisfied.
  • 54% favor "a proposal that provided every American with health insurance, even if it meant your taxes or health care premiums would increase as a result;" 42% oppose.


POLL: CBS/NYT Iraq


A new CBS News/New York Times national survey (story, results) of 889 adults (conducted 7/20 through 7/22) finds:

  • 30% approve of the way George Bush is handling his job as president; 62% disapprove.
  • 26% approve of the way Congress is handling it's job; 61% disapprove.
  • 12% say the U.S. should "increase the number of U.S. troops in Iraq;" 15% say keep the same number of troops; 30% say decrease the number; 36% say remove all troops.
  • 19% say the increase of 20,000 troops "is making the situation in Iraq better;" 20% say it's making it worse, 53% say it's having no impact so far.


POLL: ARG Bush Approval


A new American Research Group national survey of 1,100 adults (conducted 7/18 through 7/21) finds:

  • 25% approve of the job George Bush is doing as president, 71% disapprove.
  • "This is the highest level of disapproval and lowest level of approval for the Bush presidency recorded in monthly surveys by the American Research Group."


POLL: Quinnipiac FL Survey


A new Quinnipiac University statewide survey of 1,106 registered voters in Florida (conducted 7/12 through 7/16) finds:

  • Among 438 Democrats, Sen. Hillary Clinton (at 36%) leads Sen. Barack Obama and former V.P. Al Gore (both at 14%) in a statewide primary; former Sen. John Edwards trails at 9%. All other candidates recieve less than five percent.
  • Among 433 Republicans, former Mayor Rudy Giuliani (at 30%) leads former Sen. Fred Thompson (18%) in a statewide primary; Sen. John McCain trails at 10%, former Gov. Mitt Romney at 9%, former Speaker Newt Gingrich at 5%. All other candidates receive less than five percent.
  • General election match-ups:

    Giuliani 46%, Clinton 44%
    Giuliani 39%, Clinton 41%, Bloomberg 9%
    Giuliani 47%, Obama 39%
    McCain 40%, Clinton 46%
    McCain 38%, Obama 42%

View all Florida Primary poll data at Pollster.com:

AFLTopDems190.png AFLTopReps190.png
Democrats Republicans


POLL: ABC/Post Democratic Primary


A new ABC News/Washington Post national survey (ABC story, results; Post story, results) of 1,125 adults (conducted 7/18 through 7/21) finds:

  • Among Democratic voters and those who lean Democratic, Sen. Hillary Clinton leads Sen. Barack Obama (39% to 28%) in a national primary; former Sen. Al Gore trails at 14%, former Sen. John Edwards at 9%. Without Gore, Clinton leads Obama 45% to 30%. All other candidates receive less than five percent.
  • Among those who support Clinton, 68% "strongly" support her while 32% "somewhat" support her. Among those who support Obama, 56% strongly support him while 43% somewhat support him.
  • 83% are satisfied "with the choice of candidates for the Democratic nomination for president this year;" 16% are dissatisfied.


CHIS Via Cell Phone

Topics: Cell Phones

An update from the Los Angeles Times on plans by the California Health Interview Survey (CHIS) to reach cell-phone only households with interviews conducted over cell phones:

The biannual health study, which provides data on more than 42,000 of the state's roughly 12 million households, is used to "drive decision-making and to drive the recommendations we make to the Legislature or to the governor," said Sandra Shewry, director of the California Department of Health Care Services.

If data are inaccurate, a survey paints a nonrepresentative picture of populations, which can in turn be used as the basis for an improper allocation of funds or just bad decisions, she said.

"The implications are that if we do not include people with cellphones only, then we are likely to be underreporting un-insurance rates, and we're likely to underreport smoking prevalence rates in the state," survey director David Grant said. The numbers won't be dramatic, he said, "but it will introduce some level of bias in our estimates.

For more information, see my recent posts on the challenges of the missing cell phone only households and lessons learned from pilot studies conducted via cell phone.


Pollsters of the Future

Topics: Pollsters

OK, more accurately, that should probably read statisticians of the future. Whatever. Either way, I couldn't be prouder (via the AAPOR member-only listserv):


 

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR