September 23, 2007 - September 29, 2007



Gary Langer remembers the numbers for Newt Gingrich when he was Speaker.

Frank Newport looks at Gallup's numbers on the Democratic presidential race with and without Al Gore.

Jennifer Agiesta looks at what the Fox News poll has to say about how closely Americans followed the Moveon.org "General Betray-Us" controversy.

Jay Cost considers how voter inattention combines with horserace coverage to drive early trial-heat poll numbers.

Marc Ambinder and Jonathan Martin report the latest poll memo from the Romney campaign.

David Hill looks back at what polls in 1994 had to say about Hillary Clinton's health care initiative.

Mark Mellman looks at how much results can change when pollsters weight the data.

Ben Smith has video of a verbal smackdown between Clinton pollster Mark Penn and Romney media consultant Alex Castellano.

And Pollster.com reader "sjc" notices something I overlooked: How many more respondents are opting for the Democratic primary in New Hampshire in the latest CNN/WMUR/UNH poll now as compared to 2000.

Disclosure Project Update

Topics: Disclosure

Since kicking off our Disclosure Project on Monday, we are pleased to report some very favorable early mentions and links from a variety of bloggers including The Atlantic's Marc Ambinder, Time's Ana Marie Cox, USA Today's Memmott and Lawrence, Politico Ben Smith, MSNBC's Clicked, DailyKos' DemFromCt, MyDD's Jerome Armstrong and Jonathan Singer, The Democratic Strategist's Ed Kilgore. Other names you may not recognize have left comments or endorsed the efforts on their blogs.

As of yesterday, we can add an important name to that list: My colleague Nancy Mathiowetz, the current president of the American Association for Public Opinion Research (AAPOR), left the following comment here at Pollster.com:

As President of the American Association for Public Opinion Research, I believe that this is an excellent opportunity for public opinion researchers to help improve the public understanding of polling methodology and interpretation.

Pollster.com is to be applauded for this effort.

More information from AAPOR about disclosure:


Regular readers will know that I serve with Mathiowetz on AAPOR's Executive Council, but it is nonetheless an honor to have her support.

I want to recognize the prompt and complete replies we have already received to our queries from the pollsters at ABC News/Washington Post and LA Times/Bloomberg and Time. Other organizations have requested more time to gather and report the data we requested. Given that we are doing something new here while also "debugging" our own process, we are going to allow as many organizations as possible to respond before publishing the first set of replies (and before sending out similar queries for polls done in New Hampshire, South Carolina and the nation as a whole).

I have also been in contact with the pollsters at Quinnipiac, Mason-Dixon and InsiderAdvantage regarding my post last Friday on their recent polls in Florida and will have more on that subject very soon.

I can say that all of those that have responded appear to be making a good faith effort to be transparent and, as Nancy Mathiowetz put it, "help improve public understanding of polling methodology and interpretation." For that we are grateful.

However, not all of the pollsters have responded to our queries, which is your why your support is important. As Ana Marie Cox put it:

The questions will probably seem unbearably wonky to many, but the reason for wanting the answers is important: They'll help reporters (and readers) better judge the accuracy and importance of these primary state polls.

Or to quote DemfromCT from DailyKos:

The bottom line is that if you want better data to analyze, then we, the consumers of all things political, ought to support pollster.com in asking for it. And if we expect and appreciate the analysis done by pollster.com, Swing State Project, Open Left, Slate, Real Clear Politics or any of the other sites that digest and analyze polling data, let's help make the data a bit more "open source" and transparent.

If you can comment or blog your endorsement of this, we would greatly appreciate it.

POLL: Fox National Survey

A new Fox News/Opinion Dynamics national survey (story, results; OJ story, results) of 900 registered voters (conducted 9/25 through 9/26) finds:

  • 34% approve of the job George Bush is doing as president; 58% disapprove.
  • 45% of Republicans are "hoping there is someone new who you haven't heard about yet who may enter the 2008 presidential race;" 49% are not. 36% of Democrats are hoping for it; 60% are not.
  • General election match-ups

    Clinton 46%, Giuliani 39%
    Clinton 42%, Giuliani 32%, Bloomberg 7%
    Clinton 48%, Thompson 35%
    Clinton 46%, McCain 39%
    Obama 41%, Giuliani 40%
    Obama 45%, Thompson 33%
    Obama 40%, McCain 38%

POLL: Research 2000 Texas Senate

A new Research 2000 statewide survey of 600 likely voters in Texas (conducted 9/24 through 9/26 for Daily Kos) finds:

  • If the 2008 election for U.S> Senate were held today, 40% would re-elect Republican Sen. John Cornyn, 15% would consider voting for another candidate, and 35% would vote to replace him.
  • Cornyn leads Democratic state Rep. Rick Noriega (51% to 35%) in a hypothetical general election match-up.

Exploring the Internals of the CNN/WMUR/UNH Poll

If polls are political crack, than I've been happily snorting the latest from the CNN/WMUR New Hampshire primary polls conducted the University of New Hampshire (also known as the "Granite Poll") for the last 24 hours or so. I've become a fan of this survey because of the "internals" included in their questionnaire and the very helpful set of crosstabs (for the Democratic and Republican samples) released as always by University of New Hampshire Survey Center.

Our original posts on the two surveys highlighted the same trial heat numbers that virtually everyone else is focusing on: Clinton's lead has increased since July, while support for Mitt Romney has fallen off so that he now runs neck-and-neck with Rudy Giuliani.

However, to elaborate on a point I tried to make a week ago, the trial heat numbers in such polls (yes, the ones we track here at Pollster) may be the least useful at this stage, particularly if you interpret them as a prediction of what voters will do several months from now. Surveys are good at telling us what voters think and who they currently prefer, but those preferences are always subject to change. The latest survey of Democrats from UNH/CNN/WMUR helps put some of those preferences in perspective. Here a few highlights:

1) A preference is still not a final decision -- Just before asking the likely Democratic primary voter who they would vote for "if the election were held today," the UNH pollsters ask them how close they are to making a final decision.


So even though 91% can name a candidate they would support, only 17% are "definitely decided," 28% are just "leaning to someone" and more than half (55%) "still trying to decide." Of course, voters may ultimately decide to support their current preference, but by their own report most have not yet made that final choice.

I wrote the last two paragraphs before the Republican results had been released, but those numbers show even more evidence of uncertainty. Although 84% of Republicans express a candidate preference, 66% say they are "still trying to decide" while only 13% are definitely decided and 21% are leaning to a candidate.

2) A Big Shift on "Can Win"- One of the biggest and most noteworthy shifts on the Democratic poll involves the question about which candidate "has the best chance of beating the Republican nominee in the general election next November?" Although. Hillary Clinton's share of the vote has increased four points (from 39% to 43%) since June, we see a 17-point increase (from 37% to 54%) in assessments that she has best chance to win in November:


These results are consistent with similar findings from the most recent NBC/Wall Street Journal national survey of Democrats, which show the percentage choosing Clinton as the Democrat with the "best chance to defeat the Republican candidate" growing from 39% to 54% from April to September. Also, the Pew Research Center shows 53% of Democrats now naming Clinton as the candidate with the "the best chance of winning the general election against a Republican" (although they had not asked the question previously).

I noticed something interesting buried in the cross-tabs. Most of the shift on this measure in the UNH polls has occurred among the roughly 60% of likely Democratic primary voters that are less than "extremely interested" in the primary. That suggests a common pattern: Less attentive voters reacting mostly to the dominant campaign news story, "horserace" coverage relentlessly portraying Clinton as front runner consolidating her lead.

One thing to keep in mind though. Electability ratings are not always a great predictor of, well...of electability. The 2004 National Annenberg Election Study (NAES) tracked electability ratings of each of the Democratic candidates (how likely respondents considered each candidate to beat George Bush in November) among likely Democratic primary voters from October 2003 though February 2004. In a paper presented to the American Political Science Association annual meeting in September 2004), Annenberg's Kate Kenski reported:

Dean's electability and viability ratings were higher than those of Kerry until the Iowa caucuses. After the Iowa caucuses, Kerry's electability ratings surpassed those of Dean. Impressions of Dean's electability against Bush reached a high point of almost 53% around November 19. Two months later, on the evening of the Iowa caucuses, this perception had decreased eight points.

3) A McCain Resurgence? Support for John McCain among likely Republican primary voters, which had fallen sharply in New Hampshire as elsewhere, has increased to 17% on this survey from a low of 12% in July. McCain's support had hovered around 30% in UNH polls conducted during 2006 and early 2007.

An outlier? Perhaps, but dig deeper and the survey yields evidence of some underlying strengths for McCain in New Hampshire that may provide a foundation for a future resurgence there. His favorable rating now (63% favorable, 24% unfavorable) is slightly but not significantly better than it was in February (59% favorable, 27% unfavorable) when he had 28% of the vote and ran a point ahead of Giuliani. And McCain now leads the Republican field (with 32%) on the question of which candidate has the "right experience to be president."

It bears repeating: Eight years ago, John McCain defeated George Bush by a huge margin in New Hampshire (49% to 30%) among these same voters. Voters have a way of falling back to past preferences.

4) Favorable Ratings - Talk to campaign pollsters about the value of the trial heat results and most will tell you a similar story: Vote preference is usually the last thing to change. If you want to see evidence of the campaigning and paid advertising that candidates do, look to the movement in their favorable ratings. The table below shows the most vivid evidence of the real progress that the candidates are making in New Hampshire, starting with the Democrats:


And the Republicans:


POLL: Gallup National Democratic Primary and Clinton

New analysis aggregated from the four most recent Gallup national surveys of 1,989 Democrats and those who lean Democratic (conducted 8/3 through 9/16) finds:

  • Sen. Hillary Clinton leads Sen. Barack Obama (47% to 25%) in a national primary while former Sen. John Edwards trails at 13%. All other candidates average at less than five percent each.
  • "Clinton holds a commanding lead among nearly every major subgroup of potential Democratic primary voters."
  • A few "weak links" for Clinton: Among those in households with an annual income of $75k or more, she leads Obama 37% to 30%. Among Men aged 18-49 she narrowly leads Obama 39% to 34%.

POLL: U of Illinois Dem, Rep Primary

A new University of Illinois at Springfield statewide survey of likely primary voters in Illinois (conducted 7/24 through 9/4; released 9/13) finds:

  • Among 410 "very likely" Democratic primary voters in Illinois, Sen. Barack Obama leads Sen. Hillary Clinton (49% to 27%) in a statewide primary; former Sen. John Edwards trails at 6%.
  • Among 286 "very likely" Republican primary voters, former Mayor Rudy Giuliani (at 31%) leads Sen. John McCain (18%), former Sen. Fred Thompson (15%), and former Gov. Mitt Romney (12%).
  • All other candidates receive less than five percent each.

POLL: Strategic Vision (R) Iowa, Florida

Two new Strategic Vision (R) statewide surveys of likely primary voters in Iowa and Florida (conducted 9/21 through 9/23) find:

  • Among 600 Democrats in Iowa, Sen. Hillary Clinton (at 24%) narrowly leads former Sen. John Edwards (22%) and Sen. Barack Obama (21%) in a statewide caucus; Gov. Bill Richardson trails at 13%. Among 600 Republicans, former Gov. Mitt Romney leads former Mayor Rudy Giuliani (30% to 17%); former Sen. Fred Thompson trails at 13%, former Gov. Mike Huckabee at 8%, Sen. John McCain at 6%, and former Speaker Newt Gingrich at 5%.
  • Among Democrats in Florida, Sen. Hillary Clinton leads Sen. Barack Obama (44% to 22%) in a statewide primary; former Sen. John Edwards trails at 12%, Gov. Bill Richardson at 6%. Among Republicans, former Mayor Rudy Giuliani leads former Sen. Fred Thompson (35% to 24%); former Gov. Mitt Romney trails at 9%, Sen. John McCain at 6%, former Gov. Mike Huckabee at 5%.
  • All other candidates receive less than five percent each.

View all Iowa Caucus poll data at Pollster.com:

AIATopDems190.png AIATopReps190.png
Democrats Republicans

View all Florida Primary poll data at Pollster.com:

AFLTopDems190.png AFLTopReps190.png
Democrats Republicans

POLL: InsiderAdvantage Kentucky Gubernatorial

A new InsiderAdvantage/Majority Opinion statewide survey (results) of 743 likely voters in Kentucky (conducted 9/24 through 9/25) finds Democratic challenger Steve Beshear leading Republican Gov. Ernie Fletcher (45% to 35%) in a statewide gubernatorial general election match-up.

Kentucky Gubernatorial poll data:

PollsterDatesN PopSteve
InsiderAdvantage9/24-25/07743 LV4535
Courier-Journal9/13-18/07667 LV5535
Research 20009/10-13/07600 LV5639
SurveyUSA9/8-10/07533 LV5839
SurveyUSA8/4-6/07613 LV5837
Preston-Osborne7/25-8/2/07600 LV4931
SurveyUSA7/14-16/07560 LV5936
InsiderAdvantage7/8-9/07693 RV4138
Rasmussen5/24-25/07500 LV5135
SurveyUSA5/23-24/07609 LV6234

Note: The InsiderAdvantage survey has consistently shown a higher undecided.

POLL: CNN/WMUR New Hampshire GOP Primary

Additional results from the new CNN/WMUR Granite Poll (story, results) of 324 likely Republican primary voters in New Hampshire (conducted 9/17 through 9/24 by Univeristy of New Hampshire Survey Center) finds:

  • Former Gov. Mitt Romney narrowly leads former Mayor Rudy Giuliani (23% to 22%) in a statewide primary; Sen. John McCain trails at 17%, former Sen. Fred Thompson at 12%, former Speaker Newt Gingrich at 7%. Without Gingrich, Romney runs at 25%, Giuliani at 24%, McCain at 18%, and Thompson at 13%. All other candidates receive less than five percent each.
  • 13% have "definitely decided" who they will vote for in the New Hampshire primary, 21% are "leaning toward someone" and 66% are "still trying to decide."
  • "[Romney's] statistically-insignificant, 1 point margin is a major change from CNN/WMUR's last New Hampshire poll, taken in July, when Romney held a comfortable 14 point lead over Giuliani."

Results for the Democratic Primary are available here.

View all New Hampshire Primary poll data at Pollster.com:

ANHTopDems190.png ANHTopReps190.png
Democrats Republicans

POLL: Fabrizio, McLaughlin (R) GOP Primary

Via Jonathan Martin, a new Fabrizio, McLaughlin and Associates (R) national survey of 1,000 registered republicans (conducted 9/21 through 9/23) finds:

  • Former Mayor Rudy Giuliani leads former Sen. Fred Thompson (23% to 18%) in a six-way national primary; Sen. John McCain trails at 12%, former Gov. Mitt Romney at 10%, former Speaker Newt Gingrich at 7%.
  • 55% of Republicans say they would be "more likely" to support a candidate if he was endorsed by the president; 12% say "less likely," 31% say it would have "no affect."

POLL: SurveyUSA Domenici Approval

Additional results from the most recent SurveyUSA automated survey (story; results) of 600 adults in New Mexico (conducted 9/14 through 9/16 for KOB-TV) finds:

  • 41% approve of the job Sen. Pete Domenici is doing as U.S. Senator; 54% disapprove.
  • "This is the lowest Domenici approval rating since SurveyUSA began tracking the senator's popularity in May 2005 and the first time his approval has sunk below 50 percent. His approval number has fallen from 68 percent last November and plunged 11 percent last month."

"It Bears Repeating"

Topics: Internet Polls

Carl Bialik, the Wall Street "Journal's Numbers Guy," warns once again an online survey of online activity:

Would you be willing to give up online access for a week? Then you might not be the most likely person to answer an online survey (or to read this blog post, for that matter).

Yet a Web survey was the basis for articles by Reuters, the New York Post and InformationWeek claiming that 28% of Americans have let the Internet cut into time they spend with friends and 20% have cut back on sex to spend more time online. Just 20% of respondents said they could go without Web access for a week or more.

I've said this before, but it bears repeating: The people who answer online surveys aren't likely to be representative of Americans when it comes to online behavior. They've found or been found by survey companies, they've signed up to receive emails announcing new surveys and they are among the first to respond to those emails by filling out the surveys. Those are characteristics of people likely to use the Web more regularly and devotedly than the average American.

I've seconded Carl on this point before. The full post has more details and reactions from the company that conducted the survey.

POLL: Quinnipiac New Jersey Primary, General Election

A new Quinnipiac University statewide survey of 1,230 registered voters in New Jersey (conducted 9/18 through 9/23) finds:

  • Among 406 Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (46% to 15%) in a statewide primary; former V.P. Al Gore trails at 11%, former Sen. John Edwards at 7%. All other candidates receive less than five percent each.
  • Among 408 Republicans, former Mayor Rudy Giuliani leads former Sen. Fred Thompson (45% to 12%) in a statewide primary; Sen. John McCain trails at 8%, former Gov. Mitt Romney at 6%. All other candidates receive less than five percent each.
  • General election match-ups:

    Giuliani 45%, Clinton 44%
    Giuliani 49%, Obama 40%
    Giuliani 50%, Edwards 39%

    Clinton 46%, McCain 41%
    Obama 44%, McCain 41%
    Edwards 44%, McCain 40%

    Clinton 48%, Thompson 36%
    Obama 49%, Thompson 34%
    Edwards 48%, Thompson 34%

    Clinton 52%, Romney 33%
    Obama 51%, Romney 31%
    Edwards 51%, Romney 30%

POLL: Rasmussen Richardson vs Giuliani/Thompson

A new Rasmussen Reports automated survey of 1,200 likely voters nationwide (conducted 9/21 through 9/23) finds Gov. Bill Richardson edging out former Sen. Fred Thompson (42% to 41%) in a national general election match-ups while former Mayor Rudy Giuliani narrowly leads Richardson (43% to 40%).

POLL: CNN New Hampshire Dem Primary

A new CNN/WMUR statewide survey (story, results) of 307 likely Democratic primary voters in New Hampshire (conducted 9/17 through 9/24 by the University of New Hampshire Survey Center) finds:

  • Sen. Hillary Clinton leads Sen. Barack Obama (41% to 19%) in a statewide primary; former Sen. John Edwards trails at 11%, former V.P. Al Gore at 7%, Gov. Bill Richardson at 6%. Without Gore, Clinton runs at 43%, Obama at 20%, Edwards at 12%, Richardson at 6%. All other candidates receive less than five percent each.
  • 17% say they have "definitely decided" who they will vote for, 28% say they are "leaning toward someone" and 55% say they are "still trying to decide."
  • 93% are satisfied with the field of Democratic candidates; 8% are not satisfied.

Update: Full results are now available from University of New Hampshire.

Catching Our Breath Remainders

Politico's Elizabeth Wilner tracks the drop in Republican identification as measured by the NBC/Wall Street Journal poll.

The Des Moines Register reports on a focus group of Republican caucus-goers (via Martin)
First Read reports that Pollster Bill McInturff is leaving the McCain campaign.

Gallup Guru Frank Newport tells us ten things we may not know about recent poll results and "unspins" Hillary Clinton's recent Sunday talk show appearances.

The Corner's Kathryn Jean Lopez cautions about reading too much into early general election polls with the memory of some very wrong predictions from 1980.

Open Left's brklyngrl digs deep into the combined crosstabs released by the Cook Political Report/RT Strategies polls.

Boris Grushin, a pioneer of surveys in, yes, the Soviet Union died Tuesday in Moscow at age of 78 (New York Times via AAPOR's member's only listserv).

And finally, the "read it all" item for this list: ABC's Gary Langer answers criticism of the recent ABC/BBC/NHK survey of Iraqi citizens and provides a detailed description of the unique poll vetting operation at ABC News:

Years ago, with the support of management, we set up a polling standards and vetting operation. First we developed fair-minded but rigorous standards for what survey research we will and will not report. And then a procedure by which any survey being considered for air at ABC News goes through my unit first - or is supposed to - for a review in which we check it out, and either clear it for air or kick it out the back door.

POLL: Rasmussen Florida Primary

A new Rasmussen Reports automated survey of likely primary voters in Florida (conducted 9/19 through 9/20) finds:

  • Among 527 likely Democratic primary voters, Sen. Hillary Clinton leads Sen. Barack Obama (47% to 22%) in a statewide primary; former Sen. John Edwards trails at 11%.
  • Among 508 likely Republican primary voters, former Mayor Rudy Giuliani leads former Sen. Fred Thompson (29% to 23%) in a statewide primary; Sen. John McCain trails at 12%, former Gov. Mitt Romney at 11%.
  • All other candidates receive less than five percent each.

View all Florida Primary poll data at Pollster.com:

AFLTopDems190.png AFLTopReps190.png
Democrats Republicans

POLL: Gallup Party Favorability

New analysis from the most recent Gallup national survey of 1,010 adults (conducted 9/14 through 9/16) finds:

  • 53% rate the Democratic party favorably; 43% rate it unfavorably. 38% rate the Republican party favorably, 59% rate it unfavorably.
  • 54% say the Democratic party "will do a better job of keeping the country prosperous;" 34% say the Republican party will.
  • 47% say the Democratic party "will do a better job of protecting the country from international terrorism and military threats;" 42% say the Republican party will.

POLL: SurveyUSA VA, MO, IA, NM General Election Match-ups

Four new SurveyUSA automated surveys in Virginia, New Mexico, Iowa, Missouri of about 500 registered voters in each state (conducted 9/14 through 9/16) finds:

  • Virginia:
    Clinton 50%, Giuliani 44%
    Clinton 50%, Thompson 43%
    Clinton 53%, Romney 38%
    Obama 46%, Giuliani 45%
    Thompson 47%, Obama 45%
    Obama 50%, Romney 38%
    Edwards 48%, Giuliani 43%
    Edwards 49%, Thompson 39%
    Edwards 52%, Romney 33%
  • Missouri:
    Giuliani 48%, Clinton 45%
    Clinton 48%, Thompson 45%
    Clinton 51%, Romney 40%
    Obama 46%, Giuliani 44%
    Obama 48%, Thompson 45%
    Obama 51%, Romney 40%
    Edwards 47%, Giuliani 42%
    Edwards 50%, Thompson 40%
    Edwards 56%, Romney 32%

Full crosstabs also available for Iowa and New Mexico.

On Plouffe's Memo and the Disclosure Project

Topics: 2008 , Disclosure , The 2008 Race

A public memo circulated today by the Obama campaign and authored by campaign manager David Plouffe (via Marc Ambinder) argues that "Iowa is fundamentally a close three-way race with Obama, Clinton and Edwards all within the same range in most public polling." His characterization is reasonable, especially if one defines that "same range" as roughly seven percentage points wide. Our own trend estimates for Iowa based on all available public polls show both Clinton and Edwards running a few points apart in the mid twenties (Clinton does slightly worse and Edwards slightly better if we exclude the polls by the American Research Group) with Obama trailing at roughly 20%.

But Plouffe goes on to make an assertion that is harder to evaluate:

[P]olls consistently under-represent in Iowa, and elsewhere, the strength of Barack's support among younger voters for at least three reasons. In more than one survey, Barack's support among Iowa young voters exceeded the support of all the other candidates combined. First, young voters are dramatically less likely to have caucused or voted regularly in primaries in the past, so pollsters heavily under-represent them. Second, young voters are more mobile and are much less likely to be at home in the early evening and thus less likely to be interviewed in any survey. Third, young voters are much less likely to have a landline phone and much more likely to rely exclusively upon cell phones, which are automatically excluded from phone surveys. So all of these state and national surveys have and will continue to under-represent Barack's core support – in effect, his hidden vote in each of these pivotal early states. Of course, there are organizational challenges associated with maximizing this support, but we are heavily focused on that task.

Each of the Plouffe's three arguments is at least theoretically plausible, particularly in Iowa, but hard to prove or disprove conclusively with the data available.

Consider the cell phone effect. We know that younger adults are much more likely to live in cell-phone only households, that unweighted national poll samples tend to skew older as a result but that age-related bias tends to fade to just a percentage point or two (at most) when pollsters adjust their adult samples to match census age estimates. However, in a state like Iowa, the big polling challenge is to select the "likely caucus goers" that will hopefully represent the tiny sliver of adults that will choose to participate in the caucus. The “census norms” available for all adults are of much less utility when trying to determine the appropriate demographic composition of the one-in-ten voters that we hope will represent likely caucus-goers.

Pollsters will argue and disagree among themselves about the best way to model and weight likely voters in a state like Iowa. We will not be able to resolve those arguments here. What the rest of us should be looking for, at least, is whether the various public polls are showing variation in their age composition and whether any such variation is making a tangible difference in the results.

Although Plouffe may be cherry-picking an unusally favorable result, the national surveys consistently show Obama doing better among younger voters. For example, in a combined crosstabulation of its five most recent national polls (conducted since June), the Cook Political Report/RT Strategies survey shows Obama receiving 30% of the vote among 18-24 year olds, 24% among 25 to 49 year olds and only 17% among those over 50. So if early state polls are under representing younger voters, they may be slightly understating Obama's support.

But how much is the age difference in Iowa and how much do the Iowa polls (or any of the other early states) vary by vary in their age composition? Who knows? As far as I can tell, only the Time survey conducted by in late August has reported its composition by age and other demographics.

All of which brings me back to the Pollster.com disclosure project. One of the most important reasons why we are requesting additional details on the polls conducted in Iowa and the other early states is to allow us all to better evaluate arguments like the one Obama's campaign manager made today. So please read my post from earlier today and comment or blog if you think this is a worthy idea. We would appreciate your support.

The Pollster.com Disclosure Project

Topics: 2008 , Iowa , Likely Voters , The 2008 Race

Over the last few months I have written a series of posts that examined the remarkably limited methodological information released about pre-election polls in the early presidential primary states (here, here and here, plus related items here). The gist is that these surveys often show considerable variation in the types of "likely voters" they select yet disclose little about the population they sample beyond the words "likely voter." More often than not, the pollsters release next to nothing about how tightly they screen or about the demographic composition of their primary voter samples.

Why do so many pollsters disclose so little? A few continue to cite proprietary interests. Some release their data solely through their media sponsors, which in the past limited the space or airtime available for methodological details (limits now largely moot given the Internet sites now maintained by virtually all media outlets and pollsters). And while none say so publicly, my sense is that many withhold these details to avoid the nit-picking and second guessing that inevitably comes from unhappy partisans hoping to discredit the results.

Do pollsters have an ethical obligation to report methodological details about who they sampled? Absolutely (and more on that below), and as we have learned, most will disclose these details on request as per the ethical codes of the American Association for Public Opinion Research (AAPOR) and the National Council on Public Polls (NCPP). Regular readers will know that we have received prompt replies from many pollsters in response to such requests (some pertinent examples here, here, here and here).

The problem with my occasional ad hoc requests is that they arbitrarily single out particular pollsters, holding their work up to scrutiny (and potential criticism) while letting others off the hook. My post a few weeks back, for example, focused on results from Iowa polls conducted by the American Research Group (ARG) that seemed contrary to other polls. Yet as one alert reader commented, I made no mention of a recent Zogby poll with results consistent with ARG. And while tempting, speculating about details withheld from public view (as I did, incorrectly, in the first ARG post), is even less fair to the pollsters and our readers.

So I have come to this conclusion: Starting today we will begin to formally request answers to a limited but fundamental set of methodological questions for every public poll asking about the primary election released in, for now, a limited set of states: Iowa, New Hampshire, South Carolina or for the nation as a whole. We are starting today with requests emailed to the Iowa pollsters and will work our way through the other early states and national polls over the next few weeks, expanding to other states as our time and resources allow.

These are our questions:

  • Describe the questions or procedures used to select or define likely voters or likely caucus goers (essentially the same questions I asked of pollsters just before the 2004 general election).
  • The question that, as Gary Langer of ABC News puts it, "anyone producing a poll of 'likely voters' should be prepared to answer:" What share of the voting-age population do they represent? (The specific information will vary from poll to poll; more details on that below).
  • We will ask pollsters to provide the results to demographic questions and key attributes measures among the likely primary voter samples. In other words, what is the composition of each primary voter sample (or subgroup) in terms of gender, age, race, etc.?
  • What was the sample frame (random digit dial, registered voter list, listed telephone directory, etc)? Did the sample frame include or exclude cell phones?
  • What was the mode of interview (telephone using live interviewers, telephone using an automated, interactive voice response [IVR] methodology, in-person, Internet, mail-in)?
  • And in the few instances where pollsters do not already provide it, what was the verbatim text of the trial heat vote question or questions?

Our goal is to both collect this information and post it alongside the survey results on our poll summary pages, as a regular ongoing feature of Pollster.com. Obviously, some pollsters may choose to ignore some or all of our requests, but if they do our summary table will show it. We are starting with Iowa, followed by New Hampshire, South Carolina and the national surveys, in order to keep this task manageable and to determine the feasibility of making such requests for every survey we track.

Again, keep in mind that the ethical codes of the professional organizations of survey researchers require that pollsters adequately describe both the population they surveyed and the "sample frame" used to sample it. The Code of Ethics of the American Association for Public Opinion Research, for example, lists "certain essential information" about a poll's methodology that should be disclosed or made available whenever a survey report is released. The relevant information includes:

The exact wording of questions asked . . . A definition of the population under study, and a description of the sampling frame used to identify this population . . . A description of the sample design, giving a clear indication of the method by which the respondents were selected by the researcher . . . Sample sizes and, where appropriate, eligibility criteria [and] screening procedures.

The Principles of Disclosure of the National Council on Public Polls (NCPP) and the Code of Standards and Ethics of the Council of American Survey Research Organizations (CASRO) include very similar disclosure requirements.

We should make it clear that we could ask many more questions that might help assess the quality of the survey or help identify methodological differences that might influence the results. We are not asking, for example, about response rates, the method used to select respondents within each household, the degree to which the pollster persists with follow-up calls to unavailable respondents or the time of the day in which they conduct interviews. We have limited our requests to try to make it easier for pollsters to respond while also focusing on the issues that seem of greatest importance to the pre-primary polls.

What can you do? Frankly, we would appreciate your support. If you have a blog, please post something about the Pollster Disclosure Project and link back to this entry (and if you do, please send us an email so we can keep a list of supportive blogs). If not, we would appreciate supportive comments below. And of course, criticism or suggestions on what we might do differently are also always welcome.

(After the jump - a more exhaustive list of the questions that we will use to determine the percentage of the voting age population represented by each sample)

Continue reading "The Pollster.com Disclosure Project"

POLL: Rasmussen Obama vs, Georgia Senate

Two new Rasmussen Reports automated surveys find:

  • Among 800 likely voters nationwide (conducted 9/12 through 9/20), Sen. Barack Obama narrowly leads former Gov. Mitt Romney (46% to 43%) and Sen. John McCain (46% to 41%) in national general election match-ups.
  • Among 500 likely voters in Georgia (conducted 9/12), Sen. Saxby Chambliss leads Democratic candidates Dale Caldwell (49% to 33%) and Vernon Jones (53% to 28%) in statewide general election match-ups for U.S. Senate.

POLL: American Research Group Bush Approval

A new American Research Group national survey of 1,100 adults (conducted 9/18 through 9/21) finds:

  • 34% approve of the way George Bush is handling his job as president; 60% disapprove.
  • 33% approve of the way Bush is handling the economy; 58% disapprove.

POLL: Lake Research (D) Vulnerable Dem Seats

Via the Washington Post, a recently released Latino Policy Coalition survey of 1,200 likely general election voters in 31 "vulnerable" Congressional Districts currently held by Democrats (conducted 7/31 through 8/5 by Lake Research Partners (D) ) finds:

  • 23% say "things in the country are going in the right direction;" 65% feel "they have gotten pretty seriously off on the wrong track."
  • Among a half sample, former Mayor Rudy Giuliani leads Sen. Hillary Clinton (49% to 39%) and narrowly leads Sen. Barack Obama (41% to 40%) in general election match-ups.
  • 51% would vote to re-elect their Democratic candidate for U.S. Congress (name given) if the election were held today; 32% would vote for the unnamed Republican.
  • When prompted that the Democratic incumbent "is a strong supporter" of Clinton or Obama and will support their "liberal agenda of big government and higher taxes" and will be a "rubber stamp" and "forget the values of our district," the named Democratic candidate leads the unnamed Republican candidate by six percentage points.

Note: Lake Research Partners also polls on behalf of Democratic presidential candidate Joe Biden.