September 30, 2007 - October 6, 2007


Tribe Time Remainders*

Kathy Frankovic revisits the classic "forbid or allow" question experiment in the context of the Mahmoud Ahmadinejad speech at Columbia University.

David Hill sees hope for conservatives in a recent Gallup poll result on the Supreme Court.

Adam Nagourney reviews the data on New Hampshire's independent voters.

Frank Newport says the national polls should "cause significant concern" for the Romney and Obama campaigns.

Marjorie Connelly (polling director at the New York Times) examines how much voters "consider the candidates' personal lives" as well as their records.

Matt Stearns quotes "polling experts" who say Hillary Clinton's negatives "may not be politically fatal or even much of a drag."

Greg Sargent has yet another polling and strategy memo from the Giuliani campaign.

Doug Schoen argues that Edwards is the most electable (via Ben Smith) in a column that may surprise Glenn Greenwald.

Carl Bialik examines the way web sites featuring user ratings of "rate beer, restaurants, board games and movies" compile their rankings (for fans of Thomas Bayes).

*Because you can take the boy out of Cleveland...

Yet More on Undecided vs. "Decided"

Topics: 2008 , The 2008 Race

Earlier this morning, I posted a response to a comment on yesterday's entry from reader Danielle Clark that on reflection is worth discussing further on the main page. Her comments concern the ways that pollsters handle "undecided" voters:

For those wondering why Obama's poll numbers seem surprisingly low, check the raw results as opposed to the press releases. If the raw results aren't posted on the polling company's site, there's probably a reason for that. ;-)

Awhile back I learned it's considered acceptable to drop, re-allocate or report "Don't Know" answers - it all depends on the polling company's choice, and they don't tend to disclose which they did. This can cause misleading results, especially at this point when there are still a lot of people who don't know enough about Obama (and other candidates) to want to commit to one.

For example, a poll may find 5% support Obama, 10% Clinton, 10% others, and 75% don't know. It's considered acceptable to report this as 20% support Obama, 40% Clinton, and 40% others - which implies everyone asked had an opinion of some sort.

Another reason to insist on seeing the raw data is to know what questions were asked before the question one's interested in - those can clearly shape the answer.

She also linked to a useful paper on "Interpreting Polls " put out by a parliamentary librarian in Australia. Aside from our regular commentary here, I would also recommend the resources available on the new web page of the American Association for Public Opinion Research (AAPOR).

Danielle was right to recommend examining the verbatim text of survey questions and the way pollsters calculate their results. Where available, we include links to the most complete report of results in our "poll update" posts and in the tables that appear below the charts in our Polls section. The Polling Report also provides verbatim text for poll questions. Follow those links to see the full results and verbatim text when available.

However, we need to be careful to distinguish between tables or references in news stories that omits the results for the undecided category and those that reallocate undecided voters as Danielle describes.

In the US, virtually all polls are now reporting trial heat results based on calculations that include the undecided or "don't know" category. Some pollsters will put out "projections" based on their final poll before the election that use various methods to reallocate undecided voters (the Polling Report has a handy set of tables showing both "vote projections" and "final trial heats" from 2004 side by side). But I know of no pollsters putting out any such projections right now.

One way to check, if the undecided category is omitted, is to add up the results to each candidate. If they total 100%, then the results have almost certainly been recalculated.

More to the point, all of the results posted and plotted in the charts and tables in the Polls section of Pollster.com -- including the numbers discussed in yesterday's post -- are based on calculations that include the undecided category.

It is also important to distinguish between the undecided category on a trial heat question and separate questions that probe how certain or "decided" voters say they are. The undecided category tends to understate the real potential for change in a political contest. In recent weeks have written about questions asked on recent LA Times/Bloomberg and CNN/WMUR/UNH polls showing huge number of voters in Iowa and New Hampshire that are still less than certain about their choice.

Danielle may be thinking about computations that take those "uncertain" voters into account that Obama himself has been touting on the campaign trail. For example, NBC's First Read reported the following recent comments by Barack Obama at a rally in New Hampshire:

[Obama] also referred to a recent poll that had Sen. Clinton in the lead by twenty points, and said that it meant nothing besides what percentage of people were supporting which candidate. "Twenty percent of voters are with Sen. Clinton, 10 percent are with me, Edwards has about 5 percent and the rest of the candidates have less than that."

Citing that same poll, Obama said the most important fact overlooked by the pundits were the number of undecideds still up for grabs. "There are 55 percent of voters who are undecided. That's 55 to 60 percent of the people who are waiting to hear from you," he told the crowd of campaign volunteers made up of students from colleges in Massachusetts and across New Hampshire as well as local volunteers.

Obama was probably referring to the recent CNN/WMUR/UNH poll that gave Clinton a 22 point lead over Obama (with only 9% "undecided"), but also showed 55% of New Hampshire likely Democratic primary voters saying on a separate question that they are "still trying to decide" which candidate to support. If I set aside the uncertain 55% and recalculate vote preference numbers among those who say they are "definitely decided" or "leaning," I get 23% for Clinton, 10% for Obama and 6% for Edwards.

ABC/Post Poll: Surge or Outlier?

Topics: 2008 , Iowa , The 2008 Race

About that new ABC News/Washington Post poll released yesterday, the one showing Hillary Clinton winning the support of 53% of "leaned Democrats" nationally to 20% for Barack Obama, 13% for John Edwards and single digits for the other candidates. Since this survey was the first to show a majority supporting Clinton, a lot of bloggers let loose with the adjectives, writing that she "pulls away" with a lead that is "crushing" or "explodes," telling us that we have reached "knockout punch time," that "she's killin' it," or declaring simply, "oh baby, when she moves, she moves." My friend Chris Cillizza parsed Clinton's "surge" in his Washingtonpost.com blog and concluded:

It doesn't take a polling expert to understand that Sen. Hillary Rodham Clinton's showing in the latest Washington Post/ABC News poll is a major moment in the campaign.

There is obviously no denying that the poll has created the perception of a major change in the race, but has a Clinton really experienced a "surge" of support over the last month, as indicated by the 12-point increase (from 41% to 53%) in her support on the Post/ABC poll? I'm not so sure.

I checked our national chart for the Democrats late yesterday and saw that this new ABC/Post poll appears to be very different from the others conducted over the last month. I quickly realized, however that this particular comparison highlights a frustrating limitation of our chart, which is that about half of the national polls plotted include Al Gore as an option and about half do not. Since Clinton typically picks up roughly five points on polls - including the new ABC/Post survey - that do not offer Gore as a choice, our standard chart may make the ABC/Post survey look like more of an outlier than it is.

So we decided to devote some time today to producing a chart that displays only the results for polls without Gore (using recalculated votes based on second choice when available). The result below plots 66 national polls released so far this year (about a third fewer than our standard version - as not all provide a "vote without Gore"):


The chart makes a few things clear. First, the 53% result for new Post/ABC poll is more of an "outlier" from the regression trend line than any poll conducted this year (it's the purple dot at the extreme top right of plot area). At 53%, the polls estimate of Clinton's support falls a full ten percentage points higher than our current estimate of the trend (42.5%) even without Gore in the race.

Second, the addition of the new poll has not budged our trend line. That is mostly by design. Professor Franklin set the sensitivity of the regression model to prevent the inevitable outlier from causing wild gyrations in the trend line. That is one of the reasons we prefer to use regression estimates over simple rolling averages.

One of the reasons we put so much effort into these plots is that they paint a picture of the random variation that is inherent in polling. You can actually see the random variation. While it typically appears as a random and predictable "cloud" of points, bigger outliers still occur from time to time. The more experience you have looking at poll numbers, the more you learn to understand that even with well designed polls, outliers happen. Our advice to poll consumers - as reflected in the design of the charts -- is to try to avoid over-reacting to any one poll.

Of course, we do not yet know whether this poll is really a statistical outlier. Other polls have obviously been showing a more gradual increase in her support recently, and it is still theoretically possible that Clinton's support suddenly lurched up ten points last week. Senator Clinton had appeared on all of the Sunday morning talk shows on September 23, just four days before this poll went into the field. Earlier that week sho also rolled out her plan to reform health care, an issue that ABC/Post and other polls confirm as one her great perceived strengths. But a sudden upward surge of this magnitude does not seem very likely, if only because no other news event so far this year has caused anywhere near as much change in the Democratic race.

We should know soon enough. Unfortunately, the AP-IPSOS survey released yesterday does not help resolve this issue, as Clinton's Sunday talk show tour de force occurred in the middle of their field period and (I'm told) after they had completed most of the interviews on the main sample.

So we will wait and see. But I'll wager that a month from now the real trend will not look nearly as dramatic as the one suggested by yesterday's news.

PS: It will probably take a few days, but we will have a full, regularly updating version of the national "vote without Gore" up soon.

UPDATE (10-16): More recent polls do move the Clinton trend line up and make the Post/ABC poll look much less like an outlier. More here.

POLL: InsiderAdvantage Republican Primaries in Iowa, New Hampshire, South Carolina, Florida, and Michigan

Five new InsiderAdvantage statewide surveys of likely Republican primary voters in Iowa, New Hampshire, South Carolina, Florida, and Michigan (conducted 10/2 through 10/3) finds:

  • Among 1,339 likely caucus-goers in Iowa, former Gov. Mitt Romney leads former Mayor Rudy Giuliani (24% to 16%) in a statewide caucus; former Sen. Fred Thompson and former Gov. Mike Huckabee both trail at 13%, Sen. John McCain at 10%, and Sen. Sam Brownback at 5%.
  • Among 1,822 likely primary voters in New Hampshire, Romney leads Giuliani (28% to 20%) while McCain trails at 17%, Thompson and Huckabee both at 8%, and Rep. Ron Paul at 6%.
  • Among 1,281 in South Carolina, Thompson (at 21%) leads Giuliani, Romney, and McCain (all at 16%) while Huckabee trails at 11%.
  • Among 725 in Florida, Giuliani leads Thompson (29% to 19%) while Romney trails at 16%, McCain at 10%, and Huckabee at 6%.
  • Among 1,190 in Michigan, Giuliani edges out Romney (19% to 16%), while McCain trails at 15%, Thompson at 14%, Huckabee at 6%, and Paul at 5%.
  • All other candidates receive less than five percent each.

We will add links once they are available.

POLL: Gallup Dem Candidates

New analysis from the most recent Gallup national survey of 537 Democrats and those who lean Democratic (conducted 9/24 through 9/27) asks whether Sen. Hillary Clinton, Sen. Barack Obama, or former Sen. John Edwards would do the best job when faced with 17 different challenges as president.

The analysis finds:

  • "While Clinton dominates on core policy issues, Sen. Barack Obama does relatively well on the handful of items included that tap into the candidates' ability to relate to people and heal divisions in the country. Democrats do not consider former Sen. John Edwards the best candidate on any issue."
  • "The fact that Edwards receives his highest rating on the commander-in-chief dimension could say more about what Democrats think of Clinton and Obama on this issue, than what they think of Edwards."

POLL: NBC/WSJ National Republican Survey

A new NBC News/Wall Street Journal national survey (NBC story, results; WSJ story, results) of 606 likely Republican primary voters (conducted 9/28 through 9/30) finds:

  • Former Mayor Rudy Giuliani leads former Sen. Fred Thompson (30% to 23%) in a national primary; Sen. John McCain trails at 15%, former Gov. Mitt Romney at 10%. All other candidates receive less than five percent each.
  • 38% of Republican think the next president should "take an approach similar to that of George W. Bush:" 48% think the next president should take a different approach.
  • 59% of Republicans agree that "foreign trade has been bad for the U.S. economy, because imports from abroad have reduced demand for American-made good, cost jobs here at home, and produced protentially unsafe products;" 32% choose the alternate statement that "foreign trade has been good for the U.S. economy, becuase demand for U.S. products abroad has resulted in economic growth and jobs for Americans here at home and provided more choices for consumers."

POLL: CNN/WMUR/UNH New Hampshire Senate

Additional results from the recent CNN/WMUR/UNH statewide survey of 508 adults in New Hampshire (conducted 9/17 through 9/24) find:

  • Former Gov. Jeanne Shaheen leads Sen. John Sununu (54% to 38%) in a statewide general election match-up for U.S. Senate.
  • 56% rate Shaheen favorably, 25% rate her unfavorably.
  • 40% rate Sununu favorably, 37% rate him unfavorably.

POLL: AP-Ipsos National Primary

A new AP-Ipsos national survey (story, results) of 1,317 adults (conducted 9/21 through 9/25; released 10/3) finds:

  • Among 482 Democrats and those who lean Democratic, Sen. Hillary Clinton leads Sen. Barack Obama (40% to 26%) in a national primary; former Sen. John Edwards trails at 12%. When former V.P. Al Gore is included, Clinton runs at 35%, Obama at 23%, GOre at 12%, and Edwards at 9%. All other candidates receive less than five percent each.
  • Among 358 Republicans and those who lean Republican, former Mayor Rudy Giuliani (at 21%) runs slightly ahead of former sen. Fred Thompson and Sen. John McCain (both at 17%) in a national primary; former Gov. Mitt Romney trails at 10%, former Gov. Mike Huckabee at 7%, and former Speaker Newt Gingrich at 5%. All other candidates receive less than five percent each.
  • "Clinton led Obama among whites by 35 percent to 18 percent, Blacks were essentially evenly divided, 40 percent for Obama and 38 percent for Clinton."

POLL: Democracy Corps (D) SCHIP, Bush

A new national survey (memo, results) from Democracy Corps (D), the joing product of Democratic pollster Stan Greenberg and Democratic consultants James Carville and Bob Shrum, has released a new national survey of 1,000 likely voters (conducted 9/16 through 9/18) focusing on the debate over the Children's Health Care (S-Chip) Bill and finds:

  • 38% approve of the way George Bush is handling his job as president; 57% disapprove.
  • "In the latest Democracy Corps survey, we examined a full battery of economic worries, but health care tops everything else, rising dramtically in the most recent period."

The Other Numbers You Should Watch

Topics: 2008 , The 2008 Race

The most useful news about the presidential primary races I came across in the last 24 hours - especially for those of us who watch the survey data - comes not from any new poll but rather from a little known media researcher named Evan Tracey. Even better news for political junkies (as I learned via links by First Read and Ben Smith), Tracey blogs on Advertising Age's "Campaign Trial."

Who is Even Tracey and why should you read him? Tracey is the founder and chief operating offer of something called the Campaign Media Analysis Group (CMAG) that performs a unique service for its very high paying political clients. It systematically monitors the transmissions of the national broadcast networks, 25 national cable networks and local advertising in the countries top 100 markets, uses software to electronically identify and code unique advertisements. They are also now apparently tracking Internet advertising as well. Through this process, CMAG compiles detailed reports on what candidates are spending on broadcast and cable advertising, what ads they are running and how many times viewers are seeing them (more details on their methodology here and here).

As a campaign pollster, I had the opportunity to be on the receiving end of CMAG's tracking service just once, and it was absolutely phenomenal. No media tracking report I saw in 20 years as a campaign pollster was anywhere near as throughout or detailed. Unfortunately (for all of us), Tracey charges his clients a high price for his data and is understandably reluctant to give away his bread and butter.

Simply put, no one has better data on the candidates' ad spending, and this is why I am excited to see Tracey sharing a smattering of his valuable data online. More often than not, the ad expenditures by the candidates explain the trends we see in the early state charts here at Pollster.com.

Yesterday, for example, Tracey tells us that on the Republican side:

Mitt Romney has aired nearly 10,000 TV spots since late February and spent close to $8 million dollars with a majority of his spending in Iowa and New Hampshire. He is now expanding this strategy to South Carolina and Florida.

Meanwhile, he writes, the other Republicans have either placed token ad buys or run nothing at all. McCain's new ad has been airing for just the last week. Take a look at our charts for Iowa and New Hampshire Republicans and you will see the sharp increase for Romney in the second and third quarters of 2007 that does not occur in South Carolina or Florida. But check that just released ARG survey in South Carolina that shows a huge upward spike (from 9% to 26%) for Romney. That change may turn out to be an outlier, but it would not be a surprising change given what Tracey reports.

Of course, observers should also keep in mind that until McCain's buy started last week, none of the other Republicans have run ads. So a huge question for the early states is whether Romney's lead will hold as other candidates begin to air their own advertisements.

Now consider what Tracey tells us about the Democrats:

Bill Richardson leads the pack, airing over 4,000 commercials in an effort to move up in the polls in the early states of Iowa and New Hampshire. Barack Obama is close behind with close to $2 million invested in TV time, mostly in Iowa. Hillary Clinton and John Edwards, because of their profiles, were able to skip the typical bio spots and have spent far less. The Edwards campaign has made relatively small buys in Iowa and New Hampshire, but has also used ad buys tactically on cable TV and in Washington to inject himself into the Iraq war debate. Clinton, on the other hand, has spent over $1 million on TV in just five weeks and her latest buys on South Carolina radio signal the campaign is serious about running the table on the early primaries.

Others on the Democratic side of the ledger have gotten in on the act with the Biden and Dodd campaigns combining for over $1 million dollars in TV ad spending with compelling messages about the war and global warming.

Not surprisingly, the poll trends follow the money: Richardson has seen a modest bump up in his support to roughly 12% in Iowa and 10% in New Hampshire. Obama, Tracey tells us, has run ads "mostly in Iowa." Thus, Obama's trend line is up in Iowa, but trending slightly downward in New Hampshire and elsewhere. And while he's a little unclear on exactly where Clinton has spent that $1 million on the last five weeks, he implies she's buying television in both Iowa and New Hampshire and radio in South Carolina. Her trend lines are on the rise in all three states, as they are nationally.

Also, as I pointed out last week, we can see clearer evidence of the ad buys in the favorable ratings of initially lesser known candidates. The latest CNN/WMUR/UNH poll of New Hampshire shows that Richardson, Biden, Dodd and Obama have seen double digit jump in their favorable ratings since June. Richardson's favorable rating on that poll has nearly doubled (from 27% to 53%).


Definitely go read all of Tracey's post - he is providing us with invaluable data on the most important part of the campaign getting underway in the early states.

PS: Marc Ambinder has also been a good source of reports on media buys, posting most recently on advertising by the Democratic candidates in Iowa and on Clinton's radio buy in South Carolina.

POLL: Quinnipiac New York Primary, General Election

A new Quinnipiac University statewide survey of 1,504 registered voters in New York State (conducted 9/24 through 9/30) finds:

  • Among 637 Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (47% to 15%) in a statewide primary; former V.P. Al Gore trails at 9%, former Sen. John Edwards trails at 7%.
  • Among 476 Republicans, former Mayor Rudy Giuliani (at 48%) leads Sen. John McCain and former Sen. Fred Thompson (both at 8%) in a statewide primary; former Speaker Newt Gingrich and former Gov. Mitt Romney both trails at 5%.
  • All other candidates receive less than five percent each.
  • General election match-ups:

    Clinton 52%, Giuliani 41%
    Obama 45%, Giuliani 45% (tie)
    Giuliani 45%, Edwards 44%

POLL: NewsMax/Zogby GOP New Hampshire

New results from the most recent NewsMax/Zogby statewide survey of 502 likely Republican primary voters in New Hampshire (conducted 9/26 through 9/28) finds former Gov. Mitt Romney narrowly leading former Mayor Rudy Giuliani (24% to 21%) in a statewide primary; Sen. John McCain trails at 16%, former Sen. Fred Thompson at 7%, and former Gov. Mike Huckabee at 5%. All other candidates receive less than five percent each.

POLL: ABC/POST Bush, Primary, Iraq

A new ABC News/Washington Post national survey (Post story, results; ABC story, results) of 1,114 adults (conducted 9/27 through 9/30) finds:

  • 33% approve of the way George W. Bush is handling his job as president -- "equal to his career low in Post-ABC polls;" 64% disapprove.
  • Among 592 Democrats and those who lean Democratic, Sen. Hillary Clinton leads Sen. Barack Obama (53% to 20%) in a national primary; former Sen. John Edwards trails at 13%. All other candidates trail at less than five percent each.
  • Among 398 Republicans and those who lean Republican, former Mayor Rudy Giuliani leads former Sen. Fred Thompson (34% to 17%) in a national primary; Sen. John McCain trails at 12%, former Gov. Mitt Romney at 11%, and former Gov. Mike Huckabee at 8%. All other candidates trail at less than five percent each.
  • 67% say Congress should reduce the Bush administration's request for 190 billion dollars to fund the wars and related U.S. activites in Iraq and Afghanistan over the next year, which "is about 40 billion dollars more than first estimated;" 27% say Congress should approve the request.

Update: ABC story and results on the 2008 national primary now available.

When Pollsters Attack: Epilogue

Topics: 2008 , Disclosure , The 2008 Race

Nearly two weeks ago, just before we kicked off our Disclosure Project, InsiderAdvantage pollster Matt Towery used a syndicated column headlined "What's a Quinnipiac?" to attack the Florida polls conducted by the Quinnipiac University Polling Institute. Towery not only highlighted how his polls differed from a recent Quinnipiac survey but also commissioned Mason-Dixon Polling and Research to conduct a parallel poll to prove his point. Towery's unusually jocular broadside amounted to "one pollster whack[ing] the other upside the head," as Politico's Jonathan Martin put it.

"At the very least," Towery argued,

Quinnipiac numbers should stop being taken at face value as the paragon of accuracy in Florida. Somewhere in their methodology they continue to misread the state they claim to know so intimately.

When I looked at the four polls of Florida Republicans conducted recently by InsiderAdvanage, Quinnipiac and Mason-Dixon, the differences between them seem explained mostly by the inclusion of Newt Gingrich as a candidate in the Quinnipiac poll and the fact that Fred Thompson's announcement occurred during the fielding of the Quinnipiac poll but before the others. Still, Towery's suggestion that the Quinnipiac differences might be found "somewhere in their methodology" led me to ask the same kinds of methodological questions as we have been asking as part of our Disclosure Project. Their responses follow, and the difference in sampling methodology adds another possible explanation. Quinnipiac's sample of "registered Republicans" samples a population roughly four times the size of the "likely voters" surveyed by InsiderAdvantage and Mason-Dixon.

Interview Dates - One question I asked only of Quinnipiac was to provide the number of interviews conducted before and after Fred Thompson's announcement of candidacy. Doug Schwartz at Quinnipiac reports that 199 (or 45%) of their 438 interviews were conducted on or before the evening of September 5. Thompson declared his intentions later that night and received a burst of positive coverage in the week that followed. While the methodologies of these surveys differ, it is worth remembering that the other polls by InsiderAdvantage and Mason-Dixon were fielded in their entirety after September 5.

Sample Frame - Although the term is a bit wonky, one of the most important ways these polls differ is in what pollsters call the "sampling frame." Put more simply, the issue is the source for the random sample of voters called by each pollster.

Quinnipiac uses a random digit dialing (RDD) methodology that contacts a random sample of all the working landline telephone numbers in Florida and then uses screen questions to select a random sample of registered Republicans. In this case, both InsiderAdvantage and Mason-Dixon selects voters at random from the list of registered Republicans provided by the Secretary of State, using both actual vote history and screen questions to identify and interview "likely" Republican primary voters.

For more information on the debate about RDD versus list sampling, see our prior posts here and here.

How Did They Select Republican Registered or Likely Voters? - The pollsters at Quinnipiac provided a complete and relatively straightforward answer. They asked two questions about vote registration and party affiliation:

Some people are registered to vote and others are not. Are you registered to vote in the election district where you now live, or aren't you?

[IF REGISTERED] Are you registered as a Republican, Democrat, some other party or are you not affiliated with any party?

Both Towery and Brad Coker at Mason-Dixon were both initially reluctant to describe the specifics of their likely voter selection procedures, citing the need to protect "proprietary" methods. After a bit of email back-and-forth, however, both were willing to describe their methods in general terms. Let's start with Coker, answering on behalf of Mason-Dixon:

Our sample design and screening method takes into account voter registration, party registration, past primary voting history and likeliness to vote in the primary. Other factors that were taken into account were the age, county, gender and race of the voting population based on previous Republican primary elections.

What that means -- as I read it -- is that Mason-Dixon uses information on past vote history on the voter list to draw a random sample of a subset of Republicans that they consider most likely to vote. They ask those sampled individuals questions on "likeliness to vote in the primary" and screen out unlikely voters. Finally, they weight the demographics of the final sample based on the demographics of voters in previous Republican primaries.

Towery reports using a similar procedure at InsiderAdvantage: "We do poll off of [a list of] registered voters, but we do then cull that number down based on a voting history that gives us a more likely voter sample." They then ask a screen question to identify likely primary voters: "are you likely to vote in the_____presidential primary to be held ____."

What Percentage of the Voting Age Population Did Each Poll Represent? - The calculation for the Quinnipiac poll is relatively straightforward: They report starting with a random sample of 1,325 adults and using the questions above to identify and interview 438 registered Republicans. So the Quinnipiac Republican sample amounted to 33% of Florida adults (438 divided by 1,325).

Again, Coker and Towery were initially reluctant about sharing specific numbers, but ultimately provided the information necessary to answer my question. Let's start again with Coker and Mason-Dixon:

The population we were trying to capture was the roughly 1 million Republican voters who will be most likely to vote in January. In a universe of approximately 3.8 million registered Republicans, we targeted a population of about 1.2 million Republican voters and had an incidence of 83%.

So Mason-Dixon interviewed a sample designed to represent approximately 1 million voters (1.2 million * .83) out of 14.2 million Florida adults, or 7.0% of Florida adults.

Next, Towery and InsiderAdvantage:

The [target] universe based on our sample system was around 1.6 million. This reflects the slightly higher than normal turnout you see in a Presidential primary. Incidence rate, based on data I just received was around 75%. Based on your description this would mean a final "universe" of around 1.2 million voters (all registered) which I believe reflects the likely turnout for a GOP Pres. primary turnout.

So the Insider-Advantage interviewed as ample designed to reflect approximately 1.2 million adults, or 8.5% of Florida adults.

As should be obvious, Mason-Dixon and InsiderAdvantage sampled significantly narrower populations of Republican voters than Quinnipiac. Via email Brad Coker argues that a narrower "screen" is more appropriate to a pre-election poll aimed at projecting the preferences of likely voters: "I would question the validity of a poll of ‘registered Republican voters' simply on the grounds that 75% of those sampled probably won't be voting in January."

According to the Florida Secretary of State, the vote for Republican primary candidates totaled roughly 1 million in 2006 (Governor), 1.2 million in 2004 (Senate) and 700,000 in the 2000 presidential primary.

In response to my initial questions, Quinnipiac's Doug Schwartz sent this statement:

The methodology used by the Quinnipiac poll is similar to that of all the other major polling operations in the country. It has correctly predicted the outcome of every major race it has polled on in Florida during the past three years. For details on the methodology used, contact Doug Schwartz or visit http://www.quinnipiac.edu/x271.xml

That is true, but we should note that the final pre-election survey conducted by Quinnipiac in 2006 reported the results among "likely voters" rather than all registrants. So their primary voter "methodology" may shift as we get closer to Election Day. Pollsters continue to debate the merits of various likely voter models months prior to the election, something I covered in great detail in 2004 in the context of general elections. Putting that debate aside, however, the point is that the universes sampled in this instance are very different.

What Are the Demographics? - I asked each pollster to provide the results to demographic questions asked of their Republican samples. Both Quinnipiac and Mason-Dixon were quick to respond. The table below shows that the Quinnipiac sample is a bit younger. This is not surprising given that voters are typically older than non-voters.


Both Quinnipiac and Mason-Dixon also included the regional composition of their samples. While their regions were not identical, their definition of South Florida came close. Mason-Dixon had fewer Republicans in their Southeast Florida region (18% in Palm Beach, Broward, Dade and Monroe Counties) than Quinnipiac (23%; although the Quinnipac South Florida region also includes Hendry County, which accounts for just 0.1% of registered Republicans statewide).

This last difference is important because, according to the Mason-Dixon cross-tabulations that Coker also provided, Rudy Giuliani ran far ahead of Fred Thompson (33% to 11%; n=70) in Southeast Florida, but trailed Thompson narrowly elsewhere (22% to 26%; n=330). So the fact that Quinnipiac had a greater percentage of respondents in South Florida provides yet another explanation for Giuliani doing better statewide in their poll.

Coker also provided this information:

Since we have the Florida voter file, we know the precise demographic profile of those who have voted in previous elections (at least in terms of county, age, gender and stated race/ethnicity). Our sample matches it within 1-2% of the actual figures from the average of 2004 & 2006 GOP primary turn-outs. Deaths and out-migration could easily account for any differences.

Towery, on the other hand, was more reticent:

We don't give out our weighting percentages or our demographic regional breakdowns because those are proprietary and if we did so, it would be like Coke giving away the secret formula, well not that big, but important to us!

Which brings us back to the whole point of our Disclosure Project. We should congratulate all three pollsters for providing the "incidence" data necessary to help us answer, in essence, not just "What is a Quinnipiac?" (to borrow Towery's headline) but also, "what is a Mason-Dixon?" and "what is an InsiderAdvantage? As a result of their disclosure, we can see how different the "target populations" were and take those differences into account in assessing the results.

It took some coaxing, to be sure. Coker has previously refused similar requests on the grounds of protecting proprietary interests. Given his extensive experience in Florida (he tells me he has conducted more than 200 statewide polls in Florida since 1984), Coker was understandably reluctant about responding in this instance. So his cooperation here is noteworthy. Hopefully, other pollsters follow his lead because the general descriptions and incidence calculations provided above could be easily replicated by every pollster and released online for every poll. Similarly, a demographic composition table, like the one above, would be an easy addition to the online documentation virtually every pollster and news organization makes available for every poll.

On the other hand, Towery's "secret formula" dodge has a fundamental flaw. Coke need not give away its "secret formula" when it prints on every can, as required by law, a list of ingredients, the number of calories and the grams of carbohydrate and other nutrients contained in each serving. As should be obvious, the our Constitution's First Amendment precludes the sort of mandatory labeling for pollsters that the FDA requires for food. However, pollsters like Towery ought to start thinking about how to better label their own products in terms of their sample composition, lest some snarky blogger ask, "What's an InsiderAdvantage?"

POLL: Zogby/Newsmax New Hampshire Democratic Primary

A new NewsMax/Zogby statewide survey (story, results) of 505 likely Democratic primary voters in New Hampshire (conducted 9/26 through 9/28) finds Sen. Hillary Clinton leading Sen. Barack Obama (38% to 23%) in a statewide primary; former Sen. John Edwards trails at 12%, Gov. Bill Richardson at 8%. All other candidates receive less than five percent each.

A Different Perspective on Disclosure

Topics: Disclosure

Last Friday, in my semi-regular "remainders" wrap of interesting poll blogs of the week, I neglected to link to an interesting column from Kathy Frankovic of CBS News, which has an interesting twist on our recent focus on disclosure.

She writes of the laws in "at least 30 countries" other countries that prohibit the publication of pre-election poll results, but then also points out the unusual new law in Greece:

In Greece, however, the restriction on reporting pre-election polls was brand new, and it also carried disclosure requirements. A published opinion poll there has to be based on at least 1,000 interviews; and the questionnaire, the collected data and the survey report must be deposited with a special public committee.

After noting that the First Amendment of the U.S. Constitution would prevent any such prior restraint in our country, she considers that disclosure requirement:

The Greek law's requirement of disclosure is something that professional survey research organizations have long desired. The American Association for Public Opinion Research (AAPOR), the National Council on Public Polls (NCPP), and the World Association for Public Opinion Research (WAPOR), among others, require disclosure of information that would allow a reader or listener to judge the value of a poll.

However, these organizations also oppose government restrictions on publication of pre-election polls. (The Internet has made those restrictions more difficult to enforce. How could the French government enforce its law prohibiting the publication of poll results, if those results appeared on Web site based in Switzerland?)

She goes on to consider the implications of a lack of pre-election polling in the last two weeks of the Greek campaign. It's worth reading in full.

POLL: Rasmussen South Carolina Primary

A new Rasmussen Reports automated survey of likely primary voters in South Carolina (conducted 9/26 through 9/27) finds:

  • Among 567 Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (43% to 30%) in a statewide primary; Sen. John Edwards trails at 10%.
  • Among 863 Republicans, former Sen. Fred Thompson narrowly leads former Mayor Rudy Giuliani (20% to 24%) in a statewide primary; former Gov. Mitt Romney trails at 15%, Sen. John McCain at 11%.
  • All other candidates receive less than five percent each.

More on the Newsweek Poll

Topics: 2008 , Disclosure , The 2008 Race

Family obligations and a nagging cold virus kept me mostly off the grid this weekend while the blogs were abuzz over the latest Newsweek poll of Iowa likely caucus goers. So while late, let me add a few thoughts to those already offered over elsewhere.

First, the margins of sampling error reported by Newsweek -- +/- 7% for the likely Democratic caucus goers and +/- 9% for he Republicans - means that statistically meaningful conclusions are all but impossible regarding Barack Obama's "slight edge" (28% to 24%) over Hillary Clinton. Strictly speaking, even Mitt Romney's 9 point advantage does not attain the usual 95% confidence level that pollsters require to describe a lead as "statistically significant."

Noam Scheiber wonders about what the pollsters could say about the probability of an Obama lead among likely caucus goers, if not 95%. My best guess (assuming that the reported margins of error were based on the usual 95% confidence level) is that the probability of an Obama lead based on the Newsweek poll is about 50%. In other words, the odds of Obama "leading" on this poll are no better than a coin-flip, if we were to take repeated samplings of exactly the same design.

But Matt Yglesias makes the more important point:

It seems to me that there's no real point in arguing about the significance of the rather large +/- 7 points margin of error on this Newsweek poll . . . For something like this, uncertainty about the likely voter screen are probably going to be a bigger problem than sampling error anyway.

He is exactly right. Since July we have seen 12 public polls released in Iowa by 9 different organizations, and each appears to define and sample the likely caucus-goer universe differently. To the extent that pollsters have revealed the details, their snapshots of the electorate are poles apart, to say nothing of the candidates that those voters support. A month ago, for example, I found the percentage of first-time caucus-goers reported on four different polls of Democrats varying from 3% to 43%, with Edwards doing worse (and Clinton better) as the percentage of newcomers increased. The Newsweek survey reports 36% of likely Democratic caucus goers saying "this would be your first caucus."

Unfortunately, the Newsweek release omits many of the same methodological details left out of the other Iowa polling releases (including, remarkably enough, the number of interviews conducted with likely Democratic and likely Republican caucus goers). I have emailed Newsweek's pollsters the same questions we sent last week to the other Iowa pollsters and will include their responses when we begin reporting on the Disclosure Project

By the way, Yglesias also makes another important point: In a truly close race, the ultimate winner among the Democrats may depend on the second choices forced by the convoluted Caucus rules on those whose first choice fails to achieve "viability" (usually 15% of the vote) in their precinct. Remember that the official results for the Democrats will not be a head-count of the first preference of all caucus goers (as in a poll) but rather the estimated share of state delegates won by each candidate based on the final choices at the end of the night. So even if pollsters agreed on how to sample "likely caucus goers," the numbers would still be inconclusive in a close race.

Update:: Slate's Christopher Beam, who called just before I wrote this item, has more

POLL: American Research Group Iowa, New Hampshire, South Carolina

Three new American Research Group statewide surveys of likely caucus/primary voters in Iowa, New Hampshire, and South Carolina (conducted 9/26 through 9/29) find:

  • Among 600 likely Democratic caucus-goers in Iowa, Sen. Hillary Clinton leads Sen. Barack Obama (30% to 24%); former Sen. John Edwards trails at 19%, Gov. Bill Richardson at 10%. Among 600 Republicans, former Gov. Mitt Romney narrowly leads former Mayor Rudy Giuliani (22% to 21%); former Sen. Fred Thompson trails at 16%, Sen. John McCain at 11%, former Speaker Newt Gingrich at 5%.
  • Among 600 likely Democratic primary voters in New Hampshire, Clinton leads Obama (41% to 22%); Edwards trails at 10%, Richardson at 8%. Among 600 Republicans, Romney (at 24%) narrowly leads McCain and Giuliani (both at 20%); Thompson trails at 8%, Gingrich at 6%.
  • Among 600 likely Democratic primary voters in South Carolina, Clinton leads Obama (41% to 30%); Edwards trails at 7%, Richardson at 5%. Among 600 Republicans, Romney edges out Giuliani (26% to 23%) while McCain, Thompson, and Gingrich trail at 15%, 10%, and 7% respectively.
  • All other candidates trail at less than five percent each.

POLL: Newsweek Iowa Caucus

A new Newsweek statewide survey (story, results) of 1,215 registered voters in Iowa (conducted 9/26 through 9/27) finds:

  • Among all registered Democrats (+/- 5% margin of sampling error), Sen. Hillary Clinton narrowly leads Sen. Barack Obama (31% to 25%) with former Sen. Edwards at 21% and Gov. Bill Richardson at 6%. Among likely Democratic caucus-goers (+/- 7% margin of sampling error), Obama edges out Clinton (28% to 24%); Edwards runs at 22%, Richardson at 10%, and Sen. Joe Biden at 5%.
  • Among all registered Republicans (+/- 6% margin of sampling error), former Gov. Mitt Romney runs at 25%, former Sen. Fred Thompson at 16%, former Mayor Rudy Giuliani at 15%, Sen. John McCain at 7%, and former Gov. Mike Huckabee at 6%. Among likely Republican caucus-goers (+/- 9% margin of sampling error), Romney narrowly leads Thompson (24% to16%) while Giuliani trails at 13%, Huckabee at 12%, McCain at 9%, and Rep. Ron Paul at 5%.
  • All other candidates receive less than five percent each.

POLL: Gallup Republican Primary and Giuliani

New analysis aggregated from the four most recent Gallup national surveys of 1,690 Republicans and those who lean Republican (conducted 8/3 through 9/16) finds "[Giuliani] is no worse than tied for the lead among all major Republican constituencies, including those from groups that are most likely to disagree with him on some of these issues. And among Republicans who are most likely to vote in the primary, his lead is eight points over Thompson, 32% to 24%."