September 30, 2007 - October 6, 2007
Frankovic revisits the classic "forbid or allow" question experiment in the
context of the Mahmoud Ahmadinejad speech at Columbia University.
Hill sees hope for conservatives in a recent Gallup poll result on the Supreme Court.
Nagourney reviews the data on New
Hampshire's independent voters.
Newport says the national polls should "cause significant concern" for the Romney
and Obama campaigns.
Connelly (polling director at the New York Times) examines how much voters "consider
the candidates' personal lives" as well as their records.
quotes "polling experts" who say Hillary Clinton's negatives "may not be
politically fatal or even much of a drag."
Sargent has yet another polling and strategy memo from the Giuliani
Schoen argues that Edwards is the most electable (via Ben
Smith) in a column that may surprise Glenn
examines the way web sites featuring user ratings of "rate beer, restaurants,
board games and movies" compile their rankings (for fans of Thomas Bayes).
*Because you can take the boy out of Cleveland...
Earlier this morning, I posted a response to a comment
on yesterday's entry from reader Danielle Clark that on reflection is worth
discussing further on the main page. Her comments concern the ways that
pollsters handle "undecided" voters:
For those wondering why Obama's poll numbers seem
surprisingly low, check the raw results as opposed to the press releases. If
the raw results aren't posted on the polling company's site, there's probably a
reason for that. ;-)
Awhile back I learned it's considered acceptable to
drop, re-allocate or report "Don't Know" answers - it all depends on
the polling company's choice, and they don't tend to disclose which they did.
This can cause misleading results, especially at this point when there are
still a lot of people who don't know enough about Obama (and other candidates)
to want to commit to one.
For example, a poll may find 5% support Obama, 10% Clinton, 10% others, and
75% don't know. It's considered acceptable to report this as 20% support Obama,
and 40% others - which implies everyone asked had an opinion of some sort.
Another reason to insist on seeing the raw data is
to know what questions were asked before the question one's interested in -
those can clearly shape the answer.
She also linked to a useful paper on "Interpreting Polls " put out by a
parliamentary librarian in Australia.
Aside from our regular commentary here, I would also recommend the resources available on the new web page of the
American Association for Public Opinion Research (AAPOR).
Danielle was right to recommend examining the verbatim text of survey
questions and the way pollsters calculate their results. Where available, we
include links to the most complete report of results in our "poll update" posts
and in the tables that appear below the charts in our Polls section. The Polling Report also provides verbatim text for poll questions. Follow those links to
see the full results and verbatim text when available.
However, we need to be careful to distinguish between tables or references
in news stories that omits the results for the undecided category and
those that reallocate undecided voters as Danielle describes.
In the US,
virtually all polls are now reporting trial heat results based on calculations
that include the undecided or "don't know" category. Some pollsters
will put out "projections" based on their final poll before the election that
use various methods to reallocate undecided voters (the Polling Report has a
handy set of tables
showing both "vote projections" and "final trial heats" from 2004 side by
side). But I know of no pollsters putting out any such projections right now.
One way to check, if the undecided category is omitted, is to add up the
results to each candidate. If they total 100%, then the results have almost
certainly been recalculated.
More to the point, all of the results posted and plotted in the
charts and tables in the Polls
section of Pollster.com -- including the numbers discussed in yesterday's post
-- are based on calculations that include the undecided category.
It is also important to distinguish between the undecided category on a trial heat question and separate questions that probe how
certain or "decided" voters say they are. The undecided category tends to
understate the real potential for change in a political contest. In recent
weeks have written about questions asked on recent LA
Times/Bloomberg and CNN/WMUR/UNH
polls showing huge number of voters in Iowa and New Hampshire that are still less
than certain about their choice.
Danielle may be thinking about computations that take those "uncertain"
voters into account that Obama himself has been touting on the campaign trail. For
example, NBC's First Read
reported the following recent comments by Barack Obama at a rally in New Hampshire:
[Obama] also referred to a recent poll that had
Sen. Clinton in the lead by twenty points, and said that it meant nothing
besides what percentage of people were supporting which candidate. "Twenty
percent of voters are with Sen. Clinton, 10 percent are with me, Edwards has
about 5 percent and the rest of the candidates have less than that."
Citing that same poll, Obama said the most important fact overlooked by the
pundits were the number of undecideds still up for grabs. "There are 55
percent of voters who are undecided. That's 55 to 60 percent of the people who
are waiting to hear from you," he told the crowd of campaign volunteers
made up of students from colleges in Massachusetts
and across New Hampshire
as well as local volunteers.
Obama was probably referring to the recent CNN/WMUR/UNH
poll that gave Clinton a 22 point lead over Obama (with only 9%
"undecided"), but also showed 55% of New Hampshire likely Democratic primary
voters saying on a separate question that they are "still trying to decide"
which candidate to support. If I set aside the uncertain 55% and recalculate
vote preference numbers among those who say they are "definitely decided" or
"leaning," I get 23% for Clinton,
10% for Obama and 6% for Edwards.
About that new ABC
News/Washington Post poll released
yesterday, the one showing Hillary Clinton winning the support of 53% of "leaned
Democrats" nationally to 20% for Barack Obama, 13% for John Edwards and single
digits for the other candidates. Since this survey was the first to show a majority
supporting Clinton, a lot of bloggers let loose with the adjectives, writing that
away" with a lead that is "crushing" or "explodes," telling us that we
have reached "knockout
punch time," that "she's killin' it,"
or declaring simply, "oh
baby, when she moves, she moves." My friend Chris Cillizza parsed Clinton's "surge"
in his Washingtonpost.com blog and concluded:
It doesn't take a polling expert to
understand that Sen. Hillary Rodham Clinton's showing in the latest Washington
Post/ABC News poll is a major moment in the campaign.
There is obviously no denying that the poll has created the perception of a major change in the
race, but has a Clinton really experienced a "surge" of support over the last
month, as indicated by the 12-point increase (from 41% to 53%) in her support
on the Post/ABC poll? I'm not so sure.
I checked our national chart for the Democrats late
yesterday and saw that this new ABC/Post poll appears to be very different from
the others conducted over the last month. I quickly realized, however that this
particular comparison highlights a frustrating limitation of our chart, which
is that about half of the national polls plotted include Al Gore as an option
and about half do not. Since Clinton typically picks up roughly five points on
polls - including the new ABC/Post survey - that do not offer Gore as a choice,
our standard chart may make the ABC/Post survey look like more of an outlier
than it is.
So we decided to devote some time today to producing a chart
that displays only the results for polls without Gore (using recalculated votes
based on second choice when available). The result below plots 66 national polls
released so far this year (about a third fewer than our standard version - as not
all provide a "vote without Gore"):
The chart makes a few things clear. First, the 53% result for new Post/ABC poll is more of an "outlier"
from the regression trend line than any poll conducted this year (it's the purple dot at the extreme top right of plot area). At 53%, the
polls estimate of Clinton's
support falls a full ten percentage points higher than our current estimate of
the trend (42.5%) even without Gore in the race.
Second, the addition of the new poll has not budged our
trend line. That is mostly by design. Professor Franklin set the sensitivity of
the regression model to prevent the inevitable outlier from causing wild gyrations
in the trend line. That is one of the reasons we prefer to use regression
estimates over simple rolling averages.
One of the reasons we put so much effort into these plots is
that they paint a picture of the random variation that is inherent in polling. You
can actually see the random variation. While it typically appears as a random and
predictable "cloud" of points, bigger outliers still occur from time to time. The
more experience you have looking at poll numbers, the more you learn to
understand that even with well designed polls, outliers happen. Our advice to poll
consumers - as reflected in the design of the charts -- is to try to avoid over-reacting
to any one poll.
Of course, we do not yet know whether this poll is really a
statistical outlier. Other polls have obviously been showing a more gradual
increase in her support recently, and it is still theoretically possible that Clinton's
support suddenly lurched up ten points last week. Senator Clinton had appeared
on all of the Sunday morning talk shows on September 23, just four days before
this poll went into the field. Earlier that week sho also rolled out her plan
to reform health care, an issue that ABC/Post and other polls confirm as one her
great perceived strengths. But a sudden upward surge of this magnitude does not
seem very likely, if only because no other news event so far this year has caused
anywhere near as much change in the Democratic race.
We should know soon enough. Unfortunately, the AP-IPSOS
survey released yesterday does not help resolve this issue, as Clinton's Sunday talk show
tour de force occurred in the middle of their field period and (I'm told) after
they had completed most of the interviews on the main sample.
So we will wait and see. But I'll wager that a month from
now the real trend will not look nearly as dramatic as the one suggested by yesterday's
PS: It will probably take a few days, but we will have a
full, regularly updating version of the national "vote without Gore" up soon.
UPDATE (10-16): More recent polls do move the Clinton trend line up and make the Post/ABC poll look much less like an outlier. More here.
Five new InsiderAdvantage statewide surveys of likely Republican primary voters in Iowa, New Hampshire, South Carolina, Florida, and Michigan (conducted 10/2 through 10/3) finds:
- Among 1,339 likely caucus-goers in Iowa, former Gov. Mitt Romney leads former Mayor Rudy Giuliani (24% to 16%) in a statewide caucus; former Sen. Fred Thompson and former Gov. Mike Huckabee both trail at 13%, Sen. John McCain at 10%, and Sen. Sam Brownback at 5%.
- Among 1,822 likely primary voters in New Hampshire, Romney leads Giuliani (28% to 20%) while McCain trails at 17%, Thompson and Huckabee both at 8%, and Rep. Ron Paul at 6%.
- Among 1,281 in South Carolina, Thompson (at 21%) leads Giuliani, Romney, and McCain (all at 16%) while Huckabee trails at 11%.
- Among 725 in Florida, Giuliani leads Thompson (29% to 19%) while Romney trails at 16%, McCain at 10%, and Huckabee at 6%.
- Among 1,190 in Michigan, Giuliani edges out Romney (19% to 16%), while McCain trails at 15%, Thompson at 14%, Huckabee at 6%, and Paul at 5%.
- All other candidates receive less than five percent each.
We will add links once they are available.
New analysis from the most recent Gallup national survey of 537 Democrats and those who lean Democratic (conducted 9/24 through 9/27) asks whether Sen. Hillary Clinton, Sen. Barack Obama, or former Sen. John Edwards would do the best job when faced with 17 different challenges as president.
The analysis finds:
- "While Clinton dominates on core policy issues, Sen. Barack Obama does relatively well on the handful of items included that tap into the candidates' ability to relate to people and heal divisions in the country. Democrats do not consider former Sen. John Edwards the best candidate on any issue."
- "The fact that Edwards receives his highest rating on the commander-in-chief dimension could say more about what Democrats think of Clinton and Obama on this issue, than what they think of Edwards."
A new NBC News/Wall Street Journal national survey (NBC story, results; WSJ story, results) of 606 likely Republican primary voters (conducted 9/28 through 9/30) finds:
- Former Mayor Rudy Giuliani leads former Sen. Fred Thompson (30% to 23%) in a national primary; Sen. John McCain trails at 15%, former Gov. Mitt Romney at 10%. All other candidates receive less than five percent each.
- 38% of Republican think the next president should "take an approach similar to that of George W. Bush:" 48% think the next president should take a different approach.
- 59% of Republicans agree that "foreign trade has been bad for the U.S. economy, because imports from abroad have reduced demand for American-made good, cost jobs here at home, and produced protentially unsafe products;" 32% choose the alternate statement that "foreign trade has been good for the U.S. economy, becuase demand for U.S. products abroad has resulted in economic growth and jobs for Americans here at home and provided more choices for consumers."
Additional results from the recent CNN/WMUR/UNH statewide
survey of 508 adults in New Hampshire (conducted
9/17 through 9/24) find:
- Former Gov. Jeanne Shaheen leads Sen. John Sununu (54% to
38%) in a statewide general election match-up for U.S.
- 56% rate Shaheen favorably, 25% rate her unfavorably.
- 40% rate Sununu favorably, 37% rate him unfavorably.
A new AP-Ipsos national survey (story, results) of 1,317 adults (conducted 9/21 through 9/25; released 10/3) finds:
- Among 482 Democrats and those who lean Democratic, Sen. Hillary Clinton leads Sen. Barack Obama (40% to 26%) in a national primary; former Sen. John Edwards trails at 12%. When former V.P. Al Gore is included, Clinton runs at 35%, Obama at 23%, GOre at 12%, and Edwards at 9%. All other candidates receive less than five percent each.
- Among 358 Republicans and those who lean Republican, former Mayor Rudy Giuliani (at 21%) runs slightly ahead of former sen. Fred Thompson and Sen. John McCain (both at 17%) in a national primary; former Gov. Mitt Romney trails at 10%, former Gov. Mike Huckabee at 7%, and former Speaker Newt Gingrich at 5%. All other candidates receive less than five percent each.
- "Clinton led Obama among whites by 35 percent to 18 percent, Blacks were essentially evenly divided, 40 percent for Obama and 38 percent for Clinton."
A new national survey (memo, results) from Democracy Corps (D), the joing product of Democratic pollster Stan Greenberg and Democratic consultants James Carville and Bob Shrum, has released a new national survey of 1,000 likely voters (conducted 9/16 through 9/18) focusing on the debate over the Children's Health Care (S-Chip) Bill and finds:
- 38% approve of the way George Bush is handling his job as president; 57% disapprove.
- "In the latest Democracy Corps survey, we examined a full battery of economic worries, but health care tops everything else, rising dramtically in the most recent period."
The most useful news about the presidential primary races I
came across in the last 24 hours - especially for those of us who watch the survey
data - comes not from any new poll but rather from a little known media researcher
named Evan Tracey. Even better news for political junkies (as I learned via
links by First
Read and Ben
Smith), Tracey blogs on Advertising Age's "Campaign Trial."
Who is Even Tracey and why should you read him? Tracey is
the founder and chief operating offer of something called the Campaign Media Analysis Group
(CMAG) that performs a unique service for its very high paying political clients.
It systematically monitors the transmissions of the national broadcast
networks, 25 national cable networks and local advertising in the countries top
100 markets, uses software to electronically identify and code unique
advertisements. They are also now apparently tracking Internet advertising as
well. Through this process, CMAG compiles detailed reports on what candidates
are spending on broadcast and cable advertising, what ads they are running and
how many times viewers are seeing them (more details on their methodology here and here).
As a campaign pollster, I had the opportunity to be on the
receiving end of CMAG's tracking service just once, and it was absolutely
phenomenal. No media tracking report I saw in 20 years as a campaign pollster
was anywhere near as throughout or detailed. Unfortunately (for all of us),
Tracey charges his clients a high price for his data and is understandably
reluctant to give away his bread and butter.
Simply put, no one has better data on the candidates' ad
spending, and this is why I am excited to see Tracey sharing a smattering of
his valuable data online. More often than not, the ad expenditures by the
candidates explain the trends we see in the early state charts here at
Yesterday, for example, Tracey tells us that
on the Republican side:
Mitt Romney has aired nearly 10,000
TV spots since late February and spent close to $8 million dollars with a
majority of his spending in Iowa and New Hampshire. He is now
expanding this strategy to South Carolina and Florida.
Meanwhile, he writes, the other Republicans have either
placed token ad buys or run nothing at all. McCain's new ad
has been airing for just the last week. Take a look at our charts for Iowa and New Hampshire
Republicans and you will see the sharp increase for Romney in the second and
third quarters of 2007 that does not occur in South Carolina or
check that just released ARG survey in South Carolina that
shows a huge upward spike (from 9% to 26%) for Romney. That change may turn out
to be an outlier, but it would not be a surprising change given what Tracey
Of course, observers should also keep in mind that until
McCain's buy started last week, none of the other Republicans have run ads. So
a huge question for the early states is whether Romney's lead will hold as
other candidates begin to air their own advertisements.
Now consider what Tracey tells us about the Democrats:
Bill Richardson leads the pack,
airing over 4,000 commercials in an effort to move up in the polls in the early
states of Iowa and New Hampshire. Barack Obama is close behind
with close to $2 million invested in TV time, mostly in Iowa. Hillary Clinton and John Edwards,
because of their profiles, were able to skip the typical bio spots and have
spent far less. The Edwards campaign has made relatively small buys in Iowa and New Hampshire,
but has also used ad buys tactically on cable TV and in Washington
to inject himself into the Iraq
war debate. Clinton, on the other hand, has
spent over $1 million on TV in just five weeks and her latest buys on South Carolina radio
signal the campaign is serious about running the table on the early primaries.
Others on the Democratic side of
the ledger have gotten in on the act with the Biden and Dodd campaigns
combining for over $1 million dollars in TV ad spending with compelling
messages about the war and global warming.
Not surprisingly, the poll trends follow the money: Richardson has seen a modest bump up in his support to
roughly 12% in Iowa
and 10% in New
Hampshire. Obama, Tracey tells us, has run ads "mostly in Iowa." Thus, Obama's trend line is up in Iowa, but trending slightly downward in New Hampshire and elsewhere. And while he's
a little unclear on exactly where Clinton has
spent that $1 million on the last five weeks, he implies she's buying
television in both Iowa and New
Hampshire and radio in South Carolina. Her
trend lines are on the rise in all three states, as they are nationally.
Also, as I pointed
out last week, we can see clearer evidence of the ad buys in the favorable
ratings of initially lesser known candidates. The latest CNN/WMUR/UNH
poll of New Hampshire
shows that Richardson, Biden, Dodd and Obama have seen double digit jump in
their favorable ratings since June. Richardson's
favorable rating on that poll has nearly doubled (from 27% to 53%).
Definitely go read all of Tracey's post - he is providing us
with invaluable data on the most important part of the campaign getting underway
in the early states.
Ambinder has also been a good source of reports on media buys, posting most
recently on advertising by the Democratic candidates in Iowa
and on Clinton's radio buy in South
A new Quinnipiac University statewide survey of 1,504 registered voters in New York State (conducted 9/24 through 9/30) finds:
- Among 637 Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (47% to 15%) in a statewide primary; former V.P. Al Gore trails at 9%, former Sen. John Edwards trails at 7%.
- Among 476 Republicans, former Mayor Rudy Giuliani (at 48%) leads Sen. John McCain and former Sen. Fred Thompson (both at 8%) in a statewide primary; former Speaker Newt Gingrich and former Gov. Mitt Romney both trails at 5%.
- All other candidates receive less than five percent each.
- General election match-ups:
Clinton 52%, Giuliani 41%
Obama 45%, Giuliani 45% (tie)
Giuliani 45%, Edwards 44%
New results from the most recent NewsMax/Zogby statewide survey of 502 likely Republican primary voters in New Hampshire (conducted 9/26 through 9/28) finds former Gov. Mitt Romney narrowly leading former Mayor Rudy Giuliani (24% to 21%) in a statewide primary; Sen. John McCain trails at 16%, former Sen. Fred Thompson at 7%, and former Gov. Mike Huckabee at 5%. All other candidates receive less than five percent each.
A new ABC News/Washington Post national survey (Post story, results; ABC story, results) of 1,114 adults (conducted 9/27 through 9/30) finds:
- 33% approve of the way George W. Bush is handling his job as president -- "equal to his career low in Post-ABC polls;" 64% disapprove.
- Among 592 Democrats and those who lean Democratic, Sen. Hillary Clinton leads Sen. Barack Obama (53% to 20%) in a national primary; former Sen. John Edwards trails at 13%. All other candidates trail at less than five percent each.
- Among 398 Republicans and those who lean Republican, former Mayor Rudy Giuliani leads former Sen. Fred Thompson (34% to 17%) in a national primary; Sen. John McCain trails at 12%, former Gov. Mitt Romney at 11%, and former Gov. Mike Huckabee at 8%. All other candidates trail at less than five percent each.
- 67% say Congress should reduce the Bush administration's request for 190 billion dollars to fund the wars and related U.S. activites in Iraq and Afghanistan over the next year, which "is about 40 billion dollars more than first estimated;" 27% say Congress should approve the request.
Update: ABC story and results on the 2008 national primary now available.
Nearly two weeks ago, just before we kicked off our Disclosure
Project, InsiderAdvantage pollster Matt Towery used a syndicated column headlined
a Quinnipiac?" to attack the Florida
polls conducted by the Quinnipiac University Polling Institute. Towery not only
highlighted how his polls differed from a recent Quinnipiac survey but also commissioned
Mason-Dixon Polling and Research to conduct a parallel poll to prove his point.
Towery's unusually jocular broadside amounted to "one pollster whack[ing] the
other upside the head," as Politico's Jonathan Martin put
"At the very least,"
Quinnipiac numbers should stop
being taken at face value as the paragon of accuracy in Florida. Somewhere in their methodology they
continue to misread the state they claim to know so intimately.
When I looked at
the four polls of Florida Republicans conducted recently by InsiderAdvanage,
Quinnipiac and Mason-Dixon, the differences between them seem explained mostly
by the inclusion of Newt Gingrich as a candidate in the Quinnipiac poll and the
fact that Fred Thompson's announcement occurred during the fielding of the
Quinnipiac poll but before the others. Still, Towery's suggestion that the
Quinnipiac differences might be found "somewhere in their methodology" led me
to ask the same kinds of methodological questions as we have been asking as
part of our Disclosure Project. Their responses follow, and the difference in
sampling methodology adds another possible explanation. Quinnipiac's sample of
"registered Republicans" samples a population roughly four times the size of
the "likely voters" surveyed by InsiderAdvantage and Mason-Dixon.
Interview Dates -
One question I asked only of Quinnipiac was to provide the number of interviews
conducted before and after Fred Thompson's announcement of candidacy. Doug
Schwartz at Quinnipiac reports that 199 (or 45%) of their 438 interviews were
conducted on or before the evening of September 5. Thompson declared his
intentions later that night and received a burst of positive coverage in the
week that followed. While the methodologies of these surveys differ, it is
worth remembering that the other polls by InsiderAdvantage and Mason-Dixon were
fielded in their entirety after September 5.
Sample Frame -
Although the term is a bit wonky, one of the most important ways these polls
differ is in what pollsters call the "sampling frame." Put more simply, the
issue is the source for the random sample of voters called by each pollster.
Quinnipiac uses a random digit dialing (RDD) methodology
that contacts a random sample of all the working landline telephone numbers in Florida and then uses
screen questions to select a random sample of registered Republicans. In this
case, both InsiderAdvantage and Mason-Dixon selects voters at random from the
list of registered Republicans provided by the Secretary of State, using both
actual vote history and screen questions to identify and interview "likely"
Republican primary voters.
For more information on the debate about RDD versus list
sampling, see our prior posts here
How Did They Select Republican
Registered or Likely Voters? - The pollsters at Quinnipiac provided a
complete and relatively straightforward answer. They asked two questions about
vote registration and party affiliation:
Some people are registered to vote and others
are not. Are you registered to vote in the election district where you now
live, or aren't you?
[IF REGISTERED] Are you registered
as a Republican, Democrat, some other party or are you not affiliated with any
Both Towery and Brad Coker at Mason-Dixon were both
initially reluctant to describe the specifics of their likely voter selection
procedures, citing the need to protect "proprietary" methods. After a bit of email
back-and-forth, however, both were willing to describe their methods in general
terms. Let's start with Coker, answering on behalf of Mason-Dixon:
Our sample design and screening method takes
into account voter registration, party registration, past primary voting
history and likeliness to vote in the primary. Other factors that were taken
into account were the age, county, gender and race of the voting population
based on previous Republican primary elections.
What that means -- as I read it -- is that Mason-Dixon uses
information on past vote history on the voter list to draw a random sample of a
subset of Republicans that they consider most likely to vote. They ask those
sampled individuals questions on "likeliness to vote in the primary" and screen
out unlikely voters. Finally, they weight the demographics of the final sample
based on the demographics of voters in previous Republican primaries.
Towery reports using a similar procedure at InsiderAdvantage:
"We do poll off of [a list of] registered voters, but we do then cull that
number down based on a voting history that gives us a more likely voter
sample." They then ask a screen question to identify likely primary voters:
"are you likely to vote in the_____presidential primary to be held ____."
What Percentage of
the Voting Age Population Did Each Poll Represent? - The calculation for
the Quinnipiac poll is relatively straightforward: They report starting with a
random sample of 1,325 adults and using the questions above to identify and
interview 438 registered Republicans. So the Quinnipiac Republican sample
amounted to 33% of Florida adults (438 divided by 1,325).
Again, Coker and Towery were initially reluctant about
sharing specific numbers, but ultimately provided the information necessary to
answer my question. Let's start again with Coker and Mason-Dixon:
The population we were trying to capture was
the roughly 1 million Republican voters who will be most likely to vote in
January. In a universe of approximately 3.8 million registered Republicans, we
targeted a population of about 1.2 million Republican voters and had an
incidence of 83%.
interviewed a sample designed to represent approximately 1 million voters (1.2
million * .83) out of 14.2 million Florida
adults, or 7.0% of Florida adults.
Next, Towery and InsiderAdvantage:
The [target] universe based on our sample
system was around 1.6 million. This reflects the slightly higher than normal
turnout you see in a Presidential primary. Incidence rate, based on data I just
received was around 75%. Based on your description this would mean a final
"universe" of around 1.2 million voters (all registered) which I believe
reflects the likely turnout for a GOP Pres. primary turnout.
Insider-Advantage interviewed as ample designed to reflect approximately 1.2
million adults, or 8.5% of Florida adults.
As should be
obvious, Mason-Dixon and InsiderAdvantage sampled significantly narrower
populations of Republican voters than Quinnipiac. Via email Brad Coker argues that a narrower "screen" is more appropriate to a
pre-election poll aimed at projecting the preferences of likely voters: "I
would question the validity of a poll of ‘registered Republican voters' simply
on the grounds that 75% of those sampled probably won't be voting in January."
According to the Florida
Secretary of State, the vote for Republican primary candidates totaled
roughly 1 million in 2006 (Governor), 1.2 million in 2004 (Senate) and 700,000
in the 2000 presidential primary.
In response to my
initial questions, Quinnipiac's Doug Schwartz sent this statement:
The methodology used by the Quinnipiac poll is similar to
that of all the other major polling operations in the country. It has correctly
predicted the outcome of every major race it has polled on in Florida during the past three years. For
details on the methodology used, contact Doug Schwartz or visit http://www.quinnipiac.edu/x271.xml
That is true, but we should note that the final pre-election survey conducted by Quinnipiac in 2006 reported the
results among "likely voters" rather than all registrants. So their
primary voter "methodology" may shift as we get closer to
Election Day. Pollsters continue to debate the merits of various likely voter
models months prior to the election, something I covered in great
detail in 2004 in the context of general elections. Putting that debate aside,
however, the point is that the universes sampled in this instance are very different.
What Are the Demographics? - I asked
each pollster to provide the results to demographic questions asked of their
Republican samples. Both Quinnipiac and Mason-Dixon were quick to respond. The
table below shows that the Quinnipiac sample is a bit younger. This is not
surprising given that voters are typically older than non-voters.
and Mason-Dixon also included the regional composition of their samples. While
their regions were not identical, their definition of South
Florida came close. Mason-Dixon had fewer Republicans in their
Southeast Florida region (18% in Palm Beach, Broward, Dade and Monroe Counties)
than Quinnipiac (23%; although the Quinnipac South Florida region also includes
Hendry County, which accounts for just 0.1% of registered Republicans
difference is important because, according to the Mason-Dixon cross-tabulations
that Coker also provided, Rudy Giuliani ran far ahead of Fred Thompson (33% to
11%; n=70) in Southeast Florida, but trailed Thompson narrowly elsewhere (22%
to 26%; n=330). So the fact that Quinnipiac had a greater percentage of respondents
in South Florida provides yet another explanation
for Giuliani doing better statewide in their poll.
provided this information:
Since we have the Florida voter file, we know the precise
demographic profile of those who have voted in previous elections (at least in
terms of county, age, gender and stated race/ethnicity). Our sample matches it
within 1-2% of the actual figures from the average of 2004 & 2006 GOP
primary turn-outs. Deaths and out-migration could easily account for any
Towery, on the
other hand, was more reticent:
We don't give out our weighting percentages or
our demographic regional breakdowns because those are proprietary and if we did
so, it would be like Coke giving away the secret formula, well not that big,
but important to us!
Which brings us
back to the whole point of our Disclosure Project. We should congratulate all
three pollsters for providing the "incidence" data necessary to help us answer,
in essence, not just "What is a Quinnipiac?" (to borrow Towery's headline) but
also, "what is a Mason-Dixon?" and "what is an InsiderAdvantage? As a result of
their disclosure, we can see how different the "target populations" were and
take those differences into account in assessing the results.
It took some
coaxing, to be sure. Coker has previously
refused similar requests on the grounds of protecting proprietary interests.
Given his extensive experience in Florida (he
tells me he has conducted more than 200 statewide polls in Florida since 1984), Coker was
understandably reluctant about responding in this instance. So his cooperation
here is noteworthy. Hopefully, other pollsters follow his lead because the
general descriptions and incidence calculations provided above could be easily
replicated by every pollster and released online for every poll. Similarly, a
demographic composition table, like the one above, would be an easy addition to
the online documentation virtually every pollster and news organization makes
available for every poll.
On the other hand,
Towery's "secret formula" dodge has a fundamental flaw. Coke need not give away
its "secret formula" when it prints on every can, as required by law, a list of ingredients, the number of calories and the grams of carbohydrate and other
nutrients contained in each serving. As should be obvious, the our
Constitution's First Amendment precludes the sort of mandatory labeling for pollsters that the
FDA requires for food. However, pollsters like Towery ought to start thinking
about how to better label their own products in terms of their sample
composition, lest some snarky blogger ask, "What's an InsiderAdvantage?"
A new NewsMax/Zogby statewide survey (story, results) of 505 likely Democratic primary voters in New Hampshire (conducted 9/26 through 9/28) finds Sen. Hillary Clinton leading Sen. Barack Obama (38% to 23%) in a statewide primary; former Sen. John Edwards trails at 12%, Gov. Bill Richardson at 8%. All other candidates receive less than five percent each.
Last Friday, in my semi-regular "remainders" wrap of interesting poll
blogs of the week, I neglected to link to an interesting
column from Kathy Frankovic of CBS News, which has an interesting twist on our
recent focus on disclosure.
She writes of the laws in "at
least 30 countries" other countries that prohibit the publication of pre-election
poll results, but then also points out the unusual new law in Greece:
In Greece, however, the restriction on
reporting pre-election polls was brand new, and it also carried disclosure
requirements. A published opinion poll there has to be based on at least 1,000
interviews; and the questionnaire, the collected data and the survey report
must be deposited with a special public committee.
After noting that the First Amendment of the U.S. Constitution
would prevent any such prior restraint in our country, she considers that
The Greek law's requirement of disclosure is something that professional survey
research organizations have long desired. The American Association for Public
Opinion Research (AAPOR), the
National Council on Public Polls (NCPP),
and the World Association for Public Opinion Research (WAPOR), among others, require
disclosure of information that would allow a reader or listener to judge the
value of a poll.
However, these organizations also
oppose government restrictions on publication of pre-election polls. (The
Internet has made those restrictions more difficult to enforce. How could the
French government enforce its law prohibiting the publication of poll results,
if those results appeared on Web site based in Switzerland?)
She goes on to consider the implications of a lack of pre-election
polling in the last two weeks of the Greek campaign. It's worth reading
A new Rasmussen Reports automated survey of likely primary voters in South Carolina (conducted 9/26 through 9/27) finds:
- Among 567 Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (43% to 30%) in a statewide primary; Sen. John Edwards trails at 10%.
- Among 863 Republicans, former Sen. Fred Thompson narrowly leads former Mayor Rudy Giuliani (20% to 24%) in a statewide primary; former Gov. Mitt Romney trails at 15%, Sen. John McCain at 11%.
- All other candidates receive less than five percent each.
Family obligations and a nagging cold virus kept me mostly off
the grid this weekend while the blogs were abuzz over the latest Newsweek poll of Iowa likely caucus goers. So while late, let
me add a few thoughts to those already offered over elsewhere.
First, the margins of sampling error reported by Newsweek -- +/- 7% for the likely
Democratic caucus goers and +/- 9% for he Republicans - means that statistically
meaningful conclusions are all but impossible regarding Barack Obama's "slight
edge" (28% to 24%) over Hillary Clinton. Strictly speaking, even Mitt Romney's
9 point advantage does not attain the usual 95% confidence level that pollsters
require to describe a lead as "statistically significant."
Scheiber wonders about what the pollsters could say about the probability
of an Obama lead among likely caucus goers, if not 95%. My best guess (assuming
that the reported margins of error were based on the usual 95% confidence
level) is that the probability of an Obama lead based on the Newsweek poll is about 50%. In other
words, the odds of Obama "leading" on this poll are no better than a coin-flip,
if we were to take repeated samplings of exactly the same design.
Yglesias makes the more important point:
It seems to me that there's no real
point in arguing
about the significance
of the rather large +/- 7 points margin of error on this Newsweek poll
. . . For something like this, uncertainty about the likely voter screen are
probably going to be a bigger problem than sampling error anyway.
He is exactly right. Since July we have seen 12 public polls
released in Iowa
by 9 different organizations, and each appears to define and sample the likely
caucus-goer universe differently. To the extent that pollsters have revealed
the details, their snapshots of the electorate are poles apart, to say nothing
of the candidates that those voters support. A month ago, for example, I found
the percentage of first-time caucus-goers reported on four different polls
of Democrats varying from 3% to 43%, with Edwards doing worse (and Clinton better)
as the percentage of newcomers increased. The Newsweek survey reports 36% of likely Democratic caucus goers saying
"this would be your first caucus."
Unfortunately, the Newsweek
release omits many of the same methodological details left out of the other
Iowa polling releases (including, remarkably enough, the number of interviews conducted with likely Democratic and likely
Republican caucus goers). I have emailed Newsweek's
pollsters the same questions we sent last week to the other Iowa pollsters and will include their responses
when we begin reporting on the Disclosure Project
By the way, Yglesias also makes another important point: In
a truly close race, the ultimate winner among the Democrats may depend on the
second choices forced by the convoluted Caucus rules
on those whose first choice fails to achieve "viability" (usually 15% of the
vote) in their precinct. Remember that the official results for the Democrats
will not be a head-count of the first
preference of all caucus goers (as in a poll) but rather the estimated share of
state delegates won by each candidate based on the final choices at the end of
the night. So even if pollsters agreed on how to sample "likely caucus goers,"
the numbers would still be inconclusive in a close race.
Update:: Slate's Christopher Beam, who called just before I wrote this item, has more
Three new American Research Group statewide surveys of likely caucus/primary voters in Iowa, New Hampshire, and South Carolina (conducted 9/26 through 9/29) find:
- Among 600 likely Democratic caucus-goers in Iowa, Sen. Hillary Clinton leads Sen. Barack Obama (30% to 24%); former Sen. John Edwards trails at 19%, Gov. Bill Richardson at 10%. Among 600 Republicans, former Gov. Mitt Romney narrowly leads former Mayor Rudy Giuliani (22% to 21%); former Sen. Fred Thompson trails at 16%, Sen. John McCain at 11%, former Speaker Newt Gingrich at 5%.
- Among 600 likely Democratic primary voters in New Hampshire, Clinton leads Obama (41% to 22%); Edwards trails at 10%, Richardson at 8%. Among 600 Republicans, Romney (at 24%) narrowly leads McCain and Giuliani (both at 20%); Thompson trails at 8%, Gingrich at 6%.
- Among 600 likely Democratic primary voters in South Carolina, Clinton leads Obama (41% to 30%); Edwards trails at 7%, Richardson at 5%. Among 600 Republicans, Romney edges out Giuliani (26% to 23%) while McCain, Thompson, and Gingrich trail at 15%, 10%, and 7% respectively.
- All other candidates trail at less than five percent each.
A new Newsweek statewide survey (story, results) of 1,215 registered voters in Iowa (conducted 9/26 through 9/27) finds:
- Among all registered Democrats (+/- 5% margin of sampling error), Sen. Hillary Clinton narrowly leads Sen. Barack Obama (31% to 25%) with former Sen. Edwards at 21% and Gov. Bill Richardson at 6%. Among likely Democratic caucus-goers (+/- 7% margin of sampling error), Obama edges out Clinton (28% to 24%); Edwards runs at 22%, Richardson at 10%, and Sen. Joe Biden at 5%.
- Among all registered Republicans (+/- 6% margin of sampling error), former Gov. Mitt Romney runs at 25%, former Sen. Fred Thompson at 16%, former Mayor Rudy Giuliani at 15%, Sen. John McCain at 7%, and former Gov. Mike Huckabee at 6%. Among likely Republican caucus-goers (+/- 9% margin of sampling error), Romney narrowly leads Thompson (24% to16%) while Giuliani trails at 13%, Huckabee at 12%, McCain at 9%, and Rep. Ron Paul at 5%.
- All other candidates receive less than five percent each.
New analysis aggregated from the four most recent Gallup
national surveys of 1,690 Republicans and those who lean
Republican (conducted 8/3 through 9/16) finds "[Giuliani]
is no worse than tied for the lead among all major
Republican constituencies, including those from groups that
are most likely to disagree with him on some of these
issues. And among Republicans who are most likely to vote
in the primary, his lead is eight points over Thompson, 32%