May 27, 2007 - June 2, 2007
A new automated survey (via MyDD) from the company IVR Polls of registered Democrats who regularly vote in general and Democratic primary elections in Florida (conducted 5/31) finds:
- Among 487 registered Democrats, Sen. Hillary Clinton (at 45%) leads Sen. Barack Obama (18%) and former Sen. John Edwards (14%) in a Democratic primary.
- Among 259 registered Democrats who regularly vote in Democratic primary elections, Clinton (at 51%) leads Edwards (17%) and Obama (11%).
Note: The survey was based on a registration based sample (RBS) that used actual vote history to identify regular primary voters.
Rasmussen Reports has released their party identification summary for 15,000 interviews conducted in May. "For the fourth month straight, the number of those identifying themselves as Republicans has decreased" (now 31%); those who identify themselves as Democrats has also decreased (now 36%).
In a post
on Wednesday on the recent CBS News/New
York Times poll on immigration, I argued that survey respondents will often
answer questions about complex public policy issues for which they lack
pre-existing opinions. Respondents frequently form on-the-spot opinions, drawing
on other previously held attitudes or values cued by the question text. To
illustrate that point, I used a cross-tabulation from a recent SurveyUSA
automated poll in Charleston, South Carolina and in so doing, inadvertently
stumbled into a more complicated question: Whether the relatively small number
that are closely following news about the immigration reform bill may be more
or less supportive of the bill than other Americans.
The intriguing discrepancy at the heart of both issues is
that while majorities of Americans react positively to the "the major
provisions" of the immigration reform bill as described
by the CBS/New York Times poll, the Rasmussen
automated survey shows
a two-to-one (48% to 26%) plurality of likely voters opposed to something described only as the "immigration reform
proposal agreed to last week." A recent SurveyUSA sampling
of adults in the Charleston,
South Carolina media market
showed a similar result.
I argued -- and continue to believe --that the vast majority
of Americans have little sense of the details of the immigration reform bill.
Thus, I speculated on Wednesday that the negative reaction measured by the
automated pollsters is mostly a reflection of Americans' underlying attitudes
toward illegal immigration (most say it is a big problem) and the way the
government seems to be handling the issue (most say badly).
To support that speculation I offered a crosstabulation from
the SurveyUSA Charleston study that showed lower levels of support for "the
immigration reform bill" among those who said that they did not understand the
provisions of the bill well.
The problem is that in a release I had not yet seen,
Rasmussen Reports provided a similar tabulation showing essentially the
opposite result. While all voters in his survey oppose the immigration bill by
a 48% to 26% margin, the 37% that say they are following news about the bill
"very closely" oppose the legislation by an even bigger margin (69% to 23%). While
Rasmussen did not include the complete cross-tabulation, we can extrapolate
that the less attentive respondents divided more closely, with 28% in favor, 32%
opposed and 40% unsure.
So we have another puzzle. One possibility, of course, is
that adults in the Charleston,
SC media market have a different
perspective on immigration than adults nationally. But a more convincing
explanation comes via email from Mickey Kaus, who argues that saying you "understand
the provisions" of the bill is different from saying you are "following news
stories" about it: "Neither accurately captures whether someone does or does
not understand the bill. In fact, the more you follow it the more you realize
you don't understand it."
So for both reasons, let's set aside that SurveyUSA result. But
that leaves us to consider the more complicated question I inadvertently
stumbled into: Does paying more attention to news about the immigration debate
make you more opposed to the bill? It might, but unfortunately that question is
virtually impossible to resolve with the data available because -- as any good
researcher will tell you -- correlation is not causation. If the most attentive
tend to be more negative about the immigration bill, that may be because their
greater exposure soured them or because those who pay more attention were more
inclined to oppose an "immigration reform" bill from the beginning.
The CBS/New York Times
data provide a hint that this pattern may be the results of the latter phenomenon.
A tabulation provided to Pollster.com by Kathy Frankovic, the polling director
at CBS, shows that those who have heard or read "a lot" about "changing the
laws of immigration" are significantly more likely to agree that "our
immigration policy has so much wrong with it that we need to completely rebuild
it" (56%) than those who have heard "some" (47%) or "not much/nothing at all"
Even this result provides only a clue, and complicating it
all further, the Rasmussen and CBS/NYT polls are about as different as two
surveys can be (including mode, population, field dates and question wording,
to name the most obvious). Without a news exposure experiment or at least a set
of surveys that tracks changing attitudes over time, we really cannot know for
certain either way.
Before moving on, I want to take this discussion back from
the minutia: The polling results worth watching most closely are those that
measure pre-existing attitudes about immigration in general, illegal
immigration in particular and the job the government seems to be doing handling
both (but especially the flood of illegal immigrants). These questions -- as
well as those that probe reactions to words like "amnesty" and "deportation" --
will provide the best guide to the way public opinion on this issue will drive
both the legislative debate and the upcoming presidential campaign.
Frankovic's column this week is also deals with the immigration poll and
focuses on the unique way that attitudes on this subject cross party lines.
A new Gallup Panel national survey of 1,007 adults (conducted 5/21 through 5/24) finds:
- 83% think Sen. Hillary Clinton has an "excellent" or "good" chance of winning the Democratic party's nomination; 77% say the same about Sen. Barack Obama, 53% about former Sen. John Edwards, 35% about former V.P. Al Gore.
- 76% think former Mayor Rudy Giuliani has an "excellent" or "good" chance of winning the Republican nomination; 60% say the same about Sen. John McCain, 42% about former Gov. Mitt Romney, 32% about former Sen. Fred Thompson, 28% about for former Speaker Newt Gingrich.
A new Winthrop/ETV statewide survey of 670 registered voters in South Carolina (conducted 5/16 through 5/27) finds:
- Among Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (29% to 21%) in a statewide primary; former Sen. John Edwards trails at 11%.
- Among Republicans, former Mayor Rudy Giuliani runs at 19%, Sen. John McCain at 14%, and former Gov. Mitt Romney at 12% in a statewide primary.
- 84% (90% of Democrats, 78% of Republicans) believe a woman "should be able to obtain a legal abortion in certain circumstances."
Let's start, once again, with the unavoidable conflict: Pollster.com
is owned and sponsored by Polimetrix, a company that conducts online surveys. We
are, however, walled off in a way that allows us the editorial freedom to write
whatever we choose to write (about online surveys and anything else), while
also keeping us ignorant of the work that Polimetrix does for its clients.
Case in point is the must-read
article on online surveys that appears in today's New York Times. I heard about it not from my corporate overlords,
but from a colleague who posted it on the member-only listserv of the American
Association for Public Opinion Research. The key quote:
Despite the strong skepticism, Internet-based
survey results are likely to get some publicity during the 2008 elections, and
executives from companies that conduct these surveys hope that they can use the
attention to gain credibility for their methods.
YouGov, for example, has formed a partnership with
Polimetrix, an online survey company based in Palo Alto,
Calif., for surveys in the United States.
Polimetrix, with a panel of one million people, plans to track the 2008
presidential election with a 50-state survey covering a minimum of 1,000
panelists in each state.
"State-by-state election results are an important
way for us to prove that our methodology delivers accurate results," said
Douglas Rivers, a Stanford University
political science professor who founded Polimetrix in 2004. "You can be lucky
once, but not 50 times."
Professor Rivers said that the margin of error for
Polimetrix surveys is similar to that of polls conducted by telephone. YouGov
said that its own results in recent British elections were as close or closer
to the actual votes than traditional polling methods.
Oh, so many conflicts, here. I also serve on the executive
committee of the American Association for Public Opinion Research (AAPOR), which
has condemned as
"misleading" the reporting of a margin of sampling error associated with opt-in
panels. I supported that statement and, as regular readers know, have been critical
of online surveys that report a "margin of error. So on what basis can Rivers
claim that "the margin of error for Polimetrix surveys is similar to that of
polls conducted by telephone?"
Leaping at the opportunity to bite the hand that feeds me, I
emailed Doug Rivers for his comment. His response on the narrower issue of
sampling error is a bit technical, and I have copied it below. His larger argument
is about how all surveys deal with bias. Telephone surveys that begin with
random samples of working telephone numbers suffer some bias in representing
all adults due to those who lack landline phone service (coverage) or refused
to participate in the survey (non-response). We know, for example that they
tend to under-represent younger Americans and those who live in more urban
areas. To try to reduce or eliminate this bias, pollsters currently weight by
demographic variables (such as age, race, education level and urban/rural
As a pool of potential respondents "opt-in" Internet panels
also suffer a bias -- how big is also a matter of some debate -- because panel
members must have Internet access, discover the survey panel (usually through an
advertisement on a web site) and volunteer to participate. Rivers believes his "sample
matching" technique will ultimately do a better job reducing the bias of
the opt-in panel universe than standard weighting does to reduce the bias in
standard telephone surveys. That is the crux of his argument.
Like a lot of my colleagues, I remain skeptical, but nonetheless committed
to an empirical evaluation. As I wrote
in Public Opinion Quarterly in 2005 (well
before I had any business relationship with Polimetrix):
At what point, if ever, might we
place greater trust in surveys drawn from opt-in panels? The only
way we will know is by continued experimentation, disclosure, and
attempts to evaluate the results through the Total Survey Error
framework. Opt-in panels are gaining popularity, whether we approve
or not. We should encourage those who procure and consume such research
to do so with great caution and to demand full disclosure of methods
and results. If nonprobability sampling can ever routinely deliver
results empirically proven more valid or reliable, we will need to
understand what produces such a result.
In a few days, Charles Franklin and I will begin posting the
findings we presented at the AAPOR conference, which include an assessment of how
the Polimetrix and other panel surveys did in 2006. This is obviously a debate
that will continue in the world of survey research, and we will try to follow
along, conflicts and all.
PS: Doug Rivers response regarding the "margin of error:"
Standard errors measure sampling
variability & sampling variability is easy to calculate when observations
are independent, which they definitely are for large opt-in panels and phone
surveys. (The standard error calculation is more complicated for cluster
samples, where the observations aren't independent, but these samples aren't
clustered; matching introduces another source of variability, but the effect on
standard errors is relatively small.) The MSE [mean squared error] calculations
involve squared bias and my ambition is to beat your average phone survey in
MSE by bias reduction.
One great value of long term data archives for polling information is the ability to compare trends over long spans of time and across different question wording. The Roper Center at the University of Connecticut provides one of these archives. Its subscription only iPoll database includes over 500,000 questions that pollsters have asked since 1935. This is a uniquely valuable resource for opinion research.
Public support for the death penalty over time is a good example of the power of this collection. The plot above looks at death penalty support and opposition since 1936 in Gallup polls. Gallup has maintained a constant question wording over that span giving us the best long term look at this issue. Gallup has a summary of these data on a no-subscription required page here. It provides more data than I look at here.
My point today is two-fold. First, just to look at the trends over time. The standard Gallup question plotted above is
Are you in favor of the death penalty for a person convicted of murder?
Support for the death penalty declined to a low in 1960-75, with support around 50% and opposition around 40% (with some bouncing around a bit.) Following 1975 there was a long growth in support for the death penalty, with support rising to as high as 80% and opposition falling into the teens.
Since the decline in homicide rates in the 1990s, support has declined by about 10 percentage points since 2000, to around 70%, with opposition rising to the mid-20s.
The public overwhelmingly supports "the death penalty for a person convicted of murder."
But now let's consider what happens when we offer a specific alternative to the death penalty. In the classic Gallup question, there is no explicit alternative to the death penalty, so respondents must imagine for themselves what the alternative might be.
Since 1985, Gallup has also asked an alternative question which includes a specific alternative sentence:
If you could choose between the following two approaches, which do you think is the better penalty for murder--the death penalty or life imprisonment, with absolutely no possibility of parole?
This reflects a shift in the policy debate, as no-parole sentences were raised as an alternative to execution.
Presented with this alternative, opinion shifts substantially:
With the "life without parole" option, support for execution falls to just over 50%, while support for life terms rises to the mid 40s. Still a majority in favor of the death penalty, but a substantially more closely divided public than with the classic question.
Two points from this. First, questions define the alternatives respondents are encouraged to consider. The classic question offers one option and leaves it to the respondent to imagine others. The death vs life introduces an explicit alternative, and finds much more support for a penalty short of death. However, we could imagine a question with a third option: life but with a possible parole. Or even specific sentence lengths (10 years? 20? 50?). In part the question reflects the policy debate. But survey questions must also necessarily limit the range of options under consideration. In this, there can be no escape from question wording effects, and no end to argument about whether questions "really" capture all the issues. The fact is every question includes some and excludes other options.
The second point is the political one. If one opposed the death penalty, one might be more likely to find support by arguing forcefully for hard time and no parole rather than arguing against the death penalty in principle.
Cross-posted at Political Arithmetik.
A new Siena Research Institute statewide survey of 620 registered voters in New York State (conducted 5/18 through 5/25) finds:
- Among 301 Democrats, Sen. Hillary Clinton (at 42%) leads Sen. Barack Obama and former V.P. Al Gore (both at 13%) in a statewide primary; former Sen. John Edwards trails at 7%.
- Among 174 Republicans, former Mayor Rudy Giuliani (at 52%) leads Sen. John McCain (14%) and former Gov. Mitt Romney (7%) in a statewide primary. When asked to chose from a list of five candidates, Giuliani runs at 50%, McCain at 12%, former Sen. Fred Thompson at 8%, and Romney and former Speaker Newt Gingrich both at 7%.
- General election match-ups:
Clinton 52%, Giuliani 39%
Clinton 54%, McCain 36%
Clinton 57%, Thompson 29%
Obama 50%, Giuliani 40%
Obama 50%, McCain 33%
Gore 56%, Thompson 28%
A new Quinnipiac University statewide survey (08 results, issues results) of 1,318 registered voters in Pennsylvania (conducted 5/22 through 5/28) finds:
- Among 585 Democrats, Sen. Hillary Clinton (at 33%) leads former V.P. Al Gore (16%), Sen. Barack Obama (13%), and former Sen. John Edwards (11%) in a statewide primary.
- Among 575 Republicans, former Mayor Rudy Giuliani (at 28%) leads Sen. John McCain (11%), former Sen. Fred Thompson (10%), former Gov. Mitt Romney (9%), and former Speaker Newt Gingrich (8%) in a statewide primary.
- General election match-ups:
Giuliani 47%, Clinton 43%
Clinton Obama 40%
Giuliani 44%, Gore 45%
McCain 43%, Clinton 45%
McCain 42%, Obama 41%
McCain 44%, Gore 44%
Last week's CBS News/New
York Times survey (CBS story,
is a classic example of the way public opinion pollsters will try to learn all
they can about one complex but critical public policy issue. The survey focused
special attention on immigration, asking 38 questions that probed the issue
from almost every angle. Not surprisingly, the results to some of those
questions have generated controversy. While some of my pollster colleagues will
be quick to dismiss the complaints as partisan grousing, this survey provides some
helpful lessons about the limitations of what polls can tell us when asking
about subjects with which most Americans are unfamiliar.
I find it helpful to divide the immigration questions on the
CBS/New York Times poll into two
sections. The first twenty or so items (beginning with #41 in either the CBS
or the NYTimes
releases) use simple neutral questions to probe the presumably pre-existing
attitudes that most Americans have about immigration, legal and illegal. We
learn, for example (and quoting from the CBS release):
have a positive impression of immigrants generally," yet "are highly
critical of current U.S.
immigration policy, especially when it comes to illegal immigration."
believe most immigrants are "here illegally"
- 82% "think
the U.S. could be doing more along its borders to keep illegal immigrants
from crossing over into the U.S.; just 14% think the U.S. is now doing all
These initial results have stirred little controversy,
perhaps because they are not particularly new (in fact, as the Times release
shows, most have been asked before), and thus, not particularly newsworthy.
Needless to say, what interests everyone -- including those
most critical of the CBS/NY Times
poll -- is what Americans think of the immigration reform bill now before
Congress. So starting at about item #60, the poll asks for reactions to a
series of policy proposals. These questions introduce new and sometimes complex
information about the various potential immigration policy alternatives. The
results, which lead the coverage by both organizations, show "broad support"
for "the major provisions" in the immigration bill now before Congress" (quoting
the Times story).
These results -- and especially the wording of the questions
involved -- have drawn intense criticism from opponents of the immigration bill
(see Kaus, Allahpundit, Mark Krikorian). I will not try
to referee the various critiques here, except to point out that the results
from the second set of questions on the CBS/New York Times poll are better
characterized as a measurement of how Americans react to descriptions of
the immigration proposals than a measurement of Americans' current opinions on those proposals.
How much do Americans really know about the immigration reforms currently
being debated in Congress? One question from the CBS/NYTimes poll suggests not
very much: Only 26 % say they have read or heard "a lot" about
"changing the laws regarding immigration in the United States;" 51% have heard
"some," 18% not much, 4% nothing at all.
Moreover, when other pollsters have asked about the
immigration bill in general terms (without providing specific descriptions), reactions
are more negative than they are to the provisions tested on the CBS/NYTimes
poll. For example, the Rasmussen Reports automated survey asked:
"From what you know about the agreement, do you favor or oppose the immigration
reform proposal agreed to last Week?" Roughly half (48%) opposed the agreement,
about a quarter (26%) favored it with just as many (26%) unable to answer.
Survey researchers have long understood that respondents
will readily answer questions about unfamiliar issues, typically drawing on
their real attitudes or values concerning the subject of the question. Americans
do have a sense of how their government is handling immigration (badly - see
above), and so it is not surprising to see skepticism about whatever Congress
is up to from those not following the immigration debate closely (see the analysis
by Scott Rasmussen).
Two questions from a recent automated survey
conducted in the Charleston,
South Carolina media market by
SurveyUSA make this point more vividly:
if they "understand the provisions" of a "proposed immigration reform
bill" resulting from a "deal" reached by "the Senate and the White House,"
only 10% say they understand it very well, 34% somewhat well, 38% not very
well and 17% not at all.
- 47% say
they oppose the "immigration reform bill," 36% support it and 18% are
Put those two questions together and we see that the less informed respondents are less supportive
of the bill. Three out of four who understand little or nothing about the bill
offer an opinion, and they oppose the immigration bill by a margin of roughly
two to one (49% to 25%). The better informed respondents divide evenly (46%
favor, 44% oppose).
So back to the CBS/ New
York Times results showing positive reactions to the various specific
"provisions" of the immigration reform bill. What should we make of those? Although
many respondents are probably forming opinions on the fly in reaction to the
text of the questions, their reactions have meaning. They can tell us a lot
about how Americans may react, should
the news media or the campaigns focus far more attention on the immigration
debate than they have thus far. But I would urge a few cautions:
First, it is entirely appropriate to scrutinize the language
of the questions as the critics have, and compare and contrast results when
pollsters use different language to ask similar questions. Follow the advice of
Newport and my old anonymous friend, Professor
M, and consider the various results as the best measure of potential range of support for the legislation.
Second, be realistic about how much more closely Americans
will follow the immigration issue and what new information they are most likely
to receive. We will certainly hear arguments about whether the immigration bill
amounts to "amnesty,"
but will Americans ever consider it in the level of detail provided by the questions
on the CBS/New York Times poll?
Put another way, we might want to think about what similar
survey results foretold about previous legislative debates. Consider, for
example, the Clinton
health care proposal of the early 1990s. In September 1993, on the eve of President
Clinton's address to Congress describing his plan, the CBS/New York Times poll
tested its widely anticipated elements. Here is how the Robin Toner described
the poll results in the New York Times
poll story on September
23, 1993 (sub. req.):
The survey shows overwhelming
majorities support the idea of assuring health coverage to all Americans and
guaranteeing that no one ever loses their insurance when they switch jobs or
suffer a medical catastrophe. Sixty-one percent said they were willing to pay
higher taxes to achieve those goals, and more than half said they were willing
to have the Government require employers to pay most of the health insurance
premiums to cover their workers -- a centerpiece of the Clinton plan.
Of course, despite these favorable reactions, the Clinton plan collapsed under an assault by opponents who
played to cynicism about government programs, fears of health care rationing and disapproval of the Clinton administration's
early missteps. Those underlying, pre-existing attitudes, which were also in
evidence in that September 1993 poll, turned out to be the ones that mattered
UPDATE: Rasmussen Reports published updated results on immigration this morning (via Kaus). These latest results show a different pattern among the most attentive respondents than what SurveyUSA showed in Charleston:
Overall, despite a major push by the President and others over the past week, support for the Senate bill has not increased at all. In polling conducted last night (Tuesday, May 29), 26% of voters favor passage of the bill. That’s unchanged from the 26% support found in polling conducted the previous Monday and Tuesday. Forty-eight percent (48%) of voters remain opposed.
Eighty-one percent (81%) of American voters are closely following news stories about the issue, including 37% who are following it Very Closely. Those with the highest interest in the issue oppose the legislation by a 3-to-1 margin (69% to 23%). By a 55% to 15% margin, those following the story Very Closely believe the bill will lead to increased levels of illegal immigration.
A new Rasmussen Reports automated survey of 500 likely voters in Kentucky (conducted 5/24 through 5/25) finds Steve Beshear (D) leading Gov. Ernie Fletcher (R) 51% to 35% in a gubernatorial general election match-up.
Three new American Research Group statewide surveys of likely primary voters in Iowa, New Hampshire, and South Carolina (conducted 5/23 through 5/25) finds:
- Among 600 Democrats in each state, Sen. Hillary Clinton leads Sen. John Edwards 34% to
25% 18% in New Hampshire and edges him out by six points in Iowa and by four points in South Carolina; Obama trails with less than 20% in all three states.
- Among 600 Republicans, Sen. John McCain leads former Mayor Rudy Giuliani 32% to 23% in South Carolina and leads former Gov. Mitt Romney 30% to 23% in New Hampshire. In Iowa McCain edges out Giuliani 25% to 23%.
New Rasmussen Reports automated surveys find:
- Among 559 likely Democratic primary voters, Sen. Hillary Clinton leads Sen. Barack Obama 35% to 26% in a national primary; former Sen. John Edwards trails at 14% (conducted 5/21 through 5/23).
- Among 469 likely Republican primary voters, former Mayor Rudy Giuliani leads former Gov. Mitt Romney 25% to
26% 16%; Sen. John McCain trails at 15%, former Sen. Fred Thompson at 12% (conducted 5/21 through 5/23).
- Among 500 likely voters in New York State, Clinton leads Giuliani 50% to 29% in a statewide general election match-up; Mayor Michael Bloomberg trails at 15% (conducted 5/20).
- Among 500 likely voters in New Jersey, Clinton edges out Giuliani 42% to 39%; Bloomberg trails at 13% (conducted 5/19).
- Among 500 likely voters in Connecticut, Clinton and Giuliani both run at 40%; Bloomberg trails at 9%.
Additional results from Gallup Poll's recent Values and Beliefs survey (analysis, video) of 1,003 adults (conducted 5/10 through 5/13) finds:
- 59% think homosexual relations between consenting adults should be legal; 37% say it should not.
- 49% believe homosexual relations are morally wrong; 47% believe they are morally acceptable.
- 42% believe homosexuality is "something a person is born with;" 35% believe it is "due to factors such as upbringing and environment."
A new WCPO-TV/WHAS-TV automated survey of 609 likely voters in Kentucky (conducted 5/23 through 5/24 by SurveyUSA) finds Democrat Steve Beshear leading Republican Gov. Ernie Fletcher 62% to 34% in a gubernatorial general election match-up.