April 1, 2007 - April 7, 2007


Zogby, Hillary and the Judicial Watch Poll

Today brings another controversy involving pollster John Zogby with two potential lessons, first, about a set of transparently biased and leading questions and, second, on the limits of such efforts to manipulate opinions.

This morning, the Washington Post's Dana Milbank tells the story of a new poll conducted by Zogby Interactive and sponsored by Judicial Watch, a group that "back in the day filed drawers full of lawsuits alleging Clinton corruption." Milbank describes the poll as "rather loaded in its language:"

"Some people believe that the Bill Clinton administration was corrupt," one question begins. In another question about Hillary Clinton, every answer included the word "corrupt," and the question was not asked about other candidates so that a comparison could be made.

The pollster, John Zogby, defended the questions as "balanced" -- a label Fitton [president of Judicial Watch] made no attempt to earn. As he presented the results yesterday, he announced that Bill Clinton's financial conflicts of interest "make the issues of Halliburton and Dick Cheney . . . pale in comparison."

Let's take a look at the first two questions:

304. Some people believe that the Bill Clinton administration was corrupt. Whether or not you believe the Clinton administration was corrupt, how concerned are you that there will be high levels of corruption in the White House if Hillary Clinton is elected President in 2008?

26% Very concerned
19% Somewhat concerned
20% Not very concerned
33% Not at all concerned
1% Not sure

305. When thinking about Hillary Clinton as a politician, which of the following best describes her?

17% Very corrupt
25% Somewhat corrupt
21% Not very corrupt
30% 51% Not at all corrupt
7% Not sure

You can pretty much stop after the first sentence. The suggestion that "some believe the Clinton administration was corrupt" is an obvious effort to lead the respondents to the desired answer. The drumbeat of "corrupt" and "corruption" that follows - implying that the issue is not whether Clinton is corrupt but how much - makes the bias almost comic. MyDD's Jonathan Singer has it exactly right:

[T]he apparently unbalanced wording of the polling conducted by Zogby International belies the notion that the organization is serious about coming up with results that actually reflect the views of the American public rather than just the views of those who paid for its services. To harp on one example, beginning a question on the scruples of a politician by saying that some people believe his or her spouse was corrupt inserts such a bias to void the results of the question -- and perhaps even the questions that follow. Simply put, the questions in the poll were not, as Zogby insists, "balanced."

But this episode also raises a second issue. How effective were these leading questions in producing the desired response? Put another way, did Judicial Watch get their money's worth?

Putting aside the obvious - that a 53% majority is not concerned about corruption in a Hillary Clinton White House - consider how the Zogby results compare to a set of balanced (though somewhat dated) questions about honesty and trust (via Polling Report):

ABC News/Washington Post (May 11-15, 2006. n=1,103 adults) - Please tell me if the following statements apply to Hillary Clinton or not... She is honest and trustworthy

52% applies
42% does not apply
6% unsure

CNN/USA Today/Gallup (Aug. 5-7, 2005. n=1,004 adults) Thinking about the following characteristics and qualities, please say whether you think each applies or doesn't apply to Hillary Clinton. How about...Is honest and trustworthy?

53% applies
43% does not apply
4% unsure

So a year (or more) ago, roughly the same percentage of Americans considered Hillary Clinton "honest and trustworthy" as expressed little or no concern about Clinton corruption in the Zogby/Judicial Watch survey. While the comparison is obviously imperfect, the lesson here may be that well developed opinions tend to be more resistant to manipulation by leading questions. If you are convinced that Hillary Clinton is honest (or dishonest), the leading language is unlikely to alter your answer either way. 

Don't get me wrong. I am not defending the Zogby questions, which are obviously and comically biased. However, the similarity in results when compared to fairly worded questions about honesty and trust suggest that opinions toward Hillary Clinton are well developed and resist manipulation. Voters have a pretty clear sense of who Hillary Clinton is, and those opinions may be difficult for either Clinton or her foes to change.

UPDATE: Nancy Mathiowetz, the president elect of the American Association for Public Opinion Research (AAPOR) just sent out the following release concerning the Zogby/Judicial Watch poll (interests disclosed - I serve on AAPOR's Executive Council):

It's always disappointing when pollsters who are internationally known and widely quoted engage in practices that are so clearly out of line with industry standards -- like using loaded and biased questions. There's no other way to describe the questions in the Zogby poll performed for Judicial Watch.

The good news is that it did not fly under the radar -- The Washington Post was quick to point out the flagrant disregard for accepted survey standardsin the poll - A number of blogs whose authors are well versed in industry best practices and standards also wrote about the poll.

Industry standards, including the American Association for Public Opinion Research's Best Practices , make it clear --the manner in which questions are asked as well as the response categories provided, can greatly affect the results of a survey.

That's why question wording and order are some of the toughest parts of designing a good survey or poll, and thoughtful practitioners will spend a significant amount of time trying to ensure that they are balanced, simple, direct and clear

Nancy Mathiowetz
American Association for Public Opinion Research

Bush Approval: Two polls at 35%, Trend at 33.8%

Topics: George Bush


The new Hotline poll, taken 3/29-4/1/07 finds approval at 35% and disapproval at 61%. The AP/Ipsos poll done 4/2-4/4/07 has approval at 35% and disapproval at 62%.

These move the approval trend estimate to 33.8%, well within the range of recent variation of the estimator.

There are still no recent outliers, so the dynamics appear unchanged since the last discussion of results here.





Cross-posted at Political Arithmetik.

POLL: Rasmussen Clinton vs, with and without Bloomberg

A new Rasmussen Reports automated survey of 800 likely voters nationwide (conducted 4/2 through 4/3) finds:

  • Sen. Hillary Clinton runs at a one-point advantage over Sen. John McCain (47% to 46%) in a general election match-up. In a three-way race with Mayor Michael Bloomberg running as an independent candidate, Clinton leads McCain 46% to 37% with Bloomberg trailing at 9%.
  • Former Mayor Rudy Giuliani edges out Clinton 48% to 47% in a seperate general election match-up. When Bloomberg is included, Hillary leads Giuliani 46% to 37% with Bloomberg trailing at 9%.

POLL: AP-Ipsos National Survey

A new AP-Ipsos national survey of 1,000 adults (conducted 4/2 through 4/4) finds:

  • 35% of Americans approve of the way Bush is handling his job as president; 62% disapprove.
  • 40% approve of the way Congress is handling its job; 57% disapprove.
  • 50% think the Iraq war is a "hopeless cause;" 42% call it a "worthy cause."

POLL: SurveyUSA Kentucky Gubernatorial Primary

A new SurveyUSA automated survey of likely gubernatorial primary voters in Kentucky (conducted 3/31 through 4/2 for WHAS-TV) finds:

  • Among 546 Democrats, former Lt. Gov. Steve Henry and businessman Bruce Lunsford both run at 20%, former Lt. Gov. Steve Beshear at 15%, and Speaker Jody Richards at 12% in a statewide gubernatorial primary.
  • Among 408 Republicans, Gov. Ernie Fletcher leads former Rep. Anne Northup (40% to 31%) in a statewide gubernatorial primary; businessman Billy Harper trails with 16%.

Taylor's Column on Internet Polling

Topics: Internet Polls

In my post on Monday, I mentioned an article by Humphrey Taylor of Harris Interactive that appeared in the recent subscriber-only print edition of the Polling Report. We asked nicely, and the powers-that-be at the Polling Report have provided free access to Taylor's piece.

Taylor's piece is obviously not the last word on Internet polling, but his argument is well worth the click. The debate over "non-probability" sampling and Internet methodology is certainly one worth having.

Finally, in that Monday item, I neglected to include one bit of full disclosure that should accompany any commentary here on Internet polling: The primary sponsor of Pollster.com is the research firm Polimetrix, Inc. which conducts online panel surveys.

POLL: Zogby New Hampshire Primary

A new Zogby telephone survey of likely primary voters in New Hampshire (conducted 4/2 through 4/3) finds:

  • Among 504 Democrats, Sen. Hillary Clinton runs at 29% while Sen. Barack Obama and former Sen. John Edwards are tied at 23%.
  • Among 502 Republicans, former Gov. Mitt Romney and Sen. John McCain are tied at 25%, while former Mayor Rudy Giuliani trails at 19%.

Diageo-Hotline's Generic Presidential Vote

Topics: 2008 , The 2008 Race

Last week I received a great question from astute reader JG:

I'm an avid but amateur follower of presidential polling, and I've been wondering why all the polls to date, at least the ones reported here and at pollingreport.com, are horse-race types, matching each of the presidential candidates against each other. Why haven't any firms asked a question about a generic Democrat-Republican matchup, that is, whether voters are likely to vote for the Democratic or a Republican presidential candidate, whoever that is? Wouldn't that be more informative about the state of voters than all the horse-race questions?

While I thought this was a good question to put to the media pollsters that conduct most of the national surveys, I emailed JG back that my guess was that most prefer to use candidate names whenever feasible. The main reason they ask a "generic" vote at all is that trying to identify and administer 435 different Congressional match-ups is simply to complex a task for an RDD telephone survey.

Well, the Diageo/Hotline poll is about to provide JG with a better answer. The Hotline is telling its paid subscribers that it will release a survey later today that will show "a generic Dem candidate leading a generic GOP candidate 47%-29% in a WH '08 matchup." Results should be posted at diageohotlinepoll.com sometime this afternoon.

Regular readers will recall that I have never been a big fan of the generic vote (see commentary here, here and here). This far out, I believe that a generic vote question tells us mostly about the way voters perceive the national political parties. While those images apparently give the Democrats a huge early advantage - a finding that is certainly informative about the voters' current attitudes - the ultimate nominees of each party and their campaign messages will likely reshape those images. So, for my money, the generic vote remains something of questionable value in tracking where the race will be in 18 months. But for now (or whenever the Hotline posts the numbers)...have at it.

PS: The comment by reader Tlaloc below reminds me that one pollster - the Democracy Corps project led by Democrat Stan Greenberg - did create a "generic" vote question that inserted candidate names into a sample of 50 competitive congressional districts (see Q27). Greenberg's question actually debuted on an NPR survey he conducted along with Republican Glenn Bolger (that I wrote about here). Most pollsters now conduct surveys using "computer assisted telephone interviewing "(CATi) software that makes it feasible, with a bit of programming, to insert of candidate names for different districts.

The bigger limitation has to do with the sample. Greenberg's 50 district sample used 50 small samples drawn from registered voter lists, so it was a relatively simple matter to match telephone numbers to districts. However, most national surveys use a random digit dial (RDD) method that picks telephone numbers at random from working telephone exchanges (the first three digits of the seven digit phone number). Since phone exchanges do not correspond neatly with congressional district boundaries, it is impossible to precisely match telephone numbers with districts in an RDD sample.

POLL: CNN/WMUR/UNH New Hampshire Survey

Additional results to the recent CNN/WMUR statewide survey (CNN story, results; UNH analysis) of 303 likely Republican primary voters in New Hampshire (conduction 3/27 through 4/3 by the University of New Hampshire) finds:

  • Sen. John McCain and former Mayor Rudy Giuliani are tied at 29% in a statewide Republican primary; former Gov. Mitt Romney trails at 17%.
  • The survey measures favorable ratings of 72% for Giuliani, 69% for McCain, and 59% for Romney.

POLL: Cook/RT 2008 Survey

A new Cook Political Report/RT Strategies national survey of 807 registered voters (conducted 3/29 through 4/1) finds:

  • 28% of Americans think the new Democratic majority in Congress is doing better than the previous Republican majority, 17% say worse, and 45% say about the same.
  • Among 355 Democrats, Sen. Hillary Clinton (at 41%) leads former Sen. John Edwards (19%) and Sen. Barack Obama (17%) in a national primary.
  • Among 290 Republicans, former Mayor Rudy Giuliani (at 34%) leads Sen. John McCain (17%), former Sen. Fred Thompson (10%), former Speaker Newt Gingrich (9%), and former Gov. Mitt Romney (6%) in a national primary.

POLL: Quinnipiac NY Survey

A new Quinnipiac University statewide survey (Issues results, 2008 results) of 1,548 registered voters in New York State (conducted 3/28 through 4/2) finds:

  • Among Democrats, Sen. Hillary Clinton (at 44%) leads Sen. Barack Obama (14%), former V.P. Al Gore (14%), and former Sen. John Edwards (9%) in a statewide primary.
  • Among Republicans, former Mayor Rudy Giuliani leads Sen. John McCain (52% to 13%) in a statewide primary.
  • In general election match-ups, Giuliani trails Clinton (42% to 50%) while he runs within the margin of sampling error of Obama and Edwards; McCain trails all three Democratic candidates by more than ten points each.

POLL: Strategic Vision (R) Iowa Caucus

A new Strategic Vision (R) statewide survey of likely Republican and Democratic caucus-goers in Iowa (conducted 3/30 through 4/1) finds:

  • Among 600 Republicans, former Mayor Rudy Giuliani runs at 25%, Sen. John McCain at 20%, former Sen. Fred Thompson at 11%, former Gov. Mitt Romney at 8%, and former Speaker Newt Gingrich at 6% in a statewide caucus.
  • Among 600 Democrats, former Sen. John Edwards (at 27%) leads Sen. Barack Obama (20%), and Sen. Hillary Clinton (19%) in a statewide caucus.

Congressional Job Approval, 1990-2007


(Click once or twice on the graphic for a full resolution view. It is big.)

The Democratic congress was swept into office due in part to the low approval of the old Republican congress. Initially approval of Congress improved but is now beginning to slide down a bit. With a confrontation on Iraq war funding looming, it is time to take a look at congressional approval.

As of polling completed 3/28/07, approval of Congress' handling of its job is at 30.8%, compared to 33.3% for President Bush. But Americans love to hate Congress even as we usually love our district's member of Congress. So how does the current 30.8% approval compare with the past?

Over the long haul, from 1990, current approval is rather low, though not at rock bottom. Congressional approval bottomed out in 1992 during the House banking scandal in which members were allowed to maintain negative balances in their accounts. While trivial in comparison to national policy issues, this scandal was a potent symbol of congressional privilege and members who had bounced checks were seriously affected in the 1992 elections. Of the 47 members who bounced over 100 checks, 55% (26) were gone from the 103rd Congress in 1993. Even among those with less than 100 bounces the turnover rate was 43% higher than among those with no check problems. (Data from Jacobson and Dimock, American Journal of Political Science, August 1994, pp 601-624.)

At the low point, my estimate of congressional approval fell to an almost unbelievable 17.5% on June 7, 1992.

Approval improved in 1993 and the first half of 1994, but stayed mostly in the 20s, breaking 30% briefly in early 1994. But this small improvement was followed by another substantial collapse during the fall campaign, to 23.8% by election day. By comparison, in 2006 congressional job approval was at 27.5% on election day.

Following the 1994 Republican revolution, approval improved significantly for the first half of 1995, topping 35% for the first time since early 1990. However, this improvement reached its limit as the Republican Congress confronted a Democratic president over budget decisions. With the confrontation and the eventual shutdown of the federal government, congressional job approval turned down to the high 20s again, bottoming out as the budget crisis was eventually resolved.

Congress then enjoyed two and a half years of improving public approval. Interestingly, this occurred despite a Democratic president whose popularity was also increasing during this time. Approval reached 50% in early 1998, and remained there until after the 1998 elections.

But impeachment was a mistake, at least from the perspective of approval of Congress, with approval dropping sharply by some 7 points during and immediately after the impeachment proceedings. Congress remained below 45% until once more improving to 50% approval during the 2000 election contest.

As with presidential approval, 9/11 produced a significant upturn in congressional approval, though this fell back to around 47% by the 2002 elections.

The start of the Iraq war saw a substantial increase in approval of President Bush, but there is no evidence of a similar rally for Congress in early 2003. From 2003 through the 2004 elections, approval slowly declined to about 40% before turning up slightly at the end of 2004.

That was the last good news for congressional approval under Republican control. The spring of 2005 saw a very sharp drop in approval, from 41% to 33% in the first half of the year, followed by a slower but steady decline until mid-2006. Approval finally bottomed out around April 2006, and held relatively steady through the election, ending at 27.5% on election day.

With the start of the 110th Congress in January, the Democrats enjoyed a modest, but nonetheless significant upturn in approval, to 32.6%.

Since then, the trend has been slowly but steadily downward, reaching the current 30.8% approval as of March 28th.


Compared to 1994, the Democratic gain has been modest. Following the 1994 election, the new Republican Congress enjoyed an immediate upturn of over 7 points. The improvement peaked at 35.9% approval on March 30, 1995. The current Democratic gain was limited to 5.1%, and has since declined rather than continued to improve, as in early 1995.

And so we come to the first major confrontation of a president at 33.3% approval and a Congress at 30.8%. Given the results from 1995, a Congress should worry about confronting even an unpopular president. On the other hand, the budget battles of 1995 were quite different from the Iraq policy battles of 2007. President Bush seems to be betting that a cut off of funds, or even a temporary lapse of funding, will redound to his benefit. But recent polls have also found that more of the public support a withdrawal deadline than take the President's position. How this issue is shaped (starting a withdrawal versus cutting off the troops in the field) will have a significant effect on the outcome of the public debate. But it is a debate in which neither of the primary actors have a substantial advantage in public approval.

(I should note that ratings of "Democrats" or "Democratic leaders" in Congress are a bit above those of Congress as a whole, the variable of interest here. In the few polls that have asked about Democrats or Democratic leaders since the new Congress started, the approval ratings have mostly been in the low 40s, with one poll above 50. This is relevant to the debate over Iraq, and I will turn to this element of evaluation in a subsequent post.)

Cross-posted at Political Arithmetik.

Order Effects in The Fox News Poll?

A series of unusual questions near the end of a Fox News/Opinion Dynamics survey released last week generated criticism from the left side of the blogosphere (see Jonathan Singer on MyDD, TPM Cafe, Carpetbagger Report, Keith Olbermann via Crooks and Liars). As Carpetbagger's Steve Benen puts it, "FNC [Fox News Channel] is practically offering professors of quantitative analysis case studies in what not to do in a poll." He has a point. The wording and ordering of some of the question in the Fox News survey are a lesson in things pollsters should avoid.

Let's start near the end of the survey with a question about the recent congressional vote on the Iraq War (the Fox poll was fielded March 27-28 among 900 registered voters nationwide):

40. Last week the U.S. House voted to remove U.S. troops from Iraq by no later than September 2008 -- would you describe this as a correct and good decision or a dangerous and bad decision?

44% Correct and good
45% Dangerous and bad
11% (Don't know)

Now compare these results to findings from four other recent surveys (each sampled all adults rather than registered voters - text and results via Polling Report):

Newsweek Poll, March 28-29, 2007, n=1,004 adults - Do you support or oppose the legislation passed this week by the U.S. Senate calling for the withdrawal of U.S. troops from Iraq by March 2008?

57% Support
36% Oppose
7% Unsure

CBS News Poll, March 26-27, 2006, n=831 adults - Do you think the United States should or should not set a timetable for the withdrawal of U.S. troops from Iraq that would have MOST troops out by September 2008?

59% Should
37% Should not
4% Unsure

USA Today/Gallup, March 23-25, 2007, n=1,007 adults - Would you favor or oppose Congress taking each of the following actions in regards to the war in Iraq? How about setting a time-table for withdrawing all U.S. troops from Iraq no later than the fall of 2008?

60% Favor
38% Oppose
2% Unsure

Pew Research Center, March 22-25, 2007, n=1,245 adults - And thinking about a specific proposal: The Congress is now debating future funding for the war in Iraq. Would you like to see your congressional representative vote FOR or AGAINST a bill that calls for a withdrawal of troops from Iraq to be completed by August of 2008? .

59% Vote For
33% Vote Against
8% Unsure

So we have four slightly different questions from four different survey organizations asked during roughly the same period that provide very similar results. Support for the congressional vote to withdraw troops from Iraq in 2008 varies between 57% and 60%. But the Fox News poll shows only 45% support. Why the difference?

One contributing factor may be the oddly double-barreled answer categories on the Fox question that imply "danger" in the congressional vote. Put another way, the Fox item asks two questions: Was the vote "good" or "bad," and was the vote "correct" or "dangerous?" Good pollsters avoid double barreled questions, but even then, the wording of this question is odd. "Dangerous" is not exactly the converse of "correct." Some Americans may see the vote is both good and dangerous.** How should they answer the question?

But the bigger potential problem has to do with question order. Consider the question that Fox asked just before the item on the Congressional vote:

39. Who do you trust more to decide when U.S. troops should leave Iraq -- U.S. military commanders or Members of Congress? (ROTATE)

69% Commanders
18% Congress
7% (Both)
3% (Neither)
3% (Don't know)

40. Last week the U.S. House voted to remove U.S. troops from Iraq by no later than September 2008 -- would you describe this as a correct and good decision or a dangerous and bad decision?

44% Correct and good
45% Dangerous and bad
11% (Don't know)

While I can only speculate, I suspect that the order of these two questions primed respondents with the notion that the House vote was at odds with the recommendations of military commanders (although the initial item certainly said nothing about the views of those commanders). The only way to know for certain would be to conduct a controlled experiment that divides respondents into two random samples, asking Q39 first for half the respondents and Q40 first for the other half. And even that sort of experiment would tell us nothing about the potential impact of the 38 questions that came first.

Academic methodologists have conducted many such experiments over the years showing that the order of questions can sometimes affect results, often in very subtle and unexpected ways. In fact, the academic evidence on these "order effects: tends to involve far more subtle examples of priming than the one theoretically at work above (for examples, see Chapter 2 of the seminal text by Howard Schuman and Stanley Presser, Questions and Answers in Attitude Surveys).

I have sympathy for the task media pollsters often face in ordering questions on long surveys covering a wide range of topics that change from month to month. They seem to struggle to resolve goals that often come into conflict: On the one hand, they want to preserve the order of questions asked previously. On the other hand, inserting new items can risk an order effect in which the older items asked earlier bias the new questions inserted later. So they often face no-win choices.

In this case, however, the intent of the author of these questions seems clear. Check the MoveOn.org question (Q36) that nearly the liberal bloggers cited as a self-evident example of partisan bias:

36. After the 2004 presidential election, the president of the left-wing Moveon.org political action committee made the following comment about the Democratic Party, "In the last year, grassroots contributors like us gave more than $300 million to the Kerry campaign and the DNC, and proved that the Party doesn't need corporate cash to be competitive. Now it's our Party: we bought it, we own it and we're going to take it back." Do you think the Democratic Party should allow a grassroots organization like Moveon.org to take it over or should it resist this type of takeover?

16% Yes, allow a grassroots organization to run the party
61% No, don't allow a grassroots organization to run the party
24% Don't know

Could this clearly partisan "message test" have had an order effect of its own on the subsequent questions on the congressional vote on Iraq? Your guess is as good as mine, but either way, the items at the end of this Fox News survey are not exactly "by the book."

**See for example, the last two questions on this same Fox poll. On Q41, 59% agree that the "killing or capturing of Usama bin Laden" would be a "huge" or "major" accomplishment. But on Q42, 47% go on to say that bin Laden's capture would also "encourage additional terrorist attacks," while only 24% say it would "discourage" further terrorism. Many Americans apparently consider that outcome both desirable and dangerous.

POLL: Public Policy Polling (D) NC Primary

A new Public Policy Polling (D) automated survey of of 649 likely Republican primary voters and 500 likely Democratic primary voters in North Carolina (conducted 4/2) finds:

  • Among Republicans asked to choose between three candidates, former Mayor Rudy Giuliani (at 30%) leads Sen. John McCain (19%) and former Gov. Mitt Romney (14%) in a statewide primary; 25% would choose "a different candidate."
  • Among Democrats asked to choose between three candidates, former Sen. John Edwards (at 39%) leads Sen. Hillary Clinton (25%) and Sen. Barack Obama (20%) in a statewide primary; 8% would choose "a different candidate."

Bush Approval: Zogby at 32%, Trend at 33.3%

Topics: George Bush


A slightly stale Zogby poll taken 3/22-26/07 has approval at 32% and disapproval at 67%. The poll released results for other items, but the approval item did not appear until 4/3 on Hotline. With this result added, the approval trend stands at 33.3%.

Zogby tends to run below the trend, though in this case by only a small amount.

See my earlier post on the newer Newsweek poll for a discussion which is not changed by this revision of the trend estimate.





Cross-posted at Political Arithmetik.

Bush Approval: Newsweek at 33%, Trend at 33.5%

Topics: George Bush


A Newsweek poll taken 3/28-29/07 finds approval at 33%, disapproval at 60%. With this addition the approval trend estimate now stands at 33.5%.

The Newsweek poll, which often comes in with a below trend house effect, is very close to trend, so has little impact on the estimate. We've now seen several polls in the 33%-34% approval range, plus one at 37%.

The trend estimate has remained between 33% and 35% approval over the last 20 polls. While there appears to be a consistent but small negative slope in approval since the November elections, the rate of change is small enough that it has been hard to distinguish from stable approval. Individual polls, with a margin of error around 5 points about the trend estimate, cannot hope to reliably detect such a small rate of change. Even the trend estimate, with an uncertainty of about +/- 2 points is hard pressed to unequivocally call the current period a downward trend, though the bootstrap plot shows a small downtrend is a pretty good bet.

It is interesting that approval has not suffered a sharp downturn in the aftermath of the November elections and the Democratic takeover of Congress. Rather, it appears that public dissatisfaction with President Bush largely spent itself in the elections. Since then, the new opposition between Republican President and Democratic Congress appears to have stabilized opinion of Bush's handling of his job. As we saw in yesterday's presidential comments on funding for Iraq, the Democratic Congress provides a target for Presidential rhetoric which was lacking when Republicans controlled the Congress. The struggle over the emergency funding bill will provide an interesting lesson in the relative strengths of President and Congress.

The residuals remain well behaved, so no recent polls are currently "outliers".





Cross-posted at Political Arithmetik.

POLL: Field CA Primary

A new Field Poll statewide survey (Dem results, GOP results) of 802 likely voters in California (conducted 3/20 through 3/21) finds:

  • Among 417 likely Democratic primary voters, Sen. Hillary Clinton (at 41%) leads Sen. Barack Obama (28%) and former Sen. John Edwards (13%) in a California primary. With former V.P. Al Gore on the ballot, Clinton runs at 31%, Gore at 25%, Obama at 21%, and Edwards at 8%.
  • Among 315 likely Republican primary voters, former Mayor Rudy Giuliani (At 36%) leads Sen. John McCain (24%) and former Gov. Mitt Romney (7%). With former Speaker Newt Gingrich and former Sen. Fred Thompson on the ballot, Giuliani runs at 29%, McCain at 21%, Gingrich at 9%, and Thompson and Romney both at 7%.
  • In general election match-ups, Giuliani trails Clinton, Edwards, and Obama each by 10 or more points. McCain trails Obama and Edwards, but runs within the margin of sampling error Clinton (43% to 48% respectively).

POLL: University of Iowa Caucus

A new University of Iowa statewide survey of registered voters in Iowa (conducted 3/19 through 3/31) finds:

  • Among 298 likely Democratic caucus-goers asked to choose between three Democratic candidates, former Sen. John Edwards runs at 34%, Sen. Hillary Clinton runs at 29%, and Sen. Barack Obama runs at 19%.
  • Among 178 likely Republican caucus-goers asked to choose between five Republican candidates, Sen. John McCain runs at 21%, former Mayor Rudy Giuliani runs at 20%, and former Gov. Mitt Romney runs at 17%.


A new CNN/WMUR Granite State Poll (CNN story, results; UNH analysis) of 339 likely Democratic primary voters in New Hampshire (conducted 3/27 through 4/2 by the University of New Hampshire) finds:

  • Sen. Hillary Clinton runs at 27%, former Sen. John Edwards at 21%, Sen. Barack Obama at 20%, and former V.P. Al Gore at 11% in a statewide Democratic primary.
  • The survey measures favorable ratings of 80% for Edwards, 69% for both Gore and Obama and 64% for Clinton.

POLL: 4 ARG Statewide Surveys

Four new American Research Group statewide surveys of 600 likely voters each in Colorado, New Jersey, New York, and West Virginia find:

  • Sen. Hillary Clinton leads Sen. Barack Obama and former Sen. John Edwards by more than ten points in all four state primaries.
  • Former Mayor Rudy Giuliani leads Sen. John McCain in New Jersey and New York. Giuliani's advantage over McCain in Colorado (25% to 23%) and McCain's edge over Giuliani in West Virginia (33% to 29%) are both within the margin of sampling error.

POLL: 2 Rasmussen Nationwide Surveys

Two new nationwide Rasmussen automated surveys find:

  • Among 800 likely voters (conducted 3/28 through 3/29), Sen. Barack Obama leads former Gov. Mike Huckabee (52% to 32%), while he runs within the margin of sampling error of former Mayor Rudy Giuliani (43% to 44%).
  • Among 799 likely Democratic primary voters (conducted 3/26 through 3/29), Sen. Hillary Clinton (at 33%) leads Sen. Barack Obama (26%) and Sen. John Edwards (17%) in a national primary.
  • Among 590 likely Republican primary voters (conducted 3/26 through 3/29), former Mayor Rudy Giuliani (at 26%) leads Sen. John McCain (16%), former Sen. Fred Thompson (14%), former Speaker Newt Gingrich (11%), and former Gov. Mitt Romney (8%).

POLL: Quinnipiac Florida Survey

A new Quinnipiac University statewide survey (Issues results, 2008 results) of 1,061 registered voters in Florida (conducted 3/21 through 3/27) finds:

  • Among Democrats, Sen. Hillary Clinton (at 36%) leads former V.P. Al Gore (16%), Sen. Barack Obama (13%), and former Sen. John Edwards (11%) in a statewide Democratic primary.
  • Among Republicans, former Mayor Rudy Giuliani (at 35%) leads Sen. John McCain (15%) and former Speaker Newt Gingrich (11%) in a statewide Republican primary.
  • Giuliani leads Clinton, Obama and Edwards by ten or more points in general election match-ups; McCain leads Obama (45% to 39%) but runs within the margin of error of Clinton and Edwards.

Because You Asked...

Topics: Internet Polls

Pollster reader Petey posted a critical and incisive comment to my post last week showing that a new Harris Interactive Internet panel survey produced ratings of Hillary Clinton that were not wildly inconsistent with other recent conventional surveys:

Am I missing something, Mark?

Just because the Harris results are somewhat in line with random sampling poll results doesn't mean they should be treated as a real data point.

The Zogby Interactive results in the last election likewise were somewhat in line with other results, but they were still basically un-useful as polls.

No matter how good the weighting, self-selected polling is always going to be fundamentally flakey, no?

Petey -- assuming this is the same guy who often leaves astute comments on Pollster and Mystery Pollster -- does not miss much. His pointed question made me realize that a few lines of that post could have been written better. Specifically, by writing that a side-by-side test of the Clinton item using both the Internet panel and a traditional phone survey might "help resolve the sampling question," I implied that such a test might establish the validity of the Harris method. What I meant, more narrowly, as that it might resolve whether the Harris method produced a sample more hostile to Hillary Clinton in this instance (although since that post we now have another data point that allows a better comparison - see below). The larger issue of the merits of Internet panel surveys will not be resolved by any one test or blog item.

The problem with Internet panel surveys generally is their departure from random sampling. Traditional surveys begin by drawing a random sample from a "frame" (the pool of all wired telephone numbers, for example) that allows all or most of the population of interest a chance of being selected. Internet panels draw their samples from a pool of individuals who have volunteered to participate in online surveys, usually by responding to a banner advertisement. That fundamental difference, to answer Petey's question, is something that should make us more skeptical of Internet panel surveys generally.

However, it is important to remember that traditional telephone surveys now face fundamental challenges of their own. Response rates have fallen to below 30% on the best of public polls (and far lower on many others in the public domain), while the rapid growth of mobile phone usage has reduced the coverage of wired random digit dial (RDD) telephone samples below 90%. Since the true "science" of random probability sampling requires the assumption of 100% coverage and response - a goal that no public opinion poll comes anywhere close to reaching - we have to realize that any survey is now theoretically "flakey."

So how do we evaluate the relative "flakiness" of the competing methods? When it comes to non-probability Internet samples, some of my colleagues in the American Association for Public Opinion Research (AAPOR) like to remind me that (as one put it on Friday), "we can't accept a survey as valid just because we like the results." In other words, echoing Petey's point, if a survey lacks a sound theoretical basis, it should not be trusted just because it produces results that are consistent with other polls.

The problem with that argument, unfortunately, is that our continued trust in traditional surveys (despite their fundamental flaws) rests on studies that evaluate the results. We find reassurance in the way conventional surveys continue to predict election outcomes about as well as in past years. And the most rigorous academic studies -- such as those I saw presented at a workshop sponsored last Friday by the Washington DC chapter of AAPOR -- find few examples of bias due to low response rates once they are weighted demographically. For example, the influential study conducted by Scott Keeter and his colleagues at the Pew Research Center that used a side-by-side experiment to compare their standard methodology to a more rigorous design that produced a much higher response rate. The result? "[W]ithin the limits of experimental conditions, non-response did not introduce substantial biases in the estimates."

In other words, despite fundamental theoretical flaws in our traditional methods, we still like the results.

As such, those who produce surveys based on non-probability samples are right to ask whether their methods should also be evaluated on their "track record" (as Harris Interactive's Humphrey Taylor argued in an article in the January 15 print edition of the Polling Report). And comparing track records is what we aim to do here at Pollster.

The Zogby "Interactive" polls released in 2006 suffered from more than just theoretical questions Their results were far more variable and less valid (when compared to election results) than other conventional polls. And the public Internet panel surveys conducted during the 2004 U.S. presidential campaign by all sources (Zogby, Harris and YouGov) all showed a small but a consistent bias toward the Democrats (Professor Franklin and I been crunching the numbers on 2004 and 2006 and will have some posts to share on this subject very soon).

But all Internet panel surveys are not created equal, and past performance may not be a guarantee of future results - for any survey method. All of which brings me back to the Harris poll on Hillary Clinton. Since my post last week, Time released its latest survey which, by chance, included a question with a more similar wording and structure to the Harris item than those other surveys I looked at last week.

Time/SRBI - [Asked of 1,102 registered voters only, March 23-26] If the following candidate were to run for president and the election was being held today, how much would you support this candidate, definitely support, probably support, probably not support definitely not support?

Harris Interactive - [2,223 adults via Internet panel, Marc 6-14] If Hillary Clinton was the Democratic nominee for President, which is closest to the way you think? I definitely would vote for her, I probably would vote for her, I probably would not vote for her, I definitely would not vote for her, I wouldn't vote at all


In this case, the Time survey shows a bigger number (46%) in the two support categories than Harris (37%), although two big differences between the measurements (other than sampling) remain. The first is that the Harris item offered respondents the explicit choice of not voting at all, while the Time question offered only four categories of support. The second is that Time asked the question of registered voters only, while Harris asked it of all adults. We could get a closer comparison if we could tabulate the Harris result among self-reported registered voters, but no such results appear in the Harris release.

Nonetheless, we have at least a hint here that those who volunteer to participate in Internet panels may be less supportive of Hillary Clinton than respondents interviewed by conventional methods. And that is something we need to keep an eye on.

[Update - Interests disclosed: The primary sponsor of Pollster.com is the research firm Polimetrix, Inc. which conducts online panel surveys].

Bush Approval: Five new polls, Trend at 33.6%

Topics: George Bush


Five new polls have arrived. Fox (3/27-28/07) has approval at 33%, disapproval at 61%. Time (3/23-26/07) is at 33%/60%. Gallup/USAToday (3/23-25/07) is 34%/62%. Pew (3/21-25/07) has it 33%/58%. And Democracy Corps (3/20-25/07) is at 37%/58%. The Democracy Corps samples likely voters and has a consistently positive house effect for presidential approval. With these five new polls my approval trend estimate stands at 33.6%.

Based on the sensitivity analysis below, the variability in the trend estimate has remained between 33% and 35% approval over the last 20 polls, with only a slight negative trend. So far, at least, there is little evidence that the Gonzalez/prosecutor hearings are making a substantial impact on presidential approval. With further hearings, and a looming showdown over the war funding bill/veto, things could change with new developments in DC. More on that latter.





Cross-posted at Political Arithmetik.