February 11, 2007 - February 17, 2007


Mystery R&R

Just a quick note that the "Mystery Family" and I will be taking a week off for some much needed R&R. In my absence, the incomparable Eric will be posting updates, and Professor Franklin should be posting more often then usual. So don't touch that dial! See you on Feb 26.

ABCs Langer on Misreporting of Survey Data

Gary Langer, the Director of Polling at ABC News, has posted a review of four recent examples of "flat footed" misreporting of "scientific" survey data. It should be required reading for reporters, bloggers, students and anyone else that writes about survey data.

Langer's examples involve recent studies on napping, adoption and spouseless women that all had a "common point:"

[T]hey appear to have been well-constructed and based on good-quality data. That elevates them above all the manufactured junk data that clamors at our doors and all too often insinuates itself into our news reports. But still the reporting on these studies is not what it could have been.

One take-away is that news organizations -- including this one -- need to sharpen their efforts to report scientific studies accurately. Another is that, as a news consumer, when you see a report on a study that's of particular interest, you might take advantage of the beauty of the Internet -- and click through to the study itself.

Amen. And go read it all.

Fort those who looking to follow Gary Langer's advice, here are links to the relevant surveys from the American Sociological Review (adoption), the 2005 American Community Study (spouseless women) and the Archives of Internal Medicine (napping).

Pew Center on Past Presidential Trial Heats

The Pew Research Center's Nilanthi Samaranayake and Scott Keeter have posted another useful review examining the reliability of early trial heat polls in past presidential elections. Their verdict: Not so good. While Republican front-runners have typically won their party's nomination, "the picture is more mixed" historically for Democrats and early general election trial-heats "have a poor track record." The Pew analysts also point out that the past success of Republican front-runners is of little value now since the 2008 field features "not one but two GOP frontrunners" (John McCain and Rudy Giuliani).

Observers of the early Democratic field have been arguing about whether Sen. Hilary Clinton's early lead in many primary trial heats makes her nomination "inevitable." The Pew analysis provides little resolution. On the one hand, they tell us, "early Democratic poll leaders won four out of eight open contests between 1960 and 2004." On the other, two of the four that did not win (Cuomo in 1991 and Hart in 1987) "withdrew from the race for reasons other than lagging support in the polls." So make of that history what you will.

And if that were not enough, Samaranayake and Keeter offer another source of uncertainty for those handicapping the 2008 contests:

[T]he past polling history may be less relevant today. The process is starting earlier than ever this year and while there are some well known contenders, the public's level of familiarity with the overall field of candidates is still very low. And the increased front-loading of the primaries and the growing importance of early fundraising means that the dynamics of the nominating process are apt to be somewhat different this election cycle, making comparisons with past elections less useful.

The Pew report includes a compilation of twelve early general election trial heats ("conducted in the first quarter of the year preceding the election" over a span of 35 years) that were "mostly wrong," either about the winner or margin of victory. They advise that early general election trial heats have little predictive value: "History suggests the political climate is almost certain to change between now and November 2008."

So hold on to your hats, frontrunners, it's going to be bumpy ride.

Pres08: Willingness to vote for candidates

Topics: 2008 , The 2008 Race


Fox/Opinion Dynamics has a nice new poll out this week. One striking result is that NONE of the six leading candidates enjoys having more strong supporters than strong opponents. Fox asked:

Now I am going to read a list of possible candidates for the next presidential race. For each one, please tell me whether you would definitely vote for that candidate for president, if you might vote for that candidate, or if under no conditions would you vote for that candidate.

The "don't know" rate for all these candidates is pretty low, 10% or less.

The result, plotted above, shows that all six have considerably more voters saying they would never vote for the candidate than saying they definitely would vote for them. In this figure being towards the upper left is bad news, lower right is good news. As in previous polls, former Speaker Newt Gingrich fares worst among these candidates, with very substantial numbers saying they would not vote for him under any circumstances. The rest of the candidates are somewhat bunched together.

The further a candidate is from the diagonal line (or closer to the lower left corner) the more people said they "might" vote for them. (Recall the DK rate is both low and pretty even across candidates, so this is mostly a true statement.) Senator Clinton is the candidate with the fewest "might" vote for responses, but even in her case 34% said they would consider her. Obama shows the most room for growth with 45% saying "might" vote for him. The others range from 39-44% saying might.

So these front runners have a long way to go, and must all face stiff strong opposition from significant chunks of the electorate. Or put differently, those persuadable voters will matter a lot.

Cross-posted at Political Arithmetik.

POLL: Rasmussen Gingrich/Putin

Additional analysis from a recent Rasmussen Reports automated survey of 800 likely voters nationwide (conducted 2/12 through 2/13) finds:

  • Former House Speaker Newt Gingrich trails former Vice Pres. Al Gore (40% to 51%) and runs slightly behind Sen. Hillary Clinton (43% to 50%) in general election match-ups.
  • 66% of Americans have an unfavorable impression of Russian President Vladimir Putin, while 74% view Russia as being "somewhere in between" an ally or an enemy.

POLL: Additional USA Today/Gallup Analysis

Additional analysis from a recent USA Today/Gallup national survey of 1006 adults (conducted 2/9 through 2/11) finds:

  • 65% of American adults say former Mayor Rudy Giuliani would make a "good president;" 60% say both Sen. Hillary Clinton and Sen. John McCain would make a good president; 53% say Sen. Barack Obama would.
  • Former Vice Pres. Al Gore's favorable rating stands at 52%, while Gore's unfavorable rating (45%) "has not changed at all" since December of 2002.

POLL: Fox National Survey

A new Fox News/Opinion Dynamics national survey (story, results) of 900 registered voters (conducted 2/13 through 2/14) finds:

  • 35% approve of the job Bush is doing; 56% disapprove.
  • Former Mayor Rudy Giuliani leads both Sen. Hillary Clinton (49% to 40%) and Sen. Barack Obama (45% to 39%) in general election match-ups.
  • 54% would vote against "funding the increase in troops" in Iraq, if he/she were a member of Congress.

POLL: Rasmussen McCain vs.

Additional analysis from a recent Rasmussen Reports automated survey of 800 likely voters (conducted 2/7 through 2/8) finds Sen. John McCain within the margin of sampling error when pitted against former Sen. John Edwards (44% to 45% respectively) and Sen. Barack Obama (44% to 44%).

POLL: UNH Statewide Survey

Additional analysis from a recent University of New Hampshire WMUR Granite State Poll of 538 adults in New Hampshire (conducted 2/1 through 2/5) finds:

  • 77% of New Hampshire adults approve of Gov. John Lynch's job handling; 6 percent disapprove.
  • A 23% plurality say "education funding" is the most important problem facing New Hampshire; 16% say "high taxes and desires to change the tax system" is the most important problem.

POLL: Pew National Survey

A new Pew Research national survey (analysis, results) of 1509 adults (conducted 2/7 through 2/11) finds:

  • 33% approve of how Bush is handling his job as president; 56% disapprove.
  • 72% think Bush "doesn't have a clear plan" for bringing the situation in Iraq to a "successful conclusion;" 21% think he does.
  • Of those who oppose, 72% think "Congress should try to block Bush's plan [to send more troops to Iraq] by withholding funding for the additional forces;" 20% think Congress should not.

POLL: More USA Today/Gallup Analysis

Additional analysis from a recent USA Today/Gallup national survey (analysis, video) conducted 2/9 through 2/11 looks at general election trial heats between Democratic Sens. Hillary Clinton and Barack Obama and Republicans Sen. John McCain and former Mayor Rudy Giuliani.

Additional analysis also finds former Pres. Bill Clinton at his highest favorability rating (63%) since February 1998.

POLL: Coleman vs. Franken

A new SurveyUSA automated survey of 632 registered voters in Minnesota (conducted 2/12 through 2/13) finds Sen. Norm Coleman leading both Al Franken (57% to 35%) and Mike Ciresi (57% to 34%) for the '08 Minnesota Senate seat.


A new Public Policy Polling automated survey (release, results) of 448 likely voters in North Carolina (conducted 2/12) finds:

  • In a hypothetical match-up for the '08 North Carolina Senate election, incumbent Sen. Elizabeth Dole (R) leads Rep. Bob Etheridge (D) 45% to 30%.
  • 43% of likely North Carolina voters "want our next president" to be a Democrat; 41% want a Republican.
  • When asked to choose, 51% prefer "a universal health insurance program, where everyone is covered under a program like Medicare that is run by the government and financed by taxpayers;" 37% prefer "the current health insurance system, where most people get their health insurance from their emploters, but some people have no insurance."

Valentine's Day Round-Up

A new Monmouth University/Gannett New Jersey statewide survey of 590 New Jersey adults (conducted 2/8 through 2/12) finds:

  • 95% of New Jersey adults are "satisfied" with their current relationship.
  • 29% say they "flirt with other people;" 24% say their parter does.
  • 68% think most men would "have a one night stand if the opportunity presented itself and there was no fear of getting caught;" 40% think most women would.

A recent Rasmussen Reports automated survey of 1000 adults (conducted 2/4 through 2/5) finds:

  • 36% of unmarried Americans are "looking forward to" Valentine's Day; 22% are "dreading it."
  • Of the 58% who will be sending a card on Valentine's Day, 6% will send an e-card while 89% will send a traditional greeting card.

POLL: Rasmussen National Primary (GOP)

Additional analysis from a recent Rasmussen Reports automated survey of 507 likely Republican primary voters nationwide (conducted 2/5 through 2/8) finds former Mayor Rudy Giuliani (at 32%) leading Sen. John McCain (18%), Newt Gingrich (10%), and Mitt Romney (8%).

POLL: CBS News Mormonism

Additional analysis (story, results) from a recent CBS News national survey of 1142 adults (conducted 2/8 through 2/11) finds:

  • 63% of Americans would "vote for a qualified Mormon candidate" if he/she were a Republican; 66% would if he/she were a Democrat.
  • 25% have a favorable view of the Mormon religion, 30% have an unfavorable view, and 39% haven't heard enough to say.

POLL: USA Today/Gallup National Survey (Part II)

Additional analysis from yesterday's USA Today/Gallup national survey (USA Today story, results; Gallup analysis) of 1006 adults (conducted 2/9 through 2/11) finds:

  • Among 425 Republicans and Republican leaners, former Mayor Rudy Giuliani leads Sen. John McCain (40% to 24%) in a national Republican primary.
  • Among 495 Democrats and Democratic leaners, Sen. Hillary Clinton (at 40%) leads Sen. Barack Obama (21%), former Vice President Al Gore (14%), and former Sen. John Edwards (13%) in a national Democratic primary.
  • General election match-ups pitting Clinton against McCain (50% to 47%), Clinton againt Giuliani (48% to 50%), and Obama against McCain (both at 48%) are all within the margin of sampling error.

POLL: Harris Interactive National Survey

A new Harris Interactive online survey of 3423 adults (conducted 2/2 through 2/8) finds:

  • From a list of Republican and Democratic presidential candidates, a 20% plurality of Americans choose Sen. Hillary Clinton as their "one candidate" they "would most likely vote for;" 10% say Sen. Barack Obama and 8% say former Mayor Rudy Giuliani.
  • 45% say their first choice for President would be a Democratic candidate, while 31% would choose a Republican.

POLL: Quinnipiac NY Survey

A new Quinnipiac University statewide survey of 1049 registered voters in New York State (conducted 2/6 through 2/11) finds:

  • Among 485 registered Democrats, Sen. Hillary Clinton (at 47%) leads Sen. Barack Obama (16%) and former Vice President Al Gore (11%) in the Democratic primary.
  • Among 324 registered Republicans, former Mayor Rudy Giuliani (at 51%) leads Sen. John McCain (17%) in the Republican primary.
  • In general election match-ups, McCain trails Clinton (35% to 56%), Obama (35% to 46%), and Edwards (35% to 48%).

When Is a "Lead" Really a Lead?

In a post last week that presented an automated survey of North Carolina voters, we described a three-point lead for John Edwards over Hillary Clinton (34% to 31%) among Democrats as "statistically insignificant" and said that a six-point advantage meant that Rudy Giuliani "runs ahead" of Newt Gingrich among Republicans (31% to 26%). But reader "Thomas" asked a good question:

When I look at the results for the Republican candidates, there's a 6-point gap between Giuliani and Gingrich. But the size of the sample is only 735. Do you think this gap between the two candidates is really statistically more significant than the gap between the two Democrats candidates? I'm especially concerned with the size of the samples, and the way the interviews were conducted (automatically).

Thomas' question gets at an important issue for pre-election polls: How do we know when a lead is really a lead?

Let's get to the heart of the matter: The PPP survey of North Carolina Republicans reported a "margin of error of +/- 3.6%." Presumably, Thomas doubled that margin (getting +/- 7.2%) and compared it to the 6 point margin separating Guiliani and Gingrich. That's the right instinct, because the reported "margin of error" applies to each percentage separately. Looking at it that way, if you apply the margin of error to each candidate's percentage, you get a set of ranges that overlaps: somewhere between 27.4% and 34.6% for Giuliani and 22.4% and 29.6% for Gingrich. So how can that be a significantly meaningful lead?

The issue gets a bit technical, but the bottom line is that the statistical formula for a confidence interval (the formal term for "margin of error") for the difference of two percentages from the same sample produces something slightly smaller than just doubling the reported margin of error. I'll let my colleague, Prof. Charles Franklin, explain:

While [doubling the margin of error] is the correct conclusion when there are only two possible survey responses, it is not correct when there are more than two possible responses, which is in fact virtually always the case. The difference between the "twice the margin of error" rule and the correct calculation for the confidence interval of a difference of multinomial proportions will depend on how large are the proportion of survey responses other than that of the top two candidates combined.

Franklin's paper** has the complete formula and more details for those interested (see also Kish, Survey Sampling , 1965, p. 498-501), but the bottom line is that the margin of error for a difference of two percentages gets slightly smaller as the percentage falling into other categories (undecided or third candidates) gets larger. Franklin illustrates that point with the following graphic. The horizontal blue lines represent the reported margins of error (times two) for various sample sizes. The diagonal purple lines show how the margin of error for the difference of two percentages declines as the total of the percentages on which they are based ("p1 + p2") decline.


In this case, the margin of error for the 31% to 25% Giuliani lead is +/- 5.43, which would be just barely significant. So what do we make of that? Thomas' question implies that we should be skeptical about "barely significant" differences given that, in this case, the survey was automated. Let's consider that.

First, we need to keep in mind that this sort of significance test only takes into account the purely random variation that comes from drawing a sample rather than interviewing the entire population. Other potential errors could come from low rates of coverage or response (provided that the missing respondents have different opinions than those interviewed) or from the wording of the questions or their order. Unfortunately, the "margin of error" as we know it is not a measure of total error. So while other sources of error may not alter that "statistical significance" the result might still be wrong. Poll consumers should keep that in mind.

Also, the error margins calculated above assume a "simple random sample," but most political polls involve some weighting and other minor deviations from pure random sampling, which increase the error margin slightly.

Finally, keep in mind that the reported margin assumes a 95% level of confidence. That is, we are 95% certain a 31% to 25% lead on simple random sample of 735 respondents did not occur by chance alone. But there is nothing magic about 95%, it is just the common accepted standard used by most public opinion pollsters. If we wanted to be 99% certain, that 6 point lead would just miss "statistical significance."

All of which brings us to a lesson: As Professor Franklin likes to put it, we gain little by getting obsessed with "statistical significance," except when we are a few days before an election (and even then, it helps to look at many surveys, as we do here on pollster, rather than few). For a survey like this one, the concept of statistical significance provides an objective check, but it is more of a guide than a source of absolute rules.

**Charles wanted to make a few small revisions to his paper, which we should have posted soon.

POLL: Further UNH Analysis

Additional analysis on a recent University of New Hampshire Granite State Poll of 538 adults in New Hampshire (conducted 2/1 through 2/5) finds:

  • 31% of adults in New Hampshire approve of the job Bush is doing; 32% have a favorable opinion of him.
  • 25% approve of how Bush has handled the situation in Iraq; 72% disapprove.
  • 41% approve of Bush's handling of the economy; 53% disapprove.

POLL: CBS News National Survey

A new CBS News national survey (story, results) of 1142 adults (conducted 2/8 through 2/11) finds:

  • 67% do not think the U.S. military can be "effective in lessening fighting between Iraqis.”
  • 45% think Congress should not pass the "non-binding resolution against sending more troops to Iraq;" 44% think Congress should pass it.
  • 50% think the U.S. is "likely" to succeed in Iraq; 47% think it is "unlikely."
  • 32% approve of the job Bush is doing as president; 27% approve of his handling of Iraq

POLL: USA Today/Gallup National Survey

A new USA Today/Gallup national survey (USA Today story, results; Gallup Iraq analysis, job approvals analysis) of 1007 adults (conducted 2/9 through 2/11) finds:

  • 63% of Americans favor "setting a time-table for withdrawing all U.S. troops from Iraq by the end of next year."
  • 58% oppose "denying the funding needed to send any additional U.S. troops to Iraq."
  • 37% approve of the job Bush is doing as president
  • 41% approve of the job Democrats in Congress are doing; 33% approve of the job Republicans in Congress are doing.

POLL: ABC News College Costs

A new ABC News national survey (story, results) of 520 adults who have children under the age of 18 (conducted 1/26 through 2/4) finds:

  • 90% of Americans think it is likely their child/children "will attend a four-year college."
  • 42% think they'll be able to contribute to all or most of the cost of their child's/children's college education; 44% think a scholarship or grant will be used to pay for it.
  • 79% are confident that they ultimately "will be able to save enough to help pay for" their child's/children's college education.

POLL: Rasmussen National Primary (Dems)

A new Rasmussen Reports automated survey of 435 likely Democratic primary voters nationwide (conducted 2/5 through 2/8) finds:

  • Sen. Hillary Clinton (at 28%) runs five points ahead of Sen. Barack Obama (23%), while former Sen. John Edwards and former Vice President Al Gore trail at 13% and 8% respectively.
  • Clinton's margin over Obama was 16 points in last week's survey, but was as small as one point in early January.