July 29, 2007 - August 4, 2007


POLL: Newsweek National Survey

A new Newsweek national survey of 1,002 adults (conducted 8/1) finds:

  • 29% approve of the job George Bush is doing as president, 63% disapprove.
  • Among 324 registered Republicans and those who lean Republican, former Mayor Rudy Giuliani leads former Sen. Fred Thompson (30% to 22%) in a national primary; Sen. John McCain and former Gov. Mitt Romney trails at 13%. All other candidates receive less than 5% each.
  • Among 422 registered Democrats and those who lean Democratic, Sen. Hillary Clinton leads Sen. Barack Obama (44% to 23%) in a national primary; former Sen. John Edwards trails at 14%. All other candidates receive less than 5% each.

Bush Approval: DemoCorps 36-Pew 29-Trend 30.8

Topics: George Bush


A new Greenberg Quinlan Rosner/Democracy Corps poll, taken 7/25-29/07 finds approval of President Bush at 36%, with disapproval at 60%. The new Pew Center poll, also taken 7/25-29/07 puts approval at 29%, disapproval at 61%.

With these two new polls the trend estimate of approval now stands at 30.8%.

The GQR/Democracy Corps poll is a statistical outlier, falling above the 95% confidence interval around the approval trend. At the moment we have two outliers-- ARG's 25% is below the confidence interval.

GQR/Democracy Corps polls are of likely voters, rather than adults as with most polls here. As a result, they consistently estimate approval levels higher than for the adult population and hence above the trend line as well. This is clear from the plot below showing "Greenberg" surveys tracking high. This is a "design decision" to survey likely voters, and should not be considered a defect in the poll. Inferences are to a different population from that of the general population.


The "house effect" estimate for GQR/Democracy Corps surveys is just below 3 percentage points, with a confidence interval from just under 2 to just under 4 points. Despite this substantial house effect, GQR surveys have only rarely exceeded the confidence interval for presidential approval.

As the number of recent polls has increased, the standard "blue line" estimator of support has increased it's "bend" and is coming closer to the more sensitive "red line" estimator, further evidence that approval trends changed direction around June 28, when the red estimator put approval at 28.7%. Currently the blue estimator agrees with that date as the turning point, though the estimate will continue to change until enough data are available for a stable estimate of the turning point.

While the change in support is now pretty clear, the reasons are less so. The timing coincides with a series of Supreme Court decisions, all carried by the majority created with Bush's appointments of Roberts and Alito. I don't think the Court is salient enough with the general population to be a strong driver of public opinion, especially of presidential approval. However, it is possible that the most politically involved conservatives were both aware of the decisions and gave Bush credit for his appointments, helping arrest his decline in the polls.

The other event about this time was the commutation of Scooter Libby's prison sentence on July 2. Initial reaction among the general population and even many self-described conservatives was disapproving of this decision. But here too the effect may have been positive among conservative Republicans and served to shore up support among that constituency.

But that said, unlike previous sharp turning points that have corresponded to major presidential speeches, this one is harder to account for with "obvious" actions of the President (or of Congress, for that matter.) I invite your speculation on this.




Cross-posted at Political Arithmetik.

POLL: ARG California, Alabama Primary

Two new statewide surveys from American Research Group of likely voters in California and Alabama (conducted 7/30 through 8/2) finds:

  • Among 600 Democrats in California, Sen. Hillary Clinton leads Sen. Barack Obama (35% to 22%); former Sen. John Edwards trails at 16%, Sen. Joe Biden and Gov. Bill Richardson both at 5%. Among 600 Democrats in Alabama, Clinton (at 38%) leads both Edwards (19%) and Obama (17%). All other candidates receive less than 5% each.
  • Among 600 Republicans in California, former Mayor Rudy Giuliani (at 30%) leads former Sen. Fred Thompson and former Gov. Mitt Romney (both at 18%); Sen. John McCain and former Speaker Newt Gingrich both trail at 7%. Among 600 Republicans in Alabama, Thompson leads Giuliani (31% to 26%); McCain trails at 16%, Gingrich at 8%. All other candidates receive less than 5% each.

View all California Primary poll data at Pollster.com here:

Democrats Republicans

POLL: ABC/Post Iowa Dem Caucus

A new ABC News/Washington Post statewide survey (ABC story, results; Post story, results) of 500 likely** Democratic caucus goers in Iowa (conducted 7/26 through 7/31) finds a "three-way tie" between Sen. Barack Obama (27%), Sen. Hillary Clinton (26%) and former Sen. John Edwards (26%); Gov Bill Richardson trails at 11%. All other candidates receive less than 5% each.

** Likely voters were chosen if the respondents said they were registered to vote in Iowa at their present address, if they were certain to attend or probably will attend the Iowa caucus, and if they would vote specifically in the Democratic Party caucus.


How Tight is the Screen? Part II

Topics: 2008 , Disclosure , Likely Voters , The 2008 Race

I want to pick up where I left off on Tuesday, when I wrote about the way national surveys screen for primary voters. How well have the pollsters in early primary states done in disclosing how tightly they "screen" to identify the voters that will actually turn out to vote (or caucus)? Not very well, unfortunately.

For those just dropping in, here is the basic dilemma: Voter turnout in primary elections and, especially in caucus states like Iowa, is typically much lower than in the general election. A pre-election survey that aims to track and ultimately project the outcome of the "horse-race" -- the measure of voter preferences "if the election were held today" -- needs to represent the population of "likely voters." When the expected turnout is very low, that becomes a difficult task, especially when polling many months before an election.

And in Iowa and South Carolina, if history is a guide, that turnout will be a very small fraction of eligible adults,** as the following table shows:


When a pollster uses a random digit telephone methodology, they begin by randomly sampling adults in all households with landline telephone service. They need to use some mechanism to identify a probable electorate from within a sample of all adults. If recent history is a guide, the probable electorate in Iowa -- Democrats and Republicans -- will fall in the high single digits as a percentage of eligible adults. South Carolina's turnout is better, but is still unlikely to exceed 30% of adults. And while the New Hampshire primary typically draws the highest turnout of any of the presidential primaries, it still attracts less than half of the eligible adults in the state. Despite all the attention the New Hampshire primary receives, many voters that ultimately cast ballots in the November general election (roughly 30% in 2000) choose to skip their states' storied primary.

A pollster may not want to "screen" so that the size of their likely voter matches the exact level of turnout. Most campaign pollsters I have worked with prefer to shoot for a slightly more expansive universe, both to capture those genuinely uncertain about whether they will vote and to account for the presumption that "refusals" (those who hang up on their own before answering any questions) are more likely to be non-voters.

Nonetheless, the degree to which pollsters screen matters a great deal. If, hypothetically, one Democratic primary poll captures 10% of eligible adults while another captures 40%, the results could easily be very different (and I'll definitely put more faith in the first).

It also matters greatly how the pollster go about identifying likely voters. I wrote quite a bit about that process in October 2004 as it applies to random digit dial (RDD) surveys of general election voters. In extremely low turnout contests, such as the Iowa caucuses, most campaign pollsters now rely on samples drawn from lists of registered voters that include the vote history of individual voters. Most of the Democratic pollsters I know agree with Mark Mellman, who asserted in a must-read column in The Hill earlier this year that, "the only accurate way to poll the Iowa caucuses starts with the party's voter file."

So, based on the information they routinely release, what do we know about way the recent polls in Iowa, New Hampshire and South Carolina screened for likely voters? As the many questions marks in the tables below show, not much.


The gold star for disclosure goes to the automated pollster SurveyUSA. Of 22 survey organizations active so far in these states, they are the only organization that routinely releases (and makes available on their web site) all of the information necessary to determine how tightly they screen. Every release includes a simple statement like the one from their May poll of New Hampshire voters:

Filtering: 2,000 state of New Hampshire adults were interviewed by SurveyUSA 05/04/07 through 05/06/07. . . Of the 2,000 NH adults, 1,756 were registered to vote. Of them, 551 were identified by SurveyUSA as likely to vote in the Republican NH Primary, 589 were identified by SurveyUSA as likely to vote in the Democratic NH Primary, and were included in this survey.

I did the simple math using the number above (which are weighted values). For SurveyUSA's May survey, Democratic likely voters represented 29% of adults and Republican likely voters represented 28%, for a total of 57% of all New Hampshire adults. Their screen is a very reasonable fit for a survey fielded eight months before the primary.


Honorable mention for disclosure also goes to two Iowa polls. First, the Des Moines Register poll conducted by Selzer and Company. Ann Selzer provided me with very complete information upon request last year. Her first Iowa caucus survey last year used a registered voter list sample and screened reach a population that represents roughly 11% of the eligible adults (assuming 2.0 million registered voters in Iowa and 2.2 million eligible adults).

Second, the poll conducted in March by the University of Iowa. While their survey asked an open-ended vote question (rendering the results incomparable with those included in our Iowa chart), their release did at least provide the basic numbers concerning their likely voter screen. They interviewed 298 Democratic likely caucus goers and 178 Republican caucus-goers out of 1,290 "registered Iowa voters" (for an incidence of 37% of registered voters). Unfortunately, they did not specify whether they used a registered voter list or a random digit sample, although given the incidence of registered voters in Iowa, we can assume that the percentage of eligible adults that passed the screen was probably in the low 30s.


And speaking of the sampling frame, only 6 of 22 organizations SurveyUSA, Des Moines Register/Selzer, Fox News, Rasmussen Reports, Zogby, and Winthrop University specified the sampling method they used (random digit dial, RBS or listed telephone directory). I will give honorable mention to two more organizations -- Chernoff Newman/ MarketSearch and the partnership of Hamilton Beattie (D) and Ayres McHenry (R) -- that disclosed their sample method to me upon request earlier this year.

The obfuscation of this information by the remaining 14 pollsters is particularly stunning given that the ethical codes of both the American Association for Public Opinion Research (AAPOR) and the National Council on Public Polls (NCPP) include explicitly require the disclosure of the sampling method, also known as the sample "frame." The NCPP's principles of disclosure requires the following for its member organizations for "all reports of survey findings issued for public release:"

Sampling method employed (for example, random-digit dialed telephone sample, list-based telephone sample, area probability sample, probability mail sample, other probability sample, opt-in internet panel, non-probability convenience sample, use of any oversampling).

The AAPOR code mandates disclosure of:

A definition of the population under study, and a description of the sampling frame used to identify this population.

Finally, while virtually all of these surveys told us how many "likely primary voters" they selected, very few provided details on how they determined that voters (or caucus goers) were in fact "likely" to participate. The most notable exceptions were the Hamilton Beattie (D) Ayres McHenry (R) and Chernoff Newman/ MarketSearch polls in South Carolina, and the News 7/Suffolk University poll in New Hampshire. All of these included the questions used to screen for likely primary voters in the "filled-in" questionnaires that included full results.

So what should an educated poll consumer do? I have one more category of diagnostic questions to review, and then I want to propose something we might be able to do about the very limited methodological information available to us. For now, here's two-word hint of what I have in mind: "upon request."

Stay tuned.

**Political scientists typically use two statistics to calculate turnout among adults: all adults of voting age (also known as the voting age population or VAP), or all adults who are eligible to vote (or the voter eligible population or VEP). George Mason University Professor Michael McDonald has helped popularize VEP as a better way to calculate voter turnout, because it excludes adults ineligible for voting such as non-citizens and ineligible felons. The perfect statistic for comparison to telephone surveys of adults would fall somewhere in between, because adult telephone samples do not reach those living in institutions or who do not speak English, but might still include non-citizens that speak English (or Spanish where pollsters use bilingual interviewers).

In a state like California, with a large non-citizen population, VAP is probably the better statistic for comparisons to the way polls screen for likely voters. In Iowa, New Hampshire and South Carolina, however, the choice has very little impact. Had I used VAP rather than VEP above, the turnout statistics in the table would have been roughly a half a percentage point lower.

CORRECTION: Due to an error in my spreadsheet, the original version of the turnout table above incorrectly displayed turnout as a percentage of VAP rather than VEP. For reference, the table below has turnout as a percentage of VAP.


POLL: Pew National Survey

A new Pew Research Center national survey (summary, Congress, SCOTUS, Iraq, 08; results) of 1,503 adults (conducted 7/25 through 7/29 by Schulman, Ronca & Bucuvalas, Inc.) finds:

  • 29% approve of the job George Bush is doing as president; 61% disapprove.
  • Among 623 Democratic voters and those who lean Democratic, Sen. Hillary Clinton leads Sen. Barack Obama (40% to 21%) in a national primary -- widening her advantage to "nearly two-to-one;" former V.P. Al Gore trails at 12%, former Sen. John Edwards at 11%. All other candidates receive less than 5% each.
  • Among 546 Republican voters and those who lean Republican, former Mayor Rudy Giuliani (at 27%) leads former Sen. Fred Thompson (18%) and Sen. John McCain (16%) in a national primary; former Gov. Mitt Romney trails at 10%, former Speaker Newt Gingrich at 8%. All other candidates receive less than 5% each.
  • 43% say the U.S. will "definitely" or "probably" succeed in achieving its goals in Iraq; 49% say it will "definitely" or "probably" fail.

Bush Approval: Two New polls, Trend at 30.2%

Topics: George Bush


New polls from Hotline and NBC/WSJ continue to raise the trend estimate for approval of President Bush, taking the estimate above 30% for the first time in a while.

Hotline, taken 7/19-22/07 has approval at 33%, disapproval at 63%. The NBC/Wall Street Journal poll done 7/27-30/07 has approval at 31 and disapproval at 63%.

With these new data, the approval trend now stands at 30.2%. This is in keeping with the upturn in approval first noted here, and confirmed by recent polls. Of course, 30.2% is nothing to brag about, but it is a tad better than being in the 20s.

The exact timing of the change remains unclear from my standard estimator, but the more sensitive "red" estimator thinks the turn was sharp and the rise has been substantial. Remember, "red" is often fooled by a few polls, so retain some skepticism about the red estimator, but we've got enough recent polls to make me pretty confident "red" has the story right, if not the exact timing or extent of the rise.

With the new trend estimate, the ARG poll at 25% has now moved clearly into outlier territory. This is a good example of how determining what is or is not an outlier can shift as more complete data arrive.





Cross-posted at Political Arithmetik.

Does One Bad Pollster Spoil the Trend Estimate?

Topics: Divergent Polls



Yesterday, in response to this post, readers raised a number of excellent questions about the effects of individual polls in our trend estimates of candidate support (and just about everything else here as well, including presidential and congressional approval, support for the war and more.) Much of the discussion was about how to detect and exclude "bad" polls, which is a topic that covers a huge range of issues including "house effects" (the tendency of polling organizations to poll consistently high or low on some questions), outliers (single polls that are far from the rest) and more. The discussion will provide fodder for a number of posts to come later this month as I review our methods and try to clarify these and other issues. So there is a lot to do. Consider this a down-payment on the rest.

To paraphrase one question: "Why not exclude a polling organization if it consistently produces results out of line with everyone else?"

We could approach this in several ways. For example, suppose a pollster was consistently 4 percentage points high but their polls moved in synch with the trend in all the other polls. Movement in those polls would tell you a lot about dynamics of opinion even if the pollster were "biased" by 4 points. If the bias were consistent, then we could just subtract 4 points and have an excellent estimate of the trend. A simple shift of the average poll result above or below the overall trend is not in and of itself a clinching argument for excluding a pollster. I'll come back to this issue in much more detail later in this series of posts.

A simpler and more direct way to approach the question is to ask what difference does it make if we do include all polls, rather than exclude supposedly "bad" ones? Of course we'd have a major problem if the trend estimator were quite sensitive to individual polls or all the polls by a particular pollster. Happily, this is an empirical question, so we can answer it. And we don't have to know which pollster is "bad" to begin with.

The plots above show the trend estimate, using our standard estimator, as the black line. This uses all the polls we have available for the national nomination contests in both parties. The light blue lines are trend estimates that result when I drop each of the 19 different polling organizations, one at a time. Though the lines are indistinguishable, there are 19 different blue ones for each candidate in the figures. If the impact of individual organizations on the trend estimate were large, some of these blue lines would diverge sharply from the black overall trend line and we'd be seriously concerned about those polls that were responsible for the divergent results.

But that isn't what actually happens. The blue lines all fall within +/- 1 percentage point of the overall trend estimate and the vast majority are within less than +/- 0.5 points. There is no evidence that excluding any single organization has more than a trivial effect on the estimated trend. This alone is strong evidence that whatever problems specific pollsters or individual polls may have, they do not seriously disturb the trend estimates we use here at Political Arithmetik and Pollster.com.

It is interesting that the variation around the top candidates in both parties, Clinton and Giuliani, is larger than it is among the third place candidates, Edwards and Romney, while variation for the middle candidates falls in between. This is a possible clue to one aspect of "house effects". One well known source of house effects is due to how hard the interviewer pushes for an answer. Some organizations now routinely find 20% or more unable or unwilling to pick a candidate. Other organizations have less than 5% failing to choose a candidate. Now imagine yourself asked to pick, but lacking an actual preference. When pushed, who do you most likely "settle" for in order to placate the interviewer? I'd bet on the best known names. If that were the case, we'd see the greater variation around Clinton and Giuliani substantially explained by differences in how hard pollsters push for answers on the vote preference question. This is one more topic for another day.

The fact that we find little effect on the trend estimate due to excluding each pollster could mean one or both of two things: either no pollster is biased or discrepant enough to actually raise a problem in the first place, or the trend estimator we are using is statistically robust enough that it resists the influence of unusual pollsters or polls. The second possibility is true by design. I've chosen an estimation method and designed the approach we take so that the trend estimator should be resistant to bias due to a single organization or a single poll. While it can be fooled under the right circumstances, those should be both rare and short lived, rather than common and long term.

We are not in a position today to reach a conclusion about the first possibility, that none of our pollsters are consistently out of line with the others. That could be, but it could also be that one or more pollsters are in fact out of step but that the estimator successfully resists their influence. To address this more interesting question, will require more work and a separate post (or series of posts). It does seem to me that there are clearly systematic differences across polling organizations. I've done a good many posts in the past on "house effects" and on individual outliers, and will do more of that in the coming weeks. But it also disturbs me that many complaints are hurled at specific polling organizations with little or no effort to support the claims empirically and systematically. That is a job we'll begin to undertake here as a route to clarifying what the actual evidence is for which polls are less "reliable" than others, and what exactly that means. Stay tuned.

Cross-posted at Political Arithmetik.

POLL: Gallup Independents

New analysis from recent Gallup national surveys of adults that during the second quarter of 2007, finds 37% of adults call themselves independents, "a small -- but significant -- increase" since 2006.


POLL: Rasmussen Clinton, Kucinich vs. Giuliani, Thompson

New Rasmussen Reports automated surveys find:

  • Among 839 likely voters nationwide, both former Mayor Rudy Giuliani and former Sen. Fred Thompson edge out Sen. Hillary Clinton (46% to 45% each) in general election match-ups (conducted 7/30 through 7/31).
  • Among 1,200 likely voters nationwide, Rep. Dennis Kucinich trails both Giuliani (by 14 points) and Thompson (by 9 points) in general election match-ups.

POLL: NBC/WSJ National Survey, Part II

Additional results from the new NBC News/Wall Street Journal national survey (NBC results; WSJ story, results) of 1,005 adults (conducted 7/27 through 7/30) finds:

  • 31% approve of the job George Bush is doing as president; 63% disapprove.
  • 68% either believe the country "is currently in an economic recession" or "will be within the next twelve months;" 25% believe the country is not in a recession and will not be in the next twelve months.
  • Among likely Republican primary voters nationwide, former Mayor Rudy Giuliani (at 33%) leads former Sen. Fred Thompson (20%) and Sen. John McCain (17%) in a statewide general election match-up; former Gov. Mitt Romney trails at 11%. All other candidates receive less than 5% each.

Yesterday's results.

Mid-Week Remainders

Topics: 2008 , The 2008 Race

Republican pollster David Hill ponders the difficulty of polling in primaries and the August "Twilight Zone of presidential politics in which pollsters will have a hard time determining what's real and what is illusion in poll results."

"Gallup Guru" Frank Newport looks at Republican voter impressions of Rudy Giuliani's strengths and weaknesses and ponders why the Iraq War may not be "as much of a liability for [John] McCain as some may have thought."

CBS News polling director Kathy Frankovic ponders whether Americans would "accept a woman as commander in chief in times of international crisis?"

In a Politico op-ed, Doug Usher looks at the survey data and asks, "are voters ready for health care reform?"

The Washington Post's Jennifer Agiesta looks at attitudes on surveillance cameras in public places.

The Wall Street Journal's "Numbers Guy," Carl Bialik, picks at the methodologies of a "dubious" survey of New York City subway riders and a telephone survey sponsored by the Whole Foods and Wild Oats grocery stores being used by those companies to argue against the Federal Trade Commission's attempt to block their merger.

POLL: NBC/WSJ Democratic Primary

The first round of results from the new NBC News/Wall Street Journal national survey (WSJ story, results) of 1,005 adults (conducted 7/27 through 7/30) finds:

  • Among Democrats, Sen. Hillary Clinton leads Sen. Barack Obama 43% to 22% in a national primary; former Sen. John Edwards trails at 13%, Gov. Bill Richardson at 6%, Sen. Joe Biden at 5%. All other candidates receive less than 5% each.
  • General election match-ups

    Clinton 47%, Giuliani 41%
    Clinton 42%, Giuliani 34%, Bloomberg 11%
    Obama 45%, Giuliani 40%
  • 25% think Clinton is "very liberal" in her approach to the issues; 21% say "somewhat liberal," 35% say "moderate," 7% say "somewhat conservative," and 3% say "very conservative."

POLL: Democracy Corps (D) National Survey

A new Democracy Corps (D) national survey of 1,000 likely voters (conducted 7/25 through 7/29 by Greenberg Quinlan Rosner Research (D) ) finds:

  • 36% approve of the job George Bush is doing as president; 60% disapprove.
  • 52% approve of the way their member of Congress is handling his/her job; 22% disapprove.
  • 52% say "the gridlock in Washington leads me to want to see a Democrat elected as president;" 39% say Republican.

POLL: Zogby Iraq Confidence

A new UPI/Zogby Interactive online panel survey (via PoliticalWire) of 7,590 adults (conducted 7/13 through 7/16) looks at confidence in the president's and Congress's handling of the war in Iraq.

POLL: Siena NY Primary

A new Siena College statewide survey of registered voters in New York State (conducted 7/24 through 7/28) finds:

  • Among 176 Republicans asked to choose between five candidates, former Mayor Rudy Giuliani (at 40%) leads Sen. John McCain (13%), former Sen. Fred Thompson (11%), former Speaker Newt Gingrich (9%), and former Gov. Mitt Romney (7%) in a statewide primary.
  • Among 290 Democrats asked to choose between eight candidates, Sen. Hillary Clinton (at 48%) leads Sen. Barack Obama (14%), former V.P. Al Gore (10%), and former Sen. John Edwards (7%). All other candidates receive less that 5% each.
  • General election match-ups:

    Clinton 57%, Giuliani 36%
    Clinton 44%, Giuliani 26%, Bloomberg 22%
    Clinton 62%, Thompson 28%
    Obama 51%, Giuliani 40%
    Obama 56%, Thompson 25%

View all New York Primary poll data at Pollster.com:

ANYTopDems190.png ANYTopReps190.png
Democrats Republicans

Wilner: An NBC/WSJ Divorce?

Topics: Pollsters

Former NBC News political director Elizabeth Wilner devotes her Politico column to the state of media polling. She leads with this unhappy speculation about the NBC/Wall Street Journal poll:

Hanging in the balance of Rupert Murdoch's courtship of Dow Jones is the fate of one of the stalwart franchises in public opinion research on American politics and policy.

Competition being what it is, it's hard to imagine that a Murdoch-owned Wall Street Journal would be allowed to continue its nearly 20-year polling partnership with NBC News. Particularly when that partnership benefits business news channel CNBC, which Murdoch hopes to crush with a new, Journal-powered Fox business network.

The prospect of what Wilner later describes as a potential "NBC/Journal divorce" is a sad one for those of us who value the best in political surveys. Here is hoping that Murdoch lets this partnership continue.

Wilner has more on the way the news media covers polls. Read it all.

Update (Dept. of Unintended Irony): As it happens, the latest NBC/WSJ poll is out this morning (sub. req.), and shows Hillary Clinton widening her lead among Democrats, leading Barack Obama 43% to 22%, with John Edwards in third and other Democrats "in single digits." More details later.

On Iowa and ARG

Topics: Iowa

The release of new surveys yesterday by the American Research Group (ARG) in Iowa, New Hampshire and South Carolina has generated quite a bit of buzz as well as questions about ARG of the sort I alluded to yesterday in my post on the screens pollsters use to select likely primary voters and caucus attendees. That post is the first in a series that will look at the methodological differences among the various pollsters active in the early primary states. For now, however, let me take up two issues specific to Iowa and ARG's surveys there.

The first question comes from reader AL:

I think your averaging is wrong on the Iowa race for the democrats. I think it should be Edwards at 25.7 and Hillary at 25.4. You have the averaging mixed up. You should correct it.

I am not sure how AL arrived at those numbers, but the current Pollster.com estimates in Iowa (Clinton 25.7%, Edwards 25.4%) are not "mixed up." The confusion may arise from the fact that our estimates are regression based estimators rather than true averages. Professor Franklin explains the difference in detail here.


The second issue concerns the Iowa surveys by ARG. As commenter jsamuel put it, "ARG seems to almost always under poll Edwards." While I lack Professor Franklin's flair for graphics and regression trend lines, some simple averages show that jsamuel is right. Sen. Clinton does consistently better ARG's surveys than those from other pollsters, while former Sen. Edwards does consistently worse:


We logged in six new Iowa surveys during June and July. The two from ARG show Clinton ahead of Edwards by an average six points (31% to 25%), while the four from other pollsters give Edwards an average lead of five points (27% to 22%).

Surveys from February, March, April and May show essentially the same pattern. Clinton leads by two points (30% to 28%) in four surveys conducted by ARG, while Edwards leads by an average six points (27% to 21%) in 13 surveys by other pollsters.

Unfortunately, this is the sort of scenario in which averaging (or simple regression based estimation) can be potentially misleading. One pollster (ARG) is getting consistently different results and contributing a large number of polls to our overall estimate. So if ARG is both different and wrong, their polls are throwing off our estimates.

We know ARG's results in Iowa are different. Why? And what should we make of that difference? For some clues, stay tuned to my series on primary screens.

Pres08: Is Anyone Paying Attention? You bet!

Topics: 2008 , The 2008 Race


As everyone knows, the presidential contests started early this round. But how much attention are the races getting from ordinary people (i.e. not us junkies here.) The debates have had small audiences, but news coverage is quite substantial. How much attention are voters paying?

The Pew Center has asked a consistent question on attention to the campaign going back to 1988. Their question:

As I read a list of some stories covered by news organizations this past week, tell me if you happened to follow each news story very closely, fairly closely, not too closely, or not at all closely. News about candidates for the 2008 presidential election?

Looking at those saying they are following the news about the campaign "very closely" we find the pattern in the figure above. Around 20% of the sample of adults say they are paying this much attention to news about the election. On the one hand, that means many are not. On the other, no previous election for which we have polling this early comes close to that 20% figure. The three previous races with any polling as early as this found 10-15% paying "very" close attention to the campaign. Only about 10 months before election day, or about the start of the primaries and caucuses, do we typically see interest move up to high levels, varying from about 18-35% over the various election years.

So this year's level of interest looks quite high, compared to previous years about this time, but at an absolute level, it remains only a fifth of adults who are glued to campaign news.

If we add "very closely" with "fairly closely", the current interest is 53%, a substantial increase, but the pattern in the plot is unchanged-- 2007 stands above any previous year, but the final attention by the general election rises to over 90% when using this combined measure, so 53% now is certain to rise to near 100% by November 2008. So either analysis says current attention is well above the norm, but still has a lot of room for increase.

The most interesting races have, oddly enough, involved incumbents named Bush. The 1992 three way race and the 2004 contest provoked the greatest attention from the Pew samples. Curiously the closest of all contests, 2000, had the second lowest final attention measure of the five completed races in the plot.

Comparing attention today with past races, we appear about 5-6 months ahead of attention in previous cycles. The 20-25% paying high attention now is usually reached around January of the election year. One interesting question is how much, and how soon, attention will grow. Perhaps we are starting high this cycle, but will converge to the range of previous cycles by January. Or perhaps we'll remain above past races through November 2008. Wait and see.

The one constituency that is certainly paying more attention earlier is pollsters. At this point in the 2004 cycle, there had been 53 national nomination polls. As of July 22 this year, we had 109, more than double the previous pace.

In part this is because we started earlier this year:


The earliest consistent polling on the 2004 race began after November 2002. (I may have missed a handful of earlier polling, but certainly not a great many.) In contrast, regular polling on 2008 began immediately after November 2004. The density of polls in 2005 and especially 2006 was substantially above comparable levels in 2002-03.

However, once the contest ramped up, the rate of new polls has been quite similar between the two years. When the blue line for 2008 turns sharply up, it runs in parallel to the 2004 polling. Clearly 2008 will ultimately far outstrip 2004, but for now we are seeing comparable rates of polling, just shifted back in time.

The shift is the equivalent of six months in the campaign schedule. If I shift the 2004 data back by exactly 6 months, the result is a near perfect match, seen below (again, after the longer run up in 2004-2006).


So for those of us obsessed with campaigns and public opinion, this is great news-- the most studied race in history in going on right now, and we have box seats.

Cross-posted at Political Arithmetik.

How Tight is the Screen? Part I

Topics: 2008 , Disclosure , Likely Voters , The 2008 Race

The questions we seem to get most often here at Pollster, either in the comments or via email, concern the variability we see in the presidential primary polls, especially in the early primary states. Why is pollster A showing a result that seems consistently different than what pollster B shows? Why do the results from pollster C seem so volatile? Which results should we trust? I took up one such conflict last Friday.

Unfortunately, definitive answers to some of these questions are elusive, given the vagaries of the art of pre-election polling in relatively low turnout primaries. When confronted with such questions, political insiders tend to rely on conventional wisdom and pollster reputation. Our preference is to look at differences in how survey results were obtained and take those differences into account in analyzing the data.

At various AAPOR conferences in recent years, I have heard the most experienced pollsters repeatedly confirm my own intuition: To find the most trustworthy primary election polls, we need to look close at how tightly the pollsters "screen" for likely primary voters. In other words, primary and caucus turnout is usually low in comparison to general elections. In 2004 (by my calculations), the Democratic turnout amounted to 6% of the voting age population for the Iowa Caucuses and 22% for the New Hampshire primary. In other states, turnout averaged 9% in primary states and 1.4% in caucus states in 2004.

A pollster that begins with a sample of adults has to narrow the sample down to something resembling the likely electorate, which is not easy. As few will approach the task exactly the same way, this is an area of polling methodology that is much more about art than science. Nonetheless, in most primary polls, relatively tighter screens are preferable in trying to model a likely electorate.

Thus, to try to make sense of the polls before us we want to know two things. First, how narrowly did the pollsters screen for primary voters? Second, as no two such screens are created equal, what kind of people qualified as primary voters?

In this post, I will look at what some recent national polls have told us about how tightly they screened their samples before asking a presidential primary trial-heat question and what kinds of voters were selected. I will turn to statewide polls in Part II. The table below summarizes the available data, including the percentage of adults that get the Democratic or Republican primary vote questions (if you click on the table, you will get a pop-up version that includes the sample sizes for each survey).


Unfortunately, of the 20 national surveys checked above, only five (Gallup/USA Today, AP-IPSOS, CBS/New York Times, Cook/RT Strategies and NBC/Wall Street Journal) provide all of the information necessary to quantify the tightness of their screen question. Others fall short. Here is a brief explanation at how I arrived at the numbers above.

The calculation is easiest when the pollster reports results for a random sample of all adults as well as the weighted value the subgroups that answered the primary vote questions. In various ways, these five organizations included the necessary information in readily available public releases.

Five more organizations (CNN/ORC, Newsweek, LA Times/Bloomberg, the Pew Research Center and Time) routinely provide the subgroup sizes for respondents that answer primary vote questions, though they do not specify whether the "n-sizes" are weighted or unweighted. Pollsters typically provide unweighted counts because they are most appropriate for calculating sampling error. However, since the unweighted statistic can provide a slightly misleading estimate of the narrowness of the screen, I have labeled the percentages for these organizations as approximate.

Of those that report results among all adults, only the ABC News/Washington Post poll routinely omits information about the size of the subgroups that answer primary vote questions. Even though their articles and reports often lead with results among partisans, they have provided no information about the sub-group sizes or margin of error for party subgroups since February. While the Washington Post provided results for party identification during 2005 and 2006, that practice appears to have ended changed as of February 2007.

[CORRECTION: The June and July filled-in questionnaires available at washingtonpost.com include the party identification question, and those tables also present time series data for the February and April surveys. However, as these releases do not include the follow-up question showing the percentage that lean to either party (which had been included in Post releases during 2006), they still do not provide information sufficient to determine the size of the subgroups that answered presidential primary trial-heat questions].

Determining the tightness of the screen gets much harder when pollsters report overall results on their main sample for only registered or "likely" voters. Three more organizations (Diageo/Hotline, Fox News/Opinion Dynamics and Quinnipiac) provide overall results only for those who say they are registered to vote. For these three (denoted with a double asterisk in the table), I have calculated an estimate of the screen based on the educated guess that roughly 85% of adults typically identify themselves as registered voters on other surveys of adults.

Four more organizations (Rasmussen Reports, Zogby, and Democracy Corps and McLaughlin and Associates) report primary results as subgroups of samples of "likely voters." Since their standard releases provide no information on how narrowly they screen to select "likely voters," we have no way to estimate the tightness of their primary screens. If we simply divided the size of the subgroup by the total sample, we would overstate the size of the primary voting groups in comparison to the other surveys.

Finally, the American Research Group follows a procedure followed for many statewide surveys: It provides only the number of interviews asked the primary vote question with no information about the size of the universe called to select those respondents.

All of the discussion above concerns the first question: How narrowly did the pollsters screen? We have somewhat better information -- at least with regards to national surveys -- about the second question: how those people were selected. The last column in the table categorizes each pollster the by the way they select respondents to receive primary vote questions:

  • Leaned Partisans -- This is the approach taken by Gallup/USA Today, ABC News/Washington Post, AP-IPSOS. It includes, for each party, all adults that identify or "lean" to that party.
  • Leaned Partisan+ -- The approach taken by NBC/Wall Street Journal includes both party identifiers and leaners and those who say they typically vote in the primary election of the given party. The LA Times/Bloomberg poll takes a similar approach although its screen appears to exclude leaners.
  • RV/Leaned Partisan or RV/Partisan -- This approach is taken by a large number of pollsters. It takes only those partisans or "leaned" partisans that say they are also registered to vote. Those labeled RV/Partisan exclude party "leaners" from the subgroup.
  • Primary Voters -- This category includes the surveys that use questions about primary voting (rather than party identification) to select respondents that will be asked primary vote questions.

As should be apparent from the table, the pollsters that use the "leaned partisan" or "leaned partisan+" select partisans more broadly than those that include only registered voters or those that claim to vote in primaries. But all of these approaches are getting a much broader slice of the electorate than is likely to actually participate in a primary or caucus in 2008. As should be obvious, most of the national pollsters are not trying to model a specific electorate -- they are mostly providing data on the preferences of "Democrats" or "Republicans" (or Democratic or Republican "voters"). I wrote about that issue and its consequences back in March.

In Part II, I will turn to statewide polls in the early primary states and then discuss what to make of it all. Unfortunately, while the information discussed above is incomplete, the national polls look like a model of disclosure as compared to what we know about most of the statewide polls.

To be continued...

POLL: Gallup Candidate Confidence

A new Gallup panel survey of 1,011 adults nationwide (conducted 7/23 through 7/26) finds:

  • 55% have "a great deal" or "a fair amount" of confidence in Sen. John McCain "to do or to recommend the right thing for the war in Iraq;" 55% for former Mayor Rudy Giuliani, 54% for Barack Obama, 51% for Sen. Hillary Clinton, and 50% for Sen. John Edwards.
  • 69% have a great deal or a fair amount of confidence in Giuliani "to do or to recommend the right thing for handling terrorism;" 66% for McCain, 55% for Clinton, 53% for Obama, and 48% for Edwards.
  • 62% have a great deal or a fair amount of confidence in Obama "to do or to recommend the right thing for the economy;" 61% for Clinton, 60% for Giuliani, 53% for MCain, 51% for Edwards.

POLL: ARG Iowa, New Hampshire, South Carolina

Three new American Research Group statewide surveys of 600 likely Democratic primary voters and 600 likely Republican primary voters in Iowa, New Hampshire, and South Carolina (conducted 7/26 through 7/30) finds:

  • Among Democrats in Iowa, Sen. Hillary Clinton leads former Sen. John Edwards (30% to 21%); Sen. Barack Obama trails at 15%, Gov. Bill Richardson at 13%. Among Republicans, former Mayor Rudy Giuliani edges out former Gov. Mitt Romney (22% to 21%); Sen. John McCain follows at 17%, former Sen. Fred Thompson at 13%.
  • Among Democrats in New Hampshire, Clinton and Obama are tied at 31%; Edwards trails at 14%, Richardson at 7%. Among Republicans, Giuliani edges out Romney (27% to 26%); Thompson trails at 13%, McCain at 10%, former Speaker Newt Gingrich at 6%.
  • Among Democrats in South Carolina, Obama edges out Clinton (33% to 29%); Edwards trails at 18%. Among Republicans, Giuliani edges out thompson (28% to 27%); McCain trails at 10%; Gingrich and Romney both at 7%.

All other candidates receive less than 5% each.

POLL: SurveyUSA MN Senate

Two new SurveyUSA automated surveys in Minnesota find:

  • Among 628 registered voters, Republican Sen. Norm Coleman leads Democratic challengers Al Franken (49% to 42%), Mike Ciresi (48% to 42%), and Jim Cohen (49% to 37%) in statewide senatorial general election match-ups (conducted 7/26 through 7/29).
  • In a second survey among 600 adults, 43% approve of the job Coleman is doing, 48% disapprove (conducted 7/13 through 7/15).

Primaries Past

Topics: The 2004 Race


A quick reminder not to assume what is today will be so tomorrow.

The 2004 Democratic primary race as of late July 2003 showed a continuous first place held by Lieberman, though with a slow but steady erosion. Kerry and Gephardt locked in a long running tie, and Howard Dean a rising 4th place at about 12% support. Clark's late entry and sharp rise hadn't happened. Edwards looked like a goner as his initial 9% had sunk to about 5%. So from this, who would be the candidates left standing after Iowa?

But of course the dynamics changed. Between summer and late fall, Dean became the "inevitable" nominee, sparking talk of running mates and gaining Gore's December endorsement. Kerry by that point was under 10%.

And then Iowa happened, and support shifted dramatically to Kerry and Edwards and away from Dean. (And Gephardt and Lieberman were gone.)

To those who say this is clear evidence that early polling is useless, I'd say no-- early polling is reflecting the dynamics of the race. But the dynamics are highly fluid and the point of the polling is not to predict the winner from today's polls, but to understand how the race is moving and ultimately to look back at how we got to the final outcome.

The great mistake analysts make is to look at current polls and conclude from them that the dynamics are fixed. That Dean can't rise. That Dean is a lock. That Kerry was inevitable after all. The current Democratic race appears, as of last week's polling, to be relatively static. And compared to the Republicans that is certainly true. But let's not jump to the conclusion that the polls after Labor Day have to look like today's just because today's look like June 1. The polls are of interest for what they show about the history of the race so far and how it stands today. Not for their ability to predict what happens in a month or two.

Cross-posted at Political Arithmetik.

POLL: Rasmussen Obama, Edwards vs. Giuliani, Thompson

New Rasmussen Reports automated surveys find:

  • Among 643 likely voters, former Sen. John Edwards leads former Mayor Rudy Giuliani (49% to 42%) and leads former Sen. Fred Thompson (50% to 39%) in nationwide general election match-ups (conducted 7/25 through 7/26).
  • Among 1,472 likely voters, Sen. Barack Obama leads former Mayor Rudy Giuliani (47% to 41%) and leads former Sen. Fred Thompson (46% to 40%) in nationwide general election match-ups (conducted 7/23 through 7/24).

POLL: Rasmussen Meetings w/ Leaders, National Security

Via TPM-Election Central, a new Rasmussen Reports automated survey of 1,000 adults nationwide (conducted 7/25 through 7/26) finds:

  • 42% of Americans think "the next President should meet with the heads of nations such as Iran, Syria, and North Korea without setting any preconditions;" 34% think the next President should not.
  • Among Democrats 55% think the next President should, while 22% the next President should not.
  • 28% say they trust Sen. Hillary Clinton "the most to handle national security issues;" 20% say former Mayor Rudy Giuliani, 15% say Barack Obama, 11% say former Sen. Fred Thompson.