March 4, 2007 - March 10, 2007


Primary Polling Primer I: Turnout

Topics: 2008 , The 2008 Race

With so many polls released in recent weeks testing the various 2008 presidential primary match-ups, now seems like a good time to take a deep breath and take a broad look at the role of public opinion polls in the race for the presidency: how they work, what they do well and what they do not so well.

Before we start considering the mechanics of how opinion polls measure voter sentiment, we really need to think about the underlying process of picking a presidential nominee and what we might like to know about "likely voter" opinion - at this stage of the process -- if we could. So before saying to much about the polls, I'd like to devote a few posts to the process.

In the United States, the two major political parties pick their nominees by a complex system of state primary elections and party caucuses that choose delegates that attend the to the national nominating conventions. While the process differs in each state, by expressing a preference for a presidential candidate, voters effectively choose convention delegates pledged to support that candidate at the convention. Many books have been written about this complex and sometimes bizarre process, but the two issues most consequential to opinion polling have to do with turnout and timing. Today I want to consider turnout and consider how much it varies from state to state.

There are many ways to calculate voter turnout, but unfortunately one of the most common - turnout as a percentage of registered voters - is difficult to do because registration procedures vary across states. To try to create more appropriate comparisons, I obtained the total number of participants in the Democratic primaries and caucuses in each state in 2004 and calculated turnout in two ways: As a percentage of all eligible adults (VEP) and as a percentage of those who would later cast ballots for Democrat John Kerry in November (details on sources below).


A few patterns are immediately obvious. First, participation in party caucuses - open meetings where participants must publicly declare their support for a candidate - typically involve just a tiny sliver of the state's Democrats. In most caucus states, participation as a percentage of those who would cast ballots for John Kerry the following November was in the low single digits. Even in Iowa - the caucus state with the highest participation rate - the estimated 124,331 Democratic participants amounted to just 17% of those who later voted for John Kerry.

Second, even in primary states, the range of turnout - as a percentage of adults, varies widely, from highs of 22.8% in New Hampshire and 21.1% in Wisconsin to the low single digits, even in blue states like New York (5.8%), Rhode Island (4.8%) and New Jersey (3.8%).

Third, the overall rate of participation is still quite low. Across all states (taking the larger turnouts from DC and Idaho which held caucuses on two different dates), Cook's figures show an estimated total of 16.8 million Democratic primary or caucus participants in 2004. That turnout amounts to a little over a quarter of the Americans (28.5%) that voted for John Kerry in November, and just 8.3% of the adults that were eligible to vote for President in 2004.

We will come back to the polling mechanics in more detail in subsequent posts, but consider the implications of the above for those national polls that ask about primary vote preference. In the context of this process, what is a "likely voter" exactly? Even if the process of identifying a likely primary voter were easy (and it isn't), the point here is that we lack a one-size-fits all definition of "likely primary voter" model that can be applied with equal precision across all 50 states.

To make all of this even more confusing, national polls typically make no effort to select likely primary voters. Most simply ask the vote preference question of adults or self-described registered voters that identify or "lean" to either the Democratic or Republican Parties. Thus, for the Democratic primary preference for example, national surveys are reporting the views of 35% to 55% of their adult samples that identify as Democrats or Democratic "leaners" when the people we really care about - those whose votes will ultimately choose the nominee - are likely to amount to less than 10% of adults.

And, for better or worse, even among that 10% some voters are going to matter considerably more than others, which brings me to timing, the subject of the next post.

Notes on the turnout table - The primary and caucus participation totals were graciously provided by the estimable Rhodes Cook (author of the America Votes and Race for the Presidency volumes published by Congressional Quarterly). To calculate percentages, I used data from the voter turnout web site maintained by GMU Professor Michael McDonald. His "voter eligible population" (VEP), a variant of the Census voting age population (VAP) statistic, is explained in more detail here.

Giuliani Leads; Press finally notices

Topics: 2008 , The 2008 Race


Since November 2004, former New York Mayor Rudy Giuliani has consistently led Arizona Senator John McCain for the Republican presidential nomination. Giuliani has led in 46 of 57 national polls since then, with 5 more polls tied. McCain has led in only 10.5% of all polls. In the last two months, the size of that lead has increased substantially.

Despite this consistent polling result, newspaper stories have overwhelmingly referred to McCain as the "front runner", while seldom saying the same about Giuliani. Since May 1, 2005, 555 articles in Lexis/Nexis's database of US News have mentioned "front runner" and McCain, Giuliani or both in the same sentence. In 65% of these article, McCain is connected to front runner, while Giuliani's name has not appeared. Only 10% of articles mentioned Giuliani but not McCain, while 25% of articles mentioned both. (I made this point in a December post, here, before the current surge in Giuliani support.)


Only in the last month (and especially the first week of March, when these data end) have reporters responded to the polling data, but in a somewhat misleading form. A number of reports have framed the increased support for Giuliani as a surge from behind, when in fact he was leading all along, albeit by a small margin. Yet that small margin was quite consistent if reporters had bothered to look at the data.

A clear reason for this emphasis on McCain has been Giuliani's slow start on staff and campaigning. In addition reporter's (and informed observers') view that Giuliani's social issue positions represented a large barrier to obtaining the nomination, has led many to discount the import of the polling lead in characterizing the race. And indeed, these may ultimately doom his campaign. But that isn't what the data say now, or have been saying for over two years.

But the recent polls should serve as a warning that sometimes you should pay attention to the data. The apparent surprise at Giuliani's recent lead, expressed in many articles, should not be so surprising at all if the data were taken seriously.

I am not making an argument that Giuliani's lead now is a prediction of the future. I AM saying that to understand political dynamics you must look at them in the context of all the evidence, in this case all the polling data. Reporters would produce better informed stories if they did so.

Cross-posted at Political Arithmetik.

[Typos corrected]

POLL: POS (R) LA Senate

A new Public Opinion Strategies (R) statewide survey of 500 likely voters in Louisiana (conducted 1/22 through 1/23 for the National Republican Senatorial Committee; released 3/8) finds Rep. Bobby Jindal (R) leading incumbent Sen. Mary Landrieu (D) 55% to 39% in a senatorial match-up.

POLL: CSRC Alabama Primary

A recent Capital Survey Research Center statewide survey of registered voters in Alabama (conducted 2/19 through 2/22; released 3/9) finds:

  • Among 351 Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (35% to 19%); former Sen. John Edwards trails at 9%, former V.P. Al Gore at 8%.
  • Among 402 Republicans, former Mayor Rudy Giuliani runs at 28%, Sen. John McCain at 23%, and former House Speaker Newt Gingrich at 18%.

POLL: ARG General Election Match-ups

Additional results from a recent American Research Group national survey of 2,104 likely voters (conducted 3/2 through 3/5) find:

  • Former Mayor Rudy Giuliani leads Sen. Hillary Clinton (48% to 42%) and Sen. Barack Obama (46% to 41%) in nationwide general election match-ups.
  • Match-ups pitting Sen. John McCain against Obama (46% to 42%) and McCain against Clinton (45% to 42%) are within the margin of sampling error.

POLL: AP/Ipsos Approvals

A new AP/Ipsos national survey (story, results) of 1000 adults (conducted 3/5 through 3/7) finds:

  • 35% approve how Bush is handling his job as president; 65% disapprove.
  • 32% approve his handling of the situation in Iraq; 66% disapprove.
  • 33% approve how Congress is handling its job; 63% disapprove.

POLL: NBC/WSJ Full Results

Full results are now available from the recent NBC News/Wall Street Journal national survey of 1,007 adults (conducted 3/2 through 3/5).

POLL: Rasmussen Brownback vs.

A new Rasmussen Reports automated survey of 800 likely voters (conducted 3/5 through 3/6) finds Sen. Sam Brownback trailing Sen. Barack Obama (34% to 49%) and within the margin of sampling error of Sen. Hillary Clinton (41% to 46%) in two nationwide general election match-ups.

POLL: ARG National Primary

A new American Research Group national survey of 600 likely Republican primary voters and 600 likely Democratic primary voters (conducted 3/2 through 3/5) finds:

  • Among Democrats, Sen. Hillary Clinton (at 34%) leads Sen. Barack Obama (31%) by three statistically insignificant points in a national primary; former Sen. John Edwards trails with 15%.
  • Among Republicans, former Mayor Rudy Giuliani (at 34%) leads Sen. John McCain (30%) by a statistically insignificant margin in a national primary; former House Speaker Newt Gingrich and former Gov. Mitt Romney trail with 12% and 7% respectively.

Bush Approval: NBC/WSJ 35%, Trend at 33.4%

Topics: George Bush


The new NBC/Wall Street Journal poll, taken 3/2-5/07, finds approval of President Bush at 35% and disapproval at 60%. Those results are identical to the previous NBC/WSJ poll taken 1/17-20/07. With the new poll my estimate of approval (the blue line) now stands at 33.4%.

For a detailed explanation of the graphs here, see this post. The goal of these plots is to place each poll in the context of the estimated trend, other polls, deviations from the trend, and the variability of these estimates over time.

The most recent six polls are shown below, along with the trend estimate. To see the entire set of 19 recent polls visit the approval page here.


The deviations from the trend give a sense of when polls are within the expected range of values we would expect due to random sampling and non-sampling errors. The 95% confidence interval plotted here shows the region within which 95% of polls fall. Polls outside that range are "outliers" and should be viewed as unexpectedly far from trend. There are various reasons why a poll might fall outside this range, and a single outlier should not be considered evidence of "bad" polling. But such polls should be interpreted skeptically in terms of where approval actually stands. The trend estimate remains the best estimate of approval at any point in time.


The trend estimator itself is subject to variability due to the timing of polls by various organizations. The plot below shows the range of estimated trends based on 20,000 bootstrap replications of the trend estimator. This is a conservative estimate of how variable the trend estimator is.


Each new poll at the end of the time series exerts significant influence on the latest estimated trend. This estimate will vary as new polls come in. The plot below shows the estimates for the last 20 "latest polls." The blue trend line is our best estimator of the path of approval after looking over all the available data. The variation in the red dots shows how much this has varied over the past 20 polls.


Cross-posted at Political Arithmetik.

POLL: NBC/WSJ National Survey

A new NBC News/Wall Street Journal national survey (NBC News story, results; WSJ story, results, John Harwood story) of 1007 adults (conducted 3/2 through 3/5) finds:

  • 35% approve of the job Bush is doing as president; 60% disapprove.
  • In a national Democratic primary, Sen. Hillary Clinton leads Sen. Barack Obama 40% to 28%, with 15% for former Sen. John Edwards and 11% for other candidates. In a two-way match-up, Clinton leads Obama 47% to 39%.
  • In a national Republican primary, former Mayor Rudy Giuliani leads Sen. John McCain 38% to 24%, with 10% for former House Speaker Newt Gingrich, 8% for former Gov. Mitt Romney, and 8% for other candidates. In a two-way match-up, Giuliani leads McCain 55% to 34%.
  • 69% of Americans are "less confident" that the war in Iraq will come to a successful conclusion; 29% are "more confident."

POLL: Winthrop University SC Favorables

A new Winthrop University statewide survey (release, results) of 694 adults in South Carolina (conducted 2/7 through 2/28) finds:

  • Of the Republican presidential candidates, 55% have a favorable opinion of former Mayor Rudy Giuliani and 50% have a favorable opinion of Sen. John McCain.
  • Of the Democratic presidential candidates, 48% have a favorable opinion of former Sen. John Edwards and 44% have a favorable opinion of Sen. Barack Obama. 32% have a favorable opinion of Sen. Hillary Clinton; 47% have an unfavorable opinion of her.

POLL: PPP NC Primary

A new Public Policy Polling automated survey of 635 likely Democratic primary voters and 847 likely Republican primary voters in North Carolina (conducted 3/5) finds:

  • Among Democrats, 29% would vote for former Sen. John Edwards in a statewide Democratic primary; 25% for Sen. Barack Obama and 21% for Sen. Hillary Clinton.
  • Among Republicans, 32% would vote for former Mayor Rudy Giuliani in a statewide Republican primary; 26% for former House Speaker Newt Gingrich and 17% for Sen. John McCain (17%).

A "Survey" on Work and Power

If it walks and quacks like a duck, we are told, it must be a duck. Yesterday, MSNBC posted a story describing results from an "online survey" conducted in partnership with Elle magazine showing that "men rule" in the workplace:

Of male respondents, 41 percent said men are more likely to be good leaders, and 33 percent of women agreed. And three out of four women who expressed a preference said they would rather work for a man than a woman.

Read the article and the survey cited certainly "quacks" like the sort of projective, representative, "scientific" survey that news organizations typically report. The article used survey results to describe how "men" and "women" feel. Academic experts explained and reacted to the findings. But all appearances to the contrary, it wasn't that sort of duck at all: The "survey" was neither based on a random sample nor something even MSNBC editors considered "nationally representative."

The description of the polls methodology consists of the following paragraph that originally appeared as a sidebar toward end of the story:

About the Survey

Our online survey was completed by 61,647 people, about 50 percent male and 50 percent female. The average age was 42, 94 percent said they work full-time and 44 percent said they supervise other workers. Although the sample size is large and diverse, it is not considered nationally representative because it was largely restricted to MSNBC.com readers.

While this paragraph tells you what the survey is not ("nationally representative), it manages to leave out a few critical pieces of information: How were respondents selected? By saying that the "sample" was "largely restricted to MSNBC.com readers," the paragraph implies that it was not a "sample" at all, but rather the result of a straw poll open to anyone that noticed the invitation to participate on the MSNBC. If so, respondents were not selected, they selected themselves.**

As a pollster, that distinction is obvious to me, as it was obvious to my colleagues on the listserv of the American Association for Public Opinion Research (AAPOR) who were abuzz about the survey yesterday. Late last night, AAPOR's president elect Nancy Mathiowetz sent an open letter-to-the-editor criticizing the article for not making it clear "from the start" that the survey was "decidedly not a valid scientific, nationally-representative survey." She continued:

Questions administered to a self-selected audience are one thing; real surveys that use scientifically determined random samples in an attempt to measure a population's attitudes and behaviors accurately are something altogether different.

Indulging in the former is not just harmless fun. Labeling self-selected, online data collection efforts as legitimate survey research are at best misleading and lead to a diminution of the field as a whole.

The survey MSNBC conducted is a grand way to involve readers with their website. Many news organizations use such techniques to enrich their relationships with readers. However, these polls cannot be construed to be a poll representative of anything more than those who chose to take part, and certainly shouldn't be characterized as representative of some larger population.

Two additional points worth noting: The first comes from David Moore, a former senior analyst and vice president of the Gallup Organization. Yesterday, he reminded his colleagues on the AAPOR listserv that the basic contention of the MSNBC story is essentially what Gallup surveys found between 1982 and 2002 (quoted with permission):

Whatever the problems with the "poll," the findings are not far off from a Gallup poll conducted in April 22-24, 2002 (the last time, I believe, that Gallup asked the question: "If you were taking a new job and had your choice of a boss would you prefer to work for a man or a woman?"). Gallup, by the way, has a trend on this question that goes back to 1953.

In April 2002, among males: 29% preferred a male boss, 13% a female boss, and 57% said no preference (a volunteered response); 1% no opinion Among females: 32% preferred a male boss, 23% a female boss, and 43% had no preference; 2% no opinion.

Moore had apparently not seen an update by Gallup conducted in September 2006 that produced very similar results:

A majority of men, 56%, say it makes no difference whether their boss is a man or a woman, while those who have a preference favor a man (34%) rather than a woman (10%). Among women, 40% say they would prefer a new boss to be a man, while 26% would prefer a woman. Thirty-two percent of women say it makes no difference.

Let's be clear that Moore and Mathowietz both condemn the MSNBC poll. "MSNBC should have conducted its own contemporary poll," Moore says, "rather than rely on the admittedly unrepresentative responses it received online."

Second, although totally non-random pseudo can confuse and mislead, they are relatively easy to condemn. Unfortunately, the more difficult issue for the survey industry remains what to do about the theoretically "scientific" surveys that fail to contact or interview individuals at three out of four randomly selected households. We need to be clear that even surveys based on scientific sampling may still reflect the unique slant of those who choose to participate. Unfortunately, the line between "scientific" and "pseudo" is not as clear as it used to be.

Interests Disclosed: I currently serve as AAPOR's associate chair for publications and information, and in that capacity, saw a draft of the Mathiowetz letter before its release.

Update: Eve Tahmincioglu, author of the MSNBC story, confirms by email that the survey "was based on responses to an invitation to participate on the MSNBC.com website." So the respondents were entirely self-selected, as per my speculation above.

Typo corrected

POLL: Suffolk University New Hampshire Survey

Suffolk University released a new statewide survey of 500 likely voters in New Hampshire (conducted 2/24 through 2/28). Support breaks down as follows.

  • In a Republican primary among 199 Republicans interviewed, 37% for former Mayor Rudy Giuliani, 27% for Sen. John McCain, and 17% for former Gov. Mitt Romney.
  • In a Democratic primary among 212 Democrats interviewed 28% for Sen. Hillary Clinton, 26% for Sen. Barack Obama, and 17% for former Sen. John Edwards.
  • In statewide general election match-ups, Clinton trails both Giuliani (40% to 50%) and McCain (39% to 51%) while Obama ties both Giuliani (at 41%) and McCain (at 42%).

[Editor's note: Although Suffolk University did not report a margin of error for subgroups, the margin of sampling error for a simple random sample of 212 is ± 6.4, assuming a 95% confidence level. The margin of error for 199 interviews is ± 6.9. By our calculations, neither the Giuliani-McCain nor the Clinton-Obama margins are statistically significant. Given the sample sizes, a candidate‚Äôs lead would have to exceed roughly 11 points to achieve statistical significance.]

POLL: Quinnipiac FL/OH/PA Surveys

Three new Quinnipiac University statewide surveys of 1,125 registered voters in Florida, 1,281 voters in Ohio, and 1,134 voters in Pennsylvania (conducted 2/25 through 3/4) find:

  • Sen. Hillary Clinton leads the Democratic primary field by 25 points in Florida, 13 points in Ohio and 11 points in Pennsylvania. Former Mayor Rudy Giuliani leads the Republican primary field by 20 points in Florida, 17 points in Ohio, and 26 points in Pennsylvania.
  • In separate two-way general election match-ups, Giuliani leads Clinton, Sen. Barack Obama, and former Sen. John Edwards in Pennsylvania and leads Obama and Edwards in Florida. Match-ups pitting Giuliani against Clinton in Florida and against the three democratic candidates in Ohio are all within the margin of sampling error.
  • Sen. John McCain leads Clinton, Edwards, and Obama in Pennsylvania, while he trails Edwards by seven points in Ohio. Match-ups pitting McCain against Clinton, Edwards, and Obama in Florida and against Obama and Clinton in Ohio are all within the margin of sampling error.

POLL: USA Today/Gallup '08 Primary

Additional results from the recent USA Today/Gallup national survey (USA Today story, results; Gallup analysis, video) of 1010 adults (conducted 3/2 through 3/4) finds:

  • Among 424 Republicans and Republican-leaning independents, former Mayor Rudy Giuliani leads Sen. John McCain (44% to 20%) in a national primary; former House Speaker Newt Gingrich and former Gov. Mitt Romney trail with 9% and 8% respectively.
  • Among 482 Democrats and Democratic-leaning independents, Sen. Hillary Clinton (at 36%) leads Sen. Barack Obama (22%) and former Vice President Al Gore (18%) in a national primary. John Edwards trails with 9%.
  • Among all adults, Giuliani has the highest favorable rating (64% to 21%) of the Republican candidates, followed by McCain with a 57% to 25% rating. Of the Democratic candidates, Obama has the highest favorable rating (58% to 18%), followed by Gore with a 55% to 39% rating.

Bush Approval: 33.1% and A New Look

Topics: George Bush


Public approval of President Bush's handling of his job has fallen to 33.1% after a weekend of new polling. Polls from Newsweek (2/28-3/1/07, 31% Approval, 61% Disapproval), Zogby (3/1-2/07, 30%/69%) and Gallup (3/2-4/07, 33%/63%) have pulled approval down from a high estimate last week of 35.5%.

This post also introduces a new look at the approval estimates. The central theme of my analysis of presidential approval has been to present the results "in context". The worst failing of reporting common to polling stories is the exaggeration of results based on single polls without regard to other nearly simultaneous results. Some reports emphasize the CBS/New York Times Poll taken 2/23-27 at 29%, while others stress the ABC/Washington Post poll of 2/22-25 at 36%. The former represents "a new low" while the latter is an "upturn". Yet both are within the range of results we would expect if approval is "really" around 33% (as I predicted here.) The myopic focus on individual polls undermines the credibility of probability sample polling by ignoring the variability that sampling theory predicts.

By looking at polls in context with other polls and over time, I aim to temper our interpretations by focusing on the common trend in approval polls, rather than emphasizing extreme results (in either direction.)

The trend estimate, plotted in blue in the figure above, is a local regression fit to the approval series. The advantage of local regression is that it can flexibly fit data with lots of "bumps and wiggles". The local regression will run through the "middle" of the data, with roughly equal numbers of polls above and below the trend line.

To see the most recent polls in context, I'm going to start posting the graph below-- a plot of the six latest polls showing the blue trend estimate and the data for each of the six polls.


This plot allows easy comparison of each poll with the trend, revealing polls that tend to run above or below the trend, an example of "house effects" which reflect persistent differences among polling organizations. These difference can be due to question wording, the size of "don't know" responses, sampling frames (adults versus likely voters, for example), question order and a variety of other causes. The plots above make clear both the size of such effects and the tendency of all quality polls to move up and down with the trend line regardless of house effects. In the plot above, Fox tends to produce results a bit above the trend, while CBS/NYT tend to be a bit below the trend line. But both polls move up and down with the trend, demonstrating that they are responding to the same changes in approval, even if the house effects produce persistent shifts from the trend.

This "six poll plot" also makes it clear how much variation there has been over recent polling so whatever the latest poll is, it can be seen in relation to both the trend estimate and to five other polls and past results of the same poll. If a poll is "out of line" with others, the six poll plot will make that immediately clear.

The next question about polls is whether they remain mostly within a reasonable interval of the trend estimate. When a poll falls beyond the range we expect due to random sampling plus non-sampling errors, it should be clear that the result is "unusual". This could be due to a sudden change in approval, but is more often just a random fluke, in which case the next poll by that organization usually returns to the range of other polling.

This reasonable interval is taken here to be a 95% confidence interval. I've estimated this for the 2005-present polling. Using all polls since 2002 produces little difference. The impact of 9/11 makes the 2001 data more variable than more recent years so I exclude that year in calculations of variability. The estimate I use includes the effects of non-sampling errors as well as sampling. The typical poll here has a sampling error of about +/- 3.0% to +/-3.5%. The actual 95% confidence interval is around +/- 5%. That increase from around 3% to around 5% reflects house effects, question wording, and everything else that increases poll variability beyond what is due to sampling alone. This is a more realistic estimate of the variability of poll results.

Plotting the residuals (the observed approval minus that predicted by the trend line) over time makes clear how variable polls are, and indicates which ones fall outside the 95% confidence interval. Here I highlight and label the last 10 polls to provide context. The plot below shows this view of current polling.


When a poll falls outside of this interval, it is further away from the trend estimate than we would expect 95% of the time. However, this doesn't mean the poll is "bad" and especially doesn't mean the survey organization is of poor quality. By definition, 5% of polls will fall outside this interval, so condemning polls or pollsters that occasionally produce these "outliers" is a bit harsh. We might wish to discount such polls, until supported by more evidence, or we might worry if we see a persistent pattern of outliers from an organization. But an occasional outlier is inevitable. This plot lets us spot such outliers immediately.

The plot also lets us see the variability in the last ten polls relative to the trend estimate. This is another way to see results in context. In the current plot, it is clear that while four polls have fallen well below the trend, there have been three of the last 10 polls that are equally far above the trend estimate. This range of +/- 5 points puts the extreme polls in context of all the variability we have seen recently.

Just as it is important to look at variability across polls, it is also important to examine the variability in the trend estimate. My local regression estimate has some advantages over simple rolling averages, but it is not without its own uncertainties. For example, it takes some 10-12 polls before changes in trend have been clearly identified by the estimator. Which polls happen to be "latest" at any moment have significant influence on the estimate. For example CBS/NYT at 29% pulled the estimate down when that was the latest, while the ABC/WP at 36% pulled the estimate up. This means that the current estimate can vary depending on which polls happen to be the latest. There is also uncertainty in the trend due to which polls are observed and when they are taken. To get a look at this uncertainty in the estimate, I create a "bootstrap" estimate of 20,000 replications of the approval series. Each of these bootstrap samples draws a random selection of all the polls, but allows a poll to be selected more than once or to not be selected at all. Each poll is equally likely to be sampled. For each sample, the full trend is estimated. In the end we see 20,000 estimates of the trend which vary due to the random draws of polls to include. This gives a good estimate of the variability the blue trend estimate might have if a different set of polls had been observed.

The result of this bootstrap estimation is presented below. The gray area is the full range of 20,000 samples, while the blue line is the estimate based on the actual polls we have.


The range of estimates is generally fairly close to the trend estimate, and certainly much closer than the range of actual polls around the trend. But it is also clear that the variability at the end of the series is a bit larger than it is in the interior of the series. This reflects the influence of late observations. This influence is reduced when there are polls on both sides to stabilize the trend estimate.

Finally, the estimate of the "current" approval level, varies up and down with each new poll. To see how sensitive this estimate is, I add a new plot below. It plots the "current" estimate when each of the last 20 polls was the latest. The variability of these around the blue line shows how much uncertainty we should have about the current estimate. The blue line is always our current best estimate of the trend (and the current approval.) But new data will change that a bit. The red points below remind us how much the estimate has varied recently.


All polls are subject to random variability. By looking at polls in context--- compared to the recent past, to other polls and to themselves over time--- we can gain a clear understanding of the state of presidential approval and its dynamics over the course of an administration. The graphs above provide a consistent and systematic look at the context of polling and the variability of the trend estimate which is our best estimate of where approval stands at any moment.

Cross-posted at Political Arithmetik.

POLLS: Three from SurveyUSA

Three new SurveyUSA automated surveys find:

  • Among 562 likely Republican primary voters in California, former Mayor Rudy Giuliani (at 41%) leads Sen. John McCain (23%) and former House Speaker Newt Gingrich (13%) in a presidential primary. Among 782 likely Democratic primary voters, Sen. Hillary Clinton (at 44%) leads Sen. Barack Obama (31%) and former Sen. John Edwards (10%). (conducted 3/3 through 3/5).
  • Among 415 likely Republican primary voters in Kentucky, former Rep. Anne Northup (at 31%) runs within the margin of sampling error of Gov. Ernie Fletcher (33%) in a gubernatorial primary. Among 529 likely Democratic primary voters, Lt. Gov. Steve Henry (at 26%) leads former Lt. Gov. Steve Beshear (15%) and KY House Speaker Jody Richards (13%). (conducted 3/3 through 3/5).
  • Among 501 likely voters in Kansas City, Missouri, former City Auditor Mark Funkhouser leads Mayor Pro Tem Alvin Brooks in a general election for mayor. (conducted 3/2 through 3/4).

POLL: Southern Media and Opinion Research Louisiana Senate

A recent Southern Media and Opinion Research statewide survey of 600 likely voters in Louisiana (conducted 1/12 through 1/14) finds Sen. Mary Landrieu (D) leading Sec. of State Jay Dardenne (R) (53% to 38%) in a hypothetical match-up for U.S. Senate.

POLL: USA Today/Gallup Bush, Iraq

A new USA Today/Gallup national survey (USA Today story, results; Gallup Bush analysis, Iraq analysis, video) of 1004 adults (conducted 3/2 through 3/4) finds:

  • 33% of American approve of the job Bush is doing as president; 63% disapprove.
  • 59% think the U.S. "made a mistake in sending troops to Iraq," the highest percentage since the question was first asked in March 2003; 39% think it was not a mistake.
  • 28% think the U.S. will win the war in Iraq, 20% think it can but will not win, and 46% do not think the U.S. can win.

POLL: Rasmussen National Primary/GE

Two new additional analyses from Rasmussen Reports finds:

  • Among 611 likely Republican primary voters, former Mayor Rudy Giuliani (at 34%) continues to lead Sen. John McCain (19%), and former House Speaker Newt Gingrich (12%) in a national Republican primary (conducted 2/26 through 3/1).
  • Among 800 likely voters (conducted 2/28 through 3/1) Sen. Joe Biden trails both Giuliani (38% to 49% respectively) and McCain (36% to 49%) in national general election match-ups (conducted 2/28 through 3/1).

POLL: Zogby Bush Approval

A new Zogby telephone poll of 867 likely voters (conducted 3/1 through 3/2) finds an "all-time low" of 30% approving of the job Bush is doing as president; 69% disapprove.

National Journal 2006 Liberal/Conservative Scores



All these graphs plus large format poster-size extras are available in high resolution .pdf format here.

National Journal released their 2006 liberal-conservative roll call scores for the House and Senate last week. The National Journal report of these scores is here. National Journal's scores are interesting because they calculate separate scores for economic, social and foreign policy dimensions, plus overall scores. They also use more roll calls in each issue area than most interest groups use in their "support scores". NJ uses 30-40 roll calls each for economic and social, and 17 or 18 votes for foreign policy. That compares to as few as 10 in the typical interest group rating.

By all means go to National Journal's site for the scores of individual members and for NJ's write reports on these results. They make it easy to find individuals or to look at the entire range of scores. Plus the reporting by Richard Cohen here is a good read.

I've used "Conservatism" score here. The NJ's liberalism score is 100-Conservatism, so the scores are mirror images of each other. (This is not exactly true of the three components, but they are near-mirrors so I only use the conservative score there as well.) Using this measure makes ideology score run left-to-right as it should for the graphs.

There are 44 Democratic Senators rated for 2006. Jay Rockefeller (WV) missed more than half the votes on social issues and was therefore excluded by National Journal from the overall rating (his conservatism scores were 37 and 39 on economic and foreign issues, which would put him near Blanche Lincoln (AR), who is the 5th most conservative Democrat.)

Members of the 109th Congress who did not return to the 110th are marked with an asterisk (*).

The top two plots show the Senate parties. The location of presidential hopefuls will be of interest, as will that of Joe Lieberman (9th most conservative among Dems) followed by a not-so-distant Hillary Clinton at 13th most conservative (32nd most liberal), only 4 spots to the left of Lieberman. By contrast Barack Obama ranks as the 10th most liberal Democrat (35th most conservative). The other two current Senators are Chris Dodd (CT) at 17th most liberal (28th most conservative) and Joe Biden (DE) at 24th most liberal (21st most conservative).

On the Republican side, John McCain (AZ) earns his moderate image by scoring as the 46th most conservative of 55 Republicans. Sam Brownback (KS) is 35th and Chuck Hagel (NE) is 29th.

If we look at the three subscales National Journal uses we find a couple of interesting things.


First, the three subscales are a bit more consistent with each other for Democrats than for Republicans. The red-Republican points the the three top left plots show relatively little relationship. The Democrats also show more scatter for the Foreign-Social relationship. This could be because the areas are not as ideologically integrated for Republicans as for Democrats, or it could be that the scaling is not that stable, even with 30 something votes. (Political scientists tend to use ALL non-unanimous roll calls for such scaling efforts, rather than just a few dozen. Those efforts have consistently found that roll calls fit a single dimension quite well, and do not require separate dimensions for social, economic or foreign policy domains. If that is right, then we might expect MORE relationship between these subscales than we see. Perhaps we'll get a look at this issue in some future analysis.)

It is interesting to note Sen. Joe Lieberman's positions in the plot above. The plots show quite well that Lieberman is unusual only on the foreign dimension, where his voting record makes him the most conservative Democrat (but among the 5 most liberal Republicans on the foreign policy domain.) On economic and social issues, he appears well within the range of Democratic senators and quite clearly to the left of all Republicans. This has serious implications for talk of a possible party switch. Unless his voting behavior changed, he would be the most liberal Republican by a fair margin.

For all the talk of bipartisanship, there is little ideological overlap of Senators.


Ben Nelson (NE) is by far the most conservative Democrat. But take him out (and with the defeat of Lincoln Chafee (RI) ) and there is literally no overlap between the parties' Senators. While the least conservative Republicans (Snowe, Collins, Specter, Coleman and Smith) have shown some willingness to vote with Democrats, and the most conservative Democrats (Nelson, Landrieu, Pryor, Nelson (FL)) sometimes cross party lines, there is less basis for ideological agreement than there was in the past when more members ideologies overlapped their party affiliations. Not that anyone really expects bipartisanship to break out anytime soon.

One other interesting result in the Senate plot is the line of "perfect" social conservative scores in the plot of the subscales above. Eighteen Republican Senators took the conservative position on social issues on every vote. But in the plot you can also see that this group varies quite a bit on the economic and foreign dimensions. While this group certainly defines the most conservative end of the Republican party, there is still a fair bit of variation in their overall conservatism scores, as seen below.


On the House side, with many more members, there is also a bit more partisan overlap, though less sign of bipartisanship.


The coherence of votes across policy domains (below) look about the same for the House as for the Senate. Again the Democrats appear somewhat more correlated across domains than do Republicans. Given the image of Republican unity in the House in the 109th Congress this remains a bit of a puzzle.


The entire House is plotted by party in the high resolution .pdf file here. There are multiple plots for letter size paper plus oversized posters for all members of each party.

Cross-posted at Political Arithmetik.

Newport on Clinton, Obama and the Black Vote

Gallup Guru Frank Newport posted Friday on the issue I had been watching all week: "Clinton, Obama and the Black Vote." He reviews most of the same numbers I did, but adds a few thoughts worth noting. One involves the imprecision of these relatively small sample sizes and some discussion of their treatment by the news media.

[A]ll of these estimates of the black vote are significantly less precise than the typical statistics reported for polls representing the broad population. That doesn't mean that - in the absence of other evidence - they shouldn't be used. It means that we need to be cautious in the conclusions we reach.

Earlier in the piece, Newport notes that Time withheld "specific numbers, a commendable approach based, no doubt, on concerns about releasing exact figures based on small sample sizes." His point brings us back to a question that I asked last week when The Washington Post first put the vote-by-race results in the headline of a front page story. Did that story deserve to be on the front page given the inherent imprecision of the relatively small sample size?

Put another way, if the shift to Obama in the Post/ABC poll is statistically significant, and if a shift to Obama among blacks is newsworthy, why not put it on the front page? Perhaps pollsters are just cautious by nature, but while I would have included those results as part of the story, I would have given them far less emphasis. The problem is that despite all efforts to emphasize the underlying statistical imprecision, specific numbers inevitably take on a life of their. The narrowing of the race among all voters was more modest, and given the other results out last week, the real shift among African-Americans was likely less than the 40 point net shifts measured by the Post/ABC polls. But that did not stop one Sunday talk show I watched (Chris Matthews) from pushing the 40 point shift as it if was the definitive result (no transcript available yet).

PS: Before moving on to other subjects, an update on some overlooked data. In addition to the data I cited last week, two other recent polls provided tabulations among African Americans.

  • The Cook-RT Strategies survey is now two weeks old (Feb. 15-18), but they conducted 70 interviews among African American Democrats and found Clinton leading Obama 45% to 22% (see CPR-7A, p. 16; the margin of error would be roughly +/- 12%).
  • The slightly older Pew Research Center survey (conducted Feb. 7-11) asked a different type of vote preference question. Respondents were asked whether there is a "good chance, some chance or no chance" that they would support each candidate. Among African American Democrats (sample size not specified), "63% reported a "good chance" of supporting Clinton and 50% a "good chance" of supporting Obama.

POLL: Rasmussen Democratic Primary

A new Rasmussen Reports automated survey of 769 likely Democratic primary voters (conducted 2/26 through 3/1) finds Sen. Hillary Clinton (at 34%) continuing to lead Sen. Barack Obama (26%) and former Sen. John Edwards (15%) in a national Democratic primary.

POLL: Newsweek National Survey

A new Newsweek/PSRA national survey (story, results) of registered voters (conducted 2/28 through 3/1) finds:

  • 31% approve of the way Bush is handling his job as president; 61% disapprove.
  • Two-way presidential general election match-ups pitting, variously, Republicans Rudy Giuliani and John McCain against Democrats Hillary Clinton, Barack Obama and John Edwards all produce results within the margin of sampling error.
  • In a two-way Republican primary, Giuliani leads McCain 59% to 34% among registered Republicans and Republican leaners; among social conservatives, Giuliani leads McCain 61% to 31%.