December 10, 2006 - December 16, 2006


News Roundup: 12.15.06

  • The latest NPR survey out today (article & audio , charts , full results) reports that "vote are feeling more positive about the Democratic Party than about the Republicans." The survey of 800 likely voters was conducted, as always, by the bi-partisan team of Republican Glen Bolger (Public Opinion Strategies) and Democrat Stan Greengerg (Greenberg Quinlan Rosner Research).
  • The Wall Street Journal's (free to all) Washington Wire blog has more analysis today on the most recent WSJ/NBC poll that covers a variety of topics, including questions showing that "eight in 10 Americans would be 'comfortable' or 'enthusiastic' about an African-American or woman running for president."

Amy Simon: Random Digits or Lists

Topics: 2006 , Pollsters , Response Rates , The 2006 Race

Today's Guest Pollster's Corner contribution comes from Amy Simon, a partner at Goodwin Simon Victoria Research.

News media and academics hold up Random Digit Dialing (RDD) sampling methodology as the gold standard for survey samples for elections. Meanwhile, many top notch political pollsters have been serving their clients well for years by instead using samples selected from the official list of registered voters (the statewide voter file), often called Registration-Based Sampling (RBS).

RDD samples are created when a computer randomly generates the last four digits of a phone number. The advantage of RDD is that everyone with a working landline phone is included in the sample - it doesn't matter if your phone service was just turned on that morning or if your number is unlisted, since the sample isn't generated from a list of actual phone numbers. An obvious disadvantage is that an RDD sample also includes business numbers, fax numbers, disconnected numbers, and even numbers that have never been connected - so the costs of administering an RDD sample are higher since the built-in inefficiencies bring down your contact rate.

An RBS sample draws a sample from a list of registered voters. The obvious advantage of using voter files for survey samples - one that has been noted for years - is that voter file studies are cheaper to administer than RDD studies. RDD surveys have to churn through not only bad numbers but also have to bear the cost of screening out the large portion of adults who are not registered voters, in order to find their real interview targets: respondents who self report as registered voters and who, after applying their own likely voter models, the pollsters define post-interview as likely voters.

With RBS surveys, when you do reach an actual person on the phone, you already know -- since you ask for them by name - that you have a real live actual registered voter on the line and therefore have a better production rate. (The cost difference between the two methods is even more significant in a primary or other low-turnout election scenario, but the debate about using RDD versus RBS samples in low, medium, and high turnout elections is another topic requiring its own separate discussion.) In states that have high quality voter history showing which registrants have actually voted in different types of elections, pollsters can use a likely voter screen to draw the sample in the first place, further ensuring that they are interviewing people most likely to vote in the kind of election they are attempting to measure.

Yet the news media and academics engaged in polling question whether RBS studies can be as accurate as RDD studies, since no voter registration list is 100% up to date, nor does any voter file include 100% of the phone numbers of voters. In fact, the phone match rate for a voter registration list is not only less than 100% but it can vary significantly across a state based on geography, with suburban areas showing a higher match rate than either urban or rural areas. So drawing an RBS sample requires special expertise in terms of controlling for this and other issues about who is potentially over-or under-represented in your sample. So why is it that so many experienced political pollsters continue to use RBS samples despite these concerns about its accuracy? We do so because we find that in many instances (though certainly not all) it is just as accurate, or even more so, than RDD studies.

In fact, some academics and media outlets have been experimenting with voter file survey samples and have found this to be the case. Several have publicly shared at least some of their findings about the ways in which the results do or do not differ when using RDD versus voter file samples. Several studies worth reviewing are by Mitofsky, Lenski and Bloom, by Gerber and Green in Public Opinion Quarterly and the online archive of Gerber and Green's work maintained by the list vendor Voter Contact Services (VCS). These studies have largely shown that RBS studies can be just as accurate and in some cases, more accurate, than RDD studies. One hypothesis offered is that samples drawn from voter registration lists by definition consist of actual voters, while RDD studies rely entirely on respondents' self-reporting about whether they are in fact registered to vote. Given the larger and larger portion of the adult American population that is not registered to vote, the potential for survey error when relying on self-reported behavior may be introducing larger error than carefully designed RBS studies contain.

In one recent example, we saw virtually no differences between the results of an RDD and an RBS study. We provide here just one example from our own work as the polling firm for Ned Lamont for U.S. Senate in Connecticut. In the course of the general election, at one point in September we simultaneously conducted both an RDD study and an RBS study. The results were dramatically in sync, with a margin of error of +/- 4.0 percent on the n=600 RBS study and a margin of error of +/- 3.5 percent on the n=800 RDD study. Considering the far higher cost of using RDD samples as compared to RBS samples, these results certainly give weight to the common practice among political pollsters of using voter file samples instead of RDD samples in general election campaigns.


Daily Roundup 12.14.06

  • A new survey from the Pew Research Center says that 60% of Americans who have heard at least a little about the Baker-Hamilton Report say they "mostly agree" with its proposals, while 57% say Bush will not follow the panel's major recommendations. In addition to public opinion on Iraq, the survey includes Bush job approval and news stories Americans are following.

  • The latest NBC News/Wall Street Journal survey says 69% of Americans are less confident the Iraq war will come to a successful conclusion." The survey also shows Sen. John McCain trailing Sen. John Edwards (43% to 41%) in a Presidential match-up. (story, partial results)

  • A Harstad Stategic Research survey released this morning shows Sen. John Edwards leading Sen. Clinton (36% to 16%) among likely Democratic 2008 Iowa caucus participants. Sen. Barack Obama and Gov. Vilsack trail with 13% and 9% respectively.

  • A new automated survey from Rasmussen Reports shows Sen. John McCain and Rudy Giuliani leading Sen. Evan Bayh by more than fifteen points in a Presidential match-up.

  • Sen. Hillary Clinton's job approval is at "all-time high" (72%) among New Yorkers according to Quinnipiac University's latest survey.

  • ABC has posted additional analysis on the survey released December 12th looking at public opinion of Bush's handling of the Iraq war. (full results)

Vote for Pollster in the Weblog Awards!

The category may sound a bit odd, but Pollster.com has been nominated for a Weblog Award as the "Best of the Top 1751 - 2500 Blogs." The winner will be chosen by a popular vote at this link. Voting closes Friday at midnight. Every little bit helps, so we would appreciate your vote.




Many thanks to those that nominated us, and as always, thanks to our regular readers for your continuing support.

Bush Approval: LA Times 42%; NBC/WSJ 34%

Topics: George Bush


Two more new polls. The LA Times (12/8-11) has approval at 42%, disapproval at 56% while the NBC/WSJ poll (12/8-11) puts approval at 34% and disapproval at 61%. The LA Times result is quite an outlier compared to other polling, and helps to raise my trend estimate to 34.5%.

About 90% of polls fall within +/- 4.4% of the trend estimate, and 95% fall within +/- 5.3%. At 7.5 points above the trend line, the LA Times poll is very far from what we would expect. If the LA Times poll were excluded from the trend calculation, the estimate would be 34.0%. Given the distance the LA Times is from the trend estimate, the 34% estimate is more plausible than 34.5%.

UPDATE: For a look at the LA Times trend in comparison to my trend estimate see the comparison chart from our Presidential approval page.

Cross-posted at Political Arithmetik.

Four Pollsters on the Incumbent Rule

Topics: 2006 , Incumbent Rule , The 2006 Race

I spent yesterday morning at a post election conference sponsored by Charlie Cook's Political Report, James Carville, Congressman Tom Davis and the Northern Virginia Community College. The conference kicked off with a panel of four very experienced campaign pollsters, two Republicans and two Democrats. They covered many subjects, and I can't possibly do them all justice here, but I do want to pass along some of what the pollsters had to say on what I typically refer to as the Incumbent Rule.

Carville called it "as good a pollster panel as has ever been put together," and he wasn't kidding. The Republican pollsters were Republicans Neil Newhouse of Public Opinion Strategies and Dave Sackett of The Tarrance Group. The Democrats were Harrison Hickman of Global Strategy Group and Stan Greenberg of Greenberg, Quinlan, Rosner. Each is a principal in their own firm, and each has been involved in some of the most competitive statewide races since the 1980s, and collectively their four firms polled in over 180 races for Senate, Governor and the U.S. House in 2006. It is hard to imagine any four campaign pollsters with more comparable experience.

Carville moderated the pollster panel, and his first question concerned his observation that a "doctrine" prevalent in the 1980s among campaign mangers and consultants "that challengers close better than incumbents" in the final days of a campaign. As Michael Barone put it earlier this year, the idea is that "an incumbent is not going to get a higher percentage in an election than he got in the polls." Carville's question: Is that doctrine no longer valid?

A more complete look at the incumbent rule and remains on my to-do list for the next month or so, but in writing up this post, I took a quick look at how incumbents fared in the (still largely) unofficial election results as compared to our final last-five-poll average in the most competitive races for Senate and Governor.


As the table below shows, on average in these particular races looking only at the last five polls in each race, the rule did not apply particularly well. On average, both incumbents and their prime challengers picked up 2.4 percentage points -- an almost exact 50-50 split in the most competitive statewide races.** In some cases, such as the Pennsylvania Senate race, virtually all of the undecided went to challenger Bob Casey, but the pattern was otherwise typically muddled.

The tougher question is the one inherent in Carville's question to the pollsters: Why the recent change in what had been pollster doctrine? Here is a summary of what the four pollsters had to say:

  • Republican Neil Newhouse noted the example of his client, incumbent Republican Jim Gerlach (Pennsylvania-6), who was in a 44% to 44% tie on their final internal poll conducted a week before the election. In the "old days," Newhouse said, we would have assumed an easy Murphy victory. However, Gerlach ultimately prevailed (51% to 49%) after a closing with a final television ad featuring a personal appeal by Gerlach that Newhouse credited for the victory. As for the incumbent rule, Newhouse said, "we are seeing a bit of a change, but not much consistency." While he still tends to give challengers the "benefit of the doubt" when incumbents are under 50%, Newhouse believes it is no longer "carte blanche automatic" that the undecided vote on the final poll will all go to the challenger.
  • Republican Dave Sackett agreed and credited the much shorter "fifteen minute" news cycle for the ability of incumbents to turn the tables on challengers late in the campaign. He noted that his client Deborah Pryce (Ohio-15) as trailing "all the way through" on internal tracking polls, which would presumably include one in the final week (he noted via email that the margin had closed to within sampling error on the final poll). However, according to Sackett, the Pryce campaign outspent Democratic challenger Mary Jo Kilroy by a two-to-one margin over the final weekend, and credits her narrow victory to that final burst of communication.
  • Democrat Harrison Hickman pointed out that in the 1980s, the conventional wisdom was to avoid mention of your opponent, a habit that helped explain why challengers won much of the late undecided vote. Now, he said, the general pattern is for incumbents to vigorously attack challengers throughout the campaign. "Incumbents put so much more pressure on challengers then they used to." (See this pre-election column by Dick Meyer of CBS News that includes data Hickman gathered showing the impact of negative advertising on candidate favorable ratings since 1986).
  • Finally, Democrat Stan Greenberg agreed with his colleagues that the traditional pattern, seen as recently as 1996 when Bill Clinton got 49% in their final poll and 49% of the vote on Election Day, has changed. He speculated about another possible explanation, that elections for the House and Senate have become increasingly "nationalized" since 1994. Pointing to the increasing "partisan consistency" in pre-election polls (each party's candidates winning 90% or more of voters of that party's voters), Greenberg argued that elections now "get crystallized in a specific way nationally" and that local elections get "swept up" in a national tide that may negate the traditional last minute shift of undecided voters to challengers.

Of course, the solution to this very interesting puzzle is inherently speculative. It is also worthy of more analysis than I gave it above. Hopefully, we'll have more to come over the next month or so.

Daily Roundup 12.13.06

  • A new Los Angeles Times/Bloomberg telephone poll says 67% of Americans think the Democrats will be able to accomplish at least some of the goals in the next two years, while 42% believe that of Bush.

  • The latest USA Today/Gallup Poll states "a 54% majority says Bush will be judged as a below-average or poor president.... Just 19% expect him to be seen as outstanding or above average." (USA Today story, Gallup analysis)

Bush Approval: CBS/NYT at 31%, Trend at 33.2%

Topics: George Bush


With the addition of the CBS/NYT poll out today (approval 31%, disapproval 63%) the President's approval trend has fallen below the previous low of 33.7, to a new low of 33.2%.

While the CBS/NYT poll is in line with recent polls by Harris, Zogby and Newsweek, it is a good deal below other recent polls by Fox and CNN. The trend line is sensitive to the most recent polls, especially when trending down like this, so some caution is in order.

The worst news in the CBS/NYT poll is that approval among Republicans has fallen considerably, to 65%. Strong Republican support has been the crucial element keeping approval in the upper 30% range. If losses there continue, there could be a substantial drop in approval. Iraq approval appears to have especially suffered in this way. Approval of Bush's handling of Iraq has fallen from 29% to 21% since 11/14, and among Republicans the President's Iraq approval rating has fallen from 70% to 47%, an astonishing drop. This is in the immediate aftermath of the Iraq Study Group report, and strong negative reaction among the most conservative elements of the party. Perhaps the ultimate White House response will appease these critics, though at what price in support among other Republicans?

UPDATE: Two new polls by Gallup and ABC/WP raise the trend estimate to 34.4%. The "new low" due to CBS last night didn't last long. More commentary later.

Gallup's Newport & Jones on The Gallup Panel

Topics: Pollsters

(A post by Mark Blumenthal last week discussed results from a Gallup "panel" survey. Today, Frank Newport and Jeff Jones of the Gallup Organization respond with a Guest Pollster Corner piece that provides more information on the Gallup panel.)

The Gallup Poll Panel is a very large probability-based random sample of 18+ adults, used as the basis for a number of commercial and public opinion research projects. All members are recruited using the same random digit dial (RDD) methodology that is the basis for all "normal" national Gallup polls. The panel currently consists of over 37,000 households and over 54,000 individual members. Recruitment is conducted on an ongoing basis.

While the Gallup Poll Panel was developed primarily for commercial purposes, it is used as the basis for conducting occasional Gallup Poll studies. These Gallup Poll studies in essence are based on a random sample of a larger random sample. All interviewing is conducted by phone using standard Gallup Poll techniques. National samples drawn from the Gallup Poll panel are selected randomly, are demographically representative of the U.S. adult population, and are weighted on the back end to adjust to known population parameters, as we would any RDD poll. The Gallup Poll Panel sample is thus a projectable random sample of the adult population of the United States.

Before initiating the use of the Panel for Gallup Poll studies, Gallup conducted several carefully controlled experiments, fielding the same exact surveys at the same time using the panel sample and the standard RDD sample. Analysis showed very few substantive differences in the frequency distribution of results between RDD and panel. To the extent that there were differences, they were almost always within the margin of error (the one exception concerns asking knowledge questions; panel members seem to consistently score higher on these than RDD respondents). (For more information).

Gallup monitors very carefully the use of the Panel as the basis for nationally projectable samples, and will continue to do so in the future.

Frank Newport and Jeff Jones
The Gallup Poll

Daily Roundup 12.12.06

  • A new ABC News/Washington Post poll finds 36% percent of Americans approve Bush's job performance and 28% approve his handling of Iraq. (ABC story, Post story, results)

  • A CBS poll released last night (story, full results) shows 62% of Americans saying sending troops to Iraq was "a mistake." 60% of Americans also never believe Iraq will become a stable Democracy.

  • The latest USA Today/Gallup Poll (story, full results) reveals 76% of Americans saying Iraq is in a civil war. In addition, more Americans trust the Iraq Study Group than Pres. Bush in recommending the "right thing for the United States to do in Iraq." (Gallup analysis on job approval, Iraq)

  • Last night's SurveyUSA automated poll for TX-23 shows Rep. Henry Bonilla (R) leading Ciro Rodriguez (D) by four points (51% to 47%).

  • A new Rasmussen Reports automated survey shows Sen. Hillary Clinton (D) leading Gov. Mitt Romney (R) 48-40 in a presidential contest. The survey also found 78% of Americans are willing to vote for a woman President.

Party ID: Random Error Happens

Topics: Party Weighing

There is something about party identification -- the question that asks if voters think of themselves as Democrats, Republicans or Independents -- that always seems to inspire comment and conjecture. Last week, almost as an afterthought, I ended a post noting that the recent Gallup panel survey had a few more Republican identifiers among all adults (36%) than what the standard Gallup telephone surveys had been averaging since July (30%). One valued Pollster.com reader emailed to ask if this might be the sign of a bit of "buyer's remorse," a sign that some voters who felt unsure earlier in the year have come to regret that the Democrats have taken control of the House and Senate. In this case, it looks to be most the usual random variation to which polls are prone.

One obvious argument against any sort of buyer's remorse is that the Bush job rating continues to fall (see Professor Franklin's update from earlier today). Another is that, at least as of a month ago, the favorable ratings of both Nancy Pelosi and Harry Reid were increasing on the Gallup Poll.

However, to look more directly at the reader's question, the simplest approach is to see if any of the other national polls have shown any similar shift in party identification. Both Newsweek and AP/IPSOS, two of the organizations that conducted new surveys in December, routinely provide results for party identification. They show no increase in Republican identification. If anything, the opposite may be true. Among registered voters, both show small (and probably statistically insignificant) decreases in Republican identification since the election.


The table also includes the party identification results among adults released by Gallup on their standard telephone surveys in October and November (it does not include the recent panel survey results -- note also that the Newsweek and AP/IPSOS results are among registered voters).

The most likely explanation for the slightly higher number of Republicans on the panel survey discussed last week is an extreme example of the usual culprit: random variation.  A swing up to 36% Republican among adults would be extreme, but it happens.

PS: I frequently link to reports on GallupPoll.com.  Until recently, Gallup's analytical reports were available for free for the first 24 hours and then to subscribers only thereafter. It looks as if Gallup has recently shifted to more of an advertiser-driven model. Now, if you click on an archived Gallup reports (like the one referenced above), by watching a short advertisement you can get a time-limited "free premium pass" that provides access to everything on GallupPoll.com but the "Gallup Brain" data archive.

Daily Roundup 12.11.06

  • Newsweek's latest national survey tells us "more than two out of three Americans believe the United States is losing ground in Iraq (68 percent), versus 21 percent who say it is making progres -- the most pessimistic assessment the NEWSWEEK poll has ever recorded."

  • "By a 2-to-1 margin (57% to 30%), American voters believe that Social Security needs to be fixed rather than left alone" according to a newly released Rasmussen Reports automated survey. The same survey also asks who is winning the War on Terror.

Bush Approval Trend Slips to 34%

Topics: George Bush


President Bush's approval rating has fallen to 34% in my trend estimate. This is just barely above his all time low trend estimate on May 13, 2006. New polls included in this estimate include results from Fox (38%, 12/6), CNN (37%, 12/7), Marist College (37%, 12/3), AP (33%, 12/6), Newsweek (32%, 12/7) and Zogby (30%, 12/8). While the range from 30% to 38% may seem substantial, it is in fact comfortably within the +/-4.5% range around my trend estimate that we've seen over the Bush administration.

For a comparison of all pollsters, including these new polls, see our new feature at Pollster.com here. That feature will be updated each time I update the approval series and allows you to easily see the trend for each pollster in comparison to my estimated approval trend. (Scroll down to see the thumbnails for each pollster, and click on a thumbnail to see the full sized plot.)

The post-election decline is not surprising, given the election results. However, the White House has to worry that all the gains of the late spring and summer have now been lost. So far it seems unclear to me whether the President will be able to use the Iraq Study Commission report as an opportunity to recover some of his lost approval. And with the holidays approaching, now is perhaps not the most opportune moment to launch a new political offensive. But come January and the start of the new Congress, the President will certainly need some improvement with the public to help him confront the Democratic Congress. This assumes, of course, that the White House will have an agenda to press in the new year rather than retreat to a purely defensive stance vis a vis Congress.

On a technical note: This post marks the debut of new graphical formats for the approval series. A number of readers have asked for background grids for the charts, and I've finally found a color combination that I think works. The vertical grids are marked in quarters, with monthly tic marks on the x-axis. New years are marked with a slightly darker vertical grid line to help keep years straight. And the trend estimate is now printed right at the top of the table for convenient reference. (And yes, ego has overcome modesty and the graphs now say where they are from.)

The most important change from my point of view is that the "current" approval graph has now been scaled to run throughout the second term. I've written about why it is important to maintain a consistent scale to these graphs here. In practice I've used 2005 and 2006 for the second term up until now so the graphs wouldn't look too empty. But with the second term now half over, I think it is a good moment to shift to a full four year view. From now on, the current approval will be plotted in this perspective, with new points and trend added to the right of the figure as new data come in. The important thing is that this means the perspective of the trend will remain exactly the same for the next two years, so the rate of rise or fall can be compared just by looking at the figure. Virtually all the plots by pollsters or news organizations change the width of the horizontal axis as new polls come in. That distorts the trends over time because more data "scrunches" up the points and the trend over time. My graph will look the same for the next two years, so it is always comparable.

I've also scaled the vertical axis to run from 20% to 60% approval. We've only seen one poll below 30% in the last 6 years, but that came when the approval trend was just below 34%, as it is now. If the trend declines any more, it is likely we will see some readings in the 20s. This is NOT a forecast on my part, but in order to avoid any further rescaling of the graph, I've expanded the range.

The graph of approval over the entire Bush administration has also been spruced up with a new grid. This chart has always run the full width of a two term presidency and with approval from 20%-100%, so the scaling here has not changed. I've also added the current trend estimate to the top of the graph. The grid here is the same as for the second term: quarterly vertical grids, with darker lines for January 1, and monthly tic marks.


These graphs are saved and updated at the links listed in the top of the right hand column of the blog for easy reference. They are drawn at 768 x 768 resolution, and are best viewed at that. Some users may need to double click on the image to see it at that resolution.

There will be some more new things coming once the semester is finished and grading is done. Stay tuned.

Cross-posted at Political Arithmetik.