October 25, 2007
By Mark Blumenthal
Now that our trend estimates are appearing in the Slate Election Scorecard, with the daily twitches
of our trend lines getting extra attention, I will try to provide some running
commentary here on how the addition of new polls changes the trends from day to
day.
One intriguing example comes from the new
poll in New Hampshire
out today from St. Anselm College.
This New Hampshire
poll is the second in a row showing Hillary Clinton receiving 43% of the vote, above
our trend line but consistent with its increasing upward slope. The new poll,
and the addition of nearly two weeks to the trend since the last New Hampshire poll,
helps push her score on our estimate up to 40.5%, a 1.2 point increase since
the last update.
The continuing Clinton
upward trend aside, however, the results also indicate some potentially good news
for Obama. On the last four New
Hampshire polls - all conducted since late September -
Obama's share of the vote has been above our trend line: 22%, 21%, 23% and 22%.
The five polls before that - all conducted in the prior month - were slightly
lower: 17%, 16%, 18%, 17% and 19%. Remember, Professor Franklin set the
sensitivity of these trend lines to minimize the impact of just one poll.
So the current Obama trend line (above) reflects the slight
decline in Obama's New Hampshire
support from earlier in the year. Still, the addition of the most recent result
flattens that downward slope just slightly and, as such, increases our current
estimate of his support slightly (from 19.4% to 19.5%) since the last update.
What makes this intriguing is that Obama started buying New Hampshire television
campaign ads in earnest roughly four
weeks ago. So this is a trend worth watching very carefully.
By Mark Blumenthal on October 25, 2007 9:59 PM
| Permalink
| Comments (5)
| TrackBacks (0)
October 23, 2007
By Mark Blumenthal
As many of you know, our partnership with the online journal
Slate resumed last week, as they
kicked off their new Election
Scorecard feature, which is once again powered and provided by
Pollster.com. For now, the Slate
feature displays our most recent trend estimate for each candidate in the early
primary states as well as handy Flash graphic. The display of trend updates has
both the writers at Slate and yours truly watching the daily twitches in the numbers
more carefully. Today's movement in the national numbers for the Democratic
primary reveals an idiosyncrasy in the way our trend estimates behave that I
want to explain.
Some background: The trend lines we plot in our charts are different
from the rolling averages we plotted for the races for Senate, Governor and
U.S. House in 2006, and from the "polling averages" you see on other web sites.
A polling average makes use of data from just the most recent polls included in
the average (be it 5 polls or some other number). Our approach - developed by
our partner, Professor Charles Franklin - has been to plot line based on a "local
regression" that takes into account all available data for the current
estimate, not just the most recent 4 or 5 or 8 polls.
The key difference between trend estimates and rolling averages
is that an average produces a new estimate for each combination of polls
included in the average at any point in time. The regression line produces a
trend line - a line, rather than a point
- with a particular slope that
is either moving up, down or staying level at any point in time.
Another key issue is the level of sensitivity that Professor
Franklin built into the regression model that produces the trend line. I'll let
him explain:
I've chosen an estimation method
and designed the approach we take so that the trend estimator should be
resistant to bias due to a single organization or a single poll. While it can
be fooled under the right circumstances, those should be both rare and short
lived, rather than common and long term.
Franklin
explains the mechanics of the estimator in more detail in posts here,
here
and here.
This brings me to the most recent odd twitch in the national
averages for the Democratic presidential trial heat. Late last week, our last
update of the national Democratic numbers had Hillary Clinton at 43.2%. Yesterday,
we updated the charts with a new
national poll conducted and released by the Republican firm, Public Opinion
Strategies, that gave Clinton
40% of the Democratic vote. Yet despite showing a result for Clinton that was below her latest trend estimate, the addition
of the new poll moved her estimate up higher
by nearly a full percentage point (from 43.2% to 44.1%).
Why? It is all about the what the regression estimate tells
us about the trend evident in the last 10 or so polls. The chart makes clear
that our most recent estimate of the trend is sharply up for Clinton. As per Franklin's design, the addition of just one
new poll did not significantly lessen that upward slope. However, since the end
date of the new poll comes a full week since the last poll, the line has moved forward
in the upward direction for another week, thus producing a nearly one point increase.
The point is, we're not just adding one new poll and
dropping one old poll from a last-five or last-six poll average. We are
gradually updating a trend line based on all the data available.
By Mark Blumenthal on October 23, 2007 8:36 PM
| Permalink
| Comments (1)
| TrackBacks (0)
October 30, 2006
By Mark Blumenthal
Charlie Cook writes tonight: "With the election just eight days away, there are no signs that this wave is abating." Some supporting evidence: The overall average Democratic margin in the Slate 13 -- the 13 most competitive Senate races we have been tracking on the Slate Election Scorecard -- has increased for the sixth straight week (from +3.7 to +4.1 percentage points over the last week).
Again, the value in looking at this overall "mash-up" is that it combines a very large number of surveys, including at least 35 new statwide surveys in the 13 states released in the last week. In any one state, the averge might be a little lower or a little higher due to the "house effects" or other variation in recent surveys. By rolling up the results of many surveys, we should minmize the noise. And that approach shows now end to slow Democratic trend in Senate races since mid-September.
PS: The Slate Election Scorecard update for tonight focuses on the Senate race in New Jersey, where two new polls moved that State back to "lean" Democrat status.
By Mark Blumenthal on October 30, 2006 10:17 PM
| Permalink
| Comments (2)
| TrackBacks (0)
October 26, 2006
By Mark Blumenthal
After a long wait and a lot of hard work by the entire Pollster team, we are proud to unveil our new scorecard for races for the U.S. House of Representatives. We introduce the scorecard showing 219 seats in the Democratic column and 193 in the Republican column, with 23 seats showing neither candidate with a statistically meaningful lead. These numbers and how we got them deserve a bit more explanation, so here goes.
There are many different House scorecards available online that use a variety of different methods to try to estimate the outcome of control of the House. Some rely on subjective handicapping of individual races along with polling data, and some projective models use aggregate national data to forecast House seat counts. Our aim here is to simply summarize and aggregate the available District level survey data and let you reach your own conclusions.
Creating some sort of scorecard for the House is a lot more challenging than for races for the Senate and Governor because so few polls are available for House contests. As of today, we have collected public polls for 74 House seats, but that means that we have not been able to locate any public poll data for the remaining 361 contests (if you are aware of any public poll missing from our database, please do not hesitate to email us).
We like to average recent results here on Pollster, and House scorecard is no exception. On Senate and Governor scorecards (as well as the Election Scorecard we help produce for Slate) average the five most recent polls, and most of the closely watched Senate contests now have 5 or more new polls in the last 2 weeks. It is a very different story on the House side. Scroll through the House summary table and you will notice that roughly half of the districts three or fewer polls available for the entire campaign.
So we start by taking all of the available polling data and averaging across the most recent polls. We average the most recent polls available, but never more than the five. The "# in Avg" column in the summary table indicates the number of polls used to calculate the average.
To try to get an overall estimate of who leads in the race to control the House, we started by dividing the 435 seats into three categories.
- 85 seats considered competitive by the Cook Political Report as of October 20, 2006
- 184 non-competitive seats held by Democrats
- 166 non-competitive seats held by Republicans
For the purposes of the overall scoreboard, we have classified the non-competitive seats as "strong" Democrat or Republican based on the party of the incumbent. As of today, we have polls available for just seven of the non-competitive seats, and the results all show the incumbents with large leads. If we locate polling data indicating a real contest shaping up in any of the non-competitive districts, we will code it accordingly for the scoreboard.
We then focus more closely on the 85 competitive seats. Our classification system works the same as for our Senate and Governor scorecards. We rate races as "leaning" to a candidate if their lead is statistically meaningful (at least one standard error). If that lead is strongly significant (at least two standard errors), we rate the race as "strongly" Democrat or Republican.
For the U.S. House, we added a category named "no poll" for the competitive races for which we can find no available public polling data. Because most of these "no-poll" districts are among those considered only marginally competitive, we have classified them by the party of the incumbent member for purposes of the top-line scoreboard.
The color of the district labels on the map update automatically to reflect any changes in status. Dark blue and dark red represent races that we rate "strongly" Democratic or Republican respectively. Lighter shades indicate a lean status. States colored yellow are those we classify as "tossups"- races in which neither candidate shows a significant lead over the last five polls. No-poll competitive races are grey.
We are still at work preparing charts for all of the 74 House races. For now, links on the map or in the summary table will take you to a table in the page below with individual poll data and links to source pages.
Finally, note that you can sort the summary table by any of its data, by clicking on the column heading of the category of interest.
I'll have more posts in the next day or so explaining, as best I can, some of the limitations of this data and what it all means. Stay tuned.
By Mark Blumenthal on October 26, 2006 5:18 PM
| Permalink
| Comments (5)
| TrackBacks (0)
By Mark Blumenthal
Last night's update of our Slate Election Scorecard feature notes a big change (also reflected on our new Pollster map and scorecard): Republican George Allen's margin over Democratic challenger Jim Webb has narrowed to the point that we now rate Virginia as a toss-up. Thus, for the first time in the campaign, Democrats have an overall lead: 49 seats are currently held by Democrats or at least lean that way, 48 lean or are held by Republicans. To win control of the Senate, Democrats will need to win two of the three toss-up states: Missouri, Tennessee and Virginia.
A few additional notes about the new surveys in Virginia and Missouri:
Virginia shifted to toss-up as a result of a new LA Times-Bloomberg poll that had Webb leading by a statistically insignificant three points (47% to 44%). Since Slate posted our update, SurveyUSA released another poll showing Allen leading by three (49% to 46%). Even though the new poll has Allen slightly ahead, it replaces an outlier Zogby poll from early October that had Allen ahead by 11 (48% to 37%) and will serve to narrow Allen's lead even further on our last-five-poll average. Tomorrow's update will show Allen's margin on our summary table shrinking to only 1.8% (47.6% to 45.8%).
The Slate update also discusses two new polls in Missouri from LA Times/Bloomberg and SurveyUSA that suggest new momentum for Republican Senator Jim Talent in Missouri. Both surveys show Talent ahead by a statistically insignificant three points, but previous surveys had shown Democrat Claire McCaskill ahead by similar margins. The net result on our last-five poll average remains very close race that still easily qualifies as a toss-up.
While the small shift to Talent may well be real, poll consumers ought to keep in mind two words when it comes to surveys conducted in Missouri the week: World Series.
The World Series gives pollsters fits, especially when we have to poll states with a hometown team -- like the St. Louis Cardinals -- playing for all the marbles. The games draw huge audiences in the home markets of the contenders, and no one wants to stop watching the game to complete a telephone survey. As a result, pollsters typically experience lower response rates, particularly among younger men. Complicating all of this even further in Missouri is the high-profile exchange of television advertisements on stem cell research that aired during the World Series gaemes (an ad on behalf of Claire McCaskill featuring Michael J. Fox and a response by the Cardinals' Jeff Suppan and actors Patricia Heaton and Jim Caviezel).
What effect any of this might be having on the Missouri results (or whether it has had any effect at all) is a matter of pure speculation. However, I can tell you that some of the screwiest internal polls I have seen in my career were fielded during past World Series. So perhaps a few more grains of salt than usual are in order this week.
By Mark Blumenthal on October 26, 2006 12:34 AM
| Permalink
| Comments (0)
| TrackBacks (0)
October 24, 2006
By Mark Blumenthal
Last night's Slate Scorecard update reviewed the relatively recent polls showing new momentum for Iowa gubernatorial candidate Democrat Chet Culver, but I want to take a moment and review two things readers should know about the averages in the Slate Senate Scorecard.
First, readers should know that the averages in our new Pollster.com scorecards and summary tables (Senate and Governor) replicate the averages that we have been providing to Slate but extend them to every race for Senate and Governor. They use the last five polls in a race, but -- unlike the averages that currently appear on our chart pages -- they exclude surveys based on internet panels.
Second, regular readers of the Slate feature should be familiar with the big blue "momentum shift" meter that sits atop the Senate scorecard. It has pointed in the Democratic direction since September 15, meaning that recent trends across all 13 of the competitive Senate have been shifting in a Democratic direction. These averages summarize the results of over 190 polls conducted in those states since the summer. Some readers have wondered about why the meter has been seemingly frozen in place for over a month. Can a momentum "shift" really continue for a full month?
The numbers say it can. As the table below shows, the average Democratic lead across the 13 states we track for Slate has nearly doubled, rising from 1.9% to 3.7% since early September. The net gain is greater if we remove the Connecticut (for which we calculate the average deficit of Democrat Ned Lamont to Sen. Joe Lieberman).
And those gains have been spread out over a large number of races, with net gains in 10 of 13 races. The largest increases on the margin have been in New Jersey (+7.8), Ohio (+5.4) and Tennessee (+5.4).
The current scoreboard indicates a 49 to 49 tie in the Senate if all trends continue (assuming that Joe Lieberman caucuses with the Democrats), with Missouri and Tennessee still classified as "toss-up" races. When we started tracking for Slate on September 1, Republicans held a 50 to 46 lead. These numbers tell the story of what happened since.
By Mark Blumenthal on October 24, 2006 7:15 AM
| Permalink
| Comments (0)
| TrackBacks (0)
October 18, 2006
By Mark Blumenthal
Yesterday, our Slate update shifted to a district-by-district focus on races for the U.S. House. Our initial tally shows Democrats right on the edge of statistically meaningful leads in enough districts to take control of the House, and Democratic challengers running within the margin of error in many more. While we are working to creating a more comprehensive scorecard, I want to say a bit more about how our initial count given the limitations of the data.
For now we are focusing on non-partisan surveys conducted since late August (which allows inclusion of the first wave of Majority Watch surveys conducted August 27-29). We have excluded the surveys sponsored by campaigns or the party committees (including those conducted by my firm).
Looking at the survey averages in districts with two or more polls available, we see Democrats leading beyond the margin of error in ten districts currently held by Republicans (the number of surveys analyzed is included in parentheses):
In addition, we see statistically significant Democratic leads in four more districts held by Republicans surveyed only once by non-partisans since the summer (all four were polled by the Majority Watch project):
[Note: We inadvertently omitted Ohio-18 and North Carolina-08 from last night's initial Slate update].
Perhaps more troubling for Republicans is that we see no Republican leading in any district currently held by a Democrat. Moreover, of the 23 Republican held seats currently rated as "toss-ups" by the Cook Political Report, Democrats lead by significant margins in 9, Republicans leading in none just one (Minnesota-02).** The remaining 13 Republican "toss-up" seats look too close to call based on available data. And that says nothing of the 31 Republican seats that Cook rates at "lean" or "likely" Republican, where public polling is scarcer still.
Of course, readers should remember the limitations of these data. Any one poll can produce an odd or contradictory result and many of the polls conducted in September may already be stale. Consider, for example, Iowa's 1st District (an open seat currently held by Republican Jim Nussle). A Majority Watch poll in late August showed the Democratic candidate Bruce Braley leading Republican Mike Whalen by thirteen points (54% to 41%). Two weeks later, a DeMoines Register/Selzer poll conducted two weeks later had Braley ahead by seven (44% to 37%). Then a Zogby poll at the end of September showed Braley trailing by thirteen (34% to 47%). And finally, a poll conducted by my firm last week for the Democratic Congressional Campaign Committee had Braley ahead by 10 (48% to 37%). So over the course of six weeks, polls in one district show everything from a 13 point Democratic lead to a 13 point deficit with undecided percentages ranging from 5% to 19%. These differences are far beyond sampling error and almost certainly the result of differences in pollster methodology. And that's just one district.
Again, we are working on creating a House summary scorecard for Pollster.com, but it will be limited by both the relatively small number of public polls and their inevitable conflicts.
**Correction: I wrongly included MN-02 (rather than MN-06) among the list of seats rated a "toss-up" by the Cook Political Report. Four polls in MN-06 since mid-September give Democrat-Farm-Labor candidate Patty Wetterling an average lead of one percentage point (45.7% to 44.7%) over Republican Michelle Bachman. Apologies for the error.
By Mark Blumenthal on October 18, 2006 7:35 AM
| Permalink
| Comments (11)
| TrackBacks (0)
October 12, 2006
By Mark Blumenthal
Our Slate Election Scorecard update tonight focuses on two new polls in New Jersey that confirm recent gains by Bob Menendez and move the race to lean Democrat status. The overall scorecard tally now indicates 49 seats leaning or held by Democrats, 49 seats held or leaning Republican. Is this change indicative of a larger Democratic surge?
Two new polls out this evening from Survey USA in Ohio and Missouri both show the Democratic candidates in each state leading by much wider margins than on other recent polls. These results and the sometimes improbably wide Democratic margins on the generic House ballot in some recent national surveys leave some wondering whether, as reader Gary Kilbride put it in a comment a few hours ago, "the current poll numbers skew misleadingly toward Democrats due to the Foley scandal." He wonders if the same might be happening to the Majority Watch congressional district results released today.
I will have more to say about all of this tomorrow, but for tonight one quick note about those new Majority Watch congressional surveys. Although they released results from 32 districts today, only nine involved follow-up surveys in districts polled previously using comparable ballot tests. The table below shows the August and October results for those nine districts.
All of these Districts are currently represented by Republicans and all were rated as toss-ups by the Cook Political Report when the polls went into the field (they moved CO-07 to lean Democratic status just yesterday). While Tom Riehle's analysis made much of the apparent Republican improvement in Washington-08, Virginia-02 and Indiana-02, the overall pattern looks more random. Those Republican advances were largely offset by Democratic gains in North Carolina-11 and New Mexico-01. Overall, the average Democratic margin declined by just a single percentage point.
The bigger story may be that the average Republican percentage across these nine districts has not budged from 46% since August or that none of the Republicans in the nine districts holds a statistically significant lead. More on the meaning of these House polls tomorrow.
By Mark Blumenthal on October 12, 2006 11:57 PM
| Permalink
| Comments (1)
| TrackBacks (0)
October 9, 2006
By Mark Blumenthal
Our Slate Election Scorecard update for this evening focuses on two races for Governor: Michigan, where two new polls showing Governor Jennifer Granholm leading restore that state to "lean" Democrat status, and Oregon where Democratic Governor Ted Kulongoski's recently narrowing lead has shifted the race into the toss-up category. Read it all.
By Mark Blumenthal on October 9, 2006 11:02 PM
| Permalink
| Comments (3)
| TrackBacks (0)
October 7, 2006
By Mark Blumenthal
Our update to the Slate Election Scorecard yesterday focused on six new statewide polls from the USAToday/Gallup partnership. Their new poll in Rhode Island helped push our assessment of that state into the "lean" Democrat category. As always, we encourage you to read it all.
Separately, Gallup's Frank Newport posted an analysis on the Gallup site (free to non-subscribers through Wednesday) showing the results of an open-ended question asked on all six surveys of those supporting one of the candidates for Senate: "Why would you say you are voting for this candidate?" As is standard survey practice, the interviewers recorded responses verbatim, and Gallup later coded those responses into categories. Newport's analysis shows the results of those questions for each state, broken out by voters for each candidate.
But then it gets more interesting. I'll let Newport explain:
There are wide varieties of ways in which these data can be approached, and as a result a wide variety of conclusions that can be reached. Gallup Poll editors developed several important points about the reasons behind voters' choices for the Senate.
But in this situation, we thought some readers might have different approaches or insights, and that as a result it would be of interest to allow readers to suggest their own thoughts on these data, and to share those with us here at The Gallup Poll by responding to Talk to the Editors.
We will read all suggested responses, and post the most telling and insightful here on galluppoll.com next Wednesday.
And speaking of Frank Newport, his contribution to Guest Pollster's Corner last week provoked some interesting questions. Newport responded in the comments section yesterday. It's worth a click.
By Mark Blumenthal on October 7, 2006 7:01 AM
| Permalink
| Comments (2)
| TrackBacks (0)
October 5, 2006
By Mark Blumenthal
Our update Tonight's Slate Election Scorecard update focuses on the slew of new statewide surveys released earlier today by the Reuters/Zogby polling partnership. MP readers may especially appreciate the disucssion of the wide variation in polls in New Jersey. Read it all.
By Mark Blumenthal on October 5, 2006 11:50 PM
| Permalink
| Comments (0)
| TrackBacks (0)
October 4, 2006
By Mark Blumenthal
Our Slate Senate Scorecard update for tonight focuses on a new Rasmussen poll in Connecticut that shows Joe Lieberman leading Democratic nominee Ned Lamont by ten points (50% to 40%).
Tracking the Connecticut Senate race especially challenging because the most active pollsters in the state have shown consistent differences in their results -- at least until today. See the chart below (courtesy Charles Franklin), which shows Lieberman's margin over Lamont (Lieberman's percentage minus Lamont's percentage):
Both the Rasmussen automated surveys and the conventional, live interviewer phone polls conducted by Quinnipiac University showed Lieberman's margins narrowing since July but holding fairly steady over the last month. However, until the survey released today, the Rasmussen surveys have consistently shown a closer margin than the Quinnipiac Polls. This pattern is similar to the one we described yesterday in Tennessee, where Democrat Harold Ford is running stronger on the Rasmussen surveys than on conventional telephone interview polls conducted by Mason Dixon.
In this case it is harder to use the survey mode (live interviewer vs. automation) to explain the differences in because the house effects are inconsistent by mode. Another live interview pollster (American Research Group) has also shown a consistently closer race, while automated pollster SurveyUSA reported Lieberman ahead by 13 points in early September.
Today's result, however, brings the Rasmussen and Quinnipiac polls into agreement, at least for the moment. The last Quinnipiac poll released last week also showed Lieberman leading by 10 points. So is the latest turn in the Rasmussen trend line the sign of new Lieberman momentum, a convergence in the polls results or just an outlier result? Only time, and more surveys, will tell for sure.
By Mark Blumenthal on October 4, 2006 11:57 PM
| Permalink
| Comments (2)
| TrackBacks (0)
October 3, 2006
By Mark Blumenthal
Our Slate Election Scorecard update for tonight looks at a slew of new polls that largely confirm the status of many races but make one key change: The last five poll average now puts Virginia into toss-up status. Read it all.
By Mark Blumenthal on October 3, 2006 11:22 PM
| Permalink
| Comments (0)
| TrackBacks (0)
September 28, 2006
By Mark Blumenthal
If one story is more important than all others this year--to those of us who obsess over political polls--it is the proliferation of surveys using non-traditional methodologies, such as surveys conducted over the Internet and automated polls that use a recorded voice rather than a live interviewer. Today's release of the latest round of Zogby Internet polls will no doubt raise these questions yet again. Yet for all the questions being asked about their reliability, discussions using hard evidence are rare to non-existent. Over the next month, we are hoping to change that here on Pollster.com.
Just yesterday in his "Out There" column (subscription only), Roll Call's Louis Jacobson wrote a lengthy examination of the rapid rise of these new polling techniques and their impact on political campaigns. Without "taking sides" in the "heated debate" over their merits, Jacobson provides an impressive array of examples to document this thesis:
[I]t's hard to ignore the developing consensus among political professionals, especially outside the Beltway, that nontraditional polls have gone mainstream this year like never before. In recent months, newspapers and local broadcast outlets have been running poll results by these firms like crazy, typically without defining what makes their methodology different - something that sticks in the craw of traditionalists. And in some cases, these new-generation polls have begun to influence how campaigns are waged.
He's not kidding. Of the 1,031 poll results logged into the Pollster.com database so far in the 2006 cycle from statewide races for Senate and Governor, more than half (55%) have been done by automated pollsters Rasmussen Reports, SurveyUSA or over the Internet by Zogby International. And that does not count the surveys conducted once a month by SurveyUSA in all 50 states (450 so far this year alone). Nor does it count the automated surveys recently conducted in 30 congressional districts by Constituent Dynamics and RT Strategies.
Jacobson is also right to highlight the way these new polls "have made an especially big splash in smaller-population states and media markets, where traditional polls - which are more expensive - are considered uneconomical." He provides specific examples from states like Alaska, Kanasas and Nevada. Here is another: Our latest update of the Slate Election Scorecard (which includes the automated polls but not those conducted over the Internet) focuses on the Washington Senate race, where the last 5 polls released as of yesterday's deadline had all been conducted by Rasmussen and SurveyUSA.
Yet the striking theme in coverage of this emerging trend is the way both technologies are lumped together and dismissed as unreliable and untrustworthy by establishment insiders in both politics and survey research.
Jacobson's piece quotes a "political journalist in Sacramento, Calif," who calls these new surveys "wholly unreliable" (though he does include quotes from a handful of campaign strategists who find the new polls "helpful, within limits").
Consider also the Capital Comment feature in this month's Washingtonian, which summarizes the wisdom of "some of the city's best political minds" (unnamed) on the reliability of these new polls. Singled out for scorn were the Zogby Internet polls - "no hard evidence that the method is valid enough to be interesting" - and the automated pollsters, particularly Rasmussen:
[Rasmussen's] demographic weighting procedure is curious, and we're still not sure how he prevents the young, the confused, or the elderly from taking a survey randomly designated for someone else. Most distressing to virtually every honest person in politics: His polls are covered by the media and touted by campaigns that know better
The Washingtonian feature was kinder to the other major automated pollster:
SurveyUSA's poll seems to be on the leading edge of autodial innovation. Its numbers generally comport with other surveys and, most important, with actual votes.
[The Washingtonian piece also had praise for the work of traditional pollsters Mason-Dixon and Selzer and Co, and complaints about the Quinnipiac College polls]
Or consider the New York Times' new "Polling Standards," noted earlier this month in a Public Editor column by Jack Rosenthal (and discussed by MP here), and now available online. The Times says both methodologies fall short of their standards. While I share their caution regarding opt-in Internet panels, their treatment of Interactive Voice Response -- the more formal name for automated telephone polls -- is amazingly brusque:
Interactive voice response (IVR) polls (also known as "robo-polls") employ an automated, recorded voice to call respondents who are asked to answer questions by punching telephone keys. Anyone who can answer the phone and hit the buttons can be counted in the survey - regardless of age. Results of this type of poll are not reliable.
Skepticism about IVR polling based on theoretical concerns is certainly widespread in the survey research establishment, but one can look long and hard for hard evidence of the lack of reliability of IVR, or even Internet polling, without success. Precious little exists, and the few reviews available (such as the work of my friend, Prof. Joel Bloom, or the 2004 Slate review by David Kenner and William Saletan) indicate that the numbers produced by the IVR pollsters comport as well or better than with actual election results than those from their traditional competitors.
The issues involving these new technologies are obviously critical to those who follow political polling and require far more discussion than is possible in one blog post. So over the next six weeks, we are making it our goal here at Pollster to focus on the following questions: How reliable are these new technologies? How have their results compared to election results in recent elections? How do the current results differ from the more traditional methodologies?
On Pollster, we are deliberately collecting and reporting polls of every methodology -- traditional, IVR and Internet -- for the express purpose of helping poll consumers make better sense of them. We certainly plan to devote a big chunk of our blog commentary to these new technologies between now and Election Day. And while the tools are not yet in place, we are also hoping to give readers the ability to do their own comparisons through our charts.
More to say on all the above soon, but in the meantime, readers may want to review my article published late last year in Public Opinion Quarterly (html or pdf), which looked at the theoretical issues raised by the new methods.
Interests disclosed: The primary sponsor of Pollster.com is the research firm Polimetrix, Inc. which conducts online panel surveys.
By Mark Blumenthal on September 28, 2006 10:32 AM
| Permalink
| Comments (9)
| TrackBacks (0)
September 20, 2006
By Mark Blumenthal
Our update to the Slate Election Scorecard yesterday tries to put the results for the generic congressional vote from the USA Today/Gallup survey into some perspective. It also reintroduces the controversy over likely voter models in general with specfic attention to the Gallup likely voter model. More on that below. Given the obviously high interest in this particular survey, as reflected in the sometimes heated debate in the comments section yesterday, I want to first share some of my own reactions.
First, remember it's just one poll. One of the inherent weaknesses in political polls is that they come with a lot of built in variation. Some comes from interviewing a sample rather than the whole population. Some comes from other methodological differences across surveys. As such, it is always better to look at more surveys than few. We like to average results across polls, despite some theoretical shortcomings, for just that reason.
In hindsight, the single discordant poll my be just a random statistical outlier, but not always. Sometimes it can be the proverbial "canary in the coal mine" that warns us of some new and emerging trend. So we pay attention to polls like yesterday's Gallup Poll, even if we typically recommend caution in interpreting them.
Second, let's put aside the likely voter conundrum and focus on the larger sample of adults and compare the Bush job approval rating among all adults to trends on other surveys. I have updated the table from Monday's post below, and I averaged the two Gallup polls conducted in September to try to make the data as comparable across pollsters as possible.
The pattern is now strong and obvious: While the precise level of approval shows the usual variation across pollsters, eight of the nine pollsters show some small increase in the Bush job rating between August and September. That is a highly improbable result by chance alone, analogous to flipping a coin and having it come up heads eight of nine times (roughly 2% according to my favorite binomial calculator).
Third, consider one issue that everyone overlooked except one very alert MP reader: On previous Gallup polls, the Bush job rating came first on the questionnaire, or at least before questions about congressional vote preference. This is the first pre-election poll in which Gallup switched the order, asking the congressional ballot question first and then the Bush job rating.
This practice is not unusual. Media pollsters frequently juggle the order of questions with events, especially those that conduct surveys on a wide variety of topics year-round. They will generally try to position the most important (or newsworthy) questions first to reduce the chance of bias. The problem is that in moving questions around, they sometimes introduce some unforeseen new bias that unintentionally skews a time series trend.
We have no way to know whether that happened on the latest Gallup poll (absent a controlled experiment**), but it is certainly possible the change in question increased the Bush approval rating by a few points.
Fourth, as many comments on yesterday's post have noted, the 48% to 48% tie in the generic Congressional ballot question was based on the sometimes controversial Gallup likely voter model. Our Slate update yesterday provided a quick and dirty summary:
Ideally, pollsters and pundits prefer to watch likely voters because, well, they're more likely to vote than those who are simply registered. But identifying the likely electorate is much more difficult when an election is still months away, because respondents are less able to honestly assess whether they're really going to vote. (Getting a large enough sample of likely voters also costs more money, so media pollsters usually wait until closer to the election.)
The problem is that once pollsters start screening for likely voters, their methodologies vary widely. This produces the scattershot results we've seen recently. An AP-IPSOS poll conducted last week showed likely voters preferring the Democrats by a 14-point margin (53 percent to 39 percent). Other surveys conducted over the last two weeks by Zogby, Harris, and Fox show results that were more encouraging for Republicans but not the even split that Gallup shows. Further, the Gallup poll's likely voter model has been criticized for producing volatile results, especially when used a month or more before the election.
I have written extensively about the way pollsters choose likely voters. For those without time to read it all, the key point is that while evidence shows the Gallup likely voter model typically provides a better estimate of the vote on the final poll than looking at all registed voters, it can produce a lot of volatility before October. As Mickey Kaus put it yesterday, the model may tell us more about:
'who would vote if the election were held today' as opposed to what we really want to know, which is 'whom would the people who are going to vote on November 7 vote for if the election were held today.'
Finally, consider that we are obsessing over measures with very limited utility to predict election outcomes -- the generic congressional ballot and presidential approval rating. In the races for Senate, on the other hand, we have direct measures of the actual contests and far more surveys to compare. Our Slate Senate Scoreboard has logged 36 new state-level Senate polls released in September alone. And despite the modest increase in the Bush job rating, as of today, these continue to indicate momentum toward the Democrats. Of course, the Scorecard also shows 49 seats currently held or at least leaning Republican, with 46 held or leaning Democrat.
With seven weeks remaining until the election, the data above should leave no one feeling too confident about the outcome. All of these trends can and probably will change. We'll be watching, so stay tuned...
By Mark Blumenthal on September 20, 2006 6:11 PM
| Permalink
| Comments (12)
| TrackBacks (0)
September 19, 2006
By Mark Blumenthal
Our Slate update last night focuses on changes to the Election Scorecard for Governor's races over the last few weeks. The most notable difference is a shift of Iowa from lean Democrat to toss-up. Read it all.
By Mark Blumenthal on September 19, 2006 7:08 AM
| Permalink
| Comments (0)
| TrackBacks (0)
September 18, 2006
By Mark Blumenthal
Our last Slate Election Scorecard update on Friday night shows a shift in national momentum for the first time, based on recently improving Democratic fortunes in states like Tennessee, Virginia, Missouri and Washington. These gains have occurred despite the small upward trend in the Bush job approval rating as seen on recent national surveys, as noted in Charles Franklin's post on Friday. How could these trends be moving in opposite directions?
Our Slate national momentum meter is an average of the averages of recent horserace results in each of thirteen competitive contests we have included on the Slate Scorecard. Much of the shift appears to come from states where Democratic candidates have benefited from issues or tactical advantages specific to those states. These include George Allen's "macaca" gaffe in Virginia, Harold Ford's post-primary spending advantage in Tennessee, the DUI revelations regarding Republican Mike McGavick in Washington.
At the same time, the more recent improvements in the Bush job rating are slight and may have less immediate impact on the attitudes of likely voters in the key races states.
It is worth considering the question raised by several astute readers in comments over the weekend: Could the improvement result from a recent shift by some pollsters from reporting results among all adults to likely or registered voters?
I did a quick "apples-to-apples" comparison table to show August to September comparisons among individual pollsters reporting comparable numbers. The results in the chart below, tend to rule out that explanation:
Only Fox News reported results that are not comparable (likely voters in September but only registered voters in prior months). The other organizations all reported comparable numbers (at least on the PDF releases with complete results). As the table above shows, most of the August to September change was small - too small in most cases to be statistically significant for any one survey. However, five of the seven organizations show at least some improvement and the overall average indicates a two point increase in the Bush job rating.
Incidentally, while the Rasmussen automated survey showed an upward spike in the Bush job rating that peaked in the three days following the September 11 anniversary (47% approve, 50% disapprove), it has since subsided to roughly the same level (41%-58%) as measured in August (40%-58%). Consider that the two September surveys showing the biggest changes (Zogby & AP-IPSOS) were also conducted immediately after the 9/11 anniversary.
By Mark Blumenthal on September 18, 2006 1:40 PM
| Permalink
| Comments (7)
| TrackBacks (0)
September 12, 2006
By Mark Blumenthal
Our daily Senate Race Summary is now up on the Election Scorecard at Slate and focuses on two news polls in Arizona & Washington State.
By Mark Blumenthal on September 12, 2006 7:33 PM
| Permalink
| Comments (2)
| TrackBacks (0)
