July 3, 2008
By Mark Blumenthal
My Nationaljournal.com column, on the differences between rolling-average tracking polls and other "traditional" surveys, is now online.
Regular readers may be interested in the chart in the column created by our own Charles Franklin (see below) and spiffed up considerably by the National Journal's Reuben Dalke (see the column). I wondered how the tracking poll trends compare to standard trend estimates that you see on our national chart. The chart that Franklin created plots the trends on the Obama margin (Obama percentage minus McCain percentage) using a loess regression trend line based on the non-overlapping releases from Gallup Daily, Rasmussen Reports and all other national polls. To make for a fair comparison, all three lines are plotted with the same sensitivity.
I was also curious how the trends would look if we simply "connected the dots" between the non-overlapping tracking poll releases by Gallup and Rasmussen tracking surveys as well as the "traditional" USA Today/Gallup results (based on "likely voters") . You see that below.
The Gallup Daily line looks more variable than what you are used to seeing on Gallup's Daily release, partly because the time scale is more compressed, partly because we are plotting the Obama-McCain margin rather than separate lines for each candidate and partly because we are plotting only every third or fifth day which eliminates the "smoothing" effect of the overlapping intermediate samples.
What conclusions do you draw?
Update: In the comments, PatrickM asks:
As to the sampling process for the Gallup tracking survey: I thought the purpose of the tracking survey was to draw a discrete sample each night. Since completion quotas are set for each night, non-respondents must necessarily be "replaced" for that night's calls. Theoretically, all these replacements should balance out if non-response is random.
But Gallup seems to be taking a second bite at this apple by drawing an entirely new sample on the second night and supplementing it with non-respondents from the first night until the nightly completion quota is reached. So theoretically, the 3-day rolling results could include data from the originally drawn sample point AND its doppelganger replacement phone number.
I'm not a sampling expert, but is there anybody out there who can describe the rationale behind why this is OK?
The best explanation I have seen of "rolling cross section design" (a more technically correct term than "rolling average") is Kate Kenski's description of the National Annenberg Election Survey (NAES) in Chapter 4 in Romer, Kenski, Winneg et. al., Capturing Campaign Dynamics 2000 & 2004 .
The NAES, ongoing now for 2008 but mostly held back for academic analysis, uses the same general "tracking design" as Gallup only with far more rigor: In 2004, they protocol involved dialing non-contacts as many as 18 times over as many as 14 days.
I won't try to summarize the whole chapter, but this paragraph gets closest to answering Patrick's question:
What is important to note here is that there were strict procedures in place so that no telephone number was treated differently from any of the other numbers selected. Telephone numbers released on Tuesdays were not handled differently from telephone numbers released on Fridays. This protocol ensures that the probability of being interviewed is a random event. By stabilizing the proportion of respondents who completed an interview after having been called numerous times, the representativeness of the daily cross-sections is maximized.
Why is it important that the date of the interview be a random event? if the date of interview is random, then the characteristics of the sample on any given day will not vary systematically.
By Mark Blumenthal on July 3, 2008 10:58 AM
| Permalink
| Comments (14)
July 1, 2008
By Mark Blumenthal
From last Friday, posted by Michael McDonald:
[About the] Gallup Editors' Election Poll Analysis that attempts to place Gallup's tracking poll horserace numbers in context with other polls. Two of three contributions emphasize the "stability" of the Gallup poll, which is not something I find particularly compelling since random sampling error could produce the appearance of stability when there is instability.
Perhaps the most interesting explanation has to do with the question sequencing and wording, where Frank Newport takes issue with the LA Times/Bloomberg poll for asking the direction of the country question before the ballot test.
Still, I'd like to see Gallup pull apart their poll more. What are the demographics and support among groups in comparison to other polls? I think a beginning towards the answer why Gallup is different will lie somewhere in there.
I also find it a little disingenuous to explain the differences between Gallup and the two double-digit Obama lead polls only. I'm willing to believe the truth lies somewhere in the averages, but this begs the question why the Gallup daily tracking is different from the single-digit Obama lead polls.
By Mark Blumenthal on July 1, 2008 10:50 AM
| Permalink
| Comments (0)
June 30, 2008
By Mark Blumenthal
There has been quite a bit of speculation this week over that perennial we enjoy here at Pollster.com: Why are the polls so divergent? In the last week or so, we have had national polls showing everything from a dead even race (Gallup Daily) to a 15 point Obama lead (Newsweek) and a half-dozen or so results in between.
Here are three more sets of reactions and analyses from pollsters themselves:
- Los Angeles Times pollster Susan Pinkus, as quoted in a blog item from the Times' Don Frederick
- Richard Morin, now of the Pew Research Center, formerly polling director for The Washington Post, as interviewed by CQ Politics.
Cumulatively, this discussion covers the usual suspects: differences in party identification as measured by the surveys, as well differences in question wording and question order. I wrote about the party ID debate in my column last week, something that Pinkus, Morin and Rasmussen address at the links above.
The Gallup editors raise the issue of question wording and order. Frank Newport points out the Gallup Daily begins with the vote question, while the L.A. Times poll started with the standard question about the direction of the country. He also points out that the L.A. Times vote question includes the phrase, "or would you vote for a candidate from some other party," an option not offered explicitly by the Gallup question.
The decision of the L.A. Times pollsters to ask the vote question second, after the "right-direction, wrong track" question, struck Nate Silver as a "big no-no." It is true that most of the media pollsters have now shifted to asking the presidential vote question at the very beginning of the survey, a practice they generally follow in surveys in the last few months before the election.
But not all do. Most of the pollsters doing internal "benchmark" surveys for campaigns take a different approach that usually involves asking questions about the direction of the country and favorable ratings of the candidates before asking about vote preference. You can see examples of this philosophy in the survey out today from Democracy Corps poll conducted by Democratic firm Greenberg-Quinlan-Rosner, or the most recent NBC/Wall Street Journal poll.
The rationale for that question order is that voters do not make their choices on a whim. They cast their ballots after some reflection at the end of the campaign, not by being ambushed and asked to make an immediate decision in the middle of the summer. If neutral questions that get respondents thinking about the candidates and the direction of the country for a minute or two produce a result different than asking immediately for vote preference, then the effect gets us closer to the way they will approach the real campaign. As Silver put it in a second thought, posted after some reflection:
If the mere suggestion that the country might be on the wrong track is enough to send scores of independents into the Obama column, imagine what a concerted effort to frame the discourse that way might do.
It is always important to consider question wording and question order, although I'm dubious that the placement of the "right direction wrong track" question alone explains why the L.A. Times had a better result for Obama than most other recent polls. After all, the Newsweek poll started with the vote questions, and they showed an even bigger (15 point ) margin for Obama.
By Mark Blumenthal on June 30, 2008 1:40 PM
| Permalink
| Comments (0)
June 4, 2008
By Mark Blumenthal
A quick follow-up to my discussion yesterday of the poll from the American Research Group (ARG) in South Dakota.
ARG had the winners right in South Dakota and Montana, but that was about it. If we compare the margin between the top two finishers on the poll to the margin in the vote count (with 100% of the precincts reporting according to AP this morning), ARG had errors of 11 points on the margin in Montana and 14 points in South Dakota. Those were not ARG's worst misses of the primary season -- their final polls were off by more in Connecticut, Iowa, Illinois and South Carolina -- but these last two polls were a bit worse than their average (9) for the Democratic primaries.
Looking at all the polls in both Democratic and Republican primary contests, a SurveyUSA report card compiled in early May showed ARG with a median error of 7.0, ranking them 26 on the list of all 41 pollsters, and dead last among the nine organizations that polled in 10 or more contests this year (remember due to "regression to the mean," average errors can be much larger or smaller when a pollster does only a handful of surveys -- as the number of polls averaged goes up, the error should go down).
The most striking aspect of ARG's difficulties during this primary season was the way their surveys tended to err in Hillary Clinton's favor. We first noticed this pattern in their surveys of Iowa last summer. While not totally consistent, the ARG polls tended to show Clinton doing a few points better in Iowa than other pollsters, although at the time the pattern looked like a consistent difference, but not necessarily an error. After the final ARG poll showed Clinton leading Obama by nine percentage points (she finished third), ARG pollster Dick Bennett lashed out at critics and the "deeply flawed" Des Moines Register poll (that had correctly forecast an Obama win).
Bennett might have done better to examine his own methods, because in the primaries that followed, ARG's final poll erred in Clinton's favor in 19 of 27 contests, averaging 6.8 percentage of error on the margin in Clinton's favor. The odds of that happening by chance alone are extremely remote.
Only Bennett is in a position to explain why his surveys were further off the actual result than those of other pollsters, and why his surveys tended to err in Clinton's favor. To be clear: I am not suggesting that Bennett had a pro-Clinton agenda. Rather, I think the answer has something to do with some aspect of ARG's methodology. Unfortunately, since ARG, like all too many public pollsters, tells us so little about their methods, we can only guess.
By Mark Blumenthal on June 4, 2008 12:00 PM
| Permalink
| Comments (7)
June 3, 2008
By Mark Blumenthal
"Do you trust the ARG poll in South Dakota?"
I have been asked that question more than once over the last 24 hours. The South Dakota poll released by the American Research Group yesterday and conducted this past weekend shows Hillary Clinton leading Barack Obama by a 26 point margin (60% to 34%). It is one of only two public polls released on the South Dakota primary. The only other poll, conducted in late March by Dakota Wesleyan University, showed Obama leading by 12 points (46% to 34%).
If the question is about their record during this primary season, we can look at the "Pollster Report Card" for 2008 compiled by SurveyUSA that calculated pollster "error" by comparing the percentage point margin separating the two-two candidates on the final pre-election poll for each pollster in each race to the margin between those two candidates in the actual result. On that basis, ARG shows a median error of 7.0, ranking them 26 on the list of 41 pollsters, and dead last among the nine organizations that polled in 10 or more contests this year.
Back in April, The Wall Street Journal's Carl Bialik took a closer look at ARG after it reported two polls in Pennsylvania that appeared to show the race closing from a 20 point Clinton lead to a 45-45% tie in just one week:
Dick Bennett, president of ARG, acknowledged his firm struggled in early primaries, but told me that its polls in later, big-state primaries have done well, citing California (the final poll showed a Clinton lead of four percentage points; she won by eight), Ohio (ARG had Sen. Clinton up by 14; she won by 10) and Texas (the poll had Sen. Clinton up by three; she won the primary by 3.5 points but appears to have lost the caucuses). According to Mr. Bennett, ARG’s stumbles in states such as Connecticut and South Carolina — where the firm understated Sen. Obama’s support — were due to underestimating the likelihood that first-time voters would go to the polls.
"In the tough ones, we’ve been close,” Mr. Bennett said. “As time has gone on, we’ve gotten much better.”
The most intriguing difference between South Dakota and other recent primaries is that ARG is out all alone with its results. In previous contests -- particularly Iowa, New Hampshire, South Dakota, Ohio, Texas, Pennsylvania and North Carolina -- we had ten or more surveys released over the final weekend. Many pollsters speculate cynically about the potential in such an environment for what one friend of mine calls the "watching the rearview mirror" problem, the idea that some pollsters are watching the results of the competitors and adjusting their own numbers (or weighting targets) so they are not too far off.
The potential irony here is that the big polling story at the beginning of this primary season was the courage of Ann Selzer, pollster for the Des Moines Register, whose final pre-caucus survey showed Barack Obama surging on the basis of a surge in participation by younger and more independent voters. Other pollsters were skeptical and questioned the result. ARG's final poll showed Clinton leading by nine points. Obama ultimately won 38% of the delegates elected that night, while Clinton finished third (with 29%) just behind John Edwards (at 30%). The Iowa entrance poll, arguably a better measure of the initial vote preference, also showed Obama as the first choice of 34% of attendees, followed by 27% ready to support Clinton..
In this case, ARG's John Bennett seems to be comfortable putting out numbers that contradict the conventional wisdom. Give him credit for that, at least. Slate's Christopher Beam contacted Bennett, and Bennett defended his work:
"It’s what the voters told us," he said. "It’s the same process we’ve used in other states." The survey interviewed 600 people who represented the state’s demographics, without the need for weighting or other fancy modeling.
Bennett dismissed the notion that South Dakota will vote like the surrounding states. "Look at New Hampshire compared to Vermont and Massachusetts and New York," he said. "You can’t pick out states like that."
He also pointed out that South Dakota is the oldest of the recent states except for Pennsylvania and West Virginia, and Clinton has performed well among older voters. According to the U.S. Census, 14.2 percent of South Dakota’s population is 65 years or older. In Pennsylvania, the number was 15.2 percent; in West Virginia, 15.3 percent. The Mount Rushmore State also has a closed primary, which tends to favor Clinton—no "independent bonus voters for Obama," Bennett said—and a large proportion of likely Democratic voters are women.
Will the ARG numbers prove accurate tonight? We will soon see.
In other related news:
- The statistical models of FiveThirtyEight's Nate Silver project a 5-point win for Obama in South Dakota and an 18-point Obama margin in Montana. Silver, who was the subject of a nice profile by Carl Bialik this week, also weighs in on the ARG poll -- he's skeptical.
- Brian Schaffner projects the delegate totals with and without the ARG polls. He's also live blogging the evolving delegate count all afternoon and evening.
- I am told that the networks will have exit polls tonight for both South Dakota and Montana, and both will feature supplemental telephone interviews of early voters. So we will see you here later tonight.
By Mark Blumenthal on June 3, 2008 4:14 PM
| Permalink
| Comments (2)
May 27, 2008
By David Moore
Today's Guest Pollster article comes from David W. Moore, a senior fellow with the Carsey Institute at the University of New Hampshire. He is a former vice president and senior editor with the Gallup Poll, where he worked for 13 years, and is the founder and former director of the UNH Survey Center. He manages the blogsite, Skeptical Pollster.com.
This month, the American Association for Public Opinion Research (AAPOR) gave its most coveted honor, the AAPOR Lifetime Achievement Award, to Kathleen A. Frankovic, director the CBS News polls and a former AAPOR president. In her acceptance speech, she referred to her presidential address of a decade and a half ago, when she leveled several incisive criticisms at the media polls - criticisms that deserve to be re-examined today.
Since joining CBS News over thirty years ago, Frankovic has amassed an impressive set of accomplishments, including being president of both AAPOR and its sister organization, the World Association for Public Opinion Research (WAPOR); a member of the Market Research Council; a trustee both of The Roper Center for Public Opinion Research and, separately, the National Council on Public Polls; and a former chair of the Research Industry Coalition. She is also the author of numerous published articles and book chapters on public opinion. If anyone could be considered a pillar of the polling industry establishment, she would be it.
Yet, her writings on polls have not been effusive encomiums to the presumed benefits they bring to society. While always attributing much importance to the role of media polls in American politics, she has also expressed concerns about them, nowhere more evident than in her 1993 AAPOR presidential address, "Noise and Clamor: The Unintended Consequences of Success."1
Her theme is reflected in the title, as she raised questions about the increased frequency of polls and the lack of thought that goes into many poll questions - "Immediate response is more important than what the response is or what it really means. In other words, we may no longer have to think." She also worried about the decreased value and import of polls - "It's so easy to conduct polls now that it may actually cheapen the value of each one we do. Instead of meaning, we may just be getting noise - noise and clamor."
She noted that with the advent of scientific polls, we now have a "continuous ballot box," the dream of early democratic idealists. But is that good? Not always, apparently. In the several months prior to her presidential address, polls taken at two-week intervals had showed President Bill Clinton's approval rating bouncing all over the map, "from 58 percent to 53 percent to 59 percent to 53 percent to 57 percent to 49 percent to 57 percent to 45 percent." She lamented, "This is information, but how informative is it? It's almost like what Truman Capote once remarked about Jack Kerouac's novel, On The Road: 'That isn't writing-it's typing.' Continuing ballot boxes shouldn't bounce around so much." Indeed.
Today, the uninformative information provided by polls is even more acute, obvious to anyone who has followed the pollsters' fascination with the 2008 national Democratic electorate. It isn't the results at two-week intervals, but contemporaneous results that bounce all over the place these days. One has to look only at pollster.com from April 30 to May 4 to find five polls with three different results: Gallup by itself, reporting a dead heat (Obama up by two points); Gallup with USA Today and, separately, AP/Ipsos each reporting Clinton leading by 7 points; while CBS/NYT and Diageo/Hotline each reporting double digit leads (12 and 11 points respectively) for Obama. And this isn't the only time Gallup has contradicted itself this campaign season, or that different polling organizations have come up with contrasting results when interviewing in the same time period. (See comments by ABC's Gary Langer, Dec. 12, 2007 and Feb. 26, 2008; and Mark Blumenthal's "Dueling Gallups.")
Despite her criticisms, Frankovic proposed no remedies, nor special panels to investigate the problems, perhaps in recognition that they might entail a fundamental change in the way that polls are currently conducted. In the wake of the miscalls in the New Hampshire Democratic Primary, AAPOR's president, Nancy Mathieowetz, did in fact establish a special panel "to examine what occurred, provide a timely report of our findings, and promote future research on pre-election primary polls." No such panel has been established to examine all the subsequent conflicting polls, though the New Hampshire panel might want to consider broadening its scope. The "continuing ballot boxes" are not just bouncing around, they're running into each other going in opposite directions.
Frankovic concluded in her presidential address in 1993 that "We have achieved the ability to cut through the noise and clamor of unscientific measures, even as we risk making some noise and clamor of our own." This observation suggests that the radical question the AAPOR panel needs to address is whether the noise and clamor of "scientific" polls is any better than that of the unscientific ones.
1 Kathleen A. Frankovic, Presidential Address "Noise and Clamor: The Unintended Consequences of Success," Public Opinion Quarterly, Vol. 57, No. 3 (Autumn, 1993), pp. 441-447.
By David Moore on May 27, 2008 2:00 PM
| Permalink
| Comments (0)
| TrackBacks (0)
May 6, 2008
By Mark Blumenthal
Well, for the umpteenth time this primary season, we wake up to wide variation on the final polls for the day's primaries. Today we have polls showing Barack Obama leading Hillary Clinton by anywhere from 4 to 14 points in North Carolina. Meanwhile, poll in Indiana show everything from a 2-point Obama edge to a 12-point Clinton blowout. One big question in looking at the variation is whether the pattern of variation suggests a pattern of a larger undecided translating into a hidden Clinton vote. The evidence on this question is mixed, however, and relies mostly on the results from just one pollster.
First, rather than listing the polls, lets do a chart of the final results in the last week or so from each pollster. Start with North Carolina. The following chart simply plots each result on a grid with the Obama percentage on the vertical axis and the Clinton percentage on the horizontal axis. All of the points on the North Carolina chart are above the blue diagonal line indicating an Obama lead.
Many -- such as one of Josh Marshall's readers -- think they see a pattern (which yours truly also saw at first in Pennsylvania but that subsequently disappeared) suggesting a coming "break" of undecided voters to Clinton. Such a pattern would imply a horizontal pattern to the dots above, with all of the variation in the Clinton number and little in the Obama number. That pattern holds only with respect to the Zogby poll, the one showing Obama leading by the biggest (51% to 37% margin), but also the poll with the most respondents categorized as either undecided or as choosing "someone else."
The pattern of the other points in North Carolina is mostly circular, about as varied as we would expect given sampling error and centered around a roughly seven point Obama advantage (50% to 43%) with 7% left over as undecided or "other."
Now Indiana:
In Indiana the dots are slightly more dispersed, with Zogby again the showing the best result for Obama, in this case a 2-point Obama advantage (45% to 43%), with 12% categorized as either undecided or "other." In this case, however, two polls have shown roughly as many voters choosing an option other than Obama or Clinton, although both were about a week old: One from TeleResearch (showing Clinton leading by 10 points with 14% undecided/other) and the other from Rasmussen Reports (giving Clinton a 5-point lead with 13% undecided/other).
Again, if we set the Zogby result aside, we get most of the polls forming a circular, mostly random pattern around an average advantage of 7 points (50% to 43%) with 7% undecided or "other."
In thinking about what to make about the difference between Zogby and the other polls, it may be useful to think about what it means for a respondent to tell a pollster they are "undecided" in the days leading up to the election. There are at least three possibilities:
- They are going to vote but are still uncertain about which candidate to support
- They are going to vote, have decided which candidate they support but are not willing to share their preference with the person (or computer) on the other end of the phone line
- They are not going to vote but were mistakenly identified as a "likely voter" by the pollster
Pollsters understand that many voters hover somewhere between a final decision and being totally undecided. So most consider it good practice to "push" uncertain voters, especially near election day, as the candidate they lean to supporting is almost always the candidate they ultimately support.
So I tend to agree with Pollster readers who have expressed frustration in comments with pollsters reporting a large undecided preference. What is especially puzzling about the Zogby result, however, is the very large percentage that they have reported as favoring "someone else" -- 4% in North Carolina (down from 8% over the weekend) and 5% in Indiana (down from 7%). What does that mean? Are respondents expressing a preference for John Edwards? John McCain? Are those really non-primary voters?
At any rate, given that the demographics of Zogby's samples are not radically different from the other pollsters in the two states, there is certainly a good possibility that a harder "push" would benefit Clinton. We also have seen in exit polls that late deciders have favored Clinton in most of the primaries since Super Tuesday. More on both states -- and particularly the issue of late deciders favoring Clinton -- later today.
Update: Just noticed this helpful information posted by Dick Bennett on the ARG web site:
The Democratic ballot in Indiana has two lines (Hillary Clinton and Barack Obama), while the Democratic ballot in North Carolina has four lines (Hillary Clinton, Mike Gravel, Barack Obama, and No Preference).
Democratic primary voters in our surveys in Indiana were asked just the two candidate choices. If voters said someone other than Clinton or Obama, our interviewers were instructed to inform the voters that there are only two choices on the ballot. In most cases, voters then selected Clinton or Obama instead of saying they were undecided.
In North Carolina, our surveys gave the four choices on the ballot. Democratic primary voters selecting the "no preference" line also told us that they would never vote for Clinton or Obama. Our results combine the no preference with someone else (even though no preference will get more votes than Mike Gravel).
In watching the results tonight, be aware that "someone else" is not on the ballot in Indiana and some voters in North Carolina will vote the no preference line.
By Mark Blumenthal on May 6, 2008 12:29 PM
| Permalink
| Comments (2)
By Mark Blumenthal
Last week, Democratic pollster Mark Mellman used his column in The Hill to argue that one could use "same-poll to same-poll comparisons" to argue that the controversy over Barack Obama's "bitter" comments had either no impact or a great deal of impact on the Pennsylvania primary. "In addition to providing jobs for pollsters," he wrote, "proliferating polls now give everyone the evidence to prove their favorite theory." He drew specific examples from the lesser known pollsters active in statewide contests that are typically derided by the polling establishment and the mainstream media.
Over the last 48 hours, however, we have seen Mellman's point proved by the country's most respected and well established polls and media organizations. As Gallup editor in chief Frank Newport noted in his Gallup Guru blog:
Two different headlines today in the New York Times and USA Today about the impact of the Jeremiah Wright controversy came to different conclusions about Jeremiah Wright and Barack Obama. The Times’ headline: “ In Poll, Obama Survives Furor; but Fall Is the Test”, while USA today headlines: “Flap over pastor pulls Obama down, poll finds”.
And for those who missed it, ABC's Gary Langer briefly summarized the conflicting numbers:
Briefly: Times/CBS has Barack Obama +12 vs. Hillary Clinton, with a headline saying Obama “survives furor” over the Rev. Jeremiah Wright. USAT/Gallup has Clinton +7, saying the flap over Wright “pulls Obama down.” Adding to the mix is Gallup’s daily poll, which has Obama +4.
These polls also differ in their general election match-ups: Times/CBS has Obama +11 and Clinton +12 vs. John McCain, while USAT/Gallup has them basically tied. Gallup daily has Clinton-McCain tied, McCain +5 vs. Obama.
And just in overnight: AP/IPSOS weighs in with a national poll showing Clinton +7 over Obama, with Clinton +5 and Obama +4 over McCain, spawning a thousand different headlines, no doubt (as AP stories always do).
Why all the confusion? Langer has a nice summary of the various methodological quirks that may or may not explain the differences between the Times/CBS and Gallup polls, though his bottom line is the most important take-away point. This episode, he writes,
[Is] a reminder that all polls – even good-quality ones – are done differently, and don’t always get the same results or engender the same analysis. And that horse-race results, in the midst of a close and unsettled campaign, may be particularly vulnerable to these kinds of influences.
Another way to make the same point. Look at our chart for the national Democratic primary polls as captured this morning It plots points for every survey released since January 2007 (captured this morning).
Let your eyes focus, for a moment, not on the the lines but on the cloud of dots surrounding each line. Each dot represents an individual poll. The dots are a bit more dense lately, a pattern explained mostly by our inclusion of daily tracking polls by Gallup and Rasmussen Reports since January 2008 (note: we plot only ever third or fourth day of each survey so we their rolling average samples do not overlap). However, the pattern of variation -- the spread of points around each line -- is considerable but not any wider now than at any other point over the last 17 months.
What is different right now is that the gap between Obama and Clinton is very close. So some polls show Clinton ahead, some show Obama ahead and given the sate of the race and we are paying much more attention to the small differences in individual polls than we usually do. Random error may explain some of the variation, small differences in methodology (question wording, order, the particular sub-population that answered he question) explains the rest. Either way, the cloud of variation -- bringing with it sometimes odd and conflicting "same-poll to same-poll comparisons" -- of "is an inherent part of political polling.
The variation is also the reason why we favor looking at all the polls in this "mashed up" graphical form rather than debating endlessly over which individual poll is closest to "right" at any moment.
By Mark Blumenthal on May 6, 2008 7:57 AM
| Permalink
| Comments (2)
May 5, 2008
By Mark Blumenthal
And following-up on my post this morning on the demographics for the North Carolina polls, here is the same set of statistics, when available, for the recent surveys of Indiana. Since African-American's are a much smaller share of the population in Indiana, the Clinton-Obama results are not quite as sensitive to their percentage of the Democratic electorate as in North Carolina. However, should the Indiana result be close, the size of the African-American population will be important. Also, the tables show that the polls vary on age as much as in other states.
The following table shows demographic composition statistics for those pollsters that have released them. Click on the table to display a larger version that also includes the vote preference results for reach poll.
The table excludes statistics from pollsters that have not publicly released demographic information for their North Carolina surveys (or perhaps more accurately, have not published anywhere I could find it): LA Times/Bloomberg, Indianapolis Star/WTHR/Selzer, Howey-Gauge (and thanks again to Pollster reader jac13 for sharing the demographic profile data that Zogby makes available to paid subscribers).
In Indiana we see the same wide variation in the age distribution among pollsters seen elsewhere: Even The percentage of 18-to-29-year-olds varies from 8% to 22%, the percentage 18-to-44 varies from 26% to 51%.
With the exception of one pollster, the variation in racial composition is smaller. Most show an African-American percentage of somewhere between 9% and 12%, with Research2000 (13%) and Suffolk University (15%). The most extreme value is the Howey-Gauge survey, which reported a much higher percentage of African-Americans (20%) among likely primary voters.
Brian Schaffner noticed last week that larger percentage of African-Americans in the Howey-Gauge poll explained how they showed Obama with a two-point advantage while other firms showed Obama trailing by seven or more percentage points. He has some interesting speculation about the composition of the Indiana electorate, but ultimately I have to agree with his bottom line conclusion: Given the lack of an Indiana benchmark for past Democratic presidential primaries, "we don't really know what to expect in terms of African American turnout."
[Updated table to include new surveys from PPP and InsiderAdvantage]
By Mark Blumenthal on May 5, 2008 3:47 PM
| Permalink
| Comments (12)
By Mark Blumenthal
Time for another round-up of available poll demographics, this time from North Carolina. The most important variable in this state is the African American percentage of likely Democratic primary voters. The most recent polls -- at least among those that have disclosed their demographics -- have converged around a black percentage of 32-33%. Needless to say, given the near monolithic support that African Americans have given Barack Obama, that percentage will ultimately be critical to his share of the vote on Tuesday.
The following table shows demographic composition statistics for those pollsters that have released them. Click on the table to display a larger version that also includes the vote preference results for reach poll.
The table excludes the pollsters that have, as of yet, not publicly released demographic information for their North Carolina surveys: Mason-Dixon, Rasmussen Reports, and LA Times/Bloomberg (special thanks to readers Paul and jac13 for sharing the demographic profile data that Zogby shares with paid subscribers).
As in previous states, we see considerable variation in the kinds of voters selected as "likely primary voters." Easily the most variant likely voter sample on the list is the one from the Civitas Institute from early April, with a composition of just 28% African American and 17% under the age of 45. However, even if we set that survey aside, we still see considerable variation: from 51% to 58% female, from 39% to 55% age 18-to-44 and from 25% to 37% African American (and those last extremes come from a single pollster -- more below).
A quick review from my post on the demographics of the Pennsylvania surveys:
It is important to remember that pollsters come to these composition statistics through different paths. Some interview samples of adults, weight those demographically to match census estimates of Pennsylvania's adults, then select "likely voters" and let their demographics fall where they may. Others will weight their "likely voter" samples directly to pre-determined demographic targets. Some pollsters will not set weights or quotas for demographics, but will set such weights or quotas for geographic regions (based on past turnout and their assumptions about what might be different this time).
With that in mind, note two very striking changes from two pollsters that set pre-determined demographic targets, Public Policy Polling (PPP) and InsiderAdvantage:
- The first three surveys released in April by PPP had an African American composition of 36% or 37%. Their most recent survey, fielded last Sunday and Monday evenings, had a black composition of just 33%.
- The gyrations in the weighting by InsiderAdvantage are even more dizzying. Their first North Carolina survey in late March was 37% African American. Their next two surveys in April were only 25% African American, and their most recent poll last week bumped the black percentage back up to 33%. Notice that none of their percentages for women, 18-29-year-olds, 18-44-year-olds or those 65+ changed by a single digit, despite a 12-point variance in the black percentage.
Both pollsters put out written summaries of their results, but neither made any reference (that I could find) explaining or justifying their changing assumptions about the racial composition of the North Carolina electorate. [Update: On their final poll, PPP upped the black share to 35%, but explained their rationale]. By the way, we know that these two pollsters set predetermined demographic targets, because both have confirmed as much to me in previous communications (here for InsiderAdvantage and here for PPP).
The change in the PPP poll is important -- they should have noted it -- but relatively modest compared to the astonishingly large, significant and unexplained shifts in the African American composition in the InsiderAdvantage polls. InsiderAdvantage's Matt Towery likes to brag of his "significant experience" as a pollster, but after a number of curious episodes over the last few months, it is getting very hard to take those claims seriously.
It's also worth pointing out the relative stability in the racial composition of the SurveyUSA results, given that they do not force their samples to a pre-determined demographic profile (details on their procedures here). The percentage of African Americans in their four surveys since March have remained relatively stable, falling within the range of 30% to 33%.
Finally, one caution about the percentage reported as "unaffiliated" (having no party affiliation). Only PPP includes the full text of their party question, and it is possible other pollsters are asking about party identification (whether respondents "consider themselves" as partisans) rather than party registration.
Update: Almost forgot. Fivethirtyeight's Poblano posted a handy spreadsheet that can help you see just how much small changes in the racial composition of the North Carolina electorate can affect the potential margin between the candidates. It's well worth the click.
Update II: In posting this last night, I neglected to point out that North Carolina has been releasing reports on the demographics of early voters. As North Carolina is one of nine southern states still required by the Voting Rights Act of 1965 to track voter registration by race, racial tallies among early voters are also available. The demographic composition of early voters have been analyzed by Brian Schaffner, DailyKos diarist dean4ever and noted in comments by many of our readers over the weekend.
Overnight, GMU Professor Michael McDonald, whose academic focus is voter turnout, posted the following comment:
North Carolina is an exceptional state in that it provides near real-time updates of its voter registration file. Indeed, you can download the entire file of absentee and early in-person voters directly from the state's ftp site.
North Carolina is also an interesting state because race and gender are recorded on the voter file (birthdate appears to be supressed in the absentee file). When I crunch the numbers, out of the 397,850 persons who are listed as returning a Democratic Party ballot as of 5/03:
39.9% are African American
60.8% are women
Note, a small percentage (less than 1%) of records have missing data.
Will these percentages hold for Tuesday? That is hard to say, mostly because people who study early voting (myself included) don't know much about the characteristics of early primary voters. Added is the confounding factor that one-stop registration and voting is permitted for in-person early voters only and not for Election Day voters. Providing little further clues, African-Americans are only slightly more likely to vote early in-person, 40.6%, and women slightly less, 60.7%.
The fact that nearly 400,000 early votes have been cast so far is remarkable given past primary turnout in North Carolina. The state held a caucus in 2004 (due to a redistricting battle that delayed the primary), but 544,922 Democrats voted in the largely uncontested primary in May 2000, and 691,875 voted in May 1992 (statistics I gathered for a column noting that pollster PPP has been sampling from a total universe of 874,222). The record was 961,000 in 1984, according to the Charlotte Observer, which cites "long time N.C. political observers" guessing that "as many as 1.5 million" may vote this year. So this early vote will be a significant portion of the total votes cast, but as McDonald points out, no one knows exactly how big.
It is also worth pointing out that the Obama campaign has made early voting drives a focus of their field organizing, so it is certainly possible that the ranks of early voters are disproportionately swollen with Obama voters. Last week's poll from SurveyUSA showed Obama leading by a 18 points (57% to 39%) among early voters, but that subgroup was just 2% of their total sample. Thus, one key result to watch in the final poll releases today -- among those far sighted enough to track and report it -- will be the size and preference of the early voters.
By Mark Blumenthal on May 5, 2008 12:33 AM
| Permalink
| Comments (9)
April 22, 2008
By Mark Blumenthal
We have been busy here over the last day or two, including links to 8 new polls that interviewed through Sunday night, so I am going to try use this post to wrap things up a bit. All but one of the late surveys shows Clinton leading by margins of 5 to 13 points, so to no one's surprise, most expect Hillary Clinton to defeat Barack Obama tonight. The suspense seems to be about the size of Clinton's margin. On that score, unfortunately, the polls are not conclusive.
Why not? Here's the short version: (1) The pattern of smaller undecideds correlating with larger Clinton margin has largely disappeared over the last week, (2) tracking polls have been inconsistent about late trends and (3) the ultimate margin will depend on how well these surveys have selected likely voters. The longer version follows:
1) Do undecideds look like Clinton voters?
Maybe, maybe not.
The notion of undecided voters "breaking" to one candidate or another is something of a misnomer to begin with. It makes the implicit assumption that all surveys measure the true electorate and that all voters that express preferences on surveys are truly decided, leaving the final margin in the hands of voters that tell pollsters they are "undecided."
In reality, our "likely voter" models inevitably include some adults that end up not voting and exclude some that do. As such, the "undecided" category on the final round of polling usually includes a disproportionate share of those who are disengaged from the race and end up not voting. Also, some voters tell pollsters they have a preference even though they say they may still change their minds (9% of those with a preference in Pennsylvania, according to the final Mason-Dixon survey).
A week ago, I used my National Journal column to highlight a pattern in the surveys that suggested a hidden vote for Clinton. The Obama percentage appeared relatively stable across polls while the Clinton percentage varied considerably with the size of the undecided category. As the undecided percentage decreased, Clinton's percentage grew.
On the last round of polls, however, the pattern that I highlighted has disappeared. I updated the chart used in the column with the polls fielded over the final weekend highlighted in dark blue. The wide "spread" in the dots is gone. The previous pattern had owed largely to differences in the results from two pollsters (which both use an automated methodology): SurveyUSA showed big Clinton leads and small undecided, while Public Policy Polling (PPP) showed Obama even or slightly ahead and a larger undecided percentage.
On the last round of surveys, two important things changed. SurveyUSA, still finding very few "undecided" voters, showed the Clinton margin narrowing significantly, while PPP added a follow-up question asking undecided voters how they lean. PPP continues to show Obama with a slight lead, only with a much smaller undecided percentage. So the pattern of dots in the chart is now more circular, and the relationship between the size of the undecided category and the Clinton margin has all but disappeared (something Poblano also noticed yesterday).
Of course, the remaining undecided may still conceal a disproportionate share of Clinton voters, but hard evidence of that proposition is weak. Chuck Todd noticed that undecided voters in the MSNBC/Mason-Dixon survey were higher in subgroups where Clinton does better (among gun owners and outside of the Pittsburgh and Philadelphia media markets). Looking back at theTime/SRBI survey conducted in early April, Charles Franklin saw evidence that undecideds seem "somewhat more likely to support Clinton." However, as I look at the pattern of undecideds in the most recent SurveyUSA and Quinnipiac surveys, I see no clear pattern in the undecided either by region or demographic subgroups. On the Quinnipiac survey, for example, the percentage of undecided voters is roughly same among African Americans (6%) and white voters without a college education (5%).
Gallup's Frank Newport looked at the evidence on this question last Friday (a post worth reading) and "neatly" concluded that "undecideds either will or will not break for Clinton in Pennsylvania." That's about right.
2) Are polls showing a late trend?
Once again, unfortunately, the bottom line is maybe, maybe not.
The Zogby rolling average tracking shows the Clinton margin growing from one percentage point (46% to 45%) to ten (51% to 41%). However, other surveys that have tracked twice over the last week to ten days show no consistent trend. As the table below shows, both SurveyUSA and ARG showed essentially a trend in Obama's favor over the last week, while four other pollsters showed essentially no change. On average, these "apple-to-apple" comparisons show Clinton's percentage increasing by less than a point, Obama's by roughly two. Ignoring statistical significance, four polls showed movement in Obama's direction, two showed movement in Clinton's direction and one showed no change in the Clinton-Obama margin.
Click the thumbnail below to see a larger version with more complete data:
Our trend estimates add another wrinkle. The standard trend lines look parallel, suggesting little or no change in Clinton's 6-7 point margin over the last week. However, as Charles Franklin explained earlier this morning, the more sensitive estimate -- which gives greater weight to more recent polls (including a few that had not been tracking a week ago) -- shows a slightly bigger Clinton margin (8.4 points).
So, again unfortunately, we either have evidence of a late trend, or we do not.
The exit polls tonight will help resolve whether late deciders have have favored one candidate. Yesterday, drawing on recent exit polls, ABC's Gary Langer noted:
Late deciders have been turning to Clinton recently, but only recently. She’s done better among late deciders than among other voters in 11 contests, including the eight most recent. Going farther back, though, she’s done the same among late deciders in 10 contests, and worse in 10.
3) "It's the Turnout Stupid"
That's the way FiveThirtyEight's Poblano put it yesterday, and he's right. For all our worry about late shifts and the problems of interpreting the "undecided" category, the collective accuracy of the polls (or lack thereof) in predicting Clinton's margin probably depends even more on how well they have done selecting "likely voters."
Give a pollster 1,000 voters to interview, and our measures do a reasonable job discerning their preferences. But trying to discern the actual primary voters from a random sample of 1,000 adults is not so easy and far less accurate. Different methods of selecting "likely voters" can end up selecting different kinds of people. Since the Obama-Clinton race features large differences in vote preference by race, gender, age, socio-economic status and region, relatively small shifts in the composition of the electorate can alter the vote margin noticeably.
As I reviewed yesterday, if we look at their composition in terms of race, age, gender, and years of education, the Pennsylvania polls show meaningful variation. Given the demographic patterns in the vote, a difference of four points in the African-American contribution on most polls can lead to a three point shift in the Clinton-Obama margin. Differences of five percentage points in terms of the contribution of white voters under 35 or white voters with a college education may translate into two-point shifts in the Clinton-Obama margin. The same is probably true for the share of the vote in the Philadelphia metro area (as Virginia Centrist points out).
"I want to know the future," Pollster reader Fourth wrote yesterday. "Is that too much to ask?"
No, it's not. Unfortunately the challenge of selecting likely primary voters is what makes these pre-election polls blunt instruments as predictors. They can give us a general sense of where things stand, which way they are moving (when the movement is large) and guidance about what each candidate needs to do to maximize their support. But the problem involves too many unknown variables to try to predict the outcome with precision.
The future will be here in about 12 hours. We will know soon enough.
[Embarrassing typos repaired. Many thanks to Pollster reader JB for his unsolicited fill-in for Eric].
By Mark Blumenthal on April 22, 2008 10:37 AM
| Permalink
| Comments (3)
April 21, 2008
By Mark Blumenthal
Just before the March 4 primaries, I did posts on the demographic compositions of the polls from both Texas and Ohio. With the ever valuable Eric Dienstfrey away on vacation this week, I am doing this post in a bit of a rush (so apologies in advance for typos). I would strongly recommend reviewing my post on the Texas demographics as a companion to this piece.
I have broken the available results into two tables below. Most come from documents posted on the web. Quinnipiac provided results on request, and the Zogby numbers were shared with my colleagues at the National Journal.
The racial mix of the Pennsylvania polls is not quite as critical to the level of candidate support as in Texas, since the share of black and Latino voters is smaller. Still, since Obama typically does better among African-Americans, men, younger voters and those with college degrees or higher incomes, while Clinton does better with whites, women, older voters and those with lower incomes or without a college degree, the demographic composition of the electorate will play a role in determining the outcome of the race.
The surveys show more variation on some characteristics than others. Most, for example, show the percentage of women as somewhere between 55% and 58%, and most show the African-American percentage as somewhere in the mid-teens. Of course, with Barack obama expected to receive 80% to 90% of the black vote, the difference between an African American composition of 13% and 18% can alter Obama's vote total by 3 to 4 points.
On the other hand, we see quite a bit of difference in age. Unfortunately, the pollsters do not all use the same categories to ask about and report respondent age. Still, we can see quite a bit of difference, particularly in the percentages in the 18-to--29, 18-to-35 and 18-to-44 categories. We see that 18-to-29-year-olds are are anywhere from 4% to 16%, that 18-to-44/45-year-olds are anywhere from 22% to 43%, depending on the pollster. Given that Obama typically does much better among younger voters, and that Clinton does much better among retirees, this variation is obviously critical. [Update: Brian Schaffner also blogged on this issue today].
Socio-economic status is another critical characteristic in the Obama-Clinton race, especially in Pennsylvania (and something that I have written about often). Unfortunately, quite a few pollsters either ask or report nothing about the level of self-reported education or income of their samples. Still, we see considerable variation. The percentage of respondents with college degrees varies from 29% to 44%. I should point out that education and especially income are subject to more measurement error than other demographic items, especially if the text of the question and the number of categories differs.
Finally, since readers asked for it the last time, I have also posted one more table that includes all of the data above, plus the vote preference results. You will need to click on the graphic below to see a larger, readable version.
It is important to remember that pollsters come to these composition statistics through different paths. Some interview samples of adults, weight those demographically to match census estimates of Pennsylvania's adults, then select "likely voters" and let their demographics fall where they may. Others will weight their "likely voter" samples directly to pre-determined demographic targets. Some pollsters will not set weights or quotas for demographics, but will set such weights or quotas for geographic regions (based on past turnout and their assumptions about what might be different this time).
Trying to discern the differences in these methods is beyond our capacity today. The important thing is to remember that different pollsters conceive of "likely voters" in different ways, and the "likely voters" they reporting at are not identical.
Update: Poblano at FiveThirtyEight.com blogged some worthy thoughts about differences in likely voter models today.
Please note that, given the crunch of time I have probably not proofed the tables as well as I should have. If you catch a typo, please do not hesitate to send an email so I can correct it.
By Mark Blumenthal on April 21, 2008 3:39 PM
| Permalink
| Comments (5)
April 17, 2008
By Mark Blumenthal
My NationalJournal.com column for the week, on those widely divergent Pennsylvania polls, is now online. As a tease to motivate you to click thru, here's the graphic featured in the column (graphically enhanced by my National Journal colleagues).
I should note that while I was writing this column yesterday, Brian Schaffner was apparently in the midst of creating a scatter plot for his CCPS blog that is eerily similar to the one I did for the column. He gets all due props for getting his posted first.
[I've got some additional thoughts that were too much for the column and will update with those a little later this morning].
By Mark Blumenthal on April 17, 2008 10:18 AM
| Permalink
| Comments (5)
April 8, 2008
By Mark Blumenthal
I will admit that, like Pollster reader jsh1120, I am at a bit of a loss about the flurry of recent results from Pennsylvania. In the last week, we have seen surveys released showing everything from an 18-point Clinton lead to a 2-point Obama advantage (for all links, see our Pennsylvania chart and table). Of course, we have had days before this primary season where we saw huge spreads among pollsters in their Democratic primary results.
As before, the most likely explanation involves differences in the kinds of people pollsters are selecting as likely primary voters. In California, Texas and other states with very large minority populations (either African American, Latino or both), the variation in racial composition explained much of the difference. In Pennsylvania, however, the percentage of black voters is relatively low and the Latino population in the low single digits. As such, the mix of gender, age and socio-economic status may help explain the divergent results before us. Unfortunately, only a few pollsters routinely release their composition statistics and most are not asking respondents about their education or income levels. So we are largely left to speculate about the differences.
One way to help clarify the numbers, if only slightly, is to focus on the results among white voters for the recent polls that have tracked and released results by race over the last month. Doing so (see the table below) at least eliminates the differences due to variations in racial composition.
The polls certainly differ, but in many ways, their results are consistent. All four show Clinton with roughly 60% of the white vote in early March. All show modest increase in support (+3 to +7 points) for Barack Obama in recent weeks. They diverge mostly on the Clinton trend. Three of the four show a relatively modest decline in support for Hillary Clinton, while SurveyUSA shows no change at all for Clinton and the smallest increase (+3 points) for Obama.
I am hoping to post more on Pennsylvania -- focusing on the new Quinnipiac University data by race and education -- later today.
By Mark Blumenthal on April 8, 2008 3:24 PM
| Permalink
| Comments (12)
March 31, 2008
By Mark Blumenthal
Can you guess which presidential election gave us this headline and lead?:
POLLS: A YEAR TO BE WARY
ONCE in a while, all pollsters should take the kind of beating we took in the primaries, just to maintain equilibrium," says Don Muchmore, board chairman of Opinion Research of California, one of the many polling firms that came a cropper in one or more of this year's presidential primaries.
The phrase "came a cropper" probably gives it away. The year was 1964 when this article in appeared in Time. I was all of 17 months old (my own world had been rocked the previous day, or so I am told, by the birth of my brother...but I digress). In a bit of Internet serendipity, I stumbled on the article yesterday when Googling the name of Oliver Qualyle (Lyndon Johnson's pollster) for the previous item. It is worth reading in full, if only for the perspective it provides on 2008.
It is striking how many of the issues of polling in 1964 -- when most political surveys were conducted in person by interviewers that went door-to-door to select respondents -- remain the same in 2008. The topic of "likely voters" was just a confusing: In 1964, Time reported, pre-election polling techniques "vary considerably from pollster to pollster" and the researchers had "not yet licked" their "inability to assess the probable voter turnout." The Time article also includes a discussion of whether "people lie to the pollsters" on the issue of "race relations" in a year when that issue seemed "particularly important."
On the other hand, on at least one issue, polling has come nearly full circle. In 1964, the concept of random sampling was not yet universally accepted among American pollsters. The article cites a "trend towards 'randomization'" among the pollsters and then, perhaps unknowingly, lists various departures from random sampling in their techniques, such as Gallup's practice, circa '64, of instructing interviewers to "skip some corner houses on the theory that corner property is higher-priced and its occupants are likely to be more affluent than their neighbors."
Today the controversy over random sampling is whether rates of non-response are so high as to render true random sampling impossible, leading some pollsters to return to various departures from true "randomization" such as the use of volunteer online panels.
Back to a polling evergreen. Are polls "reliable enough" to justify their high costs?
In probing general attitudes toward candidates and issues, they undoubtedly come close enough to be of value to campaign strategists. When it comes to calling elections, most of the pollsters insist that they do not make predictions, merely measure the popularity of candidates at a given point in time. In the post-mortems they are, of course, the first to boast when they hit one right. But that seems fair enough, since they take a beating when they are wrong.
By the way, the entire Time archive of articles published since 1923 is now available online to non-subscribers. Happy searching.
By Mark Blumenthal on March 31, 2008 1:48 PM
| Permalink
| Comments (1)
March 25, 2008
By Mark Blumenthal
Update: In the comments, Chris G argues that I am "way off" to conclude that "there has been far more stability than change in the national Obama-Clinton vote preference since Super Tuesday." He writes:
[T]hat simply does not follow from the simulations. the only thing that can be inferred is that if we're looking at these 2 time series alone, any meaningful changes in support are swamped by the noise. that's all we can conclude.
Since I may have been unclear, let me try to clarify: I am not arguing that the Gallup Daily and Rasmussen Reports tracking data proof the complete absence of change in candidate preference since Super Tuesday. Chris is absolutely right: No survey can do that. The best we can do is conclude that changes have been too small to be detected with confidence.
The point I was trying to make is that the changes since Super Tuesday have been (a) short lived, (b) small enough that they are indistinguishable from random noise, or (c) both. I do not consider changes of that sort to be very meaningful substantively, though your definition of "meaningful" may differ.
I am also not arguing that we should ignore the Gallup Daily. We just need to be patient and wait to see big, persistent changes. Look back at the numbers they reported in January through early February and you can see a very large, sustained and meaningful trend toward Obama:
In the midst of writing this update, I discovered that Gallup's Frank Newport made essentially the same point in his daily video report today:
As I look at the Gallup Daily election tracking, I am struck by the fact that neither candidate, Hillary Clinton or Barack Obama has been able to move ahead to a sustained and significant lead over the other [emphasis added].
By Mark Blumenthal on March 25, 2008 10:48 PM
| Permalink
| Comments (8)
By Mark Blumenthal
In case you missed our update, the most recent Gallup Daily result on the Democratic race shows a near dead-heat, with Barack Obama ahead of Hillary Clinton by a single percentage point margin not nearly large enough to attain statistical significance (47% to 46%). That one point lead is somewhat apropos, since it is virtually identical to the average of all of Gallup's Daily releases since February 8 (Obama 46%, Clinton 45%). So the question for the day: How much of the daily variation over the last six weeks has been real and how much is random noise?
Let's start with the chart of the Gallup Daily results since their three-day track completed on February 8 (and released on February 9). That was the first three-day result collected entirely after the results from the Super Tuesday primaries were known.
While the Gallup trend has shown several "figure eights" over the last few weeks (as reader "emcee" put it), most of that variation occurs within the range that we should expect from a survey with a +/- 3 point margin of sampling error.
To illustrate that point, consider the hypothetical possibility that the preferences among Democrats have remained perfectly stable for the last six weeks. Let's assume that the average result since February 8 -- 46% to 45% favoring Obama -- has been the unchanging reality. What sort of random variation should we expect from taking a sample rather than interviewing the entire population?
First, remember that the so-called "margin of error" applies to the individual percentages, not the margin between the candidates. So under our hypothetical "no change" scenario, we would expect the the Obama percentages to fall somewhere between 43% and 49% (46% +/- 3) and the Clinton percentages to fall somewhere between 42% and 48% for Clinton (45% +/-3).
Since February 8, the results of the actual Gallup Daily have fallen outside that range on just three days:
- March 1, when Obama led 50% to 42%
- March 13, when Obama led 50% to 44%
- March 18, when Clinton led 49% to 42%
But wait. As some of you may remember, most political surveys (including Gallup) calculate the margin of error using a 95% confidence level. That assumption means that we should expect results slightly outside the margin of error for one poll in twenty.
Unfortunately, at this point our story gets a little bit more complicated, because the "one in twenty" assumption applies to statistically independent measurements. Since each Gallup Daily release is based on a three-day rolling average, there is overlap in the sample on successive days. So only the results from every third day are truly "independent." 'll skip over some even more confusing explanation and get to the bottom line: Since February 8, roughly one-in-seven independent samples from the Gallup Daily series has produced a result outside the margin of error from my hypothetical, no-change, 46-45 scenario. That's a little bit more than we would expect by chance alone, but not much more.
Having said all that, my explanation still oversimplifies. It ignores the possibility for meaningful change within the standard "margin of error" -- subtle shifts that might not attain statistical significance in a single three-day sampling, but might over the course of a week or more.
A better way to distinguish the meaningful patterns is to compare Gallup's results to those from another pollster or two. Let's start with a chart of the Rasmussen Reports daily tracking poll over the same six week period. Not surprisingly, the average of the Rasmussen data gathered since February 8 also shows Obama leading by a single percentage point (45% to 44%).
Compare the two charts (or look at the chart below, which plots a Clinton-minus-Obama margin for both polls) and you will see several features in common:
- Both show a shift from Clinton to Obama between Super Tuesday and mid-February
- Both show Obama maintaining a low single-digit lead from mid to late February
- Both show Clinton rising a few days before the March 4 primaries and falling a few days after
And yet, at about the time the news surrounding Jeremiah Wright became a full-blown media obsession (March 14), the results of the two polls appear to diverge. Why is that?
We should keep in mind that Gallup and Rasmussen collect their data differently (and ask slightly different questions -- see the postscript). Gallup uses live interviewers, makes repeated call-backs to unavailable respondents, samples cell phone numbers, and routes calls to Spanish speaking interviewers when they reach a Spanish speaking household. Rasmussen uses an automated system and recorded voice to conduct interviews, a slightly tighter screen for "likely voters," yet (as I understand it) makes no calls backs, does not call cell-phones and makes no provision for bilingual interviewing.
Some, I am sure, will readily conclude that one or more of these characteristics (or perhaps others that I've omitted) provide "obvious" explanations for the discrepancies. I am reluctant to make too much of these differences. The reasons be clearer after we look at data from a third source. I obtained it earlier today from an anonymous but trusted pollster that I'll call "Polimatic." Here is a chart of the Polimatic's tracking data for the last six weeks:
Those who notice the greater stability in the Polimatic data as compared to Gallup and Rasmussen are on to something important. Next consider how the Clinton-minus-Obama margin from the Polimatic data compares to the other pollsters:
See some interesting patterns? Starting to form theories about what type of poll Polimatic is, or how their methodology might influence their results?
Well, before you go too far, I should fess up. I fibbed. "Polimatic" is not a pollster at all. The data are based on a simulation run by our friend Mark Lindeman. Mark created a spreadsheet that generates random results consistent with a thee-day rolling average tracking sample of 1,26040 interviews and the assumption that the "true" population value remains an unchanging 46% to 45% Obama lead.
The Polimatic line is more stable, suggesting that the consistently highest highs and lowest lows of the blue and red lines probably represent real divergence. However, the purely random variation of the simulated poll trend line is frequently hard to distinguish from the real surveys.
To generate the results above, I closed my eyes and clicked the mouse to let the spreadsheet recalculate. As such, the "Polimatic" line illustrates one potential trend showing nothing but random noise around a 46% to 45% margin. I'll say it one more time to be clear: All of the variation in the Pollmatic trend lines is based on purely random chance. Any resemblance to real changes as measured by Gallup or Rasmussen is entirely coincidental.
So what can we conclude from all this?
First, there has been far more stability than change in the national Obama-Clinton vote preference since Super Tuesday, and that includes the period of last ten days. To the extent that we have seen real changes, they are barely bigger than what we might expect by chance alone.
Second, if you look closely, you will notice that the seemingly odd divergence between Gallup and Rasmussen since the Wright story broke is really not that unusual. It is comparable to similar separations in the trend lines that occurred around February 13 and February 29. Random variation will do that.
Third, and probably most important, it is far too easy to look at these rolling average tracking surveys and see compelling narratives and spin interesting theories from what is often little more than random noise.
PS: Yes, as a few readers have already suggested in prior comments, some of the stability in national Democratic vote preference may stem from the fact that most states have already held their primaries and caucuses. We had some discussion about a month ago about how Gallup alters its screen slightly to accommodate states that have already voted. However, neither Gallup nor Rasmussen alters their vote question for those who have already voted. Here is the text used by each:
Gallup: Which of these candidates would you be most likely to support for the Democratic nomination for president in 2008, or would you support someone else? [ROTATED: New York Senator, Hillary Clinton; Former Alaska Senator, Mike Gravel; Illinois Senator, Barack Obama]
Rasmussen: If the Democratic Presidential Primary were held in your state today, would you vote for Hillary Clinton or Barack Obama? [options are rotated]
PPS: While I was writing this post, Mickey Kaus blogged a theory for the divergent Gallup and Rasmussen trend lines:
The 'Bradley Effect' is Back? Gallup's national tracking poll has Obama retaking the lead over Hillary after bottoming out on the day of his big race speech. Rasmussen's robo-poll, on the other hand, shows Obama losing ground since last Tuesday. True, even Rasmussen doesn't seem to be putting a lot of emphasis on his survey's 6-point shift. But isn't this week's primary race exactly the sort of environment--i.e.., the issue of race is in the air--when robo-polling is supposed to have an advantage over the conventional human telephone polling used by Gallup? Voters wary of looking like bigots to a live operator--'and why didn't you like Obama's plea for mutual for understanding that all the editorial pages liked?'--might lie about their opinions, a phenomenon known as the Bradley Effect. But they might be more willing to tell the truth to a machine. ...
Or more likely, the apparent differences between are about random variation in one or both polls. If you average the results from data collected since March 14 (the day the Wright story exploded) they are not very different:
- Live Interviewer Gallup Daily: Clinton +2 (47% to 45)
- Automated Rasmussen Reports: Obama +1 (45% to 44%)
Kaus also links to an automated PPP survey in North Carolina that fielded on the evening of March 17, the night before the Obama speech. As such, it is consistent with Gallup's "bottoming out" for Obama, not contradictory. The SurveyUSA results I blogged about on Friday were also collected from March 14 to March 16, just after the Wright story broke but before Obama's speech.
By Mark Blumenthal on March 25, 2008 1:35 AM
| Permalink
| Comments (22)
March 7, 2008
By Mark Blumenthal
Here is a quick review of a few items of interest I neglected to link to in the aftermath of this weeks' Junior Tuesday primaries.
First, SurveyUSA has posted report cards comparing pollster performances in four contests: Ohio Democrats, Ohio Republicans, Texas Democrats and Texas Republicans. One suggestion for this already helpful feature would be some indication of whether the results for each pollster fall within the range of random sampling error of the actual result. In other words, in theory, even when polls are as right as they can be about an election result they are still subject to random sampling error. As such, a poll that is "right" should capture the actual result within its "margin of error." If all polls are "right" then the ranking of best to worst is a matter of chance.
Yes, when we compile these scores over many different elections, those random factors should cancel out, but when focusing on an individual race, it would be useful to see, at a glance, which polls captured reality within random sampling error and which did not.
Second, here's an example of the lengths pollsters will go to in trumpeting their successes. The Boston Herald's Marie Szaniszlo sent a bouquet to Suffolk University pollster David Paleologos in the form of a short piece puffing his "knack for calling races." The evidence?: A poll in Ohio that "came decidedly closer to the mark" than a survey by "polling giant Zogby," and a New Hampshire poll that showed "Obama winning by 5 compared with Zogby, which showed him leading by 13."
Adam Lewis of the Boston Phoenix responded with a blog entry calling the piece "a bit of a debacle," pointing out that Clinton, not Obama, won the New Hampshire primary, and noting several other instances this spring (the New Hampshire Republican primary and the Massachusetts and California Democratic primaries) in which the Suffolk poll had been far off the mark compared to other pollsters. His final point:
In her lede, Szaniszlo sets up a David-and-Goliath narrative, with "polling giant Zogby" showing Clinton and Obama tied just before Ohio's Democratic primary and "a small polling center based at Suffolk University" putting Clinton ahead 52-40. Clinton won by ten points. Good for Paleologos, but good for a few other pollsters, too, all of whom go unmentioned.
True, but one more thing to consider: That Ohio poll by Suffolk also forecast a Democratic electorate that was 8% African American and 38% age 65 or older. The reality, according to the exit poll, was 18% African Americana and 14% 65+. None of the other pollsters -- save for the Columbus Dispatch mail-in poll sample that interviewed only previous primary voters -- came any where close to that demographic mix.
Finally, Wall Street Journal Numbers Guy Carl Bialik blogged on Tuesday about the challenges we all face telling good polls from bad focusing on the decision by our friends at RealClearPolitics to drop polls from the American Research Group from their averages. Here is a quote from our own Charles Franklin summing up our desire to report all polls that at least claim to provide a representative sampling of "likely voters:"
“Lots of pollsters have shown volatility, not just ARG.” Prof. Franklin added, “The inclusion or exclusion of a pollster runs the perils of cherry-picking polls, something we’ve tried not to do.”
But because of another difference between Real Clear Politics and Pollster, American Research’s numbers aren’t having much of an impact; the two poll aggregators basically agree that Sen. Clinton is ahead by six or seven points in Ohio, and by two points in Texas. That’s because Pollster’s method “has always discounted the effects of outliers — the more dramatically out of line a poll is, the less weight it gets,” Prof. Franklin said.
Those looking for more details on our method might want to review this post Charles did back in August.
By Mark Blumenthal on March 7, 2008 10:36 AM
| Permalink
| Comments (1)
