The two-question sequence on health care reform asked by the highly regarded NBC News/Wall Street Journal poll -- slightly updated since I wrote about it in my column two weeks ago -- has been getting a fair amount of attention.. And rightly so. Although I've been arguing for a long time that, our chart aside, it's a bad idea to try to boil all of public opinion on health reform down to a single measure, these two questions do a good job of getting at what people know and how they react when asked about "Barack Obama's health care plan."
They begin with a question that they have tracked since April:
From what you have heard about Barack Obama's health care plan, do you think his plan is a good idea or a bad idea? If you do not have an opinion either way, please just say so.
On the most recent survey, just 36% consider the plan a good idea (up from 31% in late January), 48% consider it a bad idea (up from 46%) and 16% either have no opinion or are unsure (down from 23%). This question, which prompts to say when they have "no opinion," also shows a slow steady decline over the past year in the percentage without an opinion, from 41% last April to just 16% now.
Then they follow up:
Do you think it would be better to pass Barack Obama's health care plan and make its changes to the health care system or to not pass this plan and keep the current health care system?
Here opinion divides evenly: 46% say pass and change, 45% do not pass and keep the current system, with the rest volunteering a response of "neither" (4%) or unsure (5%).
What is even more interesting is the pattern of the result when tabulated by party identification. The tabulations below (kindly provided by the NBC/WSJ pollsters) includes party "leaners" among the partisans, so the independent group represents the 15% of adults on their most recent survey that think of themselves as "strictly independent."
Republicans are solidly, consistently opposed to Obama's health reform bill. Four out of five (83%) think its a bad idea -- a number that has not changed since January -- and almost as many (79%) would rather not pass the plan and keep the status quo.
While a majority of Democrats favor the legislation, we see an 11-point gap between the number who think it's a good idea (64%) and those who prefer to pass the bill (75%). Among independents the gap is 19 points: Only 26% are convinced the bill is a good idea, but far more want to pass the bill and change the system (45%).
So while Republicans are uniformly opposed, many Democrats have doubts, even those who prefer to see the bill pass than to do nothing. For some, these doubts are about the lack of a public option or too much compromise, but for others, the doubts stem from their perceptions (right or wrong) about the bill's cost or the increased role of government (for more, see Nate's Silver's word clouds of the very helpful Gallup open-ended data).
The slight increase in support for reform measured by most surveys in recent weeks (the just released Pew Research Center poll being an apparent exception) comes mostly from Democrats. That pattern makes perfect sense, since the intramural disagreements among Democratic leaders have faded considerably in recent weeks. Consider Glenn Greenwald's summary:
For almost a full year, scores of progressive House members vowed -- publicly and unequivocally -- that they would never support a health care bill without a robust public option...Up until a few weeks ago, many progressive opinion leaders -- such as Moulitsas, Howard Dean, Keith Olbermann and many others -- were insisting that the Senate bill was worse than the status quo and should be defeated. But now? All of those progressives House members are doing exactly what they swore they would never do -- vote for a health care bill with no public option -- and virtually every progressive opinion leader is not only now supportive of the bill, but vehemently so.
The important point: Neither of these poll questions gets at the whole of public opinion on health care reform nor provides even a complete picture of the general impressions of the legislation. If you focus exclusively on the "good idea/bad idea" question (which, incidentally, now matches almost perfectly the Pew Research "favor or oppose" results released today), you miss that three quarters of Democrats and nearly half of independents prefer to move forward with this bill than remain locked in the status quo.
But if you focus exclusively on the pass-and-change/don't-pass-don't-change question, you miss the big doubts expressed by the vast majority of true independents and nearly a third of Democrats, and the huge gap in intensity of opinion on this subject that separates Republicans and Democrats.
Finally, I received two emails yesterday taking us to task for including the first NBC/WSJ question in our chart but "completely ignoring" the second. Here's one:
Right now, you're using the "good idea/bad idea" numbers from the NBC/WSJ poll today. However, the name of the graph on your website is "Favor" or "Oppose". If someone favors something, they would say pass it.
You should be using this metric or both metrics - not the "good idea/bad idea" metric by itself, which does not fit whatsover into your chart.
Our chart admittedly flaunts a bit of polling orthodoxy by combining results from different questions using different language and response categories. The more traditional approach would stop at the sort of apples-to-apples comparisons plotted in my post yesterday. So reasonable people will likely disagree with the questions we have chosen to include or exclude on the chart. If we dropped the good idea/bad idea result from the most recent NBC/WSJ poll, and replaced it with the pass & change/don't pass-don't change result, our overall trend estimates would narrow slightly (from 43.4% favor, 48.9% oppose to 44.7% favor, 48.5% oppose).
But I disagree with the argument that the second NBC/WSJ question is obviously closer to the standard "favor or oppose" question asked by other pollsters. It does ask if the respondent wants to pass the bill, which is straightforward, but it also frames the question in terms of change versus the status quo. How many Republican leaders have you heard state that they oppose the Democratic plan because they want to "keep the current health care system" as it is now?
Moreover, the most important purpose of the chart to track trends apparent across multiple polls, not to somehow magically derive the true levels of support and opposition from multiple polls. The NBC/WSJ poll has tracked their good idea/bad idea formulation for almost a year. Abruptly switching introduces some discontinuity. For better or worse, we will stick with this measure for NBC/WSJ unless and until they start tracking something else.
When reporters asked Rep. Dennis Kucinich (D-Ohio) today about whether his intention to vote for the health reform bill might convince other doubting Democrats, he answered: "If I can vote for this bill, there's not many people who shouldn't be able to support it."
Kucinich's decision had no effect on the polls conducted earlier this week, and I do not expect his announcement alone to significantly impact public opinion. However, as PPP's Tom Jensen put it earlier today, Kucinich's "flip...is symbolic of a broader shift" that has occurred in recent weeks as support for health reform legislation has increased among liberals and Democrats. The often vigorous debate among Democratic leaders in evidence for much of the last year has faded, and the Democratic and liberal rank-and-file are slightly more likely to express support. If the bill passes, that trend may add a few percentage points of support in the coming weeks.
First, let's look at the current snapshot of the overall trend in support for health reform as measured by our trend chart (as of this writing - click to see the automatically updated, interactive version).
Now let's explode that trend a bit and consider a second chart showing apples-to-apples comparisons -- trend lines for pollsters that tracked opinion using questions whose wording did not vary over time. Six of the seven pollsters show nominal increases in support for health care reform legislation.** While only a few tracked during January, the results from all pollsters are mostly consistent with a slight decrease in support following the Massachusetts Senate election followed by a rebound during February and early January. Our trend estimate, which is based on all of the health care favor-or-oppose results and not just those plotted below, shows support increasing from a low of 40.6% on January 26 to 43.9% as of this writing.
Now consider the pattern in the opposition percentage. Four organizations (Economist/YouGov, Rasmussen, McClatchy/IPSOS and PPP) show nominal declines in opposition since mid-January, and three (Kaiser Family Foundation, AP-GfK and NBC/Wall Street Journal) show nominal increases. Our trend line shows a decline (from 52.3% to 48.8% since late January), since the organizations showing declines have polled more frequently.
It may be just a coincidence, but the nominal declines have also all occurred for organizations whose measures tend to show more opposition, while the nominal increases have occurred among those that typically report smaller opposition percentages.
Nonetheless, the overall point is that while the level of support for health reform varies widely depending on how pollsters ask the question, the trends are reasonably consistent: Most organizations have tracked modest increases in support for health reform.
Also, on some of the most recently released surveys, the increases in support have been larger for Democrats than Republicans.
- The NBC/Wall Street Journal poll shows an eight point increase in the number of Democrats saying the health reform bill is a "good idea" since late January (from 56% to 64%) compared to just a four point increase for Republicans (from 6% to 10%) .
- PPP shows a 13-point increase in support for reform among Democrats since mid-February (from 63% to 76%), but only a one point increase among Republicans (from 10% to 11%).
- PPP also shows a 16-point gain in support among liberals (from 73% to 89%), compared to a 4 point drop among conservatives (12% to 8%).
- The most recent Economist/YouGov poll shows a seven point gain in support among Democrats since late January (from 73% to 80%), while Republicans showed no change (11%).***
So the recent increases in support for reform appear to be coming mostly from Democrats. As skeptical progressive like Dennis Kucinich offer their votes for the health reform bill, that trend could continue.
**The Economist/YouGov time series omits the poll conducted in early march (2/28-3/2) that used a slightly health reform question. See my post from last week for more details.
***Thanks to Hart Research and the NBC/Wall Street Journal poll for sharing results by party.
[Note (10:17 pm): This post originally displayed an inadvertently uploaded older version of the Percent Oppose chart that omitted the new polls released this week. Apologies for the oversight].
Last week, I linked to an article on Dr. George Gallup that included a reaction from statistical consultant Dominic Lusinchi that had been posted on the members-only listserv mailing list of the American Association for Public Opinion Research (AAPOR). Lusinchi argued that the original article perpetuated two "myths" about the infamous Literary Digest poll of 1936 -- the poll whose failure helped establish competitor Gallup as a household name. No sooner had I reproduced Lusinchi's email than Jan Werner, another knowledgeable AAPOR member took to the listserv to quarrel with Lusinchi's version of the history.
While a discussion of the shortcomings of a 74-year-old poll may seem a little out of place here, it involves issues that remain highly relevant to the contemporary debates about modern surveys, including the merits of probability sampling and the errors that can result from poor coverage or a low response rate. Since I posted Lusinchi's original comments, I also want to share the full exchange, which begins after the jump. Thanks to Dominic and Jan for allowing me to reproduce their comments here.
Original comment from Dominic Lusinchi:
This story perpetuates two myths:
1) That Gallup "predicted" that the Digest poll would forecast a Landon victory and
2) That the Digest failed because its sampling frame was "skewed ... to the wealthy".
Myth 1: In a July 12, 1936 syndicated column "America Speaks", Gallup wrote:"If the Literary Digest were conducting its poll at the present time [my emphasis], following its usual procedure, Landon would be shown in the lead." (Wash. Post, Section III, p.2, col. 7, Sunday, July 12, 1936) It's one thing to say "at the present time" and another to say "when the Digest presents its final results".... It is only after the Digest poll debacle that this story morphed into a "prediction". What Gallup really predicted, at that time (7/12/1936), was that the election was going to be a close one: the title of his column "1936 Election Seen As Closest in Years".
Myth 2: The Digest poll failed because its original sample, composed mainly of telephone and/or car owners, was irretrievably skewed against Roosevelt. A close analysis of a May 1937 Gallup (yes, Gallup!) poll, which asked its respondents if they had received and returned a Digest ballot card, shows that the principal cause of the Digest poll's failure was non-response bias. As Peverill Squire wrote in POQ (vol. 52, 1988, p.125), "if all those who were polled had responded, the magazine would have, at least, correctly predicted Roosevelt the winner." In fact, its prediction (my analysis) would have been as good if not better than Gallup's - he was off by nearly 7 points of the two-party vote.
Jan Werner:
It's one thing to debunk a myth, it's something quite different to replace one myth with another.
Myth 1
What Gallup wrote in the July 12 1936 article was: "...if The Literary Digest were conducting a poll at the present time...the actual figures would be in the neighborhood of 44 per cent for Roosevelt and 56 per cent for Landon." Although it was presented as a throw-away comment, Gallup did not come up with those numbers lightly. As he explained to a journalist a few years later, he "was able to call [The Literary Digest's] shot by making a separate survey of telephone subscribers and automobile owners -- the class which would mail back most of the Digest's ballots." (Williston Rich, Jr., "The Human Yardstick," Saturday Evening Post, January 21, 1939)
The statement was clearly intended to provoke a response from The Literary Digest and thus generate publicity for Gallup's fledgling
organization. In that regard, he succeeded. On July 19, 1936, the New York Times quoted an open letter from Wilfred J. Funk, editor of The Literary Digest, as follows: "I am beginning to wish ... that the esteemed Dr. Gallup would confine his political crystal-gazing to the offices of the American Institute of Public Opinion and leave our Literary Digest and its figures politely and completely alone...We've been through many poll battles...We've been buffeted by the gales of claims and counter-claims. But never before has any one foretold what our poll was going to show before it was even started."
Thus, even if Gallup's claim might not technically be described today as a prediction, it was certainly taken as such at the time by his target, and was probably meant to be.
Myth 2
Peverill Squire did indeed write in POQ (vol. 52, 1988, p.125), "if all those who were polled had responded, the magazine would have, at least, correctly predicted Roosevelt the winner." However, he then goes on to say: "But more importantly, the initial sample was flawed; when compounded with the response bias, it produced the wildly erroneous forecast of the vote percentages." and then states that "a rough calculation of the bias produced by the sample is around 11%, with another 7% accounted for by problems with the responses." Unfortunately, there is no reason to trust the data from the May 1937 Gallup poll used to estimate the nonresponse rate (as Squire himself admits, before going on to do so). There certainly is nothing to justify this kind of precision in divvying up error among specific causes.
The fact is that The Literary Digest did not define a sampling frame and did not conduct anything resembling a representative sample, random or otherwise, of the voting population, nor did they make much of an effort to keep track of what was sent out and what was returned. It's convenient to use the spectacular failure of their straw poll as a cautionary tale, but without any real documentation of what they did, or data to analyze, any attribution of specific causality is little more than speculation.
Dominic Lusinchi:
Jan,
Thanks for your response. Here are my comments.
Your statement to the effect that "It's one thing to debunk a myth, it's something quite different to replace one myth with another" is an inaccurate characterization of what I am saying. I am not replacing one myth by another as you seem to suggest - if I am: what myth is it that I am promoting?
Myth 1
The fact that the protagonists (Funk and Gallup) believe that what Gallup said in his July 12 column is a prediction does not make it a prediction. I repeat that what Gallup was really predicting was a close race: so says the title of his column (which I hope you received). The Digest editors came to believe, and were encouraged to do so by all the praise they received regarding its "uncanny accuracy" (10/31/1936, 6), that their poll was a "forecasting machine" (8/22/1936, 3), when in reality they had no idea how their poll results were produced - in other words, the belief does not make it so.
In any case, this (the so-called "prediction") was used as a great promotional story after the Digest fiasco to show and tell: show that the new "scientific" polling was the wave of the future and tell that the Digest "straw" polling was a thing of the past. For example, Crossley wrote: "The Institute [Gallup] even went so far as to forecast the DIGEST vote, which it did with remarkable accuracy by the simple means of tabulating separately that part of its ballots which followed the DIGEST's general basis." ("Straw Polls in 1936", Archibald M. Crossley, The Public Opinion Quarterly, Vol. 1, No. 1. (Jan., 1937), p.29) The moral of the story: the "scientific" polls are superior because not only can they predict correctly the election but they can also predict what the Digest would forecast.
I agree with you when you say it was "presented" as a "throw-away comment"; but it really was a very cleverly worded statement. I also agree when you say that it was meant to "generate publicity for Gallup's fledgling organization." Think about it: if I (Gallup) am wrong, I can always say (plausible deniability) that's how things (the presidential race) were at that time (July); if I'm right, I can present it as a prediction (of the results the Digest will publish right before the election) and thereby trumpet the superiority of my "scientific" polling methods. The statement has to be analyzed within an overall strategy to dominate the opinion
polling domain.
Gallup and his fellow "scientific" pollsters were not disinterested parties searching for the cause of the failure of the DIGEST poll out of some academic interest. They wanted to be top dogs and knock the Digest off its pedestal. They used the Digest incident, especially in the early years, to promote what they considered to be a superior "product": their "scientific" polling. Of course, they were to receive a nasty shock in 1948 - although not fatal to them, as 1936 was for the Digest.
Myth 2
I had the opportunity, thanks to the Roper Center, to have access to the raw data from the May 1937 AIPO (Gallup) poll. First the data in its raw form requires some editing because the file contains some anomalies. For example, you might have noticed that in Squire's table 2 (p.130) there are 780 respondents that claim to have received the Digest ballot, while in table 3 (p.131), which reports what they did with the ballot (returned or not or don't know), there are 829 respondents! Squire analyzed the data at face value.
Second, Squire did not weight the data despite the fact that we have information that allows us to do so: each candidate's share of the actual vote, the response rate to the Digest (~24%), and the percent each candidate received from respondents to the poll. This takes care of anomalies, pointed out by Squire, such as over-report of support for FDR, over-sampling of Digest poll respondents, and over-report of support for FDR among respondents.
The weighted results confirm much of Squire's conclusions: had the Digest relied solely on telephone and car owners it would have forecast a Roosevelt victory; Digest non-respondents were strongly in favor of FDR; and, last but not least, had all those who were polled by the Digest responded, the magazine would have pointed to the correct candidate. In other words, yes the Digest original sample of 10M was biased but not sufficiently to have prevented it from calling a Roosevelt victory. In fact, a less "rough
calculation" (Squire, p.131) shows that non-response bias was the main culprit.
Ironically, the samples Gallup used throughout the 30s and 40s were also biased - in favor of Republicans. And he was called on that (e.g. Special House Committee Investigating Campaign Expenditures, 1944).
The conclusions based on the May 1937 poll (e.g. non-response as the main cause of the 1936 Digest poll failure) are consistent with other available data. I have re-analyzed the data provided by Cahalan in his Cedar Rapids study in 1936-7 reported in his 1989 POQ (vol. 53 pp.129-133) and in 1939 Psychological Record (vol.1, no.1, pp.3-11) papers. The results show that the Digest list used for that city was not biased and that respondents and non-respondents were very different in their candidate preference. Add to that the results from Allentown, Scranton and Chicago where only registered voters were polled (Digest, 14 November, 1936, p.7) and I would say we have some pretty solid "real documentation".
Does this amount to a definitive answer? No, absolutely not. The only way we could have resolved the question would have been to conduct, on the original sample of 10M, the same type of study that Cahalan performed in Cedar Rapids: select a random sample from the Digest list and ask respondents whether or not they sent in their Digest ballot and which candidate they favored at that time.
Short of that the May 1937 AIPO survey, the Cahalan study and the results from Allentown, Scranton and Chicago is the best we have. Speculation you say? Well, yes, I suppose you're right, but a lot less than simply repeating the old refrain repeated by so many: "The failure of the Literary Digest's polling approach can be explained simply. The Digest's sample of voters was drawn from lists of automobile and telephones owners." (Gallup, 1972, "Opinion Polling in a Democracy", in Statistics: A Guide to the Unknown, Judith M. Tanur et al. (eds.), San Francisco, Holden-Day, p.147.)
Jan Werner:
Some really good stuff here, but, as you suggested in an earlier (off-list) message, we are probably going to have to agree to disagree
-- My reaction is that, if anything, your additional material supports my previously expressed opinions.
So, let me clarify what I mean by replacing one set of myths with another.
Myth 1:
Original myth: Gallup predicted the Literary Digest straw poll results.
Revised myth: Gallup did not predict anything.
My take: Gallup presented what he knew full well would be interpreted as a prediction, although he worded it so as to provide him with an escape if he were wrong. Given what we know now (see Myth 2) he was very lucky to come as close as he did. Whether or not that constitutes a prediction is a semantic judgment.
Myth 2:
Original myth: The Literary Digest poll failed because it did not use a representative sample.
Revised myth: It failed because of nonresponse bias (as shown by the May 1937 Gallup data, and other sources).
My take: The Literary Digest polling procedures were a complete mess. Evidence from other sources does show that an unrepresentative sample was not the sole cause, but is not strong enough to prove anything beyond that. In particular, I don't see the May 1937 Gallup data as being reliable enough to justify giving either bad sampling or nonresponse bias pride of place, so to speak, in assigning blame.
None of this addresses what I consider to be the worst myth about the Literary Digest straw poll failure, namely that it was the reason
pollsters began using probability samples.
Dominic Lusinchi:
Well, Jan, if you consider these two statements semantically equivalent:
"If the Literary Digest were conducting its poll at the present time, following its usual procedure, Landon would be shown in the lead." - "When all the results are in the Digest poll will show Landon in the lead."
then I can't much argue about that - we've reached a dead-end.
You say that you don't consider "the May 1937 Gallup data as being reliable enough" to say anything about what caused the 1936 Digest poll to fail so miserably. I agree (along with Squire) that the Gallup survey is less than perfect. But I disagree that it cannot be used for the problem at hand. Gallup polls of the 30s and 40s have been put to good use by researchers: e.g., Baum & Kernell "Economic Class and Popular Support for Franklin Roosevelt in War and Peace," Public Opinion Quarterly 65, 2001, 198-229; and Berinsky's 2006 discussion "American Public Opinion in the 1930s and 1940s: The Analysis of Quota-Controlled Sample Survey Data," Public Opinion
Quarterly 70: 499-529. The polls have their limitations and the data have to be analyzed with that in mind.
The "conventional explanation", as it has been called, that the 1936 Digest poll failed because it was biased in favor of Landon due to the fact that it relied mainly on lists of phone owners and car owners, is pure speculation, plausible speculation perhaps, but speculation nonetheless. In contrast, the only real evidence we have, although it is limited and has flaws (the results from Allentown, Scranton, Chicago, Cedar Rapids and the Gallup data) all points to the same problem: non-response bias. My preference in describing what happened to the 1936 Digest poll leans towards the latter (warts and all). It is, of course, a tentative explanation but it has the advantage of being backed by data, however imperfect.
As for your last comment about the belief that the Literary Digest poll failure being the reason why pollsters began using probability samples is, of course, nonsense, so I agree with you on that - but since the issue was not part of the original post, I'll leave it at that.
Jan Werner:
The late John Gorman, who founded Opinion Dynamics, liked to mention a specific date in the late 1970's, which he estimated was the day on which George McGovern edged out Richard Nixon, based on trending how respondents said they voted in the 1972 presidential election in polls conducted during the years following the Watergate affair. John's estimate was also based on data, however imperfect.
As for what is, or isn't, a prediction, I'll let it go at that. Others can make up their own minds as to what Gallup intended when he wrote what he did in 1936, and what it should be called.
Regardless of whether or not we agree, I'd like to thank you for an enlightening discussion and much valuable information. Let me note that the image you sent me of the 7-12-1936 Washington Post article does not include the beginning of the article on page 1, and there is no way to tell whether the headline "1936 Election Seen As Closest in Years" was written by Gallup or by the Washington Post's editor.
Dominic Lusinchi:
Thank you, Jan, for your insights; always glad to discuss what I see as the defining moment in the genesis of modern polling.
Aside from our difference regarding the semantics of what constitutes a prediction, the perception is just as important: i.e. that the Digest editors saw it as such reveals a lot about them - specifically how convinced they were that their poll was an "uncanny" "forecasting machine".
[Typos corrected and updated to include the last exchange of comments that I omitted from the original post].
They're back. As reported yesterday by Politico, Strategic Vision, LLC posted results** from what they claim is a survey of Georgia. As per our previous entries on this subject (here and here), we will no longer publish their results as "poll updates" or in our poll charts. Yesterday's release does virtually nothing to answer of questions raised by well over 200 purported surveys released by Strategic Vision since 2004. It also falls well short of the minimal standards of disclosure that got the company into trouble in the first place.
For the uninitiated, the saga began with a rare censure by the American Association for Public Opinion Research (AAPOR) last fall resulting from Strategic Vision's failure to comply with requests for information about their response rates and weighting procedures -- information that 21 other organizations provided upon request in connection with AAPOR's investigation of the primary election polls of 2008.
Following AAPOR's action, blogger Nate Silver raised the possibility of fraud and subsequently found a pattern in the trailing digits of the percentages reported in Strategic Vision polls suggesting a "possibility of fraud." Michael Weissman, a retired professor of Physics at the University of Illinois and frequent commenter on Silver's site, did some additional number crunching (a Fourier analysis) and concluded that the odds were 1 in 5,000 that the pattern in Strategic Vision could have been produced by chance alone.
The issues raised by Silver and Weissman were highly technical and difficult for mathematical mortals to evaluate, but even more troubling was Strategic Vision's strange pattern of half-truths and evasion. Commenters on FiveThirtyEight discovered that the four offices listed on the Strategic Vision web site were UPS store mailboxes. In the wake of the initial stories, Strategic Vision CEO David Johnson announced to at least five news organizations that he would soon take legal action against AAPOR and Silver. He promised to release additional subgroup tabulations of the contested data. None of this ever happened.
"We intend to vindicate ourselves," Johnson told Politico a few days after the AAPOR Censure. If his surveys were real, if they had been conducted by live interviewers at actual call centers, Johnson should have a wealth of evidence at his disposal to silence his critics. The public polls released by Strategic Vision since 2004 (archived here and here by Harry Enten) add up to more than 200,000 interviews. That many interviews would leave a lot of witnesses: Call center managers, supervisors, probably hundreds of interviewers, any one of which could come forward to vouch for the process that produced the numbers. And there should be electronic records of the actual survey data somewhere -- at least for the most recent projects. Why hasn't Strategic Vision taken any steps to present some of this evidence and vindicate themselves?
Yesterday, Johnson tried to Strategic Vision stopped releasing public polling in September and went dark in terms of public polling until this week. Here's what he told Politico:
[Johnson said] that the lull in his firm's work had represented a deliberate choice to take some time off in the light of the allegations and let the scrutiny subside. He also said a family illness prevented him from polling the Georgia gubernatorial race earlier in the year.
"Some of the stuff was getting to me. I felt it was best to take some time off," Johnson said. "You know the old adage - lawyers should never defend themselves. I should never try to be my own PR person."
He also told Atlanta Journal Constitution columnist Jim Galloway that his libel suit threats "was me speaking in anger because I was really outraged at the time."
Yesterday's release includes two new twists. For this first time, Strategic Vision sent Galloway and other reporters a set of tables in a compressed file showing results tabulated by gender, age, race and income, and all of their percentages are computed to one decimal point. Is it more likely that these results are based on some sort of interview data, for what that's worth.
Mark Grebner, a Michigan-based, Democratic political consultant, left this comment on Pollster.com last month when Johnson started promising new surveys:
I've got a counter-intuitive guess: maybe SV-LLC will start doing real polling. It's not hard, it doesn't cost much money, and it would serve more than one purpose.
One thing is that it would restore their status as serious participants in conservative politics. A second benefit is that it would undermine the case against them, at least in the public's mind.
Maybe they'll never release another result, but if they do, I'd guess it would be genuine.
Doesn't affect the utter bogusness of everything they've done to date, of course.
He's right that new surveys, even if "genuine," do nothing to resolve the serious questions raised about Strategic Vision's previous work.
But let's ponder the meaning of "genuine" as we consider what Strategic Vision's latest release does not tell us: They say nothing about the mode of the survey (whether it used live interviewers or some automated method), the sample frame (whether telephone numbers were selected from some sort of list or via a random digit method), the weighting procedure (whether results were weighted and the variables used to weight them), and they do not identify of who conducted the survey (the call center or field-work provider, if these used one). These basic facts are part of the minimal disclosure requirements of both AAPOR and the National Council on Public Polls (NCPP).
NCPP also requires that its members describe the "size and description of the subsample, if the survey report relies primarily on less than the total sample." It is not clear whether NCPP's mandate applies to cross-tabulations, but it is very clear that Strategic Vision tables provide no information about the size of each demographic subgroup.
Both organizations also mandate that releases tell us, "who paid for the poll?" Strategic Vision's release says nothing about how this poll was paid for and, as an alert Pollster reader informs me, fails to disclose a significant conflict of interest: A search of Georgia campaign finance records shows that Strategic Vision was paid $3,500 to conduct a poll in 2009 for Ralph Hudgens, a candidate in the Republican primary contest for Insurance Commissioner tested in the new survey.
So again, for all of these reasons, we will no longer publish results by Strategic Vision, LLC on Pollster.com. But that raises a much bigger problem: Strategic Vision is not the only polling organization that has fallen far short of the minimal disclosure requirements of organizations like AAPOR and NCPP, and their results do appear on Pollster.com. That shortcoming is something I want to discuss at greater length this week. Stay tuned.
**For what it's worth: That link and the rest of the Strategic Vision, LLC web site remains inaccessible to computers in our offices and to our colleagues at the National Journal Group and Atlantic Media.
This week's column looks at the U.S. Census and the controversy over its use of television advertising and advance letters to convince Americans to fill out and return their census forms.
The column reviews how the census differs from a survey -- the census has a constitutional mandate is to to count every member of the population -- but does not make explicit what they have in common: Both involve questionnaires that both need to coax their respondents to complete. As such, the advance letters are supported by decades of methodological research proving that their worth, something the Washington Post's Jennifer Agiesta reviewed last week.
In case you missed it, last night's Colbert Report included an interview with pollster Scott Rasmussen that began with an extended metaphor on polling as a "Truth Grinder" and Colbert's own proven-to-be "scientific" online poll. Regular readers will want to watch it all.
The full "results" of the Colbert Repoll are posted here (via Alex Lundry).
Just to shake things up a bit, here's a post on a polling controversy from 1936.
Earlier this week, Investor's Business Daily ran a fascinating biographical profile of Dr. George Gallup, the founder of the Gallup poll and, for all practical purposes, the founder of political polling as we know it. The article includes some details that I had not heard before, such as the fact that Gallup's first application of market research to political campaigns was on behalf of his mother-in-law's successful campaign for secretary of state in Iowa. It is well worth a click.
That said, I want pass along some interesting commentary about the story posted earlier today on the members-only listserv of the American Association
for Public Opinion Research (and quoted with permission). The article opens by revisiting Gallup's bold prediction that Franklin Roosevelt would win reelection in 1936 in the face of well known polling by the Literary Digest magazine showing a big lead for Republican Alf Landon. The IBD story is correct that Roosevelt's ultimate victory "led to the death of the Literary Digest" and helped make Gallup "a household name." According to statistical consultant Dominic Lusinchi, however, the story "perpetuates two myths" about the infamous Literary Digest polls:
1) That Gallup "predicted" that the Digest poll would forecast a Landon victory and
2) That the Digest failed because its sampling frame was "skewed ... to the wealthy".
Myth 1: In a July 12, 1936 syndicated column "America Speaks", Gallup wrote:"If the Literary Digest were conducting its poll at the present time [my emphasis], following its usual procedure, Landon would be shown in the lead." (Wash. Post, Section III, p.2, col. 7, Sunday, July 12, 1936) It's one thing to say "at the present time" and another to say "when the Digest presents its final results".... It is only after the Digest poll debacle that this story morphed into a "prediction". What Gallup really predicted, at that time (7/12/1936), was that the election was going to be a close one: the title of his column "1936 Election Seen As Closest in Years".
Myth 2: The Digest poll failed because its original sample, composed mainly of telephone and/or car owners, was irretrievably skewed against Roosevelt. A close analysis of a May 1937 Gallup (yes, Gallup!) poll, which asked its respondents if they had received and returned a Digest ballot card, shows that the principal cause of the Digest poll's failure was non-response bias. As Peverill Squire wrote in POQ (vol. 52, 1988, p.125), "if all those who were polled had responded, the magazine would have, at least, correctly predicted Roosevelt the winner." In fact, its prediction (my analysis) would have been as good if not better than Gallup's - he was off by nearly 7 points of the two-party vote.
Why Gallup never referred to this May 1937 poll done by his organization when he commented (many many times) on the failure of the Digest poll...?
Well that would take too long... got to get back to work.
Thanks Dominic!
Update: These comments provoked a lengthy exchange with another knowledgeable AAPOR member who takes with Dominic Lusinchi's version of the history.
Our chart of the favor-or-oppose questions on health care reform has generated a fair amount of discussion this week. Both Chris Bowers and the analysts at Democracy Corps (the Democratic affiliated polling outfit) noticed a slightly tighter margin in recent weeks (support increasing and opposition decreasing), which in turn caught the attention of Jon Chait. Andrew Sullivan leaned heavily on our chart this morning in effort to refute a new Wall Street Journal op-ed by pollsters Scott Rasmussen and Doug Schoen about the "steady" nature of public opinion on health reform, which in turn drew a response from Megan McArdle taking issue with Sullivan's conclusions about which polls are "outliers" on health reform.
All of this commentary gets at two important questions: Is support for health reform growing (and opposition fading)? And do the large "house effects" among pollsters obscure our ability to see trends amidst the noise?
The version reproduced above is a snapshot of our chart as of this writing (click here to see the regularly updated, interactive version). Remember, our chart is something of a mash-up that combines different questions and surveys produced by more than 20 different pollsters. In it, we do something that many pollsters and statisticians advise against, which is to compare apples and oranges in terms of the question text and populations sampled. When we look at horse-race results for election campaigns, most pollsters use very similar questions and ultimately at least try to measure the same population (the likely electorate). In this case, the wording and format of the questions vary widely. Some sample all adults, others sample "likely voters." Look closely at the chart and you will see far more variation in the results than is typical for our horse-race charts -- between 10 and 20 points worth of variation in the favor and oppose percentages at any given time.
That variation also reflects the vague sense that many Americans have of the health care reform legislation now being debated in Congress. When the NBC/Wall Street Journal poll asks about health reform, they prompt respondents to say if they "do not have an opinion either way." As a result, roughly one-in-five adults (23% on their last survey) do not express an opinion. Other pollsters (such as ABC/Washington Post and Rasmussen) report an "unsure" percentage in the low single digits, while another (YouGov/Polimetrix) reports none at all. Thus, the degree to which pollsters push their respondents for an opinion explains some of the "house effect" variation.
All of this makes it prudent to take an apples-to-apples approach in pondering the recent trend. That's what I tried to do in the two charts that follow. I separated the lines the favor and oppose percentages into two charts to make them more legible. I also limited the plotted pollsters to the seven organizations that have updated health reform tracking over the last month. However, I also included the Pollster.com trend line from our interactive chart, which is based on all available polls, not just the seven whose dots are connected below.
Some observations:
1) The trend evident in the grey Pollster.com trend line -- a 4.4 percentage point drop in opposition and a 1.6 percent increase in support -- is more or less consistent with the trends shown by YouGov/Polimetrix and Rasmussen Reports, the two organizations that have polled most often on this topic during 2010. The results from PPP and IPSOS are also consistent with the same trend.
2) The results from Gallup's two polls appear to show a contradictory trend, although we should note that the Gallup changed their question wording slightly between January and March. Their most recent survey asks, "would you advise your representative in Congress to vote for or against a healthcare reform bill similar to the one proposed by President Obama?" In previous surveys, they asked about voting for "a health care bill this year" without reference to the President or either party. Note that Gallup's own analysis does not treat the January and March results as comparable.
3) Rasmussen shows a house effect on the oppose percentage (typically 5-6 point higher than our trend line; early January was an exception), but tends to be in the middle of the pack on the favor percentage. YouGov/Polimetrix shows a similar house effect on the favor percentage (typically 5-6 points higher than average), but not the oppose percentage. Whatever doubts you might have about their methods -- Rasmussen uses automated, recorded voice interviewing and YouGov/Polimetrix conducts online interviews sampled from an opt-in panel -- both are consistent in their respective questions and methods and both shown trends that generally track with those measured by other pollsters.
[Correction: The wording of the question asked on the most recent Economist/YouGov/Polimetrix survey, conducted February 28 to March 2, was slightly different from what they had asked before. Their previous surveys asked, "Overall, given what you know about them, do you support or oppose the proposed changes to the health care system being developed by Congress and the Obama Administration?" On their 2/28-3/2 survey, they dropped the reference to Congress and simply referred to the proposal as "being proposed by the Obama administration." That change could account for the spike in support to 53%].
4) We would still see a closing margin (increased support, falling opposition) if we use our charts filter tool to remove both Rasmussen and Polimetrix (as per the snapshot below). One reason may be the absence in recent weeks of surveys like Quinnipiac and NBC/Wall Street Journal (which typically report lower than average support percentages) and CNN (which typically reports higher than average oppose percentages). Notice how the range of dots is narrower over the last few weeks than in previous months. To be absolutely sure the trend is real, we will need to wait for updates from these organizations.
So yes, there are certainly large "house effects" in the health care favor-or-oppose results, but even though different pollsters gauge different levels of support, most pick up more or less the same trends, especially when they ask exactly the same questions on multiple surveys exactly the same way. Any way you slice it, there does appear to be a real tightening of opinion on health reform although as always, these results are snapshots and subject to change.
Update: Given the correction above regarding the small wording change on the most recent YouGov/Polimetrix survey, I thought it best to create two new charts that connect-the-dots for only those polls with consistent wording. So the two charts that follow drop the two Gallup surveys and the most recent YouGov/Polimetrix survey. The grey Pollster.com trend line, however, is still consistent with the line on our standard chart that is based on all surveys on this questions, regardless of wording. My bottom line remains the same: There does appear to be a small but real tightening of opinion.
On Sunday, the Associated Press published a lengthy report on a controversy brewing at the University of Wisconsin-Madison that involves some friends of Pollster.com.
What AP reporter Ryan Foley describes as a "fiasco" involves a year-old agreement between the University and the Wisconsin Policy Research Institute (WPRI), a conservative think-tank, to conduct statewide polls this year in partnership with the University. Under the agreement, WPRI would help fund statewide polling, including a $13,000 contract with UW political scientist Ken Goldstein. According to the AP report, however, the University never had a formal contract with WPRI. And then there are these details uncovered by a liberal activist:
Scot Ross, a liberal muckraker who runs the group One Wisconsin Now, was critical of the deal from the beginning. He said his "worst fears were confirmed" after he obtained e-mails under the open records law showing WPRI President George Lightbourn lobbied Goldstein to publicize results from one question in a way favorable to its agenda.
The question asked whether government funding should be used for school vouchers, which WPRI supports. A majority of residents statewide were opposed, but those surveyed from Milwaukee County were in favor.
Lightbourn wrote Goldstein he was concerned critics would portray the data as showing a lack of support for vouchers and asked for the Milwaukee County results to be emphasized. The university's press release read: "School choice remains popular in Milwaukee."
The AP story -- which is well worth reading in full -- includes complete details plus a reaction from Goldstein who says he is "stunned, flabbergasted, amazed -- every single adjective you can come up with" as the criticism he has received.
Our own interests in this story are as follows: Pollster.com co-creator and contributor Charles Franklin is a member of the UW-Madison political science department and a friend and colleague of Goldstein but, he tells me, was not personally involved in the WPRI polling. Also, well before the WPRI polling project, my assistant Emily Swanson worked for Goldstein as an undergraduate at UW-Madison.
If nothing else, this episode demonstrates the increasing difficulty consumers of polling data have in identifying potential conflicts in the sponsorship and funding of public polling. Simply identifying polls sponsored by a political campaign or political action committee or conducted by a campaign pollster -- something we try to do on Pollster.com -- is obviously not enough. In this case, a University of Wisconsin news release billed WPRI as a "non-partisan, non-profit think tank [that] has been conducting independent, annual polls on politics and issues for more than 20 years." Yet the Institute acknowledged to AP what their report characterized as a "free-market, limited government slant and receives funding from the Bradley Foundation, a Milwaukee group that supports numerous conservative causes."
As promised, my column for this week considers where general perceptions of health care reform currently stand, and whether they might improve should congress finally enact the legislation into law in the coming weeks.
Read it all for details, but the short version: While it is a long shot that passage will dispel many of the common misperceptions, it might not take much of a shift in the polling numbers to change the way political insiders perceive the legislation's popularity. Think of it this way: Many opponents of health reform point to our chart of general favor-or-oppose questions to argue that the legislation is "deeply unpopular," wisdom now so conventional it was the premise of a Saturday Night Live spoof over the weekend.
Yet our current trend estimate, based on all the available surveys, shows opposition leading support by eight percentage point (51% to 43%). If just 1 voter in 25 shifts from opposition to support, our estimate would show Americans evenly divided on the issue. I'm not predicting that will happen, just pointing out that it will not take a huge shift to bring these measurements to something fairly characterized as division rather than "deep" unpopularity.
I'm working on a column today about whether enactment of health care reform might affect misperceptions of the legislation and support for it generally. The column will appear on Monday, but for now I want to pass along one item that may be included but that, either way, I can't let pass without comment.
It involves the television advertisement featuring Republican Senate candidate Sue Lowden now airing in Nevada (via Ben Smith). While the ad attacks Democrat Harry Reid, I am assuming that Republican primary voters are as much the intended audience as persuadable voters in the fall election (Lowden will face Danny Tarkanian and several others in a May 6 primary). Now I worked as a political consultant for about 20 years, and I'm not naive about the, shall we say, exaggerations that are common in campaign advertising. But the absurdity of this one is really remarkable (emphasis mine):
As a mom I know one-size-fits-all clothes don't fit, aren't comfortable and are seldom a bargain. So why does Harry Reid want to force one-size-fits-all government health care on us? Harry Reid thinks Washington knows best, but I think we the people know best. Harry Reid's big government health care plan will raise taxes, put a bureaucrat between you and your doctor, weaken Medicare, kill jobs, push us further into debt. I'm Sue Lowden and I approve this message because government run health care is wrong.
Need I say more?
P.S.: Harry Reid's campaign posted a detailed rebuttal.
Blogger Mickey Kaus has taken out nominating petitions to run for Senate? Against Barbara Boxer? In a Democratic primary? Apparently so.
Let's start with the polling, which in this case is unequivocal. Whatever vulnerability Stu Rothenberg and others might see for Boxer in the general election, she has a solid base among Democrats, as several recent surveys show:
- SurveyUSA (2/12-14): 71% approve of the job Boxer has been doing as Senator, 22% disapprove, 7% not sure (among Democratic identifiers).
- The Field Poll (1/5-17): 71% have a favorable opinion, 7% unfavorable, 12% no opinion (among registered Democrats).
- PPIC (1/12-19): 80% approve of the way Boxer is handling her job as U.S. Senator, 14% disapprove, 6% don't know (among Democratic likely voters).
The PPIC result suggests that Boxer's standing is slightly better among the Democrats that are most likely to vote.
Yes, California has an open primary that allows unaffiliated voters to request a Democratic ballot, but with a huge base among Democrats it's hard to imagine a scenario where Boxer is vulnerable to any primary opponent, much less a little known conservative Democrat (though obviously of interest to political bloggers). For what it's worth, Kaus implies that he's not in this to win:
[T]he basic idea would be to argue, as a Democrat, against the party's dogma on several major issues (you can guess which ones). Likeminded Dem voters who assume they will vote for Sen. Boxer The Incumbent in the fall might value a mechanism that lets them register their dissent in the primary.
But how many anti-Labor, anti-immigration Democrats are looking to express their dissent on these issues in a California Democratic primary? How many like minded independents are willing to skip to two hotly contested Republican contests for Governor and Senate to register such a protest? I'm guessing not many, but if you can point to data showing otherwise, I'm all ears.
Interests disclosed (for those who don't know the history): Six years ago I sent occasional emails to Kaus about polling, which he frequently quoted under the moniker "Mystery Pollster." A year later, when I started blogging under that name, Kaus was a kind booster and frequent linker.
My column for this week looks at surprisingly extensive discussion of poll results at last weeks health care reform summit. My conclusion:
In some ways, the summit's polling conversation mirrors the way
pundits and partisans have talked about public opinion all along. We
have certainly not suffered from a shortage of polls. According to its
editor, Tom Silver, the nonpartisan Polling Report has
published results of health policy questions asked (or tracked) 1,168
times since March 2009.
But rather than accept the often conflicting hopes, anxieties and preferences
those polls measure, compounded by less-than-universal
awareness of the policy details, partisans prefer to cherry-pick
whatever number purports to show the "American people" on their side.
Please click through to read the whole thing.
As noted in the column, Gallup editor-in-chief Frank Newport has already published a review of the origins of the poll numbers referenced at the summit (to the extent that he could identify them). Two related items that appeared over the weekend: Yesterday's New York Times includes as assessment by Dalia Sussman of the differences that question wording can make in health reform polling. Her review drew a reaction from AAPOR member Jan Werner.
Nate Silver begins his rebuttal to Robert Moran this morning by saying, "I don't like criticizing our good friends over at Pollster.com." Well, I don't like criticizing our contributors either, but when Bob wrote on Tuesday in the context of a post on the latest results on the Ohio governor's race that incumbent candidates "get what they get in the tracking, " that it's a "fairly ironclad rule" that "incumbents tend to get trace elements of the undecideds at the end of a campaign," Nate is right and Bob is wrong.
I sympathize with Bob because my instincts led me to a similar line of argument when I started blogging five years ago. My experience as a campaign pollster, gained mostly during the late 1980s and 1990s, taught me to expect the sort of "incumbent rule" patttern that Bob refers to. It was not just an impression. Nick Panagakis, the long time pollster for the Chicago Tribune and other Midwestern media outlets, published some evidence on the Rule in a 1989 article in The Polling Report. Progressive blogger Chris Bowers updated the Panagakis data in 2004, and I summarized both a few months later. When I saw five polls in Ohio showing amazing consistency in Bush's number (46% to 47%), with considerable variation in the Kerry number (45% to 50%), I argued that we were seeing the "underlying principles of the Incumbent Rule in action."
Problem was, I was wrong. Both Bush and Kerry got a bigger percentage of the vote on Election day than they had received in the polling averages (Bush gained a little bit more in Ohio, but not much). Moreover as Bowers had, to his credit, already flagged in 2004, the Rule had been "weakening" since 1998, and by 2006 it was clear that it had largely vanished in competitive, statewide races (the same post includes comments from four campaign pollsters on why they think the "doctrine" was no longer valid). Contests like the 2009 New Jersey Governor's race have been the rare exception, and not a rule.
Since 2004, however, I have also realized that when pollsters or political junkies cite the Incumbent Rule, they sometimes mean two different things. The topic I obsessed over in 2004 involved whether we should anticipate a "break" among undecided voters toward the challenger between the final round of polling and Election Day. Bob's argument on Tuesday, however, argues something a little different: That the incumbent's percentage is unlikely to rise during the course of the campaign, that you get on Election Day what you're getting in tracking, even as far out as February.
Nate's post -- which is well worth reading in full -- attacks the second idea, and I want to stay focused on that topic for the rest of this post. He looks at 63 elections for Senate and Governor since 2006 in which there were polls conducted between January and June and where the two major party candidates ultimately won at least 90% of the combined vote. He then calculated a simple average of all polls fielded between January and June and compared those to the election results.
Nate finds that it is "extremely common for an incumbent to come back and win reelection." He finds that 19 of 30 incumbents who scored under 50 percent in the average of early polls ultimately won reelection. Further he finds that in almost every case (58 of 63) the incumbent ended up with a larger percentage of the vote than they had received in the early polling average. Those findings contradict Bob Moran's argument. .
However, Nate's data also suggests a middle ground and confirms that campaign pollsters and other political professionals are right to focus more on the incumbent's percentage than that of the challenger in early polling. First, as Nate puts it, the "corollary of Bob's hypothesis is almost always true." The incumbent won in 32 of 33 cases since 2006 where early polling showed the incumbent with more than 50% of the vote. Further, when the incumbent's percentage fell under 45%, their probability of succeeding dropped dramatically: Only 5 of 15 ultimately won. Finally,
[I]t does appear to be the case that the incumbent's share of the vote is a better predictor of the final voting margin than the challenger's share. The correlation between the incumbent's vote share in early polls and the final voting margin is .85; the correlation between the challenger's vote share and the final margin has a smaller magnitude, at (negative) .80. Interestingly, the correlation between the margin in early polls and the final margin is also just .85 -- no better than that obtained from looking at the incumbent's vote share alone. This may suggest that the opponent's vote share provides little additional informational value once the incumbent's vote share is known. As I hope I've made clear, however, this does not mean that incumbents "get what they get in the tracking"; they almost always add to their number.
The point of this post is probably obvious, but given the tendency among political junkies to assume that other Americans follow politics the way we do, it's probably worth repeating: While it is true that the health care summit will likely draw an "audience of millions," live viewing will be limited to those who watch live via C-SPAN, cable news networks or the internet. As such, that audience is likely to be a significantly smaller than the numbers that watched the Obama-McCain debates or typically tune into prime-time presidential addresses.
Let's think for a moment about what it means to have an audience of "millions." We are a nation of 304 million people (of all ages), 286 million in television households, and nearly 213 million who were eligible to vote and just over 131 million who cast a ballot int the 2008 presidential election.
Now consider the way the health care summit will be televised today: My colleagues at The Hotline tell me that live coverage will likely be limited to C-SPAN and the cable news networks (CNN, Fox News and MSNBC), plus websites that stream video over the internet.
Yesterday, a spokesperson for the Nielsen Company kindly shared the following ratings data with me. The average audiences for the cable news networks for this "season to date" is 1,235,000 million for Fox News, 495,00 for CNN and 362,000 for MSNBC (all statistics represent averages across each networks programming day for all persons including children over age 2 -- you can find similar reports at the invaluable web site, TV by the Numbers). So if we assume that the overlap between these audiences is trivial, it means that those three networks typically draw a combined average of a little over 2 million.
Sure, that doesn't include C-SPAN or the internet or any ancillary coverage on networks like CNBC or CNN Headline News, and the cable networks are likely to get a modest boost in viewing today. But let's assume the live audience tops five million. That would certainly be "millions" (plural), but still a single digit percentage of all American voters.
Compare that to the audiences for the following events that were covered live by all of the broadcast networks during prime-time evening hours:
- 48 million watched Obama's first State of the Union Address last month.
- 32.1 million watched Obama's live address on health care in September 2009.
The September health care address, combined with the news coverage that followed, did result in a brief and modest increase support for health care reform and in Obama's health care approval rating, but the audience was much larger than those likely to tune in to the health care summit today. The overwhelming majority of those that watch live today or follow news coverage of the event later tonight and tomorrow will likely be the same news junkies that have been closely following news about health care reform all along.
But this event is extremely important with one small but very crucial audience: The members of the House and Senate and the news junkies that surround and advise them. For them this event probably serves as something of a demonstration project for how the health care debate might play out in their own elections later this fall: Will the substance of the discussion help change the tone of the debate or the coverage? Will health reform supporters gain an upper hand in selling whatever plan faces further votes in the House and Senate? As such, the very small live audience is important, I just wouldn't look for a big, overnight polling surge in either direction.
Update: Ironically, after all the barbs traded over lack of final health care "negotiations" being conducted "on C-SPAN" (as candidate Obama pledged in 2008), and after this hilarious satirical trailer that bounced around the internet yesterday, the summit is not being broadcast live on either C-SPAN or C-SPAN2 since both the House and Senate remain in session. However, C-SPAN plans to re-air the entire summit tonight in prime time. My guess is that C-SPAN's prime-time audience is not large enough to significantly alter my calculus above.
I'm a little late in sharing this news, but the Pew Research Center is running a live webcast all day today of their Millennials conference, timed to coincide with their release of an new survey report that, according to their release, "contrasts the attitudes and behaviors of 18- to 29-year-olds with those of older generations."
The final session, from 12:30 a.m. to 2:30 p.m. Eastern time, will focus on whether younger Americans will turn out in and how they will affect the political landscape in 2010 and beyond. Speakers will include Matt Bai of the New York Times, Eli Pariser of MoveOn.org, Reihan Salam of the New America Foundation and Scott Keeter and Michael Dimock from the Pew Research Center. I'm guessing they will have more to say about this chart and its ramifications:
You can watch the webcast here (although caveat emptor: my video has been a bit choppy).
A regular reader wrote recently to ask if we could compare the polls we have entered at this point in the current electorate cycle (2009 to 2010) to those we entered at this point two years ago. His concern is the possibility that Rasmussen Reports, the automated pollster whose results often show a house effect favoring Republicans, might be "flooding the zone" to a greater degree than in past elections. He pointed me to this post by Swing State Project diarist blogger spiderdem, arguing that Rasmussen polls "have dominated the narratives in many of these [2010] races as a result of their sheer frequency."
I've done a crude comparison that shows considerable apparent growth on the pace of polling so far this cycle -- though not just for Rasmussen -- but I need to caution readers about the limits of this data. Since races for Governor are far more numerous in a non-presidential years, I looked only at polls that tested U.S. Senate contests. In 2008, I counted up horserace results -- one per state -- for the general election match-ups that ultimately appeared on the November ballot. Unfortunately, we were only starting to enter polling data for non-presidential races at this point in 2008, so our internal database does not include many polls for potential match-ups that failed to materialize.
The problem, of course, is that we cannot predict which candidates will run and win primaries in 2010, so it is impossible to generate a strictly comparable list. Instead, I opted to count polls for this cycle for the candidate match-up in each state that has generated the most results to date. Also, I used wikipedia (sorry, Harry) to gather a few dozen polls in the less competitive contests that we are not yet charting here on Pollster.com.
The main point: my method is fuzzy. Others might come up with slightly different counts, and mine probably exaggerates the apparent increase in polling in the current cycle. Still, it should be close enough to give a sense for whether any one pollster is "flooding the zone."
On to the data. The following table shows my count of polls conducted at this point in each cycle. It shows a huge overall increase in polling on U.S. Senate races: 160 polls conducted as of last week as compared to just 74 at this point in 2008. Yes, Rasmussen has conducted more than three times as many polls fielding (45 vs. 13), but you can see similar rates of growth for PPP (21 vs. 5), Quinnipiac University (14 vs. 0), DailyKos/Research2000 (13 vs. 5) and several others.
Equally interesting is the decline for SurveyUSA, whose surveys are usually sponsored by local television stations. They had fielded 16 surveys on general election contests for U.S. Senate at this point in the 2008 cycle but none that I counted so far this time. Similarly, the number of polls conducted by less prolific media or non-partisan pollsters fell as a percentage of the total (from 26% to 16%), although the absolute number was slightly higher (26 vs. 19).
The message I get from these numbers is that the growth in polling in Senate contests so far this cycle has been driven by non-traditional media and academic sponsors and pollsters (like Rasmussen and PPP) that routinely conduct and release surveys without sponsors for their marketing value. These numbers also imply that traditional media sponsors -- local television stations and newspapers -- have cut back on their polling budgets over the last year.
But back to the question that prompted this exercise. Yes, Rasmussen Reports has fielded far more polls so far this cycle, both in absolute terms (45 vs. 13) and as a percentage of the total (28% vs 18%). One likely explanation is the "major growth capital investment" from a private equity firm they announced this past August:
"This investment will enable Rasmussen Reports to expand and enhance all aspects of our business," said Scott Rasmussen, founder and president of Rasmussen Reports. "That includes expanding our Premium Membership service and subscription base, developing new index products and sponsorship opportunities, and exploring new research techniques."
The more difficult questions, which my data do not answer, are those that critics have often asked about Rasmussen and that others will ask about PPP, Daily Kos and other pollsters or sponsors with a demonstrable point of view: Is bias or partisanship at work in decisions about where to poll, what to ask, and what population (adults, registered voters or likely voters) to interview?
My column for this week takes up the cause of GWU Professor John Sides and looks at how the definition of "independent" used most often tends to exaggerate the degree of independence among independents. Confused by that reference? Please read the whole thing.
Update: Alan Abramowitz covered some of this material in a post in August 2009.
The strange saga of Strategic Vision, LLC and the continuing promises of its CEO David Johnson to produce data or new surveys continues. The latest installment: Johnson appeared on conservative talk radio show in Wisconsin promising a new survey there in a few weeks.
The relevant background: Back in September, following a censure from American Association for Public Opinion Research (AAPOR) and accusations of fraudulent data from blogger Nate Silver, Johnson threatened to sue everyone in sight to clear his name and promised to release cross-tabular data that reporters had requested. Five months later, as far as I know, no one has been sued, and no crosstabs have been released.
Last month, Johnson surfaced long enough to inform a columnist from the Savannah Morning News that he planned to conduct a Georgia survey during January. As of this afternoon, and no new poll results from any state have been posted to strategicvision.biz since September 2009.
Yesterday, our own Charles Franklin recorded Johnson giving an interview to conservative talk-radio host Vicki McKenna on Madison station WIBA-1310 WISN-1130 and promising yet another new survey.
McKenna: You're going back into the field here in Wisconsin in a couple of weeks, aren't you?
Johnson: Yes we are.
McKenna: Awesome. Um, yea, we need to find out just how soft Russ Feingold is. You have got a target rich environment here in Wisconsin when you start making those phone calls, David. But you have been polling elsewhere, and I guess my question to you is just a very broad question: Does the polling suck for Republicans anywhere?
Johnson: No it doesn't, we're looking at a real Tsunami...
For those interested, I have also uploaded the complete interview, which covers the full spectrum of politics from the Right but includes no further discussion of Strategic Vision or its polling.
As I wrote in December, if and when Strategic Vision resumes "making phone calls" or otherwise reporting results, absent significantly better methodological disclosure from Johnson, we will no longer include their numbers in our charts or publish them as "poll updates."
P.S. You can't make this up: Johnson apparently also appeared on camera on ESPN earlier today discussing the subject of "Tiger Woods and crisis communications."
Does the just released CNN/Opinion Research Corporation poll overstate "active support" for the Tea Party movement? They report "roughly 11 percent of all Americans say they have actively supported the tea party movement, either by donating money, attending a rally, or taking some other active step to support the movement." That number strikes Ben Smith as "surprisingly large," and Chris Bowers notes that respondents often exaggerate their true levels of activism. They are both right to caution us against interpreting these results too literally, but it's worthy considering how the CNN pollsters arrived at the 11% estimate and what it means.
In my column on polling on the Tea Party movement yesterday, I reported two findings from national polls released last week: The number of hard core Tea Party supporters is relatively small (somewhere in the mid-to-upper teens, depending on the measure), while 45% tell the Washington Post/ABC poll they at least "somewhat agree" with the Tea Party positions on issues.
The results of the CNN poll are broadly consistent:
- The CNN poll finds 15% of adults say they "strongly support" the Tea Party, very similar to the 14% who told said they "strongly agree" with the Tea Party movement's positions on issues on the Post/ABC poll and slightly less than the 18% who say they consider themselves to be "supporters" of the movement on the Times/CBS poll.
- CNN also finds a total of 35% who at least "moderately support the party. That is less than the 45% who say they agree with the Tea Party on issues in the Post/ABC post, but mostly because CNN offered the explicit choice "or don't you know enough about the Tea Party to say?
But the 11% statistic comes from three separate questions asked on the CNN survey.
The combine the results to find that a total of 11% of adults answer yes to any of the three questions. The number who say they have given money to the Tea Party movement is relatively small (2%). As Bowers guesses, a good chunk of the 11% -- probably about half -- comes from the 7% who say they "took any other active steps" to support the movement, which is obviously a pretty soft measure.
And Bowers is right that survey respondents tend to overstate all sorts of political participation, past voting, intent to vote, political giving, even how often they watch news broadcasts. So a literal interpretation of these statistics is not recommended. I would also caution against assuming that the Tea Party enthusiasts are unique on this score. I don't have a ready link, but political scientists have been studying vote over-reporting for decades and I do not recall reading about differences by political party or ideology.
The larger point here is that there is a relatively small number (10% to 20%) of Americans who express very strong sympathy to something called the Tea Party movement. The recent polls, including this new one from CNN, tell us a great deal about who those Americans are and what they believe.
While I was out shoveling snow and trying to keep my snow-bound children entertained last week, Dartmouth undergraduate Harry Enten -- our Pollster.com intern-to-be for 2010 -- was busy blogging up a storm. One item he posted last week provides yet another epilogue to an epilogue on the story of Strategic Vision LLC.
My last installment on the odd twist in this story noted that after apparently blocking access to strategicvision.biz to our offices at the National Journal Group, the web masters at Strategic Vision also sought to block the Internet Archive from displaying content previously released on strategicvision.biz.
But Harry noticed something: "The web pages can still be accessed online right now even without the Internet Archive!" How?
Well, it turns out that, despite not having one single page to display the polling data from 2005-2007 (they do for 2008 and 2009), one can still retrieve the original individual pages the polling data was displayed upon. In what can only be deemed as one of the WORST coverups of all time, Strategic Vision, LLC left the individual polling pages on its servers.
All you need to access the data is the original link to any poll. Those links are easily available from polling aggregation sites such as RealClearPolitics.com, Pollster.com, and even Wikipedia.org.
He even provides a video to show, step-by-step, how the links can be found.
And just in case our friends at Strategic Vision, LLC decide to take those not-quite-removed-pages down, Harry "downloaded every single poll file from 2005-2009 and have uploaded it in a single zip file for anyone to download."
Thanks Harry!
PS: He has a new post today that catches a prominent blogger's oversight and teaches a lesson about putting too much trust in wikipedia.
My column for this week looks at the recent polling on the Tea Party movement with a focus on how pollsters pondered this issue and then tackled it in longer form surveys released last week. Special thanks to the Washington Post's Jennifer Agiesta for providing some additional data.
For the most thorough review of perceptions of the Tea Party movement, I highly recommend the reports by CBS News, ABC News and the Washington Post's story and accompanying graphic.
I want to expand a little on one point made toward the end of the column: The number of hard core supporters is relatively small (somewhere in the mid-to-upper teens, depending on the measure), while a much larger percentage (45%) tell the Post/ABC poll they at least "somewhat agree" with the Tea Party positions on issues.
But which positions do these 45% agree with? After all, many sources will point out, there are many Tea Party movements with sometimes only vaguely articulated issue positions. Neither of the two recent surveys directly probed knowledge of the tea party positions, but it is possible to glean some sense of their perceptions by looking at other attitudes among the self-identified Tea Party sympathizers.
For example, as I noted in the column, expressed agreement with Tea Party positions is much higher with conservatives (63%), strong Republicans (67%), those who disapprove of Obama (65%) and those who express anger at Washington (69%). Here are some additional details from the ABC News report:
[Tea Party support] peaks among people who are more apt to see the government as wasting money; people who strongly agree with the movement say on average that the government wastes 63 cents out of every tax dollar it collects. People who disagree with the Tea Party see less waste, albeit still a lot - 47 cents on the dollar.
Tea Party supporters are more apt to classify themselves as anti-incumbent - 64 percent of those who strongly agree with its positions do so, as do 53 percent of those who somewhat agree, compared with 40 percent of those who disagree. And the movement's conservative, Republican base shows up in vote preferences for the midterm elections. Among registered voters who agree at least somewhat with Tea Party positions, Republicans hold the lead over Democratic congressional candidates by a very wide 70-22 percent.
Taken together, these results imply that among Americans who have heard something about it, the words "Tea Party movement" imply a politically conservative reaction against Obama, Washington and perceived waste in government spending.
But both surveys also yield evidence that most Americans know little or nothing about the movement, and some additional results suggest that those who are only "somewhat" supportive hold positions that may be at odds with those frequently associated with the movement. Here is more from the ABC News report:
While the Tea Party promotes limited government, some of its supporters have different views on government health care mandates. For example, 62 percent of those who say they agree at least somewhat with Tea Party positions also say the government should require businesses to provide health insurance for employees.
Even more, 71 percent, say government should require insurance companies to sell coverage to people regardless of pre-existing conditions. And while shy of a majority, a substantial share of Tea Party supporters, 43 percent, say government should require all Americans to have health insurance, from their employer or another source, with financial assistance for those who need it.
Last week, Andrew Sullivan wrote of his sense that people divide into two classes with respect to sleep and general exhaustion, "those with kids under ten and the rest of us." As the father of a 5 and 7-year-old, I can both confirm that observation and extend it a little: Having kids under 10 probably also introduces a similar divide in coping with a multi-day snow in.
For those without kids, getting snowed in may offer a respite, a chance to catch up on work, reading or some other long deferred project. Being confined to your house with two small children is a different experience entirely. Fifteen minutes of uninterrupted time is a luxury I have not known in many days. So apologies for the slower pace of blogging this week.
As I write, as the view from my window shows, we are experiencing another blast of snow, featuring white-out conditions and heavy winds. So far, we have avoided interruptions in power and bandwidth, but our luck may run out given today's 40-mile-an-hour gusts. So apologies in advance if I drop off the grid altogether. Ditto for Emily, who is also working from home to post poll and chart updates. Hopefully, we will all be back to normal soon.
My column this week looks at the controversy over a series of surveys conducted by SurveyUSA for the liberal web site Firedloglake. Please click through to read the whole thing.
Lost in the attack memos and other questions raised is an important question facing nearly every telephone survey conducted in House, Senate and Gubernatorial races this year: Are we at the point where the majority of true "likely voters" under the age of 35 are out of reach of landline telephone samples? And at what point is simply "weighting up" those younger voters that pollsters can still reach inadequate to solve the problem?
The table below, produced by the Pew Research Center and based on their national surveys, shows that by 2006 their unweighted landline samples were under-representing roughly a third of adults under age 35. And that was as of three years ago, when the percentage of all adults living in landline-only households was estimated at 12%, nine percentage points lower than the most recent estimate:
Now consider the estimated growth in the cell-phone-only population over the last three years. As shown in the chart below (which comes from a report last year by the National Center for Health Statistics of the Centers for Disease Control and Prevention), landline-only samples are most likely to miss voters under age 35.
Now consider this additional statistic reported on Pollster.com by Mike Mokrzycki in December. On the most recent CDC report covering the first half of 2009, nearly two thirds (63.5%) of people age 25-29 live in households with either no landline phone (45.8%) or in "cell-mostly" households (17.7%), those were "all or almost all calls are received on cell phones."
So what should a pollster do if they reach so few 18-to-34-year-old voters that they make up just 1% of the likely voters sample for an election where past turnout suggests that age group should make up roughly 10% of the electorate? If the pollster believes they have under-represented younger voters, can they simply weight to correct the problem? Not if the shortfall is that extreme. In a sample with only 400 or 500 completed interviews, such a weight would multiply 4 or 5 interviews by a factor of 10. As I wrote in the column, you don't need to be a statistician to imagine how those "super respondents" might crate greater error and volatility in the results, especially those produced by cross-tabulations of demographic subgroups.
Let's remember that we are able to pick at SurveyUSA because they were willing to disclose the weighted demographics of their sample and because they opted against any such extreme weighting in this case. So rather than beat up on SurveyUSA, we might do better to ask: How many polls have we seen in recent months that involved a similarly sparse number of younger likely voters and were simply weighted up by factors of 5 or greater to conceal the shortfall? How would we know?
Finally, whatever we want to make of the Firedoglake surveys, it is important to remember that SurveyUSA has maintained an outstanding record of final-poll accuracy, especially in U.S. House elections and in hard-to-model primary elections. For House races, the company's own scorecard -- which I have no reason to doubt -- shows that their average error on the margin in polling 27 House races in 2006 (3.4) was roughly half that of all other pollsters combined (6.3). Their error rate was also significantly lower than the three most prolific public pollsters that year, Research2000 (5.5), Zogby (5.9) and RT Strategies (5.9).
So since we have picked at their work mercilessly, I want to give SurveyUSA's Jay Leve the last word and reproduce the full email he sent me last week in response to my questions about the Firedoglake surveys:
In August 2002, SurveyUSA released a poll showing US Senator Robert Torricelli (D-NJ) trailing. No survey to that point had showed Torricelli trailing. An hour after the poll was released, SurveyUSA's client, CBS-TV in Philadelphia, called SurveyUSA and said, "Put your helmets on. The DSCC is coming after you." And the DSCC did. The DSCC found a journalist willing to write the smack that the DSCC was shoveling, and the message went forth: Nothing wrong with Robert Torricelli, plenty wrong with SurveyUSA.
A few weeks later, Torricelli dropped out of the race. Other polls had the same results as SurveyUSA.
Fast forward to today: In a poll conducted in January 2010, at a time the Democrats were losing the state of Massachusetts, SurveyUSA finds an incumbent Democrat in a tight fight in New York state. The DCCC is unhappy. Partisans start shoveling smack. "Sources" start providing willing journalists with leaked memos. Nothing wrong with Democrat Tim Bishop. Plenty wrong with SurveyUSA.
The highway to high office is littered with the road kill of political operatives who find it easier to campaign against a poll than an opponent.
Lost in the hurly burly is an opportunity for real reflection. To my knowledge, there has never (ever) been a publicly released telephone poll conducted in a U.S. congressional district that included a known subset of interviews with respondents who did not have a home (aka: landline) telephone. An acknowledged limitation of SurveyUSA's work in NY-01, and a known limitation to date of all congressional district polling, is that voters who do not have a home phone are under represented. At a statewide-level (in contrast to the CD level), only one pollster in the 2009 election included a known subset of cellphone-only respondents in its sample (at extraordinary expense, because of the theoretical justification), and that pollster's results were worse than many polling firms who did not include a known subset of cell-phone-only respondents. Whether one anticipates that in 2010 young voters will turn out in record numbers of stay home in record numbers, the problem of how to count those voters is real, and right before us.
Another update, this one on the volunteer exit poll conducted this week in Cook County Illinois by the recently launched Chicago Current. Current editor Geoff Dougherty posted a refreshingly candid postmortem on their efforts:
At 6:11 p.m. yesterday, before the polls closed, I wrote that our exit polling suggested Toni Preckwinkle had the Cook County Board president's race locked down.
And I was right. Our survey honed in on Preckwinkle's strong performance early in the day, and continued to highlight her lead as the election progressed.
And yet ... our poll was wrong. I predicted Preckwinkle would snag 69% of the vote, and noted that the poll had an 8% margin of error. Preckwinkle ended the day with 49% of the vote -- well outside that margin.
Such are the joys and pains of exit polling.
There's more, and it's worth clicking through to read the rest.
I would give the Current an "A" for effort and transparency, but we need to be realistic about the quality of the survey they ultimately produced. Dougherty says it cost just $200, "most of which went for a $100 rental car," and don't think he would argue with the conclusion that they got what they paid for. The poll managed to collect just 93 completed interviews at only 9 of 25 precincts (presumably) selected at random. As Dougherty reported at 1:32 p.m. on Tuesday:
So far we've got about 30 responses. We'll be taking a pause here as our field crew relocates to new spots and starts talking to voters.
We'd originally planned to survey 25 precincts, but logistics are interfering, and we'll probably wind up with about half that. We'd targeted 600 voters, but low turnout will probably leave us with about half of that count.
Never mind the very small sample size. How truly random was the sample? It's hard to tell from this description, but the execution clearly fell short of ideal.
Dougherty says that the "networks often pay tens of thousands of dollars for these things." That's not quite right. I'm not sure how it translates into a per-state cost, but the every-two-year National Election Pool (NEP) exit polling operation has a multi-million dollar budget (Voter News Services, VNS, the forerunner to NEP, operated in 2000 on a budget of over $35 million; my understanding is that current costs are much lower but still in the millions). Note that in most states of interest, NEP will sample 20 to 50 precincts. As the scale of what the Current was attempting in a single county was in line with the exit poll that NEP conducts in each state.
I write this post not to beat up on the Current -- again, I give them credit for enterprise and transparency -- but to remind my media colleagues that all "exit polls" are not created equal. Not by a long shot.
Update: The cost statistic I cited for VNS from 2000 is accurate but potentially misleading. VNS was responsible for both exit polls and reporting final vote counts for every race (the latter function is now provided by the Associated Press). The costs also vary considerably between presidential and off-year elections. Finally, the NEP exit operation still includes more than just exit polls, it also collects vote results at samples of key precincts and provides statistical modeling and analysis used to "call" races.
Today we have yet another odd epilogue to story of Strategic Vision, LLC. Apparently not satisfied with their history of setting the low bar for basic disclosure about the surveys they claim to have conducted since 2004, the company is now attempting something new: Attempting to retroactively withdraw previous disclosure.
Until a few weeks ago, the content published at the company's web site, strategicvision.biz, had been automatically archived by the non-profit Internet Archive along with hundreds of thousands of other web pages. In my December column, I linked to two such pages (displaying polls conducted during 2005 and 2007**). As of today, however, if you search the Internet Archive for strategicvision.biz or try either of the links I used previously (and be forewarned: their heavily trafficked site is notoriously slow), you will encounter this error message:
Robots.txt Query Exclusion.
We're sorry, access to http://www.strategicvision.biz has been blocked by the site owner via robots.txt.
What that means is that sometime in January, someone at Strategic Vision added some code ("User-agent: ia_archiver Disallow: /") to a file on their web site that specifically blocks the Internet Archive from searching and displaying pages from their company web site. Let's be clear that Strategic Vision is well within its rights in blocking such searches, and has done nothing illegal or particularly nefarious. As explained on their Frequently Asked Questions (FAQ) page, the Internet Archive "is not interested in preserving or offering access to Web sites or other Internet documents of persons who do not want their materials in the collection," and thus provide instructions on how to "exclude any historical pages."
That said, given the swirl of accusations about Strategic Vision arising from a failure to disclose basic information about their methods, this new effort to scrub previously disclosed information from what is essentially a public library for the Internet is more than a little creepy. Combined their apparent blocking of access to strategicvision.biz to me and my colleagues at the National Journal, and we get a story of a company that keeps digging a deeper and deeper hole for itself.
By the way, all credit for spotting this latest twist in the story goes to Michael Weissman, the retired University of Illinois physics professor who previously published a "Fourier analysis" of Strategic Vision's results on FiveThirtyEight.com. His son Jonathan realized that Strategic Vision might delete their archive, and thus downloaded everything he could before it disappeared. So the archived pages live on -- undoing previous disclosure is harder than it looks.
**As of this writing we were still able to load some of the 2005 page (sporadically), and if you experience as similar result it is probably because of something gone awry at archive.org. The code in the robots.txt file on the Strategic Vision site shows that they want Internet Archive to remove stop displaying their content.
I want to add a few thoughts to Emily's post earlier today on the DailyKos/Research 2000 poll of Republicans and how they might check for a skew in the sample that some argue would result from "sane Republicans" hanging up after taking offense to the questions. Another potential problem, called out today by Republican pollster Alex Lundry, is not as easy to check: The possibility of a skew in respondents' answers caused by what pollsters call "acquiescence bias."
Acquiescence bias is the tendency of some respondents to select affirmative answers where the choice is whether to affirm or reject the statement presented (including "agree or disagree," "favor or oppose" and "yes or no" formats). This topic has been the subject of decades of study and debate among social scientists, and even though pollsters continue to rely on agree-disagree questions, academic survey researchers mostly agree that this format tends to produce more apparent agreement than questions offer a choice between two competing statements.
Here is an example from Schuman and Presser's classic text, Questions and Answers in Surveys (p. 221), based on an experiment first conducted by the NORC General Social Survey in 1974: They asked a random half sample to agree or disagree with this statement: "Most men are better suited emotionally for politics than women." Slightly less than half (47.0%) agreed, 53.0% disagreed.
They asked the other random half-sample to choose between two statements (and included a middle choice):
Would you say that most men are better suited emotionally for politics than are most women, that men and women are equally suited, or that women are better suited than men in this area?
Fewer (33.1%) agreed that men were better, 4.3% said women were better suited than men, and 62.6% said they were both equally suited. Researchers at the University of Michigan's Survey Research Center replicated the experiment three times between 1974 and 1976, producing similar results. They produced consistently greater agreement that "men are better" using the agree/disagree format (ranging from 44.3% to 45.5%) than when using forced-choice format (ranging from 32.5% to 38.3%).
Another strategy to reduce this bias is to try to balance the direction of the statements, as recommended in Presser, et. al, Methods for Testing and Evaluating Survey Questions (p. 440):
Acquiescence bias can be reduced by balancing scales so that the affirming response half the time is in the direction of the construct and half the time is in the opposite direction (e.g. six agree/disagree items on national pride, with the patriotic response matching three agree and three disagree responses).
With those recommendations in mind, consider the questions asked on the DailyKos/Research2000 survey in the order in which they presented the results. The first eight present all of the more sensational, ludicrous assertions (most of which pertain to President Obama). Seven of eight ask respondents to affirm or reject the extreme statement:
- Should Barack Obama be impeached, or not?
- Do you believe Barack Obama was born in the United States, or not?
- Do you think Barack Obama is a socialist?
- Do you believe Barack Obama wants the terrorists to win?
- Do you believe ACORN stole the 2008 election?
- Do you believe Sarah Palin is more qualified to be President than Barack Obama?
- Do you believe Barack Obama is a racist who hates White people?
- Do you believe your state should secede from the United States?
They then ask 15 issue questions that do mix up the order somewhat. Eight questions -- ask respondents if they agree with a liberal policy position, five ask about a conservative policy position, and two (the questions about Christ and marriage as a partnership) force choices between two statements:
- Should Congress make it easier for workers to form and join labor unions?
- Would you favor or oppose giving illegal immigrants now living in the United States the right to live here legally if they pay a fine and learn English?
- Should openly gay men and women be allowed to serve in the military?
- Should same sex couples be allowed to marry?
- Should gay couples receive any state or federal benefits?
- Should openly gay men and women be allowed to teach in public schools?
- Should sex education be taught in the public schools?
- Should public school students be taught that the book of Genesis in the Bible explains how God created the world?
- Are marriages equal partnerships, or are men the leaders of their households?
- Should contraceptive use be outlawed?
- Do you believe the birth control pill is abortion?
- Do you consider abortion to be murder?
- Do you support the death penalty?
- Should women work outside the home?
- Do you believe that the only way for an individual to go to heaven is though Jesus Christ, or can one make it to heaven through another faith?
I don't want to overstate the consensus of pollsters -- academic or otherwise -- on this issue. Many highly regarded survey researchers continue to rely on agree/disagree questions, often because of their simplicity and brevity or because such questions are part of a long-standing time series that the pollster would rather not disrupt (good example of the latter here; for more discussion see Javeline, 1999).
So while it would be a bit unfair to condemn Research 2000 for relying on question formats that pollsters and academics continue to rely on, Lundry has a point. Acquiescence bias probably exaggerates the amount of agreement measured for some of the more ludicrous assertions about Barack Obama tested on the Kos poll.
Update: As Alex Lundry notes below, his comments about acquiescence bias earlier today came after reading a message sent by Stanford graduate student Josh Pasek to AAPOR's members only listserv. With Josh's permission, here is a portion of that message:
Given that 10-20% of respondents tend agree with any statement (likely
due to social norms), I went through the survey mentally subtracting 15
percentage points from every "yes" answer. That does leave some
shocking numbers -- particularly as acquiescence tended to indicate
support for gay rights, sex education, etc. -- but suggests that
Birthers, for instance, may be outnumbered in the party (a slight
consolation at best). I'm not saying this to suggest that the opinions
being expressed even with a correction are reasonable, but I worry that
not addressing this kind of issue is the reason so many people out there
are skeptical of survey results in the first place.
Nothing draws the Google traffic on an election day like the words "exit poll," and the enterprising folks at the Chicago Current (a political newspaper and website) are using their own reporters and journalism students from Northwestern University to conduct an exit poll in Cook County (i.e. not statewide) and appears to be asking about the Democratic contests only (via Christine Matthews).
The Current's Geoff Dougherty has a write-up of what they are doing that includes some worthy disclaimers. Short version: exit polls have random error like other surveys (actually, more given the need for a clustered sample) and is "not the final score" when based on early returns, even if flawless. They are posting initial impressions here.
I cannot say anything about the methodology for the Current poll because, frankly, I don't know a thing about it. For those who have never read it before, you might want to consult my classic primer on exit polls, though remember that my advice was based on the methods employed by the network exit polls done by the folks that invented them and have spent decades honing their techniques.
Without knowing more about the Current poll, I'd urge extreme caution in interpreting the results, especially if your goal is to "call" the result. Remember, exit polls are most useful after the votes are counted -- when results can be weighted to match actual turnout -- in helping to understand who voted and why.
If I learn more today, I'll post it here.
My column for this week looks at how Americans came to believe what they do about health care reform, with a focus on this puzzle: If American's followed news coverage about the health reform debate as closely as the Pew Research Center news interest surveys say they did, why are so many "unfamiliar with key elements of the major bills," as reported by the Kaiser Family Foundation survey last month?
The short version is that perceptions of health insurance reform appear to have been shaped by both the typically process-oriented coverage of the health care debate and the larger context of double-digit unemployment and massive increases in government spending over the last year. I end the column with the obvious conclusion: Months of more legislative wrangling on this issue is unlikely to change impressions or increase awareness of what's actually in the bill. Please click through for the whole thing.
But what about the question of greatest consequence right now? What if Congress were to quickly pass the existing legislation or, alternatively, just let it drop? How would voters react? This topic was the subject of a lot of discussion over the past week, from voices such as Megan McArdle and (since I filed the column) from Nate Silver and Jonathan Chait.
Looking forward, we are on much more shaky and speculative ground but I find Chait most persuasive in arguing that the rationale for Democrats to move forward and pass the bill is that they've already voted for it and thus "already own the downside." They will be attacked for "having voted for tax hikes and Medicare cuts and death panels" regardless of the outcome. He continues:
Suppose there's no upside at all to passing health care reform. McArdle assumes, without explicating her reasons, that walking away from the issue is a way for Democrats to cut their losses. Why, though, would that be the case? Passing the bill may or may not make it more popular, letting it die is surely going to make it less popular. If the bill dies, then it's the subject of lengthy, painful postmortem coverage detailing its flaws and mistakes. It becomes the symbol of big government run amok, and the 60 Senate Democrats and 220 House Democrats who voted for it will suffer politically all the more. Moreover, the already-demoralized liberal base would become apoplectic with the Democratic Party. 1994 was bad, but passing a bill through both chambers then sitting by and letting it die is the kind of behavior that makes even the most pragmatic Democratic voter want to punish his own party.
It's hard to guess at where public opinion will move next, but if I were still offering political advice to Democrats, I'd side with Chait.
From our Pollster-centric perch, this is a memorable comment. In answering the last question of the Q & A session with the House Republicans today in Baltimore, President Obama called out pollster Frank Luntz (emphasis added):
That's why I say if we're going to frame these debates in ways that allow us to solve them, then we can't start off by figuring out, (a), who is to blame, (b), how can we make the american people afraid of the other side? Unfortunately that's how our politics works right now. That's how a lot of our discussion works.
That's how we start off every time somebody speaks in Congress, the first thing they do, they stand up and all the talking points - I see Frank Luntz sitting in the front. He has already polled it and he said the way you're really going to -- I've done a focus group and the way we're going to really box in Obama on this one or make Pelosi look bad on that one -- I know Frank. I like Frank. We've had conversations between Frank and I, but that's how we operate. It's all tactics. It's not solving problems.
So the question is, at what point can we have a serious conversation about Medicare and its long-term liability or a serious conversation about Social Security or a serious conversation about budget and debt in which we're not simply trying to position ourselves politically? That's what I'm committed to doing. We won't agree all the time in getting it done, but I'm committed to doing it.
I'll update with video if and when I can find it (thanks to Marc Ambinder for noting this exchange via Twitter). Update: CSPAN has video here, but the video is not embeddable; the comments above begin at 84:21. Update 2: Thanks to MSNBC, here's the clip:
PS: Probably says more about the nature of inside-the-beltway chatter, but Mark Halperin apparently considered this exchange the most "extraordinary" aspect of the Q&A.
Update 2: Greg Sargent reports on a post question-time interview with Luntz (via Ben Smith):
[Luntz] conceded Obama had the advantage today -- but said he'd still
advise Republicans to debate him again, because it put them on his
"level."
Luntz also confided that Obama had approached him after
the event and joked with him about calling him out. "We had a laugh
about it," Luntz told me in an interview just now. "He said, `It's good
for business.'"
No, we are not doing any live-blogging tonight, though I might post a thought or two via Twitter.
Also, I will update this post later tonight with links to whatever quick reaction polls or focus groups get released.
Meanwhile, if you haven't yet, you might want to read my primer from earlier today on what to expect from post SOTU polls, plus my post from September on the widely varying methodology of the sort of quick reaction polls and focus groups we might say later tonight.
Update 1:
CNN's "Flash Poll" (story, results) -- CNN reported on air some results from their "flash poll" (typically conducted by re-interviewing respondents to an earlier poll that planned to watch the speech). Anchor Soledad OBrien said that, as usual, "more people who are watching the speech favor the party of the person who is giving the speech. That means, in short, more Democrats were being poll here." She did not give specific numbers, but as noted here earlier, that's a typical pattern.
The reaction was overwhelmingly positive -- 48% very positive, 30% somewhat positive, 21% negative -- although O'Brien noted that the very positive number was lower than following Obama's economic address (68% very positive) last year. It is also on the low end of the reactions recorded by Gallup during the late 90's and early 00s (see the second table here).
CNN typically posts results on CNN.com -- I'll add a link when it's up links now added.
CBS' Instant Poll (summary, report & results) - As in previous years, CBS conducted a representative online sample with Knowledge Networks among 522 speech watchers (more on the methodology here). They report that 83% of speech watchers approve of the proposals the president made in his speech tonight, but that just 42% of speech watchers think that Barack Obama will be able to accomplish all the goals he set out in his speech tonight (57% do not think he will be able to).
Democracy Corps Dial Group (
analysis,
pre/post-speech scoresheet) - Democratic pollster Stan Greenberg's Democracy Corps conducted a "dial group" -- a focus group where participants turn dials while watching the speech to indicate their reactions -- in Las Vegas tonight. Greenberg and Democracy Corps Senior Associate Andrew Bauman just provided some highlights via conference call. In addition to the dial meter readings, they also conducted before-and-after survey with the participants.
Greenberg reported a "very positive response," but warned that the "scale of shifts are always artificial" in a dial group because "people are watching him with such intensity." While they saw across the boards shifts in opinion on Obama, but the area with the biggest shift during the speech was on "bank reform and wall street and special interests."
The shifts there are very extraordinary. On the issue of whether he puts Wall Street ahead of the middle class, it was a 50 point shift on people saying that [doesn't describe him] well. There was a 40-point shift...on fighting special interests. On banking reform, on support, it was a 38 point shift in favor of that. And that's clearly, far and away the place where he showed the greatest strength and clarity.
I asked about the lack of State of the Union "bounces" for previous presidents, whether he has seen sustained movement on other internal measures following previous addresses and whether similar dial tests foretold any such shifts.
Greenberg said he could recall shifts in "thermometer approval" (favorable ratings) that held, specifically citing the movement for Obama following last year's joint session address on health care. With Obama's approval falling "within a band..averaging 48 or 49 percent," he considers big shifts unlikely. "Attributes are different," he said. "People who are not supportive don't feel they have to lock in, so there's more space there, and the view of the president is more complicated than just approval." Greenberg cited attitudes on Obama's orientation toward Wall Street as most likely to produce "sustained shifts" in opinion.
Democracy Corps plans to post a full report overnight or early Thursday (links now added above)..
Perhaps political journalists have all gotten the message by now that polling "bumps" from the annual State of the Union (SOTU) address are more myth than reality. If so, this post may be something of a cliche. But I'm not convinced, so I want to recommend this very helpful report published last night by Gallup based on their 30-year archive of pre and post SOTU polling.
If you report or comment on politics, it's a must read. If you are short on time, here are the two main points (with a little added value from our own posts over the last five years):
1) "These speeches rarely affect a president's public standing in a meaningful way, despite the amount of attention they receive."
Gallup's report includes a table showing the level of presidential approval measured immediately before and after the last 27 State of the Union addresses. "Across all presidents," they report, "the average change in approval has been less than a one percentage-point decline.
It is also keeping in mind, as I wrote on the old Mystery Pollster blog four years ago, that the one big exception to the rule -- the apparent 10 percentage point jump for Bill Clinton in 1998 -- was a very unique presidential address:
The Monica Lewinsky story had broken just a few days before. The day before that speech, Bill Clinton faced the cameras and delivered his infamous "I never had sex with that woman" quote. MP cannot find the ratings for that speech, but interest in the speech was certainly high. Ironically, the reaction to Clinton's performance - seemingly unfazed by the scandal erupting around him - help[ed] boost his numbers in a way that persisted until the impeachment trial ended with an acquittal.
So the one exception to the rule may have been less about perceptions of the speech itself and more about how the speech fit into the context of a larger event.
Bottom line: Don't expect a big bump tonight. My post from 2006 covers a lot of the same ground as the Gallup report, as does Charles' Franklin's commentary in 2007 on this graphic
2) "The audiences for the State of the Union tend to be heavily tilted toward the president's existing supporters."
This table from the report is in many ways the most useful for those of us who will be looking at the one-night-wonder polls conducted immediately after the speech.
In many ways, this is the one poll measurement that interests me tonight: Will the speech attract an audience heavily tilted toward Democrats, as is typical, or will the enthusiasm gap apparent in so many approval polls (Republicans report a greater likelihood to vote in 2010 than Democrats) make for a more balanced audience? In other words, will the audience look more like 1999 or 1995 (just days after Republicans took control of the House and Senate, installing Newt Gingrich in the seat just behind President Clinton)? While a typically Democratic skew will tell us little (notice the composition in 1994), it will be a very bad sign for Democrats if the audience consists of as many who tune in to jeer as to cheer.
On the subject of immediate reaction polls and focus groups, see my post from this past September which reviews their widely varying methodologies and the limitations inherent in this sort of survey.
Last week's Senate election in Massachusetts gave those of us who follow and report on politics an experience analogous to the movie It's a Wonderful Life. We now know what life is like without exit polls.
Unfortunately, by the time it became obvious that Scott Brown had a real chance of upsetting Martha Coakley, the lead-time necessary to conduct a true exit poll had passed. As a result, we were left with a handful of post-election telephone surveys. While the high quality survey conducted by the Washington Post and the Kaiser Family Foundation helped fill the gap, we were also treated to a number of post election polls conducted by partisans seeking to put their own spin on the results.
The day after the election, I got an email from a former network political producer who thought now would be a good time to restate the case for exit polls:
The absence of good data to help understand what happened in Mass and why is glaring...it drives me nuts that the Networks get bashed by print media and the pundits for the Exits not being perfect for prediction but never get credit for the service they provide for interpreting the results.
Very true. Too many of us assume that exit polls exist for the sole purpose of "calling" election results hours before the polls close -- an ability that has long been more myth than reality -- while the real value of the enterprise is helping us understand, once all the votes have been counted, who voted and why. Yes, we have all picked at the problems of exit polling, but if anyone has discovered a better method to survey actual voters while also correcting for apparent sampling bias, I am not aware of it.
A marginally related point: The networks do a great public service when they deposit the raw, respondent-level data in academic archives like those at the Roper Center and the Interuniversity Consortium for Political and Social Research (ICPSR). These data allow scholars to conduct all sorts of analyses that are impossible using only the simple cross-tabluations posted on election night.
Unfortunately, the raw data release takes time due to the slow process of creating accurate documentation of the samples interviewed nationwide and in all 50 states plus the District of Columbia. In the past, that process has taken many months (although the 2005 data release was expedited due to intense interest in the apparent exit poll miscues of 2004).
Scholars are still waiting on the release of the 2008 election, and the delay has produced considerable back-channel grumbling, some of which has reached me via email. Why has the release been delayed so long? That I cannot say, but I did contact the folks who are responsible for preparing the data, and they assure me that it is coming soon. The creation of documentation, they tell me, is now virtually complete, but "the files and the documentation are in the process of being reviewed [by the media sponsors], and this process should take a few more weeks to a month."
My column for this week examines an argument made last week by Mark Mellman, pollster for the Democratic Senatorial Campaign Committee (DSCC), that polls showing a close race between Scott Brown and Martha Coakley turned out to be a self-fulfilling prophecy.
The column ultimately addresses the question of why two automated polls showed a neck-and-neck race two weeks before the election, while some telephone surveys, including one conducted by Mellman's firm, showed a big Coakley lead. The column includes a statistic shared by Evan Tracey, founder of the Campaign Media Analysis Group, showing that Brown's television advertising exceeded a half million dollars while the first automated survey by PPP was still in the field.
After I filed the column on Friday, I heard from a Republican source who makes a point I did not address in the column: Brown's did not require television advertising to begin to gain on Coakley. His personal campaigning, as covered by Boston newspapers and television, helped boost his recognition and probably amplified the advertising that he ran in early January:
Massachusetts is the MOST politically aware state in the country. Behind the Pats and the Bosox, it's their blood sport. They FOLLOW it passionately. The number of verbatim comments from voters who brought stuff up that had never been advertised was amazing.
My source also passes along that by Saturday January 9, the day the first PPP poll finished interviewing, Brown's internal tracking showed that 65% of voters in the Boston market reported having seen the Brown truck ad.
The events of the week delayed my pointing to a terrific resource made available to pollsters and polling junkies over the weekend by Public Opinion Quarterly, the academic journal published by the American Association for Public Opinion Research (AAPOR). Their special issue on the 2008 Presidential Election is now posted online and access is totally free.
It leads with the article previously teased on Pollster.com by co-author Mike Mokrzycki on exit poll measurements of cell-phone only voters and implications for future polling. It also includes many more articles of interest to Pollster readers, including a review of poll performance in 2008 by Mike Traugott and Chris Wlezien, a look at how predication markets compare to polls in forecasting outcomes by David Rothschild and much, much more.
Again, POQ has made access open and free to all. Definitely worth a click.
On a morning full of big political news -- the apparent death of the health reform bill, a Supreme Court decision ending restrictions on corporate campaign spending and John Edwards admission that he fathered an out of wedlock child -- I spent the morning attending a briefing something even more exciting: What's in store for execution of the 2010 Census.
OK, maybe not that exciting, but the session sponsored by the DC chapter of the American Association for Public Opinion Research (AAPOR) and the Pew Research Center did provide a fascinating glimpse at one of the government's decennial effort to count and gather demographic data on its citizenry.
While newly installed Census Director Robert Groves did not break any news with his brief presentation, he conveyed a sense of the "massive undertaking" involved in executing the Census and passed along some information that would interest pollsters and data geeks:
- In March the Census will mail a census form to every residence in the United States, a procedure used by the Census since 1960.
- To improve the rate of response to the initial mailing, the Census will run a paid advertising campaign as it did 10 years ago. Groves added this statistic: "For every one percentage point that we raise the mail response rate through this advertising campaign, we will reduce the total cost to the Census by about $85 million," costs that they incur by sending follow-up mailings and in-person enumerators to gain full response (a video of highlights from the launch of the ad campaign is posted here).
- The Census is currently conducting a daily telephone tracking survey to monitor what Groves described as "key predictors" from prior research of what predicts likelihood of participating in the 2010 Census. They then run "predictive models" and "watch how the predictors are changing" overall and within key subgroups and, if necessary, tweak their advertising buy or messages appropriately Groves showed a tracking chart indicating that awareness started to climb following the New York City roll-out of the Census "Road Tour" (seen in the video below).
- As also indicated in the survey released by the Pew Research Center yesterday, younger Americans -- those between 18 and 29 years of age -- are the "laggard group" in terms of reported awareness and intent to participate. Groves, the noted expert on response rates in surveys, voiced a caution we rarely hear about telephone surveys: These data on younger Americans "are subject to great misinterpretation" since they are from the respondents of a random digit dial (RDD) telephone survey. "The proportion of all people sampled that became respondents is much, much, much, much, much, much lower than we'll ever get in the census, so these [younger respondents] are the cream of the crop" in terms of their willingness to participate in a survey, and thus perhaps, in the census itself. "The usefulness of this [survey]," he added, "is to watch this over time, to see if things are moving."
- Starting at the end of March, the Census will launch a tool on its web site that will allow anyone to monitor real time updates of the participation rate, featuring a thematic "heat map" that will display regional variation. You will also be able to "drill down" to see similar mapping for individual counties or zip codes, or create on screen comparisons between localities so "New York could compete with L.A." (or perhaps Washington DC with Dallas?).
- The Census is also undertaking a Census in the Schools initiative with the assistance of Sesame Street characters Rosita and Count von Count, who Groves describes wryly as his "senior technical adviser."
Not surprisingly, Groves now has a blog on the Census 2010 website.
Regular readers may recall my personal pet peeve about rushing to quick conclusions about the "most accurate pollster" in any given election. One of my objections -- all votes are typically not counted immediately -- is slightly less of a worry when applied to yesterday's Massachusetts race, as election officials there have produced an "unofficial" count based on 100% of the state's precincts. Still, the final certified count can sometimes differ slightly, sometimes enough to move the final count by a percentage point, so please take what follows as preliminary.
Another objection of mine, however, is even more valid in looking at a single state as it was back in February where we had several contests to consider:
[T[he whole notion of crowing a "big winner" based on a handful of polls in a handful of states is foolish. The final polls yesterday had random sampling error of at least +/- 3 percentage points. If a poll produces a forecast outside its margin of error, that's important. But if several polls capture the actual result within their standard error, chance alone is as likely as anything else to determine which one "nails it" and which miss by a point or two.
Let's use today's results to illustrate the problem. The following chart shows Brown's percentage of the vote as reported by the public polls conducted during the last 7 days of the campaign, with an error bar based the poll's reported margin of error. The horizontal bar represents Scott Brown's actual percentage of the vote.
What stands out most is that most of the polls produced an estimate of Brown's percentage of the vote within their own margin of error of the actual result. The exceptions are those on the left, which were conducted almost a week before the election, and if you followed our chart or read Charles Franklin's post on Monday, you know that Brown's support rocketed up over the course of January, so we should expect some of the earlier polls to show Brown's support lower than it turned out to be.
Another perennial issue with measuring the accuracy of pre-election trial heat questions is the issue of how to handle the undecided percentage. I did not allocate undecideds, and some polls had a bigger undecided percentage than others. Also, in this case, some pollsters included independent Joe Kennedy as an option, others did not. Kennedy ultimately received only 1% of the vote, but the Blue Mass Group/Research 2000 poll that missed the Brown percentage by 11 points had the biggest percentage either for Kennedy (5%) or undecided (5%).
So by and large, it is fair to say that all of the surveys conducted after Wednesday produced estimates of Brown's vote that were as accurate as a survey can be given the potential for random error.
You might reach a different conclusion, however, when you look at how they did forecasting Martha Coakley's percentage.
Here four surveys, all conducted after Wednesday, all significantly understated Coakley's percentage of the vote. Two were conducted for Pajamas Media by the Republican firm CrossTarget. The others were done by InsiderAdvantage and and by the Merriman River Group for InsideMedford.com. These four surveys are mostly responsible for the small understatement of Coakley's support in our overall trend estimate.
For what it's worth, all four used an automated IVR methodology and were completed in a single day, but all four also reported slightly higher undecided percentages than the others surveys conducted over the final weekend. So perhaps their significantly lower estimates of Coakley's support had something to do with their calling procedures, or perhaps they were not pushing undecideds hard enough.
I also produced the following table that calculates the error on each poll for each candidate and the error on the margin. The two surveys that missed the Brown's margin by the most were the Pajamas/CrossTarget and BlueMassGroup/Research 2000 polls conducted more than four days before the election - and they managed to miss in opposite directions.
Click table for full size version
A round-up of Pollster-centric reactions to the Massachusetts results:
This Boston Herald lead speaks volumes: "High turnout in Bay State 'burbs and among independent voters who flocked to the polls eclipsed a healthy turnout in staunchly Democratic Boston, fueling Republican Scott Brown's victory yesterday."
Brown pollster Neil Newhouse stays up late to offer his take on the message sent by Massachusetts voters, Brown's 12 keys to victory and a memo to Democratic leaders.
Coakley pollster Celinda Lake gives an interview to Ryan Grim defending their campaign.
The Note produces a December Coakley campaign polling memo.
Mark Mellman argues (pre-results) that early January automated (IVR) polls were a self-fulfilling prophecy.
The Republican National Committee releases a pre-election survey (via Memoli).
Rasmussen Reports conducts an election night survey.
PPP's Tom Jensen offers takeaways from Massachusetts.
Nate Silver allocates blame.
Washington Post pollster Jon Cohen reminds us of what we gets lost in the absence of exit polling and a proliferation of horse-race only polls.
Former AP polling director Mike Mokrzycki reviews what exit polls could have told us.
John Zogby predicts a Coakley win, then walks it back.
Political scientist Josh Tucker graphs the actual effect of the Massachusetts election, but Brenden Nyhan offers a different interpretation.
PS: Don't forget Charles Franklin's post below ("How Massachusetts Votes Shifted").
Late add: Research Rants adds some helpful perspective from a distance.
Other assessments of the Massachusetts polls and voter turnout from around the web:
Suffolk University shows momentum for Brown in a poll of bellwether counties.
Nate Silver sees Brown as a 3:1 favorite, gives Coakley a 25% chance of winning.
Chris Bowers gives Coakley a 35% chance to win (his assessment yesterday of the performance of polling in previous general, primary and special election is well worth reading).
Harry Enten shows Brown winning regardless of turnout.
Gallup finds the same percentage of Democratic identifiers in Massachusetts as the nation.
Martha Coakley says the polls are wrong.
So Does GregR at Blue Mass Group.
Jim Geraghty notes Brown's support among Democrats.
Reid Wilson looks at the stakes for pollsters.
WBZ reports that MA Secretary of State William Gallup expects a 40% to 55% turnout among registered voters, more reports on turnout from Ben Smith, TPM, Reid Wilson.
Patrick Ruffini sets up a Google Spreadsheet to track results at the town level (brilliant!).
PS: As I noted a few days ago on Twitter, the networks are not be conducting an exit poll in Massachusetts today. Network interest in doing one, I'm told, occurred too late for the long lead time necessary. We'll probably live-blog again tonight...check back in later for details.
Update: More...
The Cook Political Report's David Wasserman suggests some bellwethers to watch.'
Mark Ambinder posts a "what to watch for" cheat sheet with links to websites reporting results.
My column today considers the proliferation of inexpensive automated surveys and two developments that promise to fuel even more growth: Two companies that promise to offer the ability to conduct an automated telephone poll for less than a thousand dollars.
I filed the column on Friday afternoon, which in Massachusetts Senate time was seven polls ago, and those new polls underscore the message of the column. Nine of the 16 polls we logged on Pollster.com since January 1used an automated telephone methodology, including 5 of the last 7. Two of the surveys, those sponsored by Pajamas Media and and InsideMedford.com, used pollsters whose work we have not previously tracked.
Of those 15 polls, only two -- the surveys conducted by the University of New Hampshire and Suffolk University -- had traditional, mainstream media sponsors. The rest were conducted or sponsored by polling public relations, political partisans, "new media" web sites or some combination of these three.
So the vision of the future described in the column is, in many ways, already upon us.
Over the last few days, thousands of words have been written and many charts posted on Pollster and elsewhere, all trying to make sense of the sometimes divergent Massachusetts Senate polls. If you haven't yet, make sure you read Charles Franklin's tour de force review posted here a few hours ago, that walks through eighteen permutations of trend line models, all of which show Republican Scott Brown leading Martha Coakley, most showing him ahead by 4 points or more.
If you prefer a simpler summary, consider this:
- Of eight surveys completed and released since Wednesday, seven show Brown leading by at least a point. The one exception shows a dead heat Our chart of all polls shows a nearly seven point Brown gap between the trend lines for Brown and Coakley (51.2% to 44.3%).
- Browns' support on our standard trend estimate has increased by nearly twelve points (from 38.5% to 51.2%) in just the last two weeks.
A trend this strong is unusual, especially in a contest between a Democrat and a Republican. We do see such surges occasionally in primary elections -- the surprise victory by Creigh Deeds in last year's surprise victory in Virginia's Democratic primary being the most recent example -- but they are far more rare in general election contests . Over the weekend, I reviewed the most competitive contests we have tracked on Pollster.com since 2006 and found no race that produced a trend anywhere near this strong over the last few weeks of the campaign.
I am sure that there are other example, but the one that stands out for me is the victory of Democrat Harris Wofford in 1991. Wofford, appointed earlier that year to fill a vacant Senate seat, began as a virtual unknown and began trailing by more than 40 points against popular former Republican Governor Dick Thornburgh. Although the final round of public polls showed the candidates running about even, Wofford's momentum helped carry him to what turned out to be an eleven point victory margin (55 percent to 44 percent).
Of course, the same factors that make the trend toward Scott Brown so unusual also make the polling challenging and potentially misleading. Brown has moved up so rapidly partly because campaign has been truncated, but the rapid change also prompted a late avalanche of negative advertising by the Democrats directed at Brown. Because it is a special election being held on an usual date, Pollsters have no prior history to judge the size and demographics of the likely electorate. The likely voter problem is one reason why polling errors tend to be larger in special elections.
So while we have the Wofford experience on one hand, we have the lessons of the New Hampshire Democratic primary in 2008 and the special election in New York's 23rd District this past fall on the other. In both cases, candidates surged in the final polls, only to see their apparent leads disintegrate on Election Day. What those races had in common were huge surprise developments that occurred a few days days before the election (Barack Obama's Iowa victory and the withdrawal of Dede Scozzafava) that helped shake up the race, fuel the polling surge and -- perhaps -- provoke voters to focus more closely on their choices and rethink their preferences in the final hours.
Does the nationalization of the Massachusetts Senate race combined with the heavy negative advertising blitz qualify as the same sort of last minute surprise? Perhaps, but it seems like a stretch to me.
Some believe that non-response bias may have contributed to the errors in those two races, exaggerating the contribution of the most enthusiastic supporters of the surging candidate. Mike Mokrzycki developed that theory in the context of the Massachusetts Senate race here over the weekend. Some believe this phenomenon may be more acute in automated surveys, and we should not ignore that only two of the last seven public polls used live interviewers.
Yes, Coakley has done better on live interviewer surveys than the automated polls, but we saw a similar pattern in New Jersey last fall, and the robo-polls ultimately provided a closer forecast of the final margin.
Yes, the internal Coakley campaign poll numbers that have leaked out show a dead even race and perhaps a slight improvement over last week. However, there was more than one internal poll conducted by Democrats A little birdie tells me that the final tracking survey conducted by the Mellman Group for the Democratic Senatorial Campaign Committee had Brown ahead by five points.
So for me it boils down to this: I was a Democratic consultant for long enough to want to believe that Coakley can still prevail, and there is still a remote chance that the polls in this race will be as misleading as they were in New Hampshire. However, my head is not my heart. Barring another polling meltdown, Scott Brown is the likely winner.
This is the first of two posts that will wrap up my thoughts about the Massachusetts Senate race. This first one covers an admittedly narrow point. It is provoked by an email I received reacting to Nate Silver's post from Sunday -- "A Statistical Ray of Hope for Coakley" -- from Pollster reader Harry Enten (aka commenter Poughies):
[Silver's] piece makes the point that the margin between Martha Coakley and Steve Brown could be overstating Brown's lead. Silver points out that polling in close (margins of 10 or less in the polls) Senate elections since 2000 in deeply blue states (as measured by the Cook Political Partisan Index) has by an average of 3.4 points underestimated Democratic candidates' margin of victories. In deeply red states, on the other hand, polling has underestimated Republican margins by 1.9 points.
Though he used a Pollster.com average in 2008 and Real Clear Politics average in 2006 and 2004, Silver used a simple average of all non-partisan polls conducted in the final two weeks compiled by Pollingreport.com in 2002 and 2000. Perhaps, he did not know, but a Real Clear Politics average is also available for 2002 and 2000.
I was interested what, if any, effect substituting the Real Clear Politics averages in 2002 and 2000 would have on Silver's results. Therefore, I decided to create a new dataset modeled after Silver's, but using the Pollster.com data in 2008 and Real Clear Politics average* in 2006, 2004, 2002, and 2000.
Using these new rules, the underestimation of Democratic margins in blue states stays the same at 3.4 points. This result is not surprising, as the blue states part of the dataset is small, and only three results are available from 2000-2002. The underestimation of Republican margins in red states drops from 1.9 to 0.9 points.
The overall average of underestimation drops from 2.3 to 1.5 points. In only 7 of 23 contests was the underestimation above 3 points. In no contest did the polling average incorrectly predict the winner due to underestimation of Democratic candidates in blue states or Republican candidates in red states. The only incorrect winner chosen was in the South Dakota (a red state) 2002 race when the polls predicted a victory by Republican John Thune.
The bottom line is that perhaps the polls are overestimating Brown's margin in Massachusetts. The limited data involving closely polled elections in blue states suggest that Coakley might do better than the polls suggest, but when you look at the larger dataset of red states, Coakley should not expect a bump.
*In some cases (such as Alaska 2004), no Real Clear Politics "average" existed. I just averaged all the polls listed on Real Clear Politics (and in the case of Louisiana 2002) conducted in the final two weeks. I, unlike, Silver use internal polls in these cases... modeling myself after Pollster.com's inclusion of them.
P.S. The Real Clear Politics web pages can be hard to find [but are available at these links] for 2002 and 2000.
He also passed along a table (in Excel and PDF formats) with the numbers.
It's also worth taking a closer look at the six races -- there were only six -- that are the basis for the conclusion that polls understate support for Democrats:
Most interesting are the three races (Maryland and Rhode Island in 2006 and New York in 2000) that produced the biggest errors. Of these, the Rhode Island example is partly the result of an RCP average based on just two surveys conducted over the final weekend of the campaign. We used a simple last-5-poll average for Rhode Island that year, which showed Whitehouse winning by six, just one point off the actual margin. Just swapping the Pollster and RCP averages for that one race would cut the average variance to +2.5. Whatever the challenges of polling in Massachusetts, greater random error due to a shortage of final polls is not one of them.
That leaves two big errors affecting Ben Cardin in Maryland in 2006 and Hillary Clinton in New York in 2000. What's interesting about the errors in both contests was that the final round of polls had the percentage for the Republican candidate about right (within a point), but understated the support for the Democrat by about five percentage points. Even if we assume that a similar pattern will apply in Massachusetts tomorrow, the problem is that the six of the last seven surveys estimate Brown's support at 51% or greater.
One of the odd aspects of last week of the Massachusetts Senate campaign is the way Coakley "internal" polling numbers have leaked on a near daily basis, through blogger Steve Kornacki and others. Wednesday night, according to Kornacki, the Coakley campaign's own polling showed her "barely ahead, 46 to 44 percent." Thursday night's results showed her trailing, 47 to 44 percent, and conservative columnist Byron York added a quote from an unnamed but "well-connected Democratic strategist" who "heard" that "in the last two days the bottom has fallen out of her poll numbers." Then on Friday night, again according to Kornacki, Brown was ahead by just two points on Coakley's poll (47% to 45%), and a three-day average of the results from Wednesday through Friday night gave Brown the same two-point lead (47% to 45%).
These leaks produced some snickering: Via Twitter, PPP's Tom Jensen pronounced the leaks the "sign of a highly undisciplined campaign." Jay Cost asked "how lame is the Coakley campaign" to leak their internal tracking polls "EVERY DAY?" And a very smart reader emailed this morning with the observation that leaks mark Coakley's campaign "more undisciplined than a 4 year old at K-mart on a sugar high."
Let me be clear: The conclusion that Coakley's campaign -- her staff or the consultants she retains -- is responsible for these leaks is probably unfair and a bit naive. They were likely not the source.
Now, alas, I do not have any inside information and have not been the recipient of any such leaks (really, old consultant comrades, where is the love?). But I can say from my own experience as a Democratic campaign pollster that it's fairly standard practice for a Senate campaign like Coakley's to share their daily tracking results with the Democratic Senatorial Campaign Committee (DSCC), the White House political office and EMILY's List. It's really unimaginable that Coakely, or any other "disciplined" campaign in their position, would not. Also, either directly or through these organizations, the same are almost certainly going to the labor groups and other interests conducting their own campaigns on Coakley's behalf. Each of those organizations has its own pollsters, media and direct mail consultants. So the leaks could have come from damn near any "well connected strategist" in Washington.
The incentive to leak would be especially high for those who have parachuted in to help in the final week, those with great incentive lay the groundwork to take credit should Coakley "come from behind" to win. Consider that these incentives are even greater for the White House. Obama really had no choice but to come to Massachusetts to Coakley's aid (as he is doing today). A Coakley loss will be catastrophic for Obama's legislative agenda, and the White House will take some of the blame either way. So a mid-week decision to come to her assistance creates huge incentive to leak these numbers. Again, if she loses, well, the bottom had already "fallen out." If she wins, they claim credit for turning things around.
Now all of this probably speaks to a breakdown in team play or the sort of ugly finger pointing that always seems to accompany defeat. For those surprised by the wide dissemination of "internal" Coakley polling data, consider that in the fall of 2008, the Obama campaign shared polling numbers and a whole lot more on a daily consultant conference calls whose participants (I'm told) close to a hundred. Nothing of significance leaked from those calls before Election Day. One way or another, Martha Coakely and her campaign are worthy of much criticism, but piling on over these leaks is unfair.
By the way, it's more than a little crazy to be paying much attention to the random zigs and zags apparent in the relatively small one-night samples used in internal campaign polling. I certainly hope that the pollsters of record are not making decisions or recommendations on the basis of anything but the three-night rolling averages.
This morning, I'm sure of only one thing about the Massachusetts Senate race: The perception among Massachusetts voters that Democrat Martha Coakley is likely to defeat Republican Scott Brown, is not long for this world. Even the newly released Suffolk University poll shows that by a better than two-to-one margin (64% to 26%) Massachusetts still believe Coakley will win. But since Suffolk is a local University, uses live interviewers and has the Boston Herald and Boston's NBC affiliate as sponsors, their finding that Brown leads Coakley by a not quite statistically significant four points (50% to 46%) is huge news in Massachusetts this morning.
In addition to the Herald front page, the poll was also the big story on all of the local Boston television stations last night or this morning (online video available as of this writing at WCVB-5, WHDH-7, WFXT-25, WBZ-38). Watch those stories, and its hard to imagine that perceptions of a likely Coakley victory will survive the weekend. If you are a Democrat, that is probably the only silver lining in today's news.
On the other hand, the tone of these stories follows the all too typical pattern of political news coverage: an analytical focus on strategy and tactics: "What happened to Coakley's lead?" "Why is Brown surging?" In assessing tactics, they inevitably praise Brown's efforts ("Brown has been out-hustling Coakley on the campaign trail"), while dissecting the apparent failures of the Coakley campaign. If "momentum" is a factor in campaign politics, this sort of coverage is a big reason why.
Another troubling pattern for Democrats: To the extent that these stories discuss the negative advertising being run by Coakley and her Democratic allies this week, it is only as a possible explanation for her poll numbers. The WBZ story, for example, cites callers to local talk radio and emails to the station complaining about negative ads. The WBZ anchor then concludes:
That is the combination of [Coakley's] problems, visibility and negativity. You can go negative, political ads work, political consultants always say that, but only if the voter knows exactly who you are, so at this point, since there is this perception that she hasn't been out there hustling as much...since voters don't know who she is, all they see from her is negativity at this point, and at the 11th hour, that's tough to overcome.
That's not quite right. Negative advertising works when voters see its message as credible. Ad buys as heavy as the combined efforts of the Coakley campaign, the DSCC and SEIU have undoubtedly been noticed and, as such, will create some cognitive dissonance among voters still leaning to Brown. The big question is how those voters resolve the dissonance. If they come to accept the arguments the ads are making as valid, some may back away from supporting Brown. But cognitive dissonance theory says that denial and rationalization are more powerful instincts than acceptance, so it is easier for voters who already like Brown to dismiss the content of the ads as typical political "mudslinging."
The key to resolving that dissonance is the way the news media covers the campaign: If news stories focus on the substance of the ads and the debate between the candidates, there is a greater chance that the ads will have an impact. If coverage focuses on tactics alone -- as horse race stories inevitably do -- the ads are more likely to fail.
One last thing about the Suffolk Poll. One astute Pollster reader emails with a question: The poll asks "if you know when the election is (and terminates the interview if you don't have the right answer). Is that unusual for special elections?"
That question is a little unusual, in my experience, but in fairness to the Suffolk University pollsters, there really is no "usual" with likely voter screens, especially in special elections. Selecting likely voters is really where political polling is more art than science. To make matters worse, pollsters do not typically reveal the full text of their screen questions, so give the Suffolk pollsters credit for being fully transparent on that score.
I think their screen is reasonable. After all, you're not very likely to vote if you don't know the election is next week [UPDATE: but see the contrary view of reader Dan below]. The classic Gallup likely voter model includes a similar question about knowledge of your voting location (although that is just one item in a seven question scale). I would question the Suffolk screen if I believed that the Coakley campaign was poised to mount a massive weekend get-out-the-vote effort aimed to reminding identified supporters about where and when they vote. By most accounts, that is not likely.
Watching a clip of Glenn Beck interviewing Sarah Palin, Andrew Sullivan catches this comment from Beck: "I don't know yet if [Palin's] strong enough, if she's well-enough advised, or if she knows she can no longer trust anyone." Sullivan goes on to comment that "distrust of everything in politics, of every politician, of the 'system' that has been co-opted by mysterious and menacing elites, and a sense of total beleaguerment in the modern world" has become a familiar theme from "the far right."
For what it's worth, "distrust of everything" was also a prominent theme of a briefing I attended this morning on the new Allstate/National Journal poll. As my colleague Ron Brownstein put it this morning, in an economy where they are "more directly exposed to financial risk than earlier generations," Americans "don't have much confidence that any institution, government, business or the financial sector is doing much right now to help them" deal with that risk.
"As the recession has grown more prolonged," he added, "this alienation from institutions has only deepened and hardened." In follow-up interviews he conducted with survey respondents, Brownstein found "more of an edge of fear, even desperation. A sense of this economic weight settling on people and and being uncertain when the cloud is going to lift." (I've embedded the full video of the presentations below -- Brownstein's comments are at 9:55, the presentation by FD pollster Ed Reilly begins at 13:30).
The data that Brownstein had in mind came from a set of questions that asked respondents to report how much trust they have in a series of people and institutions "to help you manage the financial risks in your life." Most expressed "a lot" of trust in themselves (74%) and their spouse or family members (64%) but others ranked much lower:
Just 15 percent say they have "a lot" of trust in financial advisers to help them cope with financial risk. Labor unions ranked next in trustworthiness, at 12 percent.
Fewer than one in 10 express "a lot" of confidence in national banks, corporations, or the federal government. Strikingly, about half of those polled say they have no trust in any of those three institutions to help them. "These government plans aren't working out, and these corporations are using the government to their benefit," insisted Scott Holland, whose timber business in Garner, N.C., is bankrupt. "I really don't trust anybody at this point."
The flip side of this distrust is a sense that while Wall Street has benefited from government efforts to jump start the economy, ordinary people have seen little benefit:
At the root of the dissatisfaction crackling through the survey is the widespread belief that government's response to the economic crisis mostly benefited affluent individuals and powerful business and financial institutions--the very groups that many respondents blame for the upheaval. "There weren't enough consequences on some of the people who were part of the demise of the banks," said William Fields, a retired engineer and political independent from Villa Hills, Ky.
Who has "benefited most" from "the actions the federal government has taken to respond to the financial crisis over the last 12 months?" Three-quarters of those polled say it has been Big Business or the rich. Forty percent picked banks and investment companies, with another 20 percent identifying major corporations and 16 percent choosing wealthy individuals. Just 9 percent contend that the middle class has been the principal beneficiary, and 8 percent say that the poor have benefited most. African-Americans are more likely than whites to see middle- or lower-income families as the major beneficiaries. But in all demographic groups, the sense is widespread that over the past year most of the jam was placed on the top shelf, to paraphrase Texas populist Jim Hightower.
The survey also asked what "large financial corporations" could do to "increase the trust you have in them." The proposal that drew the most favorable response, by far, was "paying back the bailout money they received from the federal government as soon a possible." More than half (59%) said that would increase their trust "a lot," 28% said it would increase it "some" and only 11% said "not at all.".
An audience member asked about a tax increase on financial institutions proposed today by President Obama to help defray the costs of the financial bailout. Pollster Reilly responded (at 57:55 in the video) that the measure is "obviously very much aligned with where public opinion is" and that "in order to maintain any kind of a consensus moving forward politically, they have to demonstrate to these institutions that benefited from government assistance during the last year, that a fair deal has been struck and that they are paying back."
So responding to a sense of "distrust of everything in politics" is not just the province of the far right. It is a reaction to public opinion across the board.
[Special thanks to the National Journal's Ryan Morris and Reuben Dalke for sharing their beautiful graphics above that will appear in the print edition of the magazine this week].
Given the humanitarian crisis in Haiti, the internet has been buzzing with ways to help. I want to amplify this suggestion from The Nation's Peter Rothberg:
There are numerous ways to help groups already on the ground. One of the best, Partners In Health, has been operating in the country since 1987, originally to deliver health care to the residents of Haiti's mountainous Central Plateau region. PiH now also operates clinics in Port au Prince and other major Haitian cities. With hospitals and a highly trained medical staff in place, Partners In Health is already mobilizing resources and preparing plans to bring medical assistance and supplies to areas that have been hardest hit. Donations to help earthquake relief efforts will be quickly routed to the disaster.
You can make an online donation to Partners in Health on their website. According to this detailed update on what they are doing to help, their "greatest need is financial support."
I am seconding that recommendation because of a similar testimonial I heard this morning from my spouse, a physician and former colleague of Paul Farmer, Partner's executive vice president, when they both practiced at the Brigham & Women's Hospital in Boston.
Farmer's work in Haiti was the subject of Tracy Kidder's best-selling book, Mountains Beyond Mountains. To get a sense of the story, and why my wife has total confidence in Farmer's work, here are two brief excerpts of an interview Kidder gave last year:
I had met [Paul Farmer] in 1994 and found him intriguing, but I think the decisive moment was when I saw his health center in Haiti for the first time in 2000.You travel from the airport along this horrible road where you mostly notice the absence of things: Electricity, arable land, even trees. And after three hours of witnessing unremitting misery all around you --people without food, without shoes-- you come to this verdant citadel that provides high-quality medical services to everyone for miles around, regardless of their ability to pay. I remember feeling that if it was possible for this to be here, then anything was possible.
[...]
It's amazing: They now have nine sites in Haiti, including four hospitals complete with operating rooms, and they've got AIDS under control in the entire central plateau of Haiti. I mean Haiti is still in desperate shape, but this is something good that's happened there, and they're continuing to expand. They now have about three thousand staff members of Partners In Health in Haiti, almost all Haitian, and a total of about five thousand worldwide.
Take it from my family: If you want to make a donation that will be put to immediate use to help those in need in Haiti, you can do no better than Partners in Health.
Update - An even better testimonial from Kidder comes this morning in an op-ed in today's New York Times:
But there are effective aid organizations working in Haiti. At least
one has not been crippled by the earthquake. Partners in Health, or in
Haitian Creole Zanmi Lasante, has been the largest health care provider
in rural Haiti. (I serve on this organization's development committee.)
It operates, in partnership with the Haitian Ministry of Health, some
10 hospitals and clinics, all far from the capital and all still
intact. As a result of this calamity, Partners in Health probably just
became the largest health care provider still standing in all Haiti.
Fortunately,
it also offers a solid model for independence -- a model where only a
handful of Americans are involved in day-to-day operations, and
Haitians run the show. Efforts like this could provide one way for
Haiti, as it rebuilds, to renew the promise of its revolution.
The flurry of commentary on polling in the Massachusetts Senate race plus a handful of new polls -- some real, some rumor -- provide some new topics worth discussing. I'm going to take these in order, but none take away from the obvious conclusion that the race is likely to be a lot closer than most assumed a few weeks ago.
1) Recent Trend? On Sunday, I bemoaned the lack of apples-to-apples polling comparisons available over the last few weeks. Yesterday's update from Rasmussen Reports showing Scott Brown trailing Martha Coakley by just two percentage points (49% to 47%) appears to provide such a comparison, since Rasmussen's track a week ago showed Coakley leading by nine (50% to 41%). But read the fine print:
The results of this poll are not precisely comparable with last week's results because this poll includes the independent candidate by name while the previous poll simply offered the choice of "some other candidate." Additionally, the latest poll results include "leaners."
Leaners are those who don't initially have a preference for one of the major candidates but indicate that they are leaning in that direction. Without "leaners," Brown was actually ahead by a single percentage point.
The pressing of leaners is the bigger issue here, particularly given a pattern I'll discuss below. Rasmussen shows independent Joe L. Kennedy receiving just 3% of the vote (this Kennedy is unrelated to the more famous Kennedy family that includes the late Senator and his nephew Joseph P. Kennedy, who once represented Boston in Congress).
Now, I don't want to belabor this point. My colleague Marc Ambinder reported last night that the "internal Democratic tracking in MA last week had Coakley up by 15. Today, she's up by five." And there is no denying that Brown has significantly narrowed the gap since November, so it's likely that he has continued to gain in recent days.
Our chart above -- which happily compares apples and oranges without remorse, but does not include the rumored internal polling -- does indicate a narrowing margin since January 1.
2) Turnout or Persuasion? On Sunday, I argued that "turnout matters," mostly because cross-tabulations in both the Boston Globe/UNH and Rasmussen surveys show a much closer race among the most interested and likely voters than among other respondents. This latest Rasmussen survey (added to the bottom of the table below) confirms the trend:
We can assume that a special election will draw fewer voters than an off-year general election for Governor or Senator held in November, but it is not clear where to draw the line in defining the likely electorate. And efforts to increase turnout will help Coakley: And for every three previously disinterested voters who change their minds and decide to vote this week, two will be Coakley supporters..
In that context, I have to agree with Chuck Todd and company at NBC's First Read:
[I]t probably doesn't help Brown that the contest has been nationalized. All the ads Democratic and conservative groups are now airing, all the money that's now flowing into the race, and all the reminders about how health care hangs in the balance will likely boost Democratic enthusiasm.
That said, I agree with Nate Silver that the close nature of the race is "not just about turnout." As Pollster reader Harry Enten (aka Poughies) noted on Sunday, the voters identified as independent on the automated surveys support Scott Brown by margins of better than two-to-one, while the Boston Globe/University of New Hampshire poll shows a very slight Coakley advantage among both independent identifiers and the much larger group of those who report they declared no party affiliation when they registered to vote.
Having made these comparisons, let me offer some big caveats: In comparing "independents" we have two variables that might introduce house effects -- the likely voter screen and the way the pollster asks about party identification -- plus much larger random error (as these smaller subgroups typically involve much smaller sample sizes). That said, I can't account for all the difference between the Globe and the other polls on the basis of the LV screen and question wording alone. Given that, as Harry Enten notes, the surveys are much more similar in terms of the results they report for Democrats and Republicans, it looks like the differences separating the Globe and the automated surveys have a lot to do with those who consider themselves independent.
[An aside: it is likely that a more stringent likely voter screen would have a slightly disproportionate impact on independents. I checked some national pre-election polling data from 2004 and 2008 and found that while narrowing from the least restrictive to most restrictive likely voter models makes independents slightly more Republican leaning, it is hard to see how such a phenomenon alone explains the 20+ point differences in the table above].
If you accept the very close margins on the PPP and Rasmussen surveys as real, then Brown is successfully persuading a lot of non-Republicans to support him who typically vote Democratic.
Here's a hypothesis that might explain the pattern: if Brown ekes out a victory or comes within a few percentage points of winning, it will because he wins the support of a lot of voters -- most of them independent -- who typically vote Democratic. Brown has probably not yet closed the sale with these voters, given their prior vote history, but they are poised to support him. Perhaps it is harder for them to tell a live interviewer they are ready to vote Republican. Perhaps the more anonymous nature of the automated methodology better simulates the act of voting which will ultimately force a decision.
3) A 50% Coakley Ceiling? On Monday, pollster Scott Rasmussen noted that all three surveys available then "show Coakley right around the 50% mark....If Coakley is truly right around [50%], then the race is hers to lose, and Brown's best possible scenario is a very narrow victory." Rasmussen is correct about the consistency of of Coakley's support. It extends to his most recent survey and to the narrower cuts of likely and interested voters from both the Rasmussen and Globe surveys:
Consider again the theory I offered above. It may be that Brown is on the verge of winning the support of a lot of voters who typically vote for Democrats, so the differences in methodology -- how hard each pollster effectively pushes for a decision -- produce a much bigger variation in his measured support.
Sean Trende of RealClearPolitics has a different theory. He argues that the pattern evokes the much discussed "incumbent rule" (popularized by pollster Nick Panagakis) and thus expects a decisive break of undecideds likely for Brown. Trende concedes that the rule has weakened over the past decade, a development that he attributes to the faster flow of information in the internet/cable age, something Mickey Kaus has dubbed the Feiler Faster thesis.
Previously, incumbents generally enjoyed strong name recognition, while challengers were typically unknown. The reasoning behind the undecided rule," Trende says, "is that if voters haven't fallen in love with the incumbent by election day, they aren't going to vote for him (or her). The undecideds, therefore, can be expected to take a flier on the challenger." He argues that in a world where information flows faster, races for Governor, Senate and Congress now "receive a lot more scrutiny than they used to," so challengers are better known and the "rule" breaks down. At very least, the rule "is probably inapplicable as a predictive device...when you have two well-known candidates."
This special election, he says, is different, and Coakley is effectively the incumbent:
We have a sitting Attorney General who came out of a contested primary, going up against a more-or-less completely unknown state Senator. She's struggling to get above 50%. All of this points toward a very close final race -- potentially much closer than a week ago when I guessed at a 54-46 spread. Again, this is also consistent with what we're seeing in the variance in the Coakley/Brown numbers. Coakley should be worried.
Coakley should be worried -- and may well face a ceiling of support near 50% -- but I would not count on it. First, Brown is a long way from "completely unknown." Even the Globe poll, completed a week ago, found more than two thirds (69%) of likely voters able to rate Brown either favorably or unfavorably. Second, while Coakley is better known, the name recognition disparity between the two candidates is not unusual, even in the internet age.
Finally, I think Trende misses the best explanation for the incumbent rule, offered three ago by my old boss, Democratic pollster Harrison Hickman:
Hickman pointed out that in the 1980s, the conventional wisdom was to avoid mention of your opponent, a habit that helped explain why challengers won much of the late undecided vote. Now, he said, the general pattern is for incumbents to vigorously attack challengers throughout the campaign. "Incumbents put so much more pressure on challengers then they used to." (See this pre-election column by Dick Meyer of CBS News that includes data Hickman gathered showing the impact of negative advertising on candidate favorable ratings since 1986).
And of course, over the last 24 hours, the Coakley campaign, the DSCC and SEIU have all devoted significant media buys to negative attacks on Brown. Those attack ads may or may not persuade (a different issue), and will certainly be answered, but if we are truly living in a Feiler Faster world, expect the negative messages to disseminate rapidly.
Five years ago, a certain blogger noticed a similar pattern in the polling on the Bush-Kerry race in Ohio. Bush's numbers were amazingly consistent -- four polls had him at 47% and one at 46% -- while Kerry's numbers fluctuated between 45% and 50%. The blogger speculated that this pattern showed "the underlying principles of the Incumbent Rule in action" and boldly predicted that Bush "is likely headed for an Ohio defeat."
Bush carried Ohio, 50.8% to 48.7%. Incumbents did not break decisively to Kerry. If anything, Bush gained over the last two weeks, mostly because he made the race as much about Kerry as Bush. The big question hanging over the Massachusetts Senate race is whether Coakely and her allies can do the same to Brown over the next week.
An article yesterday by Larry Peterson of the Savannah Morning News rehashes much of the Strategic Vision, LLC controversy covered in my column a few weeks ago, but also features some new comments from Strategic Vision CEO David Johnson and this bit of news:
Some pollsters include cross-tabs with their results; others supply them only to paying clients. Until now, Johnson has said he's among the latter.
But [Johnson's] opted for a nod to critics who say his lack of transparency raises suspicions that he may have at least cut some corners.
He's provided the Morning News cross-tabs for a recent Georgia survey. And says he'll provide them to the news media with a survey his firm plans to conduct later this month.
No doubt, they'll be scrutinized by experts.
No doubt. If only someone would publish them. Mr. Peterson, if you have crosstabulations for a Strategic Vision poll, you may be the first, It would help if your newspaper could share with your readers whatever Johnson provided.
If Johnson's promise sounds familiar, it is probably because he made a similar pledge to the Atlanta Journal Constitution's Jim Galloway back in September:
Events of the last week have caused Strategic Vision to come to the same conclusion about its future polling.
"We're going to release all the crosstabs, and put an end to this right now," Johnson said. "That will squelch anybody from saying anything."
We're still waiting.
Finally, Peterson's article includes this curious reference to me under the heading "Partisan Flap:"
Johnson says the flap has a partisan dimension.
A conservative, he works mostly for Republicans; Blumenthal is an avowed Democrat and has served as a director of the opinion research group.
I'm not sure how an "avowed Democrat" differs from an ordinary "Democrat" (except that the former sounds more sinister), but the second half of that sentence is incorrect: I have never been the "director" or president of the American Association for Public Opinion Research (AAPOR) or of any other "opinion research group," as Peterson reports. I did serve for two years as a member of AAPOR's 15-member, all-volunteer Executive Council, something disclosed in almost every column or blog post I have written on this subject.
My column for this week looks at whether Sen. Christopher Dodd was right to insist that "any certain prediction" of his defeat in 2010, had he chosen to run for reelection, "would be absurd." While the odds of an incumbent Senator coming back from a double-digit polling deficit this early are very long, there are a few examples that I review in the column.
Thanks to the pollsters I emailed and my Twitter "tweeps" for digging into their collective memories of Senate campaigns in the not-so-recent past.
Note: most of the poll results for past races cited in the column come from the subscriber-only archives of The Hotline, thus the lack of links.
