Pollster.com

Articles and Analysis

 

Re: The Literary Digest Poll

Topics: Coverage error , Dominic Lusinchi , George Gallup , Jan Werner , Literary Digest , Pollsters , Probability samples

Last week, I linked to an article on Dr. George Gallup that included a reaction from statistical consultant Dominic Lusinchi that had been posted on the members-only listserv mailing list of the American Association for Public Opinion Research (AAPOR). Lusinchi argued that the original article perpetuated two "myths" about the infamous Literary Digest poll of 1936 -- the poll whose failure helped establish competitor Gallup as a household name. No sooner had I reproduced Lusinchi's email than Jan Werner, another knowledgeable AAPOR member took to the listserv to quarrel with Lusinchi's version of the history.

While a discussion of the shortcomings of a 74-year-old poll may seem a little out of place here, it involves issues that remain highly relevant to the contemporary debates about modern surveys, including the merits of probability sampling and the errors that can result from poor coverage or a low response rate. Since I posted Lusinchi's original comments, I also want to share the full exchange, which begins after the jump. Thanks to Dominic and Jan for allowing me to reproduce their comments here.

Original comment from Dominic Lusinchi:

This story perpetuates two myths:

1) That Gallup "predicted" that the Digest poll would forecast a Landon victory and

2) That the Digest failed because its sampling frame was "skewed ... to the wealthy".

Myth 1: In a July 12, 1936 syndicated column "America Speaks", Gallup wrote:"If the Literary Digest were conducting its poll at the present time [my emphasis], following its usual procedure, Landon would be shown in the lead." (Wash. Post, Section III, p.2, col. 7, Sunday, July 12, 1936) It's one thing to say "at the present time" and another to say "when the Digest presents its final results".... It is only after the Digest poll debacle that this story morphed into a "prediction". What Gallup really predicted, at that time (7/12/1936), was that the election was going to be a close one: the title of his column "1936 Election Seen As Closest in Years".

Myth 2: The Digest poll failed because its original sample, composed mainly of telephone and/or car owners, was irretrievably skewed against Roosevelt. A close analysis of a May 1937 Gallup (yes, Gallup!) poll, which asked its respondents if they had received and returned a Digest ballot card, shows that the principal cause of the Digest poll's failure was non-response bias. As Peverill Squire wrote in POQ (vol. 52, 1988, p.125), "if all those who were polled had responded, the magazine would have, at least, correctly predicted Roosevelt the winner." In fact, its prediction (my analysis) would have been as good if not better than Gallup's - he was off by nearly 7 points of the two-party vote.

Jan Werner:

It's one thing to debunk a myth, it's something quite different to replace one myth with another.

Myth 1

What Gallup wrote in the July 12 1936 article was: "...if The Literary Digest were conducting a poll at the present time...the actual figures would be in the neighborhood of 44 per cent for Roosevelt and 56 per cent for Landon." Although it was presented as a throw-away comment, Gallup did not come up with those numbers lightly. As he explained to a journalist a few years later, he "was able to call [The Literary Digest's] shot by making a separate survey of telephone subscribers and automobile owners -- the class which would mail back most of the Digest's ballots." (Williston Rich, Jr., "The Human Yardstick," Saturday Evening Post, January 21, 1939)

The statement was clearly intended to provoke a response from The Literary Digest and thus generate publicity for Gallup's fledgling organization. In that regard, he succeeded. On July 19, 1936, the New York Times quoted an open letter from Wilfred J. Funk, editor of The Literary Digest, as follows: "I am beginning to wish ... that the esteemed Dr. Gallup would confine his political crystal-gazing to the offices of the American Institute of Public Opinion and leave our Literary Digest and its figures politely and completely alone...We've been through many poll battles...We've been buffeted by the gales of claims and counter-claims. But never before has any one foretold what our poll was going to show before it was even started."

Thus, even if Gallup's claim might not technically be described today as a prediction, it was certainly taken as such at the time by his target, and was probably meant to be.

Myth 2

Peverill Squire did indeed write in POQ (vol. 52, 1988, p.125), "if all those who were polled had responded, the magazine would have, at least, correctly predicted Roosevelt the winner." However, he then goes on to say: "But more importantly, the initial sample was flawed; when compounded with the response bias, it produced the wildly erroneous forecast of the vote percentages." and then states that "a rough calculation of the bias produced by the sample is around 11%, with another 7% accounted for by problems with the responses." Unfortunately, there is no reason to trust the data from the May 1937 Gallup poll used to estimate the nonresponse rate (as Squire himself admits, before going on to do so). There certainly is nothing to justify this kind of precision in divvying up error among specific causes.

The fact is that The Literary Digest did not define a sampling frame and did not conduct anything resembling a representative sample, random or otherwise, of the voting population, nor did they make much of an effort to keep track of what was sent out and what was returned. It's convenient to use the spectacular failure of their straw poll as a cautionary tale, but without any real documentation of what they did, or data to analyze, any attribution of specific causality is little more than speculation.

Dominic Lusinchi:

Jan,

Thanks for your response. Here are my comments.

Your statement to the effect that "It's one thing to debunk a myth, it's something quite different to replace one myth with another" is an inaccurate characterization of what I am saying. I am not replacing one myth by another as you seem to suggest - if I am: what myth is it that I am promoting?

Myth 1

The fact that the protagonists (Funk and Gallup) believe that what Gallup said in his July 12 column is a prediction does not make it a prediction. I repeat that what Gallup was really predicting was a close race: so says the title of his column (which I hope you received). The Digest editors came to believe, and were encouraged to do so by all the praise they received regarding its "uncanny accuracy" (10/31/1936, 6), that their poll was a "forecasting machine" (8/22/1936, 3), when in reality they had no idea how their poll results were produced - in other words, the belief does not make it so.

In any case, this (the so-called "prediction") was used as a great promotional story after the Digest fiasco to show and tell: show that the new "scientific" polling was the wave of the future and tell that the Digest "straw" polling was a thing of the past. For example, Crossley wrote: "The Institute [Gallup] even went so far as to forecast the DIGEST vote, which it did with remarkable accuracy by the simple means of tabulating separately that part of its ballots which followed the DIGEST's general basis." ("Straw Polls in 1936", Archibald M. Crossley, The Public Opinion Quarterly, Vol. 1, No. 1. (Jan., 1937), p.29) The moral of the story: the "scientific" polls are superior because not only can they predict correctly the election but they can also predict what the Digest would forecast.

I agree with you when you say it was "presented" as a "throw-away comment"; but it really was a very cleverly worded statement. I also agree when you say that it was meant to "generate publicity for Gallup's fledgling organization." Think about it: if I (Gallup) am wrong, I can always say (plausible deniability) that's how things (the presidential race) were at that time (July); if I'm right, I can present it as a prediction (of the results the Digest will publish right before the election) and thereby trumpet the superiority of my "scientific" polling methods. The statement has to be analyzed within an overall strategy to dominate the opinion polling domain.

Gallup and his fellow "scientific" pollsters were not disinterested parties searching for the cause of the failure of the DIGEST poll out of some academic interest. They wanted to be top dogs and knock the Digest off its pedestal. They used the Digest incident, especially in the early years, to promote what they considered to be a superior "product": their "scientific" polling. Of course, they were to receive a nasty shock in 1948 - although not fatal to them, as 1936 was for the Digest.

Myth 2

I had the opportunity, thanks to the Roper Center, to have access to the raw data from the May 1937 AIPO (Gallup) poll. First the data in its raw form requires some editing because the file contains some anomalies. For example, you might have noticed that in Squire's table 2 (p.130) there are 780 respondents that claim to have received the Digest ballot, while in table 3 (p.131), which reports what they did with the ballot (returned or not or don't know), there are 829 respondents! Squire analyzed the data at face value.

Second, Squire did not weight the data despite the fact that we have information that allows us to do so: each candidate's share of the actual vote, the response rate to the Digest (~24%), and the percent each candidate received from respondents to the poll. This takes care of anomalies, pointed out by Squire, such as over-report of support for FDR, over-sampling of Digest poll respondents, and over-report of support for FDR among respondents.

The weighted results confirm much of Squire's conclusions: had the Digest relied solely on telephone and car owners it would have forecast a Roosevelt victory; Digest non-respondents were strongly in favor of FDR; and, last but not least, had all those who were polled by the Digest responded, the magazine would have pointed to the correct candidate. In other words, yes the Digest original sample of 10M was biased but not sufficiently to have prevented it from calling a Roosevelt victory. In fact, a less "rough calculation" (Squire, p.131) shows that non-response bias was the main culprit.

Ironically, the samples Gallup used throughout the 30s and 40s were also biased - in favor of Republicans. And he was called on that (e.g. Special House Committee Investigating Campaign Expenditures, 1944).

The conclusions based on the May 1937 poll (e.g. non-response as the main cause of the 1936 Digest poll failure) are consistent with other available data. I have re-analyzed the data provided by Cahalan in his Cedar Rapids study in 1936-7 reported in his 1989 POQ (vol. 53 pp.129-133) and in 1939 Psychological Record (vol.1, no.1, pp.3-11) papers. The results show that the Digest list used for that city was not biased and that respondents and non-respondents were very different in their candidate preference. Add to that the results from Allentown, Scranton and Chicago where only registered voters were polled (Digest, 14 November, 1936, p.7) and I would say we have some pretty solid "real documentation".

Does this amount to a definitive answer? No, absolutely not. The only way we could have resolved the question would have been to conduct, on the original sample of 10M, the same type of study that Cahalan performed in Cedar Rapids: select a random sample from the Digest list and ask respondents whether or not they sent in their Digest ballot and which candidate they favored at that time.

Short of that the May 1937 AIPO survey, the Cahalan study and the results from Allentown, Scranton and Chicago is the best we have. Speculation you say? Well, yes, I suppose you're right, but a lot less than simply repeating the old refrain repeated by so many: "The failure of the Literary Digest's polling approach can be explained simply. The Digest's sample of voters was drawn from lists of automobile and telephones owners." (Gallup, 1972, "Opinion Polling in a Democracy", in Statistics: A Guide to the Unknown, Judith M. Tanur et al. (eds.), San Francisco, Holden-Day, p.147.)

Jan Werner:

Some really good stuff here, but, as you suggested in an earlier (off-list) message, we are probably going to have to agree to disagree -- My reaction is that, if anything, your additional material supports my previously expressed opinions.

So, let me clarify what I mean by replacing one set of myths with another.

Myth 1:

Original myth: Gallup predicted the Literary Digest straw poll results.

Revised myth: Gallup did not predict anything.

My take: Gallup presented what he knew full well would be interpreted as a prediction, although he worded it so as to provide him with an escape if he were wrong. Given what we know now (see Myth 2) he was very lucky to come as close as he did. Whether or not that constitutes a prediction is a semantic judgment.

Myth 2:

Original myth: The Literary Digest poll failed because it did not use a representative sample.

Revised myth: It failed because of nonresponse bias (as shown by the May 1937 Gallup data, and other sources).

My take: The Literary Digest polling procedures were a complete mess. Evidence from other sources does show that an unrepresentative sample was not the sole cause, but is not strong enough to prove anything beyond that. In particular, I don't see the May 1937 Gallup data as being reliable enough to justify giving either bad sampling or nonresponse bias pride of place, so to speak, in assigning blame.

None of this addresses what I consider to be the worst myth about the Literary Digest straw poll failure, namely that it was the reason pollsters began using probability samples.

Dominic Lusinchi:

Well, Jan, if you consider these two statements semantically equivalent:

"If the Literary Digest were conducting its poll at the present time, following its usual procedure, Landon would be shown in the lead." - "When all the results are in the Digest poll will show Landon in the lead."

then I can't much argue about that - we've reached a dead-end.

You say that you don't consider "the May 1937 Gallup data as being reliable enough" to say anything about what caused the 1936 Digest poll to fail so miserably. I agree (along with Squire) that the Gallup survey is less than perfect. But I disagree that it cannot be used for the problem at hand. Gallup polls of the 30s and 40s have been put to good use by researchers: e.g., Baum & Kernell "Economic Class and Popular Support for Franklin Roosevelt in War and Peace," Public Opinion Quarterly 65, 2001, 198-229; and Berinsky's 2006 discussion "American Public Opinion in the 1930s and 1940s: The Analysis of Quota-Controlled Sample Survey Data," Public Opinion Quarterly 70: 499-529. The polls have their limitations and the data have to be analyzed with that in mind.

The "conventional explanation", as it has been called, that the 1936 Digest poll failed because it was biased in favor of Landon due to the fact that it relied mainly on lists of phone owners and car owners, is pure speculation, plausible speculation perhaps, but speculation nonetheless. In contrast, the only real evidence we have, although it is limited and has flaws (the results from Allentown, Scranton, Chicago, Cedar Rapids and the Gallup data) all points to the same problem: non-response bias. My preference in describing what happened to the 1936 Digest poll leans towards the latter (warts and all). It is, of course, a tentative explanation but it has the advantage of being backed by data, however imperfect.

As for your last comment about the belief that the Literary Digest poll failure being the reason why pollsters began using probability samples is, of course, nonsense, so I agree with you on that - but since the issue was not part of the original post, I'll leave it at that.

Jan Werner:

The late John Gorman, who founded Opinion Dynamics, liked to mention a specific date in the late 1970's, which he estimated was the day on which George McGovern edged out Richard Nixon, based on trending how respondents said they voted in the 1972 presidential election in polls conducted during the years following the Watergate affair. John's estimate was also based on data, however imperfect.

As for what is, or isn't, a prediction, I'll let it go at that. Others can make up their own minds as to what Gallup intended when he wrote what he did in 1936, and what it should be called.

Regardless of whether or not we agree, I'd like to thank you for an enlightening discussion and much valuable information. Let me note that the image you sent me of the 7-12-1936 Washington Post article does not include the beginning of the article on page 1, and there is no way to tell whether the headline "1936 Election Seen As Closest in Years" was written by Gallup or by the Washington Post's editor.

Dominic Lusinchi:

Thank you, Jan, for your insights; always glad to discuss what I see as the defining moment in the genesis of modern polling.

Aside from our difference regarding the semantics of what constitutes a prediction, the perception is just as important: i.e. that the Digest editors saw it as such reveals a lot about them - specifically how convinced they were that their poll was an "uncanny" "forecasting machine".

[Typos corrected and updated to include the last exchange of comments that I omitted from the original post].

 

Comments
Amy Fried:

Gallup most certainly did predict the outcome of the 1936 election. As Susan Ohmer notes in "Gallup in Hollywood" (2006, Columbia University Press), he offered a money back guarantee to publications carrying his syndicated column should he be wrong.

Gallup was also able to accurately predict what the Literary Digest would find because he had working for him a fellow named Claude Robinson who had written his dissertation on straw polls and knew a lot about them. Furthermore, Gallup had conducted surveys of various magazines' readerships, including the Literary Digest.

Of course he did not do his work as a disinterested researcher. Gallup was trying to build a business and he did so quite successfully.

Interestingly enough, Roper did a far better job than Gallup in 1936. His prediction of FDR's vote was off by only 1%, while Gallup and Crossley were off by nearly 7%. In Frank Stanton's oral history, which is held by Columbia University, Stanton claimed that he gave Roper a good deal of help in Roper's 1936 polling.

If we are to assume that newspaper headlines are meaningful, note that (as Ohmer does) that the Washington's Post headline on October 4, 1936 read, "Election will Show if Sampling Method is Accurate" and the Gallup's last piece before the election was titled, "Voting Tuesday to Test Clashing Poll Methods."

The idea that 1936 would test the different methods was a theme in other articles, including several by Hadley Cantril that were published in the New York Times. With these, the election was set up as a duel and the match between predictions and vote counts would determine the winner.

____________________

Dominic Lusinchi:

As I mentioned in my post, the title of Gallup's July 12, 1936 column was "1936 Election Seen As Closest in Years". Jan suggested that this might be the Post editor's take not Gallup's. I don't think so. Another indication of that is a Gallup article published in the November 1936 issue of Scribner's, and which, obviously, was written before the election. The article is entitled "Putting Public Opinion to Work". Gallup's first sentence reads: "The November election may prove the first close Presidential election in a generation." (p.36) Even though his polls (there were 8 in all, the first one in June) were showing Roosevelt ahead of Landon, unlike the DIGEST, he believed that the election could be a close one.

Yes, Gallup did promise a money-back guarantee to newspapers (about 80 in Nov. 1936) that carried his "America Speaks" column if his poll results for the 1936 presidential election were not more accurate than those of the LITERARY DIGEST (Saturday Evening Post, Wilton Rich, "The Human Yardstick", January 21, 1939, p.9). And he did predict the 1936 presidential election outcome, as did his fellow "scientific" pollsters - no dispute about that.

Gallup planned this "first big test" (the 1936 presidential election) for his new poll as he was planning an advertising campaign. Remember that at the time he was working for Young & Rubicam (and would do so until 1947!).

So his so-called "prediction" and his money-back guarantee must be read as elements in the promotion campaign for his (and his colleagues) "scientific" polling. The important thing is that his advertising campaign created the perception of a prediction and a big drama. He didn't want people to keep saying: "You will look in vain for the name of Dr. Gallup in Who's Who;" he and his fellow pollsters wanted to be front-page news instead of the LITERARY DIGEST. I agree with you when you write: "The election was set up as a duel" - yes, a duel between the old "straw" poll and the new "scientific" poll – set up by Gallup. As psychologists say he provided the stimulus and the subject (Funk of the Digest) responded, or more prosaically, he provided the bait and the fish (Funk) swallowed it hook, line and sinker – thereby providing Gallup a great publicity coup.

There were risks for Gallup and his fellow "scientific" pollsters but the pay-off was seen as well worth it. Roper considered this new polling as a potential "gold mine".

Robinson did not start working for Gallup at AIPO until 1938. But Gallup was no doubt well acquainted with Robinson's research on "Straw Votes", which was published in 1932. In it, the author analyzed the various biases associated with the DIGEST polls.

Gallup's 1936 election poll was based on a sample of about 125,000. He polled all 48 states; the size of Roper's sample was about 4,500 - he did not poll state by state. So Roper had an easier job than Gallup - the two are, therefore, not strictly comparable.

Part of Gallup's sample was polled by mail questionnaire, and like the DIGEST he used telephone directories and car registration lists. The rest of the sample was polled by in-person interviews. He noted the different results between the two, and surmised that the DIGEST would likely get a similar outcome - a majority in favor of Landon. Also, he must have been aware of the results of the poll the DIGEST conducted in late 1935 and whose results were reported in January 1936. The poll asked two questions: are you for or against the New Deal policies and who did you vote for in 1932. The New Deal was resoundingly condemned. Gallup must have looked closely at the data, as the keen observer he was, and noticed that the report of candidate preference in favor of Hoover was higher than the percent he received in the actual election.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR