Mark Blumenthal | April 16, 2009
More reader reaction to my post last week on the disappointing cooperation with AAPOR's Ad Hoc Committee report on the New Hampshire primary polling mishap. Or more accurately, this email from long-time survey researcher Jan Werner responds to a comment I posted from former CBS News polling director Kathy Frankovic:
Kathy Frankovic wrote that:
"In 1948, there was an accepted academic standard for survey research – probability sampling – one that was not used by the public pollsters. That – in addition to the lack of polling close to the election – was an obvious conclusion the SSRC researchers could make to explain what went wrong."
Kathy's statement is literally true, but it is also highly misleading because it fails to note that in its report, the SSRC panel led by Frederick Mosteller explicitly ruled out sampling methodology as the primary reason for the failure of the polls to predict the 1948 presidential election. The report did mention the lack of polling close to the election as one among several contributing factors, but that rationale subsequently gained prominence in large part because, as the explanation least damaging to their businesses, it could easily be endorsed by the pollsters themselves.
The Mosteller panel did admonish the pollsters for using quota rather than probability sampling, but it concluded that the incorrect predictions most likely derived from a combination of many different inadequacies in the conduct of the 1948 polls. It also noted that a major problem for the polling industry was the unrealistic expectations of accuracy that it had fostered among the public. These conclusions seem equally compelling today.
Neither the 2009 AAPOR report nor the 1949 SSRC report could satisfactorily answer the question of what went wrong, but both provide superb resources for students of public polling. AAPOR would do the profession and the public a great favor if it could help make "The Pre-Election Polls of 1948" available again, either in print or online.