Re: AAPOR's Report - 2008 vs 1948
Mark Blumenthal | April 10, 2009
Via email, Kathy Frankovic, former Director of Surveys at CBS News, sends this comment about my post yesterday on the disappointing pollster cooperation with AAPOR's ad hoc committee report on the New Hampshire primary polling mishap:
There is a big difference between 1948 and where we are today in the field of survey research. In 1948, there was an accepted academic standard for survey research – probability sampling – one that was not used by the public pollsters. That – in addition to the lack of polling close to the election – was an obvious conclusion the SSRC researchers could make to explain what went wrong. There is no such obvious methodological improvement available as an explanation for the 2008 problems in NH. It’s not cell phones, it’s not respondent selection, and it’s not ballot order. Timing (and the possibilities of last minute changes) may once again be everything. And while more organizations should have disclosed more, I think that it’s unlikely that more data would have told us anything more definitive than we learned from the report as written.
For what it's worth, the report itself specifically referenced the contrast with 1948:
The work of the committee, and hence this report, has been delayed by a slow response from many of the pollsters who collected data from the four states in which the committee focused its efforts – New Hampshire, South Carolina, Wisconsin, and California. This is quite a different situation than after the 1948 general election, when there were fewer firms engaged in public polling, the threat to the future of the industry seemed to be greater, and the polling firms were fully cooperative. In 2008, many of the firms that polled in New Hampshire had studies in the field for primaries that occurred right after that. Today, there are well-publicized standards for disclosure of information about how polls are conducted. AAPOR, an organization of individuals engaged in public opinion research; the National Council on Public Polls (NCPP), an organization of organizations that conduct public opinion research; and the Council of American Survey Research Organizations (CASRO), also an organization of organizations, have all promulgated standards of disclosure. Despite the norms, at the time this report was finalized, one-fifth of the firms from which information was requested had not provided it. For each of these four firms, we were able to retrieve some of the requested information through Internet searches, but this was incomplete at best. If additional information is received after the report’s release, the database at the Roper Center will be updated.
So if and when pollsters who did not share raw, respondent level data share it with AAPOR, it will be posted to the Roper Center's listing (which is open to anyone, not limited to member institutions). I am told that Roper will also soon add pdf reproductions of all the responses received from pollsters, not just those that shared respondent level data.
By Mark Blumenthal | April 10, 2009 7:48 AM | Permalink
