Mark Blumenthal | February 4, 2009
Topics: AAPOR , Disclosure , Gilbert Burnham , Lancet Survey
My colleagues at the American Association for Public Opinion Research (AAPOR) announced yesterday that an eight month investigation found that Dr. Gilbert Burnham violated AAPOR's Code of Professional Ethics and Practices.
At issue is the controversial study (pdf) of civilian deaths in Iraq conducted by Burhnam, a faculty member at the Johns Hopkins Bloomberg School of Public Health, and published in the journal Lancet in 2006. The study was the subject of considerable criticism because it produced a significantly higher estimate of Iraqi deaths than those of the Iraq Body Count project, the United Nations and the Iraqi Ministry of Health (for more details see the reporting by my National Journal colleagues, Slate's Fred Kaplan and the review on Wikipedia).
The AAPOR censure does not involve Burnham's methodology and renders no opinion on the substantive conclusions of the Lancet study. Instead, it focuses entirely on disclosure, or rather on Burham's failure to disclose "essential facts about his research." From the AAPOR release:
AAPOR holds that researchers must disclose, or make available for public disclosure, the wording of questions and other basic methodological details when survey findings are made public. This disclosure is important so that claims made on the basis of survey research findings can be independently evaluated. Section III of the AAPOR Code states: "Good professional practice imposes the obligation upon all public opinion researchers to include, in any report of research results, or to make available when that report is released, certain essential information about how the research was conducted."
Mary E. Losch, chair of AAPOR's Standards Committee, noted that AAPOR's investigation of Burnham began in March 2008, after receiving a complaint from a member. According to Losch, "AAPOR formally requested on more than one occasion from Dr. Burnham some basic information about his survey including, for example, the wording of the questions he used, instructions and explanations that were provided to respondents, and a summary of the outcomes for all households selected as potential participants in the survey. Dr. Burnham provided only partial information and explicitly refused to provide complete information about the basic elements of his research.”
AAPOR's President, Richard A. Kulka, added "When researchers draw important conclusions and make public statements and arguments based on survey research data, then subsequently refuse to answer even basic questions about how their research was conducted, this violates the fundamental standards of science, seriously undermines open public debate on critical issues, and undermines the credibility of all survey and public opinion research. These concerns have been at the foundation of AAPOR’s standards and professional code throughout our history, and when these principles have clearly been violated, making the public aware of these violations is in integral part of our mission and values as a professional organization."
The release also notes that Burnham is not a member of the organization. AAPOR has opted in the past to censure non-members over non-disclosure, including pollster Frank Luntz in 1997.
Update: The Associated Press could not reach Burnham for comment but reports a reaction from the Johns Hopkins Bloomberg School of Public Health:
Tim Parsons, a spokesman for the school said: "We are disappointed AAPOR has chosen to find Dr. Burnham in violation of the organization's ethics code. However, neither Dr. Burnham nor the Johns Hopkins Bloomberg School of Public Health are members of AAPOR."
ABC's Gary Langer adds more background and has an email response from Tony Kirby, the Lancet's press officer: "The Lancet is making no comment."
Update 2: Science blogger Tim Lambert asks a reasonable question: What information, specifically, did Burnham fail to provide to AAPOR?
[AAPOR's statement] seems to be more than a little misleading. Burnham has released the data from the study. This report goes into a fair bit of detail on how the survey was conducted. And here is the survey instrument which includes the "wording of the questions he used".
Lambert emailed AAPOR for comment this morning (and I did the same after reading his blog item). AAPOR's standards chair, Mary Losch, responded with the following message:
I have read your entry and would note that the links you provided did not supply the questionnaire items but rather a simple template (as noted in the heading). The Johns Hopkins report provides only superficial information about methods and significantly more detail would be needed to determine the scientific integrity of those methods -- hence our formal request to Dr. Burnham. The Hopkins website refers to data release but, in fact, no data were provided in response to our formal requests. Included in our request were full sampling information, full protocols regarding household selection, and full case dispositions -- Dr. Burnham explicitly refused to provide that information for review.
We do not provide public reports of the investigations but if there are other specific questions that I could answer, I would be happy to try to do so.
I also asked Losch why AAPOR considers the "template" of questions posted online to be something less than "the wording of the questions used." She replied that they "requested the survey instrument, (including consent information) and it was not provided. The template did not appear to be much beyond an outline and certainly was not the instrument in its entirety."
Interests disclosed: I am an AAPOR member and served on AAPOR's Executive Council for two years, concluding in May of 2008, but had no involvement in the Standards Committee's investigation of the Lancet study.