Articles and Analysis


AAPOR Censures Lancet Iraq Casualty Survey

Topics: AAPOR , Disclosure , Gilbert Burnham , Lancet Survey

My colleagues at the American Association for Public Opinion Research (AAPOR) announced yesterday that an eight month investigation found that Dr. Gilbert Burnham violated AAPOR's Code of Professional Ethics and Practices.

At issue is the controversial study (pdf) of civilian deaths in Iraq conducted by Burhnam, a faculty member at the Johns Hopkins Bloomberg School of Public Health, and published in the journal Lancet in 2006. The study was the subject of considerable criticism because it produced a significantly higher estimate of Iraqi deaths than those of the Iraq Body Count project, the United Nations and the Iraqi Ministry of Health (for more details see the reporting by my National Journal colleagues, Slate's Fred Kaplan and the review on Wikipedia).

The AAPOR censure does not involve Burnham's methodology and renders no opinion on the substantive conclusions of the Lancet study. Instead, it focuses entirely on disclosure, or rather on Burham's failure to disclose "essential facts about his research." From the AAPOR release:

AAPOR holds that researchers must disclose, or make available for public disclosure, the wording of questions and other basic methodological details when survey findings are made public. This disclosure is important so that claims made on the basis of survey research findings can be independently evaluated. Section III of the AAPOR Code states: "Good professional practice imposes the obligation upon all public opinion researchers to include, in any report of research results, or to make available when that report is released, certain essential information about how the research was conducted."

Mary E. Losch, chair of AAPOR's Standards Committee, noted that AAPOR's investigation of Burnham began in March 2008, after receiving a complaint from a member. According to Losch, "AAPOR formally requested on more than one occasion from Dr. Burnham some basic information about his survey including, for example, the wording of the questions he used, instructions and explanations that were provided to respondents, and a summary of the outcomes for all households selected as potential participants in the survey. Dr. Burnham provided only partial information and explicitly refused to provide complete information about the basic elements of his research.”

AAPOR's President, Richard A. Kulka, added "When researchers draw important conclusions and make public statements and arguments based on survey research data, then subsequently refuse to answer even basic questions about how their research was conducted, this violates the fundamental standards of science, seriously undermines open public debate on critical issues, and undermines the credibility of all survey and public opinion research. These concerns have been at the foundation of AAPOR’s standards and professional code throughout our history, and when these principles have clearly been violated, making the public aware of these violations is in integral part of our mission and values as a professional organization."

The release also notes that Burnham is not a member of the organization. AAPOR has opted in the past to censure non-members over non-disclosure, including pollster Frank Luntz in 1997.   

Update: The Associated Press could not reach Burnham for comment but reports a reaction from the Johns Hopkins Bloomberg School of Public Health:

Tim Parsons, a spokesman for the school said: "We are disappointed AAPOR has chosen to find Dr. Burnham in violation of the organization's ethics code. However, neither Dr. Burnham nor the Johns Hopkins Bloomberg School of Public Health are members of AAPOR."

ABC's Gary Langer adds more background and has an email response from Tony Kirby, the Lancet's press officer: "The Lancet is making no comment."

Update 2: Science blogger Tim Lambert asks a reasonable question: What information, specifically, did Burnham fail to provide to AAPOR?

[AAPOR's statement] seems to be more than a little misleading. Burnham has released the data from the study. This report goes into a fair bit of detail on how the survey was conducted. And here is the survey instrument which includes the "wording of the questions he used".

Lambert emailed AAPOR for comment this morning (and I did the same after reading his blog item). AAPOR's standards chair, Mary Losch, responded with the following message:


I have read your entry and would note that the links you provided did not supply the questionnaire items but rather a simple template (as noted in the heading). The Johns Hopkins report provides only superficial information about methods and significantly more detail would be needed to determine the scientific integrity of those methods -- hence our formal request to Dr. Burnham. The Hopkins website refers to data release but, in fact, no data were provided in response to our formal requests. Included in our request were full sampling information, full protocols regarding household selection, and full case dispositions -- Dr. Burnham explicitly refused to provide that information for review.

We do not provide public reports of the investigations but if there are other specific questions that I could answer, I would be happy to try to do so.

I also asked Losch why AAPOR considers the "template" of questions posted online to be something less than "the wording of the questions used." She replied that they "requested the survey instrument, (including consent information) and it was not provided. The template did not appear to be much beyond an outline and certainly was not the instrument in its entirety."

Interests disclosed: I am an AAPOR member and served on AAPOR's Executive Council for two years, concluding in May of 2008, but had no involvement in the Standards Committee's investigation of the Lancet study.



Dr. Gilbert Burnham is piece of work, I'm glad someone is taking a look at him. He was involved in three different studies in Iraq, each of which contradicted one of the other studies in significant detail.

First, he did he a study back in (IRRC) 1999 about infant mortality in Iraq under sanctions. He claimed that Iraqi infant mortality was staggeringly high and that sanctions should be removed. In 2004, he designed a study that found out that infant morality in Iraq had been good before the liberation but had exploded afterward. The first study was eventually withdraw because it was determined to be fraudulent.

Meanwhile, the 2004 study claimed 180,000+ deaths in Anbar province alone out of a population of 1.2 million. The deaths were claimed to be (1) All civilian (2) 80% women and children and (3) all caused by Allied air strikes. All this based, on a single neighborhood in Faluja (30+ houses within 100 meters of each other) and sampled using methodology outside the studies design. Based on this one outlier sample, he and his coauthors trumpeted as statistic that 80% of the [sic]100,000+ deaths in Iraq resulted form Allied airstrikes.

Worse, the 2004 study switched between two sets of data repeatedly, using a set that excluded the Falujia cluster to get a plausible number of deaths but then including it in order to lay the blame for the majority of deaths on Allied airstrikes.

The 2006 studied referenced above found no massive clumps of death in Anbar, found no huge numbers of airstrike deaths and concluded that 80% of deaths in Iraq, during the same time frame covered in the 2004 study, resulted form small arms not airstrikes.

All this just scratches the surface of the staggering incompetence, if not outright fraud, involved in Dr. Burnham's work. No one unaffiliated with Burnham has produced results anywhere close to his results. I believe he showed his true motives when he admitted he conducted and then rushed into publication the 2004 study such that it made a media splash just weeks before the 2004 election.

I believe that people like Burnham are destroying the publics trust in science. We really need to get this cleaned up.


Greg Q:

Let's see if I understand this correctly. Tim Lambert, is a "science blogger" made his reputation by criticizing the work of John Lott. In order to go after John Lott, Mr. Lambert requested John Lott give him the data that underlied the paper's Mr. Lott had published, then proceeded to attempt to prove that the data did not support Mr. Lott's conclusions.

The John's Hopkins "researchers" have pretty explicitly said that they will not give their underlying data to anyone who might try to use that data to prove them wrong. Tim Lambert sees nothing wrong with this.

There are words that can describe the kind of person who takes that position. I don't believe Mr. Blumenthal likes having those words used on his blog, so I won't say them, here.

I will just settle for point out that real scientists share the data underlying their published research with all other academics, even those ones who aren't "objective", even those ones whose sole goal is to prove the first set of researchers wrong.

The only reason to refuse to share the data is the knowledge that the data doesn't support your claims. That, in short, you have knowingly committed fraud.


In fact, the researchers have given the data to those who have tried to use it to prove them wrong. They gave it David Kane, who did the analyses cited in the linked National Journal article. And I've publicly stated that the researchers should not provide the data to anyone.

As for John Lott, he has not provided the data from his "98% brandishing" DGU survey to anyone, claiming that he lost it in a disk crash.


Greg Q:

As for John Lott, he has not provided the data from his "98% brandishing" DGU survey to anyone, claiming that he lost it in a disk crash.

And what was your commentary on that, Mr. Lambert? Further, there's a difference between "I can't give you the data, because I no longer have access to it", and "I refuse to give you the data unless you pass my political test." No?

The John's Hopkins "researchers" say right there on their web page that they won't just give their data to any academic researcher who asks, and that part of their decision process involves them doing a political assessment of the research team asking for the data.

If they had nothing to hide, they wouldn't be hiding their data. Either they believe that AAPOR might help Iraqi death squads hunt down and kill the respondents to their survey (in which case they need to get the prescriptions on their meds checked), or else they committed knowing fraud, and fear that the AAPOR would have figured that out if they turned over the data.

You don't get to pick and choose who gets to question your work. Not and call yourself a scientist.


Post a comment

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.