Articles and Analysis


A Tale of Two Doctor Polls

Topics: Disclosure , Health Care Reform , National Journal column

My National Journal column for the week explores two recent surveys of physicians that produced widely divergent results. One was published in the New England Journal of Medicine (NEJM), conducted by two researchers at he Mount Sinai School of Medicine in New York and funded by the Robert Wood Johnson Foundation. The second was conducted by Investors' Business Daily (IBD) and the TechnoMetrica Institute of Policy & Politics (TIPP).

The column focuses on the huge difference in the quality of disclosure between the two -- the NEJM study tells us much, much more about its methods and questions than the IBD/TIPP survey. The latter fails to meet even minimal standards for disclosure mandated by the National Council on Public Polls and the American Association for Public Opinion Research.

Late Update - One of the subsequent stories published by Investor's business daily on their survey of doctors now includes the following information not disclosed in their initial report:

The questionnaires were sent out Aug. 28 to 25,600 doctors nationwide. The sample was purchased from a list broker, Lake Group Media of Rye, N.Y. One hundred of those responding were retired, and their answers were not included in the final results.

If it was there last week, I missed it, but this new information tells us two things: First, the IBD/TIPP survey did not use the same sample frame as the NEJM survey. Second, as reader Franzneumann points out, the response rate is significantly lower: Even if they had included the 100 retired doctors along with 1,376 reported interviews, their response rate is only 6%. Compare that to the 43% response rate reported by the NEJM survey.

Update1:  So why do the two surveys produce such different results? As I argued in the column, without even minimal disclosure from the IBD/TIPP poll we can only speculate, but here are the most likely theories:

1) Different questions. The NEJM study asks a three-way, forced choice between a private only option (involving "tax credits or low income subsidies to buy private insurance"), a public only option (that would "eliminate private insurance and cover everyone in a single plan like medicare") and a combination offering a choice between public and private plans. It makes no reference to current legislation, the Democrats or President Obama. We do not know exactly what the IBD question asks, although it appears to be simply about "the proposed plan" or possibly about a "proposed government expansion."

We have seen some pretty big differences on samples of all adults between surveys that simply ask about health reform "proposals" being debated in Congress (without further definition) and those that attempt to describe a "public option." For example, in August, CBS News found 57% of adults in favor and 35% opposed to "the government offering everyone a government administered health insurance plan -- something like the Medicare coverage that people 65 and older get -- that would compete with private health insurance plans." At about the same time, the Pew Research Center reported 38% of adults in favor and 44% opposed to "the health care proposals being discussed in Congress."

The three-option question that the NEJM survey reported also produces a bigger percentage supporting a public option, alone (10%) in in combination with private plans (63%), than an up-or-down, favor-or-oppose question. The NEJM questionnaire actually asked about each of the three proposals separately (see the column for complete wording), then followed up with a three way choice.

They have not yet reported on the individual results, although Alex Federman, one of the two medical researchers at the Mount Sinai School of Medicine in New York who conducted the survey, tells me that 65% said they supported a choice between public and private options when asked about it separately. They plan to publish the remaining results from the survey in future articles. Federman added that they considered the three way choice most appropriate, both because of what they learned from questionnaire pretesting and because they wanted doctors to make the same choice facing legislators between a public-only option, private-only options or a choice between the two. In pre-testing, Federman says, they learned that Doctors had all sorts of opinions on health reform, but "a lot of people were more mixed" between their views of public and private options.

2) Different samples? I put a question mark on this theory because, as reviewed in the column, we know almost nothing about how the IBD/TIPP poll sampled "practicing physicians." As Nate Silver points out, they do not even define what they mean by "practicing." However, if the IBD/TIPP used a different sample frame than the AMA Physician Masterfile (a comprehensive list of all physicians, not an AMA membership list), it might well explain some of the difference in results.

[Clarification: See Late Update above -- The IBD/TIPP survey did use a different sample frame although for now we know only where it came from, not how it differed. They also imply that "practicing" means non-retired].

[One dissent worth noting: This morning, a public health research used Twitter to take issue with my observation of that the AMA file is a "very accurate" list. "Best out there?," she asked. "Yes, but problematic."]

3) Response bias? Even if the the two lists (or "sample frames") were identical, it is possible that different kinds of doctors responded to the two surveys. For example, did the IBD survey prominently identify their survey as sponsored by Investors' Business Daily? Those who know or subscribe to it know that its editorials tend to be more conservative, and its editorial criticism of the Obama health care reforms have drawn sharp rebuke. As such, reform supporters might be less likely to participate in an IBD survey. Of course, we do not know anything about how the IDB/TIPP survey recruited respondents.

[Clarification: See the Late Update above -- The IBD/TIPP survey has a response rate of roughly 6% compared the a 43% response rate on the NEJM survey].

According to Federman, the NEJM survey identified itself as coming from the Mount Sinai School of Medicine and prominently identified its support from the Robert Wood Johnson Foundation. The headline of the initial postcard they sent identified it as "The National Physician's Survey on Health Care Reform" (emphasis in original). The subhead: "Congress wants to hear from doctors on health care reform."

Some relatively recent research has shown that "interest in the survey topic, reactions to the survey sponsor, and the use of incentives" are the three most critical variables in the linkage between the response rate and the potential for non-response bias. Perhaps those interested in "health care reform" enough to do a survey are more supportive of reform than those opposed?

4) Different survey dates. The surveys did have different field periods. The first NEJM questionnaires were sent out on June 25 and they started analyzing data on September 4. The IBD poll, published on September 15, says only that it was "conducted by mail the past two weeks."

While it is possible that the field dates contributed to the difference in results, since opposition to reform has increased over the course of the year, it's unlikely. First, as Charles Franklin noted on Friday, opposition to health care reform among all adults has been mostly stable since early July, about the time when the NEJM survey started. Second, according to Federman, they sent out their questionnaires in multiple waves, analyzed the results by wave and found no trends over the course of the field period.

Again, much of this commentary is just speculation. The larger point is that when one survey discloses its methods and the other does not, we are left guessing.



If the questions in the second poll attempted to be honest: e.g. "public option," "private option" than it might be regarded as valid, contingent on other factors. However, if it uses loaded phrases like "government takeover of healthcare" than at best it's a poorly designed poll. At worst it's a push-poll In either case, it's probably not a good measure.



With the advent of the NEMJ poll showing overwhelming support among MD's for a public option, the Republicans were in desperate need of an alternative narrative. When asked about the NEMJ poll showing support, they need to be able to say "There are other polls out there showing just the opposite," without being specific.

IBD and TIPP are there to oblige. Welcome to the GOP "No integrity" zone.


Registered User:

My comment has to do with physician sample. Our firm does a lot of research with physicians mainly paid for by for-profit clients who want a pulse on various segments of the physician community. In my experience, it is very rare that the AMA masterfile gets used. More commonly, physician research conducted by my firm and others uses a convenience sample, such as an online sample provided by a sample company(e.g., GMI, ePocrates, etc.) because it is far cheaper to use the panel than to try and reach a physician via phone, despite all of the biases. Do you know for sure that the second poll was conducted via phone?

Melissa Marcello
Pursuant Research



The IBD/TIPP poll used a sample frame derived from a list obtained from a supplier specializing in direct marketing. We have no idea of how this sample frame was put together but it is highly unlikely that it was randomly recruited. We don't know the population universe either.

Further, the IBD/TIPP poll also has a screamingly low response rate - last I checked, it was about 5%.

By contrast, the NEJM survey provides extensive discussion of its methodology.

Can I trust these polls? Yes, the NEJM one, but not the IBD poll.


Mark Blumenthal:

@Melissa: All I know, from their article, is that IBD did some sort of "mail" survey. I wouldn't be surprised if they used a convenience sample as you describe, but we until they disclose more, we don't know. The column gets into the issue of the sample in more details.

Do you have any thoughts about how a mail convenience sample of doctors might differ from a mail sample of the AMA list?

@Franzneumann: Where did you find your info about the sample frame and response rate of the IBD/TIPP poll?



@Mark ... I don't have the link just now. Saw it on the weekend. Will post later. I checked out the firm that supplied the list though (just can't remember the name). The release I saw on their website (IBD), note the name of the supplier and also that they had something like ~1,300+ out of ~25K responding.



@Mark ... found it.

Here's the link: http://www.investors.com/NewsAndAnalysis/Article.aspx?id=506309

Here's the relevant statements:
"As reported Wednesday, 65% of the 1,376 practicing physicians who responded to a mailed questionnaire over the last two weeks ...

The questionnaires were sent out Aug. 28 to 25,600 doctors nationwide. The sample was purchased from a list broker, Lake Group Media of Rye, N.Y. One hundred of those responding were retired, and their answers were not included in the final results."

Lake Group Media is a supplier of direct marketing lists. Which doctors end up on such lists is anybody's guess.


Mark Blumenthal:

@Franz (and all): I found it too and updated the post above. Thanks.



Mark, I'm interested in your thoughts on the relative importance of the sampling frame source vs. the low response rate.

While it's hard to draw any conclusions without knowing much about Lake Group Media, is it not peculiar that they would use a direct marketer as opposed to a professional sampling firm?

Further, this is almost assuredly not a random probability sample frame. Thus, no ability to calculate sample error. This is only exacerbated by the fact that we don't even know the population universe (is this a subset of doctors nationally or just a wily nilly list?).

Further, if they have 25,600 "practicing physicians" in the sample frame, that's just a small fraction of doctors nationally (est. ~650K, I believe). So, sample frame is just 4% of population. That leaves it open to massive sample error potential, no?

To me, the puny response rate is just the cherry on top of the real mess.


Post a comment

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.