Mark Blumenthal | March 7, 2007
If it walks and quacks like a duck, we are told, it must be a duck. Yesterday, MSNBC posted a story describing results from an "online survey" conducted in partnership with Elle magazine showing that "men rule" in the workplace:
Of male respondents, 41 percent said men are more likely to be good leaders, and 33 percent of women agreed. And three out of four women who expressed a preference said they would rather work for a man than a woman.
Read the article and the survey cited certainly "quacks" like the sort of projective, representative, "scientific" survey that news organizations typically report. The article used survey results to describe how "men" and "women" feel. Academic experts explained and reacted to the findings. But all appearances to the contrary, it wasn't that sort of duck at all: The "survey" was neither based on a random sample nor something even MSNBC editors considered "nationally representative."
The description of the polls methodology consists of the following paragraph that originally appeared as a sidebar toward end of the story:
About the Survey
Our online survey was completed by 61,647 people, about 50 percent male and 50 percent female. The average age was 42, 94 percent said they work full-time and 44 percent said they supervise other workers. Although the sample size is large and diverse, it is not considered nationally representative because it was largely restricted to MSNBC.com readers.
While this paragraph tells you what the survey is not ("nationally representative), it manages to leave out a few critical pieces of information: How were respondents selected? By saying that the "sample" was "largely restricted to MSNBC.com readers," the paragraph implies that it was not a "sample" at all, but rather the result of a straw poll open to anyone that noticed the invitation to participate on the MSNBC. If so, respondents were not selected, they selected themselves.**
As a pollster, that distinction is obvious to me, as it was obvious to my colleagues on the listserv of the American Association for Public Opinion Research (AAPOR) who were abuzz about the survey yesterday. Late last night, AAPOR's president elect Nancy Mathiowetz sent an open letter-to-the-editor criticizing the article for not making it clear "from the start" that the survey was "decidedly not a valid scientific, nationally-representative survey." She continued:
Questions administered to a self-selected audience are one thing; real surveys that use scientifically determined random samples in an attempt to measure a population's attitudes and behaviors accurately are something altogether different.
Indulging in the former is not just harmless fun. Labeling self-selected, online data collection efforts as legitimate survey research are at best misleading and lead to a diminution of the field as a whole.
The survey MSNBC conducted is a grand way to involve readers with their website. Many news organizations use such techniques to enrich their relationships with readers. However, these polls cannot be construed to be a poll representative of anything more than those who chose to take part, and certainly shouldn't be characterized as representative of some larger population.
Two additional points worth noting: The first comes from David Moore, a former senior analyst and vice president of the Gallup Organization. Yesterday, he reminded his colleagues on the AAPOR listserv that the basic contention of the MSNBC story is essentially what Gallup surveys found between 1982 and 2002 (quoted with permission):
Whatever the problems with the "poll," the findings are not far off from a Gallup poll conducted in April 22-24, 2002 (the last time, I believe, that Gallup asked the question: "If you were taking a new job and had your choice of a boss would you prefer to work for a man or a woman?"). Gallup, by the way, has a trend on this question that goes back to 1953.
In April 2002, among males: 29% preferred a male boss, 13% a female boss, and 57% said no preference (a volunteered response); 1% no opinion Among females: 32% preferred a male boss, 23% a female boss, and 43% had no preference; 2% no opinion.
Moore had apparently not seen an update by Gallup conducted in September 2006 that produced very similar results:
A majority of men, 56%, say it makes no difference whether their boss is a man or a woman, while those who have a preference favor a man (34%) rather than a woman (10%). Among women, 40% say they would prefer a new boss to be a man, while 26% would prefer a woman. Thirty-two percent of women say it makes no difference.
Let's be clear that Moore and Mathowietz both condemn the MSNBC poll. "MSNBC should have conducted its own contemporary poll," Moore says, "rather than rely on the admittedly unrepresentative responses it received online."
Second, although totally non-random pseudo can confuse and mislead, they are relatively easy to condemn. Unfortunately, the more difficult issue for the survey industry remains what to do about the theoretically "scientific" surveys that fail to contact or interview individuals at three out of four randomly selected households. We need to be clear that even surveys based on scientific sampling may still reflect the unique slant of those who choose to participate. Unfortunately, the line between "scientific" and "pseudo" is not as clear as it used to be.
Interests Disclosed: I currently serve as AAPOR's associate chair for publications and information, and in that capacity, saw a draft of the Mathiowetz letter before its release.
Update: Eve Tahmincioglu, author of the MSNBC story, confirms by email that the survey "was based on responses to an invitation to participate on the MSNBC.com website." So the respondents were entirely self-selected, as per my speculation above.