I'm back in DC and rested, or at least, better rested than I was this time yesterday. As a wrap up to my coverage of this year's conference of the American Association for Public Opinion Research (AAPOR), here are links to all of the video interviews (with some final thoughts below):
Michael Link, AAPOR's conference chair, provides an introduction.
Masahiko Aida on a likely voter validation study conducted by Greenberg, Quinlan Rosner & Democracy Corps.
Reg Baker on AAPOR's online panel survey task force.
Chris Borick on the whether the "incumbent rule" made a comeback in 2008.
Elizabeth Dean on RTI International's efforts to recruit and interview respondents in Second Life.
Don Dillman on Address Based Sampling (ABS) and its applications for Internet surveys.
Paul Donato on the role of traditional survey research in a world of changing electronic measurement.
Tom Guterbock on his research on improving political message testing.
Lou Harris recounts experiences working for candidate John F. Kennedy in 1960.
Chase Harrison on the accuracy of pre-election polling in 2008.
Sunshine Hillygus on her study of how primary supporters of Hillary Clinton (and those of other unsuccessful presidential candidates) ended up voting in the general election.
Scott Keeter on the perils of pre-election polling in 2008.
Jon Krosnick on the challenges in assessing the quality of new survey methods.
Jennie Lai of the Nielsen Company on their research into the use of mobile devices for time use surveys.
Christopher Wlezien on the work he and Bob Erikson have done on the comparative accuracy of polls and political prediction markets.
A few things to remember about AAPOR's annual conference: First, my interviews barely scratched the surface of the breadth and depth of subjects covered and findings presented. I tend to focus more closely on topics related to pre-election polling, but the conference covered a much wider array of methodological issues. For example, there were by my count eight panels on survey non-response, six on web surveys, and five each on the address based sampling and cell phone interviewing. These tend to be highly technical and not easily conveyed via quick video interview.
Remember also that a lot of the findings presented at the conference -- including some that made their way into these interviews -- are very preliminary. The quality of the "papers" presented at the AAPOR varies widely (and I put that word in quotations because most are just Powerpoint presentations). Only a handful will eventually make their way into academic journals, and those that do are still at the beginning of a long peer review process that may ultimately lead their authors to different conclusions. Some of that review comes in the form of tough questions at the conference that my interviews cannot capture.
Finally, some words of thanks: First, thanks to o all of those I interviewed for making themselves avalable. Second, a big thanks to Michael Link, the conference chair, and Monica Evans-Lombe and her colleagues at AMP (AAPOR's management company), who helped provide important logistical support. Finally, another big thank you to Lisa Mathias at the Winston Group for creating the animation that appears at the beginning of each video (and her colleague and pollster regular Kristen Soltis for recommending her). Thank you to all!
On Saturday, I also interviewed Sunshine Hillygus, Director of Program on Survey Research at Harvard University, on her study of how primary supporters of Hillary Clinton (and those of other unsuccessful presidential candidates) ended up voting in the general election, as as presented at the annual conference of the American Association for Public Opinion Research (AAPOR).
On Saturday, I also interviewed Jennie Lai, research methodologist for The Nielsen Company, on her research into the use of mobile devices for time use surveys as presented at the annual conference of the American Association for Public Opinion Research (AAPOR).
On Saturday, I interviewed Washington State University Professor Don Dillman, author of Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, on his work on the use of Address Based Sampling (ABS) and applications for internet surveys, as presented at the annual conference of the American Association for Public Opinion Research (AAPOR).
I interviewed Elizabeth Dean, a survey research methodologist at RTI International, about their efforts to recruit and interview survey respondents in Second Life at the annual conference of the American Association for Public Opinion Research (AAPOR).
I interviewedJon Krosnick, Stanford professor of communications, political Science and psychology, on challenges in assessing the quality of newer survey methodologies, including online panel surveys, at the annual conference of the American Association for Public Opinion Research (AAPOR).
I interviewedScott Keeter, director of survey research at the Pew Research Center, on a paper he presented on the perils of pre-election polling in 2008.at the annual conference of the American Association for Public Opinion Research (AAPOR).
I interviewedChase Harrison, preceptor at the Program on Survey Research at Harvard University, on his research into the accuracy of pre-election polling forecasts in 2008 at the annual conference of the American Association for Public Opinion Research (AAPOR).
I also interviewed Harrison last year on the first stage of this research.
I interviewedTom Guterbock, director of the Survey Research Center at the University of Virginia, on a paper he presented on tests he conducted over the last year to improve message testing poll at the annual conference of the American Association for Public Opinion Research (AAPOR).
I interviewed Masahiko Aida, director of analytics for Democratic polling firm Greenberg, Quinlan Rosner (which conducts surveys for Democracy Corps) on a paper he presented on their likely voter validation study at the annual conference of the American Association for Public Opinion Research (AAPOR).
Up next in our series of interviews at the AAPOR 2009 conference is Reg Baker, Chief Operating Officer of Market Strategies International discussing his work as chair of AAPOR's Online Panel Task Force
You can see all the videos from this year's AAPOR conference here (and last year's too). I may also post occasional updates via Twitter.
The next interview in our series of interviews at the AAPOR 2009 conference is with Chris Borick, director of the Institute of Public Policy at Muhlenberg Colllege about the paper he presented on the "incumbent rule" in 2008.
You can see all the videos from this year's AAPOR conference here (and last year's too). I may also post occasional updates via Twitter.
The latest interview from our series of interviews at the AAPOR 2009 conference is with Paul Donato, Executive Vice President and Chief Research Officer for The Nielsen Company. Donato was one of two speakers at last night's Plenary Session, "The Role of Traditional Survey Research in a World of Electronic Measurement and Changing Information Needs."
Unfortunately, a combination of logistical and technical challenges prevented me from completing an interview with Columbia University Professor Kenneth Prewitt, who shared the stage with Donato.
You can see all the videos from this year's AAPOR conference here (and last year's too). I may also post occasional updates via Twitter.
Yesterday, I also interviewed to Temple University Professor Christopher Wlezien, who long-time Pollster readers may remember from the guest contribution he co-authored with Joseph Bafumi and Bob Erikson. We talked today about the work Wlezien and Erikson have done on the comparative accuracy of polls and political prediction markets:
Wlezien sent a copy of the paper he co-authored with Erikson that he presented yesterday. Their earlier article, "Are Political Markets Really Superior to Public Opinion Polls," appeared in Public Opinion Quarterly in 2008.
You can see all the videos from this year's AAPOR conference here (and last year's too). I may also post occasional updates via Twitter.
The second interview in the series I am conducting at this week's AAPOR conference has a bit of a back story. I had the real honor of interviewing 88-year-old Lou Harris, the founder of Harris and Associates in the mid-1950s and creator of the Harris Poll. Harris was also one of the first pollsters to serve as a senior advisor to a presidential candidate -- John F. Kennedy and 1960.
The unfortunate back story is that about 30 minutes after we finished our interview and just before he was scheduled to speak at the AAPOR conference, Harris fell and hurt himself badly enough that he had to be taken to a nearby hospital. I am told, however, that the injury is not serious although he is spending the night for observation. We hope he is feeling better soon.
We talked for a little over twelve minutes. In this first segment, Harris recounts the work he did for the Kennedy campaign prior to the West Virginia Democratic primary in 1960:
In this second segment, I asked Harris about the apparently apocryphal story that had candidate Kennedy snapping, ""Just give me the numbers, Lou, I know what they mean." Apparently, that story is something of a myth, although he goes on to describe Kennedy as a perceptive but demanding client:
.
You can see all the videos, including those from last year's conference, here.
Update: Twenty five years ago, the Washington Post's Louis Romano reported the following about the mythical "I know what they mean" story (via Nexis, 1/15/1984):
THE STORY IS SAID to be apocryphal, but political pollsters still love to tell it:
John F. Kennedy was soaking in a tub during the final days of the 1960 presidential campaign.
Lou
Harris, the first in a new breed of specialized presidential pollsters,
was on the edge of the tub discussing his latest survey results,
complete with a complicated package of political advice and in-depth
analysis for Kennedy.
"Just give me the numbers, Lou," Kennedy is said to have snapped. "I know what they mean."
"Oh,
we used to always meet in the bathroom," says Lou Harris today, one of
the deans of political polling. "But the story is part of the mythology
. . . I was certainly the first polltaker who . . . served on a
super-strategy committee."
This post is the first of a series of video interviews I'll be doing over the next few days at the annual conference of the American Association for Public Opinion Research (AAPOR). We begin with Michael Link, Chief Methodologist/VP for Methodological Research at The Nielsen Company, who is serving as this year's AAPOR conference chair. He explains what this conference is about and what we can expect over the next few days.
You can see all the videos, including those from last year's conference, here.
Credit for the animated graphic title that introduces each video goes to Lisa Mathias at the Winston Group (and her colleague Kristen Soltis for recommending her). Thank you Lisa and Kristen!
I am headed off to the airport in few minutes for Hollywood, Florida and this year's conference of the American Association for Public Opinion Research (AAPOR). I will be video-blogging for the rest of the week (like last year), but will otherwise try to keep updated via Twitter
The box below displays my last five Twitter updates. You can also follow posts by other AAPORites on Twitter using the hashtag #AAPOR09.
Next Thursday, I will fly to Hollywood, Florida to cover the annual conference of AAPOR -- the American Association for Public Opinion Research. That coverage will include a reprise last year's popular video interviews of those presenting on topics of interest to Pollster.com readers, as well as comments via Twitter as time permits. I will have more details on our coverage next week, but the videos will all appear right here on Pollster.com
Meanwhile, for the true polling geeks out there, we would love your input on potential interviewees. The full program is available for download, either as a full pdf file or (if you're looking for something specific) through a search form. If you are a survey geek, are there any particular topics, papers or speakers you really hope I interview? If so, please email me or leave a comment below.
In the survey world, this is very big news (from AP via The Page):
President Barack Obama is tapping Robert M. Groves, a University of Michigan professor who has pushed the use of statistical sampling, to be the next census director.
A Commerce Department official who demanded anonymity said the White House will make the announcement later Thursday.
Groves is an expert in survey methodology and statistics who served as an associate director of the Census Bureau from 1990 to 1992. He and others recommended that the 1990 census be statistically adjusted to make up for an undercount, only to be overruled by then Republican Commerce Secretary Robert Mosbacher, who called it political tampering.
It is something of an understatement to describe Robert Groves as "an expert in survey methodology." He is one of our nations' most respected survey methodologists and arguably the leading authority on the subject of non-response in surveys. He has served as the president of the American Association for Public Opinion Research (AAPOR), and won many of its awards including the career award for exceptionally distinguished achievement. Interests disclosed: I had the good fortune to study under Groves in classes I took at the University of Maryland's Joint Program in Survey Methodology (JPSM), a program that Groves helped found.
As the first three three paragraphs of the AP make clear, some see nothing but "political tampering" in any reference to "statistical sampling" regarding the census. For those tempted to label Groves as the pawn of partisans in the White House or the Democratic party, I have a warning: The notion of Bob Groves yielding to partisanship is laughable. As in rolling on the floor laughing out loud laughable. Groves is well known and universally respected among survey researchers and Census Bureau professionals alike. He is an ideal choice for this appointment.
I conducted the interview with Groves below, on the topic of non-response, at last year's AAPOR conference. The Bob Groves in this interview is the scientist and professional his students and peers know well:
Update: AP has updated their story with initial reactions to the appointment (via @AAPOR). This didn't take long:
House Republicans quickly expressed dismay Thursday over the
selection of Groves, saying Obama's choice raised serious questions
about an "ulterior political agenda."
"The fight to protect the
accuracy and independence of the 2010 census has just begun," said Rep.
Patrick McHenry of North Carolina, the top Republican on a House
subcommittee overseeing the census. "President Obama has made clear
that he intends to employ the political manipulation of census data for
partisan gain."
Also, the Washington Post has more background on Groves and the sampling controversy, including this:
Groves served as the bureau's associate director from 1990 to 1992 and currently is director of the university's Survey Research Center.
He has researched why people participate in statistical surveys, worked
to develop surveys with lower non-response errors and studied how data
is collected for surveys.
A congressional aide familiar with Census matters said Groves has
"bulletproof scientific credentials" and is "really highly regarded by
his peers as a low-key, determined guy who's been really focused on
reducing error in survey research for his whole career."
Update 2: Time's Amy Sullivan, who kindly linked to this post, reviewed the census sampling controversy when it came up in connection with Judd Gregg's withdrawl as Commerce Secretary nominee.
In early February, the American Association for Public Opinion Research (AAPOR) censured Dr. Gilbert Burnham, a faculty member at the Johns Hopkins Bloomberg School of Public Health, for violating the AAPOR ethical code for failing to disclose "essential facts about his research," a study (pdf) of civilian deaths in Iraq originally published in the journal Lancet.
When I blogged the story, one reader asked for more specifics about what exactly Burnham had failed to disclose. The response from AAPOR's standards chair, Mary Losch was a somewhat vague summary: "Included in our request were full sampling information, full protocols regarding household selection, and full case dispositions -- Dr. Burnham explicitly refused to provide that information for review."
On Tuesday, AAPOR's executive committee issued a statement (pdf) with more specifics on what they requested and how Burnham responded:
As part of the investigation, the AAPOR Standards Chair requested information from Dr. Burnham. The specific requests related to AAPOR’s finding of violation of minimum disclosure were as follows:
1. The survey sponsor(s) and sources of funding for the survey.
2. A copy of the original questionnaire or survey script used in the 2006 survey, in all languages into which it was translated.
3. The consent statement or explanation of the survey purpose.
4. A full description of the sample selection process, including any written instructions or materials from interviewer training about sample selection procedures.
5. A summary of the disposition of all sample cases.
6. How were streets selected? How were the starting street, and the starting household, selected? Once the starting point was selected, how were interviewers instructed to proceed (e.g., when they came to an intersection)? How were houses and respondents chosen at housing units?
7. The survey description says that, “The interview team were given the responsibility and authority to change to an alternate location if they perceived the level of insecurity or risk to be unacceptable.” In how many clusters did the team change location, and what were the reasons for the changes?
8. The survey description says that, “Empty houses or those that refused to participate were passed over until 40 households had been interviewed in all locations.” Were such cases included in the number of not-at-home and refusal cases counted in each cluster?
Dr. Burnham responded with the following information related to the detailed request:
• “This study was carried out using standard demographic and household survey methods.”
• “The methods we employed for this study were set out in the Lancet paper reporting our findings (Lancet, 2006;368:1421-28). The dataset from the study was released some time ago.”
Despite repeated requests from the AAPOR Standards Chair for the information detailed above, Dr. Burnham refused to provide any additional information. He did not indicate that the information was unavailable, nor did he suggest that disclosure of this information would risk revealing the identities of survey participants.
Keep in mind that AAPOR asked Burnham to disclose these details to their standards committee as a part of a confidential inquiry. They were not asking him to make these details public, at least not at that stage of their investigation. They have not provided information on the nature of the original complaint made by an AAPOR member, which may have involved aspects of the research other than disclosure. Either way, the AAPOR code is very clear about a researchers obligation to disclose such details, on request. Failure to disclose is grounds for censure.
Mary Losch will present an overview of AAPOR's code and the Burnham case and will be available for questions today at 3:00 p.m. at an event sponsored by AAPOR's DC Chapter (more details at DC-aapor.org).
Interests disclosed: I am an AAPOR member and served on AAPOR's Executive Council for two years, from May 2006 to May 2008, but was not involved in the Standards Committee's investigation of the Lancet study.
From my colleagues at the American Association for Public Opinion Research (AAPOR) come two new projects of interest:
AAPOR is again teaming up with The Poynter Institute's News University to sponsor a "webinar" on Understanding and interpreting Polls in the 2008 Election. Â The live webcast, which will be held on Thursday, September 18 at 2 p.m. Eastern time, will feature live audio and a slideshow and will allow participants to post questions. Â Access to the live or archived version will cost $24.95. Â Details are available here.
AAPOR also just launched Survey Practice, a new online publication that has been in the works for two years (including my stint as AAPOR's communications chair). Â The publication aims to be a forum for pollsters and survey researchers to share "share advances in practical survey methods, current information on conditions affecting survey research, and interesting features about surveys and people who work in survey research." Â Links to articles in the first issue is are available here.
And finally the last of my my series of brief interviews conducted at last week's AAPOR Conference, this one with Brian McCabe discussing a paper he co-authored with Jennifer Heerwig that won AAPOR's award for the best student paper presented at the conference. McCabe and Heerwig are both doctoral candidates in Sociology at New York University. The paper is entitled, "Social Desirability Bias in Estimated Support for a Black Presidential Candidate."
For those intrigued by McCabe's summary, I strongly recommend the full paper, a "terrific read," as conference discussant Murray Edelman noted after Heerwig and McCabe presented their results on Saturday. The Heerwig-McCabe paper is easily the most intriguing and currently relevant I saw presented at the AAPOR conference, and worth reading alongside the cover story ("The Big Race: Obama and the psychology of the color barrier") by John Judis in the current issue of The New Republic.
Having said that, I want to pass along two cautions. First, keep in mind that the data were collected in June 2007, when Hillary Clinton, not Barack Obama was the clear front-runner for the Democratic nomination.
Second, consider the caution that Edelman voiced on Saturday (a point that may not be clear until you read the full paper): A "mode effect" may confound the estimates of social desirability bias that Heerwig and McCabe calculate by asking the same question in different ways (explained on pp. 9-12 of their paper and shown in the comparisons of "true" to "overt" support in Table 3). Edelman cited a 2006 article in Public Opinion Quarterly by Smyth, et. al. that found that respondents tend to answer more questions more completely when asked one at a time rather than when presented in a "check all that apply" list format.
Continuing with our series of brief interviews conducted at this week's AAPOR Conference, this one with Andrew Smith, director of the University of New Hampshire (UNH) Survey Center. He discusses his AAPOR presentation on undecided voters in the New Hampshire, which drew on a question I wroteabout often:
For more on the pollster debate over RDD vs RBS sampling see our posts here and here and the AAPOR conference interview with Patrick Murray.
Continuing with our series of brief interviews conducted at this week's AAPOR Conference, this one with Mark DiCamillo, director of The Field Poll in California. A year ago, we reported on Field's use of RBS sampling. Here, DiCamillo discusses details of Field's ability to interview voters over their cell phones in California using Registration Based Sampling (RBS):
One more in series of brief interviews conducted at this week's AAPOR Conference, this one with Chase Harrison, preceptor in survey research at Harvard University. Harrison describes his analysis of the accuracy of pre-election polls during the 2008 primaries.
Yet another in series of brief interviews conducted at this week's AAPOR Conference, this one with Jeff Jones, managing editor of the Gallup Poll. Jones presented findings on the interviews Gallup has conducted by cell phone with Americans living in households with non landline phone service to supplement national surveys and polls conducted in four primary states.
Starting just after 2:00 on the video, Jones discusses the impact of those additional interviews on the general election matchups between the two Democratic contenders and John McCain. While the inclusion of cell phone only households makes little difference in the Clinton-McCain contest, it benefits Obama by a net four points: Without cell phone interviews, and weighted using Gallup's usual likely voter model, McCain would get 49% to Obama's 46% (clarification: this result combines six Gallup/USAToday surveys conducted so far during 2008). With the cell-phone interviews included, the result is Obama 48%, McCain 47%.
And don't miss the cameo appearance by ABC polling director Gary Langer.
Another in series of brief interviews conducted at this week's AAPOR Conference, this one from Patrick Murray, director of the Polling Institute at Monmouth University. His paper, co-authored by Monmouth University colleague Timothy MacKinnon, concerned an experiment with random digit dial (RDD) and registration based list sampling (RBS):
More on the pollster debate over RDD vs RBS sampling here and here.
Another in series of brief interviews conducted at this week's AAPOR Conference, this one with Sarah Dutton, deputy director of surveys for CBS News. Dutton discusses findings presented at the conference (and co-authored by CBS colleagues Jennifer De Pinto and Fred Backus) regarding the effects of race and gender of interviewers on primary polling on the 2008 Democratic presidential nomination.
Dutton discusses these findings in the context of theories about the so-called Bradley-Wilder effect. I've written about that subject here and here. Politico's Daniel Libit's story today also reviews the debate over this issue during the 2008 primaries.
Another in series of brief interviews conducted at this week's AAPOR Conference, this one with outgoing AAPOR President Nancy Mathiowetz discussing her advice to pollsters and journalists on pre-election poll coverage and thoughts on pollster cooperation with the request for data from AAPOR's Special Committee on Primary Polls:
Another in my series of interviews conducted at this week's AAPOR Conference, this one with Michael Traugott, chair of AAPOR's Special Committee on 2008 Primary Polling:
Related - Politico's Daniel Libit also filed a story today that touches on AAPOR's special committee:
In the wake of New Hampshire’s polling glitch, AAPOR convened a task force to study what had gone awry. Though its report has not been finalized, the task force’s head, University of Michigan professor Michael Traugott, says the evidence has yet to point to any “smoking gun.” [...]
AAPOR had hoped to publish the task force findings before this weekend’s convention, but Traugott says the work has been stalled by the hesitancy of pollsters to submit their methods and practices for peer review. He expects to report by mid-to-late summer, providing enough time before the general election for pollsters to tweak their methodology or improve their voter-screening questions.
“It does make for a challenge when you have public pollsters who won’t share their methods appropriately with others,” says Rob Daves, a past AAPOR president. “Science is based on transparency, and I’m not just talking about social science.”
Another in series of brief interviews conducted at this week's AAPOR Conference, this one on the topic of what to make of survey non-response rate from one of the foremost experts on the topic: Professor Robert Groves of the University of Michigan:
Another in series of brief interviews conducted at this week's AAPOR Conference, this one on the continuing growth of call phone only households with Steven Blumberg of the Division of Health Interview Statistics of the National Center for Health Statistics (NCHS):
Blumberg discusses the latest NCHS report on the continuing growth in cell-phone-only households released yesterday. The report's money quote (and chart):
Preliminary results from the July-December 2007 National Health Interview Survey (NHIS) indicate that nearly one out of every six American homes (15.8%) had only wireless telephones during the second half of 2007. In addition, more than one out of every eight American homes (13.1%) received all or almost all calls on wireless telephones despite having a landline telephone in the home.
Unfortunately, we had a bit of an audio mishap with this first interview from Frank Newport, editor-in-chief of the Gallup Poll and also AAPOR's Conference Chair for 2008. The hand-held microphone was not connected properly, so you will hear a bit more ambient sound than is optimal, but Newport's answers are clear.
Also, a verbal typo: AAPOR holds a conference, not a convention. The cause of the audio glitch (an errant adapter) slowed the pace of interviews today, but I have a few more (with better audio) that I'll be uploading shortly.
Just a quick note to say that I am in New Orleans for the 63rd Annual Conference of the American Association for Public Opinion Research (AAPOR). I will be doing video interviews and updates from the conference for the next four days. If all goes well, I will have the first posted later this afternoon, so stay tuned!
