Pollster.com: HuffPo's OfTheBus Polling Project

November 20, 2007

This morning, the Huffington Post has announced it's OffTheBus Polling Project that aims to scrutinize pollsters -- "an industry devoted to scrutinizing us" -- by created a forum for survey respondents to report and share their experiences. Here is how Arianna Huffington describes it:

Our aim is simple: to get a better understanding of how polling is being used across the country. We want to get to the bottom of how pollsters conduct their surveys, how they gather and build their stats, how they target who they contact, and, ultimately, how they reach their conclusions -- conclusions that often fuel the very races they are supposed to be analyzing.

We are launching this non-partisan effort to examine the polling industry with a wide variety of co-sponsors reaching across the political spectrum, including: Talking Points Memo, Instapundit, Politico, The Center for Independent Media, The Nation, Pajamas Media, Mother Jones, WNYC Radio, My Silver State, and Personal Democracy Forum.

Our methods are simple and direct, and stress transparency - the key ingredient missing from a lot of polling data. With the help of our co-sponsors we are looking to ask as many people as we can reach to share their polling experiences via this form, telling us exactly how they have been polled. Who called them? At what time? Did they agree to participate in the poll or refuse to (one of the least transparent aspects of polling continues to be the refusal of most polling companies to release response rates, which have plummeted in recent years to around 30 percent)? What questions were they asked? Did the questions seem fair or were they worded in a way that seemed loaded? Did they feel like they were being targeted because of their age, gender, or ethnicity? Did the pollster seem to be guiding them toward a predetermined answer?

At Pollster.com, we share the goals of greater transparency and helping survey data consumers gain a better understanding of how polls are conducted and what the data mean. Greater transparency has great potential to improve surveys, and to help reduce the abuses of the sort we have seen in recent days. Those values are also at the core of our own Disclosure Project. As such, we have signed on as formal sponsors of the HuffPost's Polling Project, and encourage our readers to participate with their own experiences.

As someone who earned his living for more than 20 years as a survey researcher, I believe the respondent is often forgotten by too many in our industry. After all, virtually every number on this site depended on respondents who donated their time to answer the pollsters questions. So having a forum for respondents to report their experiences, both good and bad, should provide a way for pollsters themselves to get a sense for what they are doing well and what not so well. I am convinced that the Huffington Post is committed to creating a resource that is both non-partisan and itself transparent.

Those in the survey industry will remember the "Partnership for a Poll Free America" that Arianna Huffington led a few years ago, and may wonder why we are sponsoring a project led by the person who said she wanted to convince "all 270 million of us collectively decided to hang up the next time some stranger from a polling company interrupted our dinner." The controversy led the American Association for Public Opinion Research (AAPOR ) to speak at its 2003 conference,** because as the conference chair put it,

I believe it is our responsibility to formulate answers [to Huffington's criticisms of polls] and to educate the public about the value and validity of our work, rather than essentially asking the public to give us the benefit of the doubt.

Moreover, as I heard it, Huffington's criticism pertained mostly to how polls are interpreted and used. She argued that pollsters need to be more transparent about their methods, that poll consumers too often fail to understand the limitations of surveys, and that politicians are too often "slaves to polls." I never agreed with her bottom-line prescription ("hang up on all polls"), as it threatened to disrupt a lot of vital non-political research, to say nothing of failing to distinguish the good from the bad of political polling. Still, I see much in her basic critique that I can agree with, and regardless, this latest Polling Project is a very positive step.

So we at Pollster.com enthusiatically support it and encourage our readers to check it out.

**Interests disclosed: I currently serve on AAPOR's Executive Council and attended Huffington's speech to the 2003 Conference.

-- Mark Blumenthal

November 20, 2007 in Disclosure