Pollster.com

Articles and Analysis

 

Unbelievable

Topics: 2006 , The 2006 Race

Here is an item published by Roll Call on Wednesday that we almost missed about two Zogby polls in New York's 25th District that two media outlets refused to run

The Post-Standard newspaper in Syracuse and WSYR-TV had asked Zogby to conduct a second poll of the race after the pollster acknowledged that his firm had improperly weighted the results of a survey last week. In that case, Zogby polled the 25th district but then weighted the data using voter registration information from the more-Republican 24th district.

Zogby promised the two media outlets that he would do a new poll from scratch, but when the results of that survey came in both declined to run them. Jim Tortora, the news director of WSYR-TV, wrote on the station's Web site that after consulting with outside polling experts, he was concerned that Zogby had conducted the second poll using the same larger sample of 5,000 likely voters as he had on the first survey.

"With respect to Mr. Zogby, we felt the questions raised ... left us with only one choice: We had to pull the poll," Tortora wrote.

Used the same sample?  Here is the explanation from WSYR's Tortora about their analysis of the second poll:

This time, the Post Standard arranged an independent expert from the University of Connecticut's Department of Public Policy to review the findings of the second Zogby poll. Late Tuesday, we discovered that some of the same people who were called for the first poll, were called again. Zogby confirmed they did indeed use the same larger sample of 5000 likely voters, to come up with this "new" poll sample of 502 likely voters. Our independent expert felt this raised a red flag...an unknown variable. The concern? How would you react if you were called twice in about a week, to answer the same questions? Would you answer differently? The same? Would you even take the call? 

And as the Syracuse Post-Standard reported, "27 people responded to both polls."  If that were not enough, even after all of this came to light, Tortora repots:

Mr. Zogby firmly stands by his findings. He insists his methodology is sound, and was prepared to join us live at 5:30pm to explain his findings and back-up his results.  He points out pollsters often disagree about each other's methods. 

In its story on the controversy, the Syracuse Post-Standard spoke to a number of other "national polling consultants," and none supported Zogby's sample recycling.

"I think it's sort of a rookie mistake if you're including people a second time from a database," said Cliff Zukin, past president of the American Association for Public Opinion Research, an industry group based in Lenexa, Kan.

A bad practice

Zukin, a professor of public policy and political science at Rutgers University, spoke before he was told who conducted the poll.

He said it's considered a bad practice to call the same people twice for "random" polls.

"The problem is the first interview activates them," Zukin said. "They follow the news differently. So the people become different from a random citizen.

"If you didn't purge those people from the database," he said, "then that is a significant methodological problem. It gives you a problem to make any inference from these data."

Yes, we often disagree, but there are limits to what can be waived off as a mere difference of opinion.  If Mr. Zogby or any other pollsters want to explain and defend the practice of reusing sample, our Guest Pollster's Corner is wide open.

 

Comments

I don't think I understand why this is an "unbelievable" bad practice.

____________________

Tom McLasky:

It strikes me that any poll in a highly competetive district with national attention (admittedly not this particular race, but many others fit this description), cannot hope to avoid polling the polled. I'm sure that there are some minor implications to heisenbergian theories on passive "pushing", but what pollster out there today screens with, have you allready been polled on the phone? And besides, how does that constitute a serious scientific difference between reusuing sample and personal exposure to friends or colleauges asking for opinions? Murky. The question will be if this poll is an outlier or not. Zogby has been a little off in some small polls this year, but this is home turf.

____________________

I don't understand this at all. If the problem was weighting the 25th district data with 24th district registration, why did he need to repoll at all? Just re-weight the raw data with the correct registration numbers, which I assume is what he did with his second set of responses. I would be intersted in knowing why he chose the 24th wieghting the first time - hard to believe it was just confusion, as that would be stunningly inept.

____________________

freepatriot:

either zogger is an amature, or he has a reason to make such an amature mistake

anybody think this is zogger's first rodeo ???

____________________

Gary Kilbride:

Zogby is not alone. Mason-Dixon made two sloppy mistakes this week in Nevada. In the attorney general race they included Independent American nominee Chris Johnson among the choices, with 4% choosing Johnson. One problem: Johnson died in August and is not on the ballot.

In the heavily publicized sheriffs race, Mason-Dixon provided a "None of These" option, with 4% selecting it. But sheriff is a Clark County race, not a statewide race. Nevada law requires "None of These" on the ballot in statewide races only. I have no idea how Mason-Dixon botched that. They poll every cycle for the Las Vegas Review-Journal and properly don't include the "None of These" category in non-statewide races, like the House districts.

Luckily for M-D, the sheriffs race is an apparent rout (60-24) so the impossible 4% has not been publicized. Attorney general is much closer, 40-30, but widening.

____________________

observer:

What Zogby did in the second poll was a bit unusual, but it's not clear it would imbalance the results any more than a million other assumptions that pollsters make. There is more going on here. Before the results were made public, Jim Walsh, the severely endangered incumbent, launched a tirade about poll methodology.

This shows at least two things: One - he has someone on the inside leaking him stuff, which is no wonder. The Post Standard has now gone on to "fix" letters from readers before publication, so as to make them more flattering to Walsh. They've just endorsed him.

Two, Walsh is very scared of the numbers the poll turned up. He's losing to Dan Maffei by about four points. For honor, Zogby actually redid the poll and released the numbers.

So if there are regularities here, Zogby has at most a share of the blame for them. The bulk of the explanation lies in the fact that thought the poll was supposed to be independently funded, it really wasn't.

____________________

Amy D:

It is unbelievable in a philisophical [or even ethical] sense that Zogby was so incapable of developing a new poll after agreeing to it. A new poll would require a new sample to draw random people from - not the same sample in hopes of calling people not previously reached/polled. NY's 25th district is easily big enough that a new sample of 5000 people could be selected and different people could be reached. While it is possible that in an election year as contentious as this, some people will get polled more than once because of the numerous polls going on, a. that number is very slim and b. those polled twice would more likely be polled by different pollsters with different surveys than the same one days later.

Regardless of who is funding a survey, as a political scientist, survey methodology should be the most important first step in the polling process. I wonder how "different" the two polls by Zogby were. Chances are if he re-used the original 5000 sample, he likely used the same questions, same order, etc... While this recycling or questions may not always be a bad thing, I think Zogby's credibility is now being challenged; that is far worse than having one poorly managed poll. In this case, transparency is important to the integrity of the poll and pollster, as well as the poll funder.

____________________

I've been called several times by Zogby for different polls. I think they continued to call back because I was cooperative and they knew I would answer their political and election questions.
I've never been called by any other polster.

____________________

Richard R:

I've also been called repeately, by Zogby and many others. I'm in Loretta Sanchez's district (Orange County, CA) and it's not bad this election - we're only getting called a couple of times a week. During the primary this year we were getting poll calls 2-3 time PER DAY. A lot of them were message testing, but a healthy percent were straight preference polls. Once I was talking to one pollster, another beeped in on call waiting, I confrenced them together and we had a 3 way. THAT was fun.

I also make them tell me about themselves before I'll talk with them - where they're calling from, if they're paid by the response and how much, how long they've been doing this, that kind of thing.

____________________

pedestrian:

Why doesn't Zogby just do all his polling at 10am on Sundays? That would give him the results the Post-Substandard is looking for. (I grew up near Syracuse.) That would also save him the trouble of having to "pre-screen" his respondents.

____________________

Scrapiron:

This is just further proof that the pollster's make the poll come out to what the think or want it to show. I agree with pedestrian. If you want a slanted poll conduct it at 10-11 AM on Sunday. That way most conservatives will be in church and you can get the democratic (athiest) answer you want. It will prove incorrect on election day but then so do the polls taken after people have voted. Ask me a question at the polls and i'll give you a lie, automatically.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR