Pollster.com

Articles and Analysis

 

When Pollsters Attack

Topics: 2008 , Disclosure , Divergent Polls , The 2008 Race

A few hours ago, we posted results from two new surveys of likely primary voters in Florida, both sponsored by the Southern Political Report. One survey was conducted by InsiderAdvantage (which is essentially part of the Southern Political Report) and one by the Mason-Dixon Polling and Research. The remarkable thing about the summary by InsiderAdvantage pollster Matt Towery has less to do with the numbers than with his unusual frontal assault on the Quinnipiac University poll.

I'll get to Towery's blast, but let's start with the survey results. The table below shows the results from the two Southern Political Report surveys, both conducted September 17-18, alongside the results from the most recent Quinnipiac University poll, conducted September 3-9. The most obvious difference is that Rudy Giuliani holds an eleven point lead over Fred Thompson (28% to 17%) in the Quinnipiac polls, but Giuliani leads Thompson by a single, statistically insignificant percentage point (24% to 23%) on the two most recent surveys sponsored by Towery's company. The previous InsiderAdvantage survey showed Thompson with an eight point lead (27% to 21%).

09-20%20florida%20reps.png

So, asks Towery, "why would Quinnipiac have shown Giuliani with an 11 point lead over Thompson -- especially a week ago, when Thompson arguably was at his apex in the Sunshine State?" Two fairly obvious and important differences jump out. First, the Quinnipiac survey includes non-candidate Newt Gingrich (who gets 6% in their poll), while the other two polls did not. That was the explanation that Towery - himself a former Gingrich campaign chair - provided in his own release last week.

Second, the Quinnipiac survey was the only one of the four fielded (at least in part) before Thompson's official declaration of candidacy on the Tonight show on the evening of September 5. The Quinnipiac poll fielded from September 3-9, so at least some of the interviews occurred before the burst of publicity that Thompson received from his announcement. The last InsiderAdvantage poll that showed Thompson "at his apex" kicked off on September 6, the day that Thompson's announcement appeared most prominently in the news.

Our chart of national Republican polls shows a continuing upward trend for Thompson. Moreover, by my hand calculations, the nine national polls conducted entirely after Thompson's announcement (by Cook/RTStrategies, Reuters/Zogby, USA Today/Gallup, Fox News, AP-IPSOS, ARG, NBC/Wall Street Journal and CNN) show his share of the Republican vote increasing by an average of four points (from 18% to 22%) as compared to the average of what the same organizations showed for Thompson before his announcement.

But for Towery the conclusion appears foregone:

In the instance of the Quinnipiac poll showing Giuliani with a monster lead over Thompson, it became all too obvious that it's time to call out this polling organization.

Maybe they're right and everybody else is wrong. But it's unlikely. At the very least, Quinnipiac numbers should stop being taken at face value as the paragon of accuracy in Florida. Somewhere in their methodology they continue to misread the state they claim to know so intimately.

In case the point is not obvious: Towery concluded a week ago that it was "time to call out" Quinnipiac and had his company take the remarkable step of sponsoring parallel studies this week to do so.

Earlier in today's release, after recalling some questionable Quinnipiac results, an old joking reference from former Governor Jeb Bush ("What's a Quinnipiac?") and a jocular answer from Mason Dixon pollster Brad Coker ("I figured [Quinnipiac] wanted more name identification in Florida so they could build recruiting for a Division 1-A football team!"), Towery focuses on what he considers the "bigger issue:"

Are universities that publish polls presumed by the media to somehow be more reliable because there are professors and students involved?

Once we get past all the snark, that's a fair question. It is also fair to ask, as Towery did last week, why Quinnipiac continues to include Gingrich as a candidate. That is especially important since, unlike most other pollsters, they ask no "second choice" question allowing for a recalculation of the results without Gingrich.

And finally, it is also fair, especially given some of the past inconsistencies, to want to probe further into the methodologies of all three pollsters. In writing this post, here are some questions I tried to answer:

  • What was the "sample frame" for these surveys? Who knows? In the past, Quinnipiac has used a random digit dial (RDD) methodology while InsiderAdvantage has sampled from voters lists. But none of the organizations specifies the sample frame used in their most recent releases.
  • How did the pollsters define the Republican electorate? Quinnipiac says they interviewed "registered Republicans" in Florida. InsiderAdvantage and Mason-Dixon say they interviewed "likely Florida primary voters." But what defined a "likely voter?" What questions were used to identify them? You won't find the answers anywhere in the various online releases.
  • More specifically, how tight was the screen? More specifically, what percentage of Florida adults qualified as registered or likely voters? Again, nothing at all from InsiderAdvantage or Mason Dixon. If we do the math on the Quinnipiac sample sizes, we can at least tell that their Republican sample amounts to 38% of Florida registered voters. However, we do not know what percentage of adults qualified as registered voters.
  • Finally, how did the composition of the samples vary demographically? InsiderAdvantage does provide results for gender, age and race, but Mason-Dixon and Quinnipiac do not.

Here is a suggestion for all three pollsters involved: Provide us with answers to all of the questions above, and we will take a closer look at what differences can be found "somewhere" in their methodologies.

 

Comments
Andrew:

Two factors contributed to Giuliani's wider lead in the Quinnipiac poll, as stated in the OP:
(1) It was conducted before Thompson's announcement and (2) it included Gingrich, who arguably split the fundamentalist, ignorant, rabidly ultra-conservative vote with Thompson.

____________________

Mark Lindeman:

Jeepers. What's that about? Is there actually a cult of Quinnipiac-worship whose existence I've overlooked?

"For example, while other pollsters showed then-Florida gubernatorial candidate Charlie Crist ready to blow his Democratic opponent Jim Davis out of the water in the 2006 governor's race, Quinnipiac reported the race to be very tight and suggested that independent voters were 'dramatically' moving away from Crist and toward Davis as the race concluded. The opposite happened, and Crist won comfortably.

"That's just one example of why many who really know the political landscape question why a Connecticut university would migrate its work to Florida...."

Why indeed? why ind-- huh? Isn't carpetbagger rhetoric a wee bit anachronistic?

And can anyone document this quotation? Here is the Quinnipiac press release:
http://www.quinnipiac.edu/x1297.xml?ReleaseID=975
The title cites a "Big Shift By Independent Voters To Davis." I don't see "dramatically" anywhere in the press release. Call it a nitpick, but I like quotations to be quoted. For that matter, can anyone document that independent voters broke to Crist, or can anyone tell what else "The opposite happened" might mean?

That said, at a glance, it does appear that Q's 2006 polls were on average a bit kind to Davis. If so, it's fair to wonder what that was about.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR