Column: Strategic Vision and the Transparency Gap
Mark Blumenthal | September 28, 2009
Topics: David Johnson , Nate Silver , National Journal column , Strategic Vision
My National Journal column for the week is now posted. Filed on Friday, it reviews the reprimand of polling firm Strategic Vision, LLC issued last week by the American Association for Public Opinion Research and the ensuing controversy sparked by Nate Silver's allegations that firm's polls "exhibit unusual patterns" that ""suggest, perhaps strongly, the possibility of fraud."
For further reading: I first reacted to Nate Silver's first "trailing digit" analysis on Friday and reviewed some of the contradictions and odd facts surrounding Johnson's reaction to AAPOR, the absence of Strategic Vision cross-tabs and the difficulty of locating a physical Strategic Vision office on Saturday. Over the weekend, Nate Silver did some additional analysis using Quinnipiac surveys as a control, raised more questions about a Strategic Vision survey of students in Oklahoma and tracked down** what appears to be a Strategic Vision office in rural Georgia. Meanwhile, Strategic Vision CEO David Johnson promises cross-tabs to the Atlanta Journal Constitution's Jim Galloway once again and tells the St. Petersburg Times that he plans to sue Nate Silver. Also, both Politico's Ben Smith and The HIll's Aaron Blake did roundups with reaction from Johnson.
**Update: Though he gave due credit to commenters both here and on FiveThirtyEight. I can say I heard it first, here, from socio-logic.
By Mark Blumenthal | September 28, 2009 8:30 AM | Permalink | Comments (7) | TrackBacks (0)
Comments
Mark -
I think the witch hunt against Strategic Vision is perhaps a bit off the mark.
It rests on 2 unproven assertions:
1. That Strategic Vision is some sort of outlier in terms of the trailing digits metric.
2. That this metric is useful in assessing "other than methodological" differences in polling.
Both are unproven.
The main line of evidence is that the trailing numbers skew in a non-random direction, that can only be explained by falsifying data using non-random techniques.
This is not particularly compelling because of the following:
1) Trailing digit patterns are largely dictated by how you treat undecideds (how hard you push them to commit) and how far in advance of the election you poll. Far in advance will produce more undecideds and thus impact trailing digits.
2) Pollsters vary GREATLY in their effort to classify undecided voters.
- Some word the question loosely to encourage soft commitment
- Some as a "leaner" follow-up (Rasmussen)
It is the strength of effort to classify undecideds and the method in doing so that will cause a non-random skew in the data.
Nate Silver's singling out of Quinnipiac Polls felt very much like cherry-picking.
Why did he not show 10, 15 or 50 pollsters trailing digit patterns?
He should have run the analysis for all major pollsters and showed us the distribution for each.
He did not do that? Why? Where is his dataset for public scrutiny?
I see none.
The fact is that some have much greater skews that Strategic Vision.
Here is my experiment.
I took all presidential approval polls from George W. Bush as archived by the Roper Center.
This produced 2,894 trailing digits.
What is good about this is that it is that pollsters are measuring:
- The same basic question
- In the same geography
- Under similar conditions
- Over the same time period
Note that the GWB approval rate ranges from 19% to 92% with an average near 50%. This almost perfectly mimics a normal distribution.
The result?
The trailing digit SKEW differed wildly across pollsters even under CONTROLLED conditions:
FIRM N %0-4 %5-9 Spread
Fox/OpDyn 282 49 51 1
Gallup 226 46 54 7
Gallup/CNN/USA 218 42 58 17
Pew 200 50 51 1
Newsweek 186 46 54 8
ABC/WP 172 52 48 5
CBS 162 49 51 2
Democracy Corp 154 52 48 4
ARG 138 38 62 25
NBC/WSJ 138 41 59 17
CBS/NYT 132 58 42 15
All 2,894 48 52 3
Even among these firms measuring...
- The same thing
- In the same geography
- Under similar conditions
- Over the same time period
We saw a spread on %0-4 Trailing Digits vs. %5-9 as high as
ARG: 25 points
NBC/WSJ: 17 points
Gallup/CNN: 17 points
CBS/NYT: 15 points
Strategic Vision's spread was 10 points, and this was...
- Different candidates
- Different geographies
- Different time periods
- Close races
Strategic Vision would be #5 if included with the pollsters above.
Even under highly controlled conditions, Nate Silver's metric proves absolutely useless.
How can it be valuable in less controlled tests.
Nate's own method indicts about 30-40% of pollsters as outright frauds.
Do you agree?
Your methodology and threshold indict about 1/3 of pollsters as frauds.
Nate Silver has turned off comments on his site.
His analysis is getting picked apart by commentors.
He still won't post his data set.
Why did he turn off comments?
Matt - who goes by the name "MidPointMan" on Silver's blog, fivethirtyeight.com, is incorrect. Comments have not been turned off, and some comments have been posted there within the last few minutes.
Matt's "controlled" experiment - which no one else has verified or duuplicated - is just as cherry-picked as he claims Nate Silver's original choice of Quinnipiac as a comparison was. Additionally, he seriously misrepresents Nate Silver's original assumptions and line of thinking.
Both sides in this debate seem intent on turning it from a dispassionate analysis into an attack on each other's motives. But that's life in the America of 2009, I guess.
Matt,
No idea what's up with Nate's comments, but, having struggled mightily with our own bug-prone, still-not-working-quite-right system, I'd give him the benefit of the doubt on that one. They've probably generated a bigger crush of comment traffic than usual, but who knows?
As for your substantive comments, I hope the tone of what I've written so far demonstrates a sense of caution. What I've seen so far seems a long way from "beyond a reasonable doubt," particularly given some of the issues raised by you and some of the commenters on fivethirtyeight. I don't pretend to have expertise at trailing digit analysis, and so I'm reluctant to weigh in as if I do.
Can you email me directly? I've got a few questions about your argument (mark at pollster dot com).
[And adding this after seeing Stan's comment, which I hadn't when I clicked "publish:" Same request to Stan -- could you email me directly?]
Matt, I think the main 'beef' against Strategic Vision LLC is the lack of transparicy in either who sponsered their polls, sampling method (or for that matter even where their office is). Mark makes that pretty clear in his article.
As to Nate Silver's theory, while it is an interesting way of using probability theory to look for discrepancies I do not think he has quite yet proven (or claims to) fraud. There still could be quite easily a mathematical reasons why the trailing figures are not uniformly distributed such as a weird way of rounding etc, of course this would be much easier to prove one way or the other if SV LLC were more open.
However I do think you are misrepresenting several points of his arguement. Firstly he took all of SV LLC polls to try and reduce the chance of a reason, specific to one particular race, to distort the distribution as much as possible. Secondly he picked quinnipaic, to compare SV, as their polling portfolio, as in what political races and public opinions and ratio of these, was closest to SV LLC. If you disagree with his choice give some reasons.
Nate's arguement was that that SVs' trailing numbers had a much greater quantity of 8s and a much lower quantity of 1s than you would normally put down to random variation (assuming that data can be modeled on uniform distribition) yet you imply it was the difference between the 0-4s and the 5-9s. This is an important point, there could well be some reasons why low numbers come up more than higher ones but much fewer why adjacent numbers are radically different.
Your own example is rather weak comparison. Each pollster only has a sample size of about 200, (compared to more than 5000 taken be Nate) and by taking the binary option you effectively nearly double the sample varation (anything which isn't a 0-4 has to be 5-9). Even assuming there is a 50/50 chance of the trailing figure, the only pollster which sticks out in your example is ARG (about 125 to 1 against that size of spread) the others would not fall outside a 99% confidence interval. In Nate's test the chance of it happening by random is over a million to one. (BTW he never indiciated a threshold)
p.s, Great article in the journal Mark!, couldn't agree more with it.
Regarding "Comments" at 538.com. They have been experimenting with a new comment function. They have not turned comments off, but the new one didn't work so well and they reverted to the old one for now. Contrary to remarks above, the comments were not overwhelmingly negative. A few radright posters did go ape, however, and posted myriad times.
Hmm... I think I actually was the one to track down the office (I posted here, on 538, and also emailed Nate Silver). Oh well, just a little part of me was hoping for some credit on behalf of citizen journalists everywhere (aka grad students looking for an excuse to procrastinate). But seriously, the analysis provided both here and on 538 has been phenomenal.
Posted on September 28, 2009 8:26 PM