Articles and Analysis


Strategic Vision: Back, But Not Here

Topics: AAPOR , David Johnson , Disclosure , Mark Grebner , Michael Weissman , Nate Silver , NCPP , Strategic Vision

They're back. As reported yesterday by Politico, Strategic Vision, LLC posted results** from what they claim is a survey of Georgia. As per our previous entries on this subject (here and here), we will no longer publish their results as "poll updates" or in our poll charts. Yesterday's release does virtually nothing to answer of questions raised by well over 200 purported surveys released by Strategic Vision since 2004. It also falls well short of the minimal standards of disclosure that got the company into trouble in the first place.

For the uninitiated, the saga began with a rare censure by the American Association for Public Opinion Research (AAPOR) last fall resulting from Strategic Vision's failure to comply with requests for information about their response rates and weighting procedures -- information that 21 other organizations provided upon request in connection with AAPOR's investigation of the primary election polls of 2008.

Following AAPOR's action, blogger Nate Silver raised the possibility of fraud and subsequently found a pattern in the trailing digits of the percentages reported in Strategic Vision polls suggesting a "possibility of fraud." Michael Weissman, a retired professor of Physics at the University of Illinois and frequent commenter on Silver's site, did some additional number crunching (a Fourier analysis) and concluded that the odds were 1 in 5,000 that the pattern in Strategic Vision could have been produced by chance alone.

The issues raised by Silver and Weissman were highly technical and difficult for mathematical mortals to evaluate, but even more troubling was Strategic Vision's strange pattern of half-truths and evasion. Commenters on FiveThirtyEight discovered that the four offices listed on the Strategic Vision web site were UPS store mailboxes. In the wake of the initial stories, Strategic Vision CEO David Johnson announced to at least five news organizations that he would soon take legal action against AAPOR and Silver. He promised to release additional subgroup tabulations of the contested data. None of this ever happened.

"We intend to vindicate ourselves," Johnson told Politico a few days after the AAPOR Censure. If his surveys were real, if they had been conducted by live interviewers at actual call centers, Johnson should have a wealth of evidence at his disposal to silence his critics. The public polls released by Strategic Vision since 2004 (archived here and here by Harry Enten) add up to more than 200,000 interviews. That many interviews would leave a lot of witnesses: Call center managers, supervisors, probably hundreds of interviewers, any one of which could come forward to vouch for the process that produced the numbers. And there should be electronic records of the actual survey data somewhere -- at least for the most recent projects. Why hasn't Strategic Vision taken any steps to present some of this evidence and vindicate themselves?

Yesterday, Johnson tried to Strategic Vision stopped releasing public polling in September and went dark in terms of public polling until this week. Here's what he told Politico:

[Johnson said] that the lull in his firm's work had represented a deliberate choice to take some time off in the light of the allegations and let the scrutiny subside. He also said a family illness prevented him from polling the Georgia gubernatorial race earlier in the year.

"Some of the stuff was getting to me. I felt it was best to take some time off," Johnson said. "You know the old adage - lawyers should never defend themselves. I should never try to be my own PR person."

He also told Atlanta Journal Constitution columnist Jim Galloway that his libel suit threats "was me speaking in anger because I was really outraged at the time."

Yesterday's release includes two new twists. For this first time, Strategic Vision sent Galloway and other reporters a set of tables in a compressed file showing results tabulated by gender, age, race and income, and all of their percentages are computed to one decimal point. Is it more likely that these results are based on some sort of interview data, for what that's worth.

Mark Grebner, a Michigan-based, Democratic political consultant, left this comment on Pollster.com last month when Johnson started promising new surveys:

I've got a counter-intuitive guess: maybe SV-LLC will start doing real polling. It's not hard, it doesn't cost much money, and it would serve more than one purpose.

One thing is that it would restore their status as serious participants in conservative politics. A second benefit is that it would undermine the case against them, at least in the public's mind.

Maybe they'll never release another result, but if they do, I'd guess it would be genuine.

Doesn't affect the utter bogusness of everything they've done to date, of course.

He's right that new surveys, even if "genuine," do nothing to resolve the serious questions raised about Strategic Vision's previous work.

But let's ponder the meaning of "genuine" as we consider what Strategic Vision's latest release does not tell us: They say nothing about the mode of the survey (whether it used live interviewers or some automated method), the sample frame (whether telephone numbers were selected from some sort of list or via a random digit method), the weighting procedure (whether results were weighted and the variables used to weight them), and they do not identify of who conducted the survey (the call center or field-work provider, if these used one). These basic facts are part of the minimal disclosure requirements of both AAPOR and the National Council on Public Polls (NCPP).

NCPP also requires that its members describe the "size and description of the subsample, if the survey report relies primarily on less than the total sample." It is not clear whether NCPP's mandate applies to cross-tabulations, but it is very clear that Strategic Vision tables provide no information about the size of each demographic subgroup.

Both organizations also mandate that releases tell us, "who paid for the poll?" Strategic Vision's release says nothing about how this poll was paid for and, as an alert Pollster reader informs me, fails to disclose a significant conflict of interest: A search of Georgia campaign finance records shows that Strategic Vision was paid $3,500 to conduct a poll in 2009 for Ralph Hudgens, a candidate in the Republican primary contest for Insurance Commissioner tested in the new survey.

So again, for all of these reasons, we will no longer publish results by Strategic Vision, LLC on Pollster.com. But that raises a much bigger problem: Strategic Vision is not the only polling organization that has fallen far short of the minimal disclosure requirements of organizations like AAPOR and NCPP, and their results do appear on Pollster.com. That shortcoming is something I want to discuss at greater length this week. Stay tuned.

**For what it's worth: That link and the rest of the Strategic Vision, LLC web site remains inaccessible to computers in our offices and to our colleagues at the National Journal Group and Atlantic Media.


Mark Grebner:

Note that SV-LLC has suddenly started posting numbers that DON'T total exactly 100.0%. In other words, they're now using some rounding technique that the rest of us can possibly imagine, so the totals for 3-option questions are sometimes 100.1% or 99.9%. Could this change in procedure have resulted from finally having to calculate percentages from actual tallies, rather than simply fabricating the percentages directly?

Also, notice for some questions they show exactly nobody selected a particular response (such as "refused") even in cases where that response would almost certainly be selected by some respondents - if it were available. This (weakly) suggests the method of interview was robo-polling, since failing to program an option would result in exactly nobody choosing it. Also, robo-polling is cheaper, and given SV-LLC's reputation, they can't be charging top-dollar for their services.

I'm now completely certain the results previously released by SV-LLC were fraudulent. Since they're back in business, I assume they've either turned over a new leaf, or at least adopted more sophisticated methods of falsifying data - methods designed to satisfy the statistical tests which caught the previous releases. The change in rounding technique would be consistent either with their becoming legit or improving their scam.

It'll be interesting to watch the release of new surveys to see if we can can observe evidence of new methods of fabrication. Just as in the worlds of counterfeit money or steroid use, we may see an arms race between better methods of concealing fraud and better methods of detecting it.


Mark W:

Ahhh, and here I was worried that they would slip away quietly in the night, never to be heard from again. Glad I was wrong on that count, and that ridicule of Strategic Vision can continue.

After all the negative press and attention given to this issue, the question of, "Who paid for this new poll," seems to be dwarfed by the question, "Who would pay for ANY Strategic Vision poll?" Who would ever be interested in purchasing results that might have even a chance of being inaccurate?

I think a little forensic accounting might be in order here. This new poll isn't for a campaign. If the poll is not legitimate, I'm guessing it wouldn't be too expensive to forge these results, and if it is legitimate, then the money has to be coming from somewhere. Mark, do you (or does anyone else) have a sense or rough estimate of what it might cost to conduct a poll such as this one?


Mark Grebner:

The SV-LLC site claims 800 completed interviews. If we guess 5% completion rate (which would be at the upper end of my experience, using a well-prepared voter file) that suggests initially calling 16000 numbers. At 4 cents - again, at the favorable end of my experience - that would come to $640 in raw vendor charges.

How much might be spent recording the initial interview and programming the decision tree, and then how much creating the tables and formatting the output would all depend on the level of expertise available in-house.

Judging by SV-LLC's previous "work", they wouldn't seem to have a very deep pool of talent to draw on. Given the streaks of zeros inappropriately reported in the cross-tabs, the standards weren't set excessively high here.

My shop could have run a similar poll for about $640 plus some elbow grease. How much SV-LLC would have spent - if this survey was actually conducted - is hard to determine but must have been at least that much.

(BTW, we need some correspondents whose name ISN'T "Mark" - it's getting confusing.)


Mark W:

I hear that. At one point on one of the pollster election night blogs, I remember noticing that about 5 comments in a row were from different "Marks". Apparently we're all political and polling junkies.

A couple of followup questions to you, Mark (Grebner). I assume that 650ish is the internal cost for a poll like this; what might a client be charged for it? Second, do your estimates include the labor costs (interviewers, management, etc.)? The reason I'm asking is that I'm trying to get a sense of the difference in profit margin (in absolute terms) between paying out vendor and labor costs, and completely forging everything (which I'm assuming could be done cheaply in a couple hours on a PC), and whether or not it would be profitable enough to take the risk of being exposed as a fraud.

I'm still not convinced that Strategic Vision is anything more than a scam, and the research I've been doing the last couple of days is just furthering my skepticism.


Mark Grebner:

If the survey was done using robo-polling, there wouldn't be any interviewers involved - just the playing of pre-recorded questions and collection of phone-key presses.

For an 800-interview robo-poll with 13 questions, I'd guess a client would pay about $4000. That would cover basic tallies and cross-tabs; any pseudo-philosophizing ruminations on the state of American democracy would be extra.


Mark Blumenthal:

Just a few thoughts to add (as well as one more "Mark" in the conversation):

The fixed costs of IVR are very low: Mostly the leasing of WATTS telephone lines and the cost of the equipment that makes calls and records responses.

Rasmussen announced back in January that they will soon offer to do an IVR poll for a flat fee of $600. The company Precision Polling also offers a similar do-it-yourself service that charges 10 cents a call (sample extra). I wrote about both in a column in January.

Both services presumably rely on automation to reduce the marginal costs of programming setup and tabulation and high volume to offset fixed costs. But their prices should give you some idea of what it costs to do IVR.


Mark W:

Mark(s): Very interesting. Thanks for your insight on the subject. I would note that in the past David Johnson has been very insistent that all of Strategic Vision's Polls were done with live interviews, which means that he's probably charging even more than $4000 (whether or not he's actually using robo polling). I'm not sure how many polls he was getting in a month, but four grand a pop with no operating costs isn't chump change.

I'm adding this event to my (very lengthy) list of discrepancies. I spent most of today digging around on this subject and I can honestly say I'm more convinced than ever that not only are Johnson's polls fake, but his whole organization is a sham. I'm putting it all together and hopefully will have it ready to post tomorrow, but suffice it to say that the fact that SV-LLC's offices aren't really in Atlanta is only one of their many problems.



This exchange raised some new issues for me. To an economist (I am an economist) the true costs of conducting a poll wouldn't normally depend on whether or not the analysis is done in house or whether it is farmed out. The idea is that you don't save money by keeping the analysis for project A in house because time spent by your analysts on project A is time that your analysts can't spend on some project B which also could have been sold to clients. So money saved by not farming out analysis A is then lost when you can't bill clients for analysis B.

Of course this whole argument presumes that the firm in question stretched near to its limits. If, instead, the firm has analysts sitting around doing nothing but collecting pay checks then the situation is totally different. In this case it is far cheaper to give the work to your idle analysts than it is to farm it out.

Thinking along these lines raises a bunch of interesting question. Did Strategic Vision lay off a lot of its staff after the AAPOR censure? Or has it been carrying and paying idle staff for some months now? Did SV have a large staff during the period when it was prolifically producing polls? If so, then either SV must have been taking quite a bath financially over the last few months or it must have layed off a lot of people. If, instead, SV operated during its prolific period largely by farming out labor then there should be a lot of people out there who have worked for SV in the past. Where are they?

Basically there should exist some large-scale combination of:

1. SV contractors.
2. Laid off SV employees.
3. Red ink.

The financial records of SV should shed a lot of light on what has gone on there.

Mike Spagat


Post a comment

Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.