Pollster.com

Articles and Analysis

 

Strategic Vision: Time for Transparency

Topics: Disclosure , Nate Silver , Strategic Vision , Transparency

Troubling new details continue to emerge about Strategic Vision following their reprimand from AAPOR for a lack of methodological disclosure earlier this week. We ought to take great care before making allegations of outright fraud, but there is now enough conflicting information -- including lack of evidence of any physical Strategic Vision office -- that the burden of proof has shifted to Johnson. His obligation is not to AAPOR but to the general public. Transparency is the first crucial ingredient that allows us to determine whether their polls, or any other, deserve our trust.

I want to set aside Nate Silver's trailing digit analysis aside for the moment (beyond my comments yesterday), as I understand that he is working on further analyses that respond to some of the questions raised by his commenters. Instead, I want to focus on the conflicting statements of Strategic Vision CEO David Johnson and other details reported by Politico's Ben Smith and others yesterday.

1) Johnson and AAPOR - Johnson told Smith that "he was refusing to cooperate with AAPOR" because the organization refuses to tell him the identity the person that filed the complaint:

"What we have asked from the very beginning was we would share all the methodology we wanted, - we wanted a copy of who filed the complaint," he said. "If they want transparency there has to be full transparency."

The problem with that statement is the phrase "from the beginning." AAPOR made multiple requests for methodological information from Strategic Vision -- and from 20 other polling organizations that did surveys in four primary states in 2008 -- starting in March 2008. These included two letters sent by Federal Express to the primary Strategic Vision mailing address listed on their web site. But these had nothing to do with any "complaint," and Johnson and Strategic Vision ignored them all.

Johnson's claim that he is "refusing to cooperate" with AAPOR contradicts two statements he made earlier in the week saying that he had cooperated fully. The journal Research reported that Johnson "said the firm had supplied AAPOR with all the information it had requested on 19 June this year." He also told ABC's Gary Langer that "I'm a little confused because we provided them the information on June 19."

AAPOR's press release this week is consistent with the more detailed statement given to ABC's Gary Langer by AAPOR Standards Chair Stephen Blumberg: After receiving a request for information in March 2009, Johnson finally "responded with an explicit refusal to provide the requested information," then after "in response to notification of AAPOR's initial findings of a violation, Mr. Johnson provided some, but not all, of the information requested." At that point, he stopped responding to their queries.

This whole episode is both puzzling and troubling. No, pollsters were not generally as cooperative with the AAPOR investigation last year as some of us had hoped. But Strategic Vision's bizarre stonewalling of AAPOR's requests this year, last year and (and of mine in 2007) were unusual. None of it makes much sense.

2) Where are The Cross-tabs? - Ben Smith writes: "Details of Strategic Vision's polls have long raised flags among pollsters, in part because it refuses -- unlike other pollsters -- to release "cross-tabs" -- the detailed demographic breakdowns of individual polls."

Strategic Vision is not the only pollster that fails to regularly post cross-tabular tables on its web site the way SurveyUSA, PPP and others do. Some hold back such tables for paying subscribers. Others prefer to report subgroup results selectively. However, Strategic Vision is the only pollster that, as far as I know, has refused to release cross-tabulations of its political surveys to anyone, including reputable journalists.

Three years ago (3/31/2006), for example, Johnson promised my colleagues at The Hotline that he would "honor requests for crosstabs and will make them available online in 4/06, when their website is revamped to handle the files." No such files ever appeared on the Strategic Vision website.

A helpful reader also alerts me to requests for cross-tabs made to Johnson by Jim Galloway of the Atlanta Journal Constitution in March 2006 and twice (here and here) in 2007, but finds no evidence that Galloway ever received any of the promised cross-tabs.

3) Where's the Office? - Some alert commenters on FiveThirtyEight discovered something that Ben Smith also reported: "[Strategic Vision's] website, as recently as last month, listed offices in Atlanta, Madison, Seattle, and Tallahassee -- all of which match the locations of UPS stores, rather than actual offices."

And to underline the point, the Atlanta mailing address (2451 Cumberland Parkway SE, Suite 3607) that, as of this hour, remains the only address on the Strategic Vision contact page is also a post office box at a UPS Store. That was the same address to which AAPOR sent its FedEx requests during 2008. I called to the UPS Store to confirm, and Suite 3607 is a mailing address that they maintain.

FiveThirtyEight reader inferno asks a good question: "[D]o they have any sort of actual physical address? i.e., an office?" If there is, its awfully hard to find any record of it.

The annual business registration for Strategic Vision filed with the Georgia Secretary of State indicates that their most recent filing -- which, notably, appears to be "active" but in "noncompliance" -- also lists the Cumberland Parkway address. The only previous address for the firm in a prior filing (2002) lists a suite in a Peachtree Street office tower that, according to a Google search, now appears to be occupied by a law office.

4) What Does It Cost? - Ben Smith gets to the crux of the issue: "Another question is how the firm pays for its polls. Its website lists at least 172 public polls, and at a stated cost of $30,000 a poll, that's an expenditure of more than $5 million -- quite a sum for a small firm."

The "stated" $30,000 cost comes from a 2006 interview Johnson gave to then Hotline polling editor Aoife McCarthy (3/31/2006). She wrote:

So how much does all of this cost? Strategic Vision uses these political polls as marketing tool for the company. Johnson says each poll costs an average of $30K to conduct. Do the math -- 39 polls in '05 and 10 polls so far in '06 at the bargain price of $30K results in nearly $1.5M in just 15 months. That does not include the first year of polling in '04.

Would it really cost Strategic Vision $30,000 to conduct surveys that typically include 800-1000 interviews? I doubt it. Not if they are really the "10-question-per-state-polls" that Johnson claimed to the Hotline in 2004 (roughly the length of most polls on their website). Not if, as Johnson reports in the 2006 Hotline interview that "callers are paid directly by Strategic Vision." I'll save the specific numbers for another post (if anyone is interested), but I have a hard time figuring out how his costs could be much higher than $5 per interview (though that still would amount to nearly a million dollars since 2004).

Why would he cite such a big number? I haven't a clue, but given all the other issues now swirling, it is a question David Johnson needs to answer. In 2004 he told the Hotline that "no client is paying," but he told Smith yesterday that some of their surveys "are piggybacked onto other polls." So which is it?   

So my bottom line: I have no idea whether Nate Silver's insinuations of fraud are real, but the burden of proof is shifting. Strategic Vision has to become considerably more transparent about their methods and data, or we will have little choice but to reach an ugly conclusion.

But there is a much bigger problem here for the rest of us. The larger issue is not whether Strategic Vision may be fabricating numbers or whether another less sensational explanation exists for all this obfuscation and contradiction. The problem is that anyone could theoretically make up a set of numbers and -- without a lot more transparency about methods and data than we now typically see -- pass it off as a real poll. The way those of us in the "new media" consume polling releases from every conceivable source, and that certainly includes Pollster.com, makes that possibility all too real. PPP's Tom Jensen has this exactly right:

I could leave PPP, start Tom Jensen Polling, put out a bunch of topline numbers the day before an election that just copied the Pollster, RCP, or Nate Silver predictions and be one of the most accurate pollsters in the country. That would be pretty darn easy and anyone could do it. And that's why public pollsters should hold themselves to a higher standard and also be held to a higher standard by the media.

That's perhaps the most extreme reason why, last month, I argued for a system of scoring pollsters for the quality of their disclosure and posting those scores online, as a matter of routine, alongside polling results. This episode makes the need for such a system more clear and more urgent than ever.

So who's with me?

 

Comments
RS:

Good round-up, Mark.
Fishier and fishier... Something definitely stinks. Just wondering - does Pollster.com still include Strategic Vision in your trend-averages? Now might be a good time to remove their data, until SV provides proof that they are on the up-and-up.
Extend that logic, and include only those polls which score high on your disclosure index.

____________________

socio-logic:

According to a facebook page I just came upon, Laura Ward Johnson (the co-founder of SV) refers to her business loc as follows:

"Tickets can be purchased at the door or in advance at Strategic Vision 22 Town Square, Suite 6 (Seasons Inn Plaza) Information: 706-781-1013"

http://www.facebook.com/group.php?gid=51297892774

____________________

Chris Peterson:

Hey, Inferno from 538 here. (My real name might show up with this.)

Anyway. I was about to post the fact that we'd tracked down SV's office over on 538, but I see I've been beaten.

I did a quick search on Google Maps and Yellow Pages. The number given corresponds to a personal phone number (which...I'm not quite comfortable with it being up if that's the case), and SV doesn't show up at the Blairsville location on Google Maps.

Furthermore, its Blairsville office is about 100 miles (and a nearly 2-hour drive) away from its Atlanta postbox. Not to say nothing, BUT...that's about the equivalent of me having my office in Mt. Pocono, PA or Haddonfield, NJ (keep in mind: Mt. Pocono is in the middle of the Poconos - like, 30 miles past the Water Gap - and Haddonfield is a Philly suburb) and a front in New York City.

(It is plausible, but...)

Also, side note:

I had hired this company to promote...‎
By SuperPages.com
‎my book and initially their customer service was adequate but toward the end of our 7 month contract they ignored my emails containing questions or concerns. They never did provide me with the information they said they would provide at the end of our contract. ..

Hmm.

(SV, LLC is also a PR firm that handles multiple business - I believe business PR, literary PR, and political PR.)

____________________

Mark Lindeman:

Just so it's named over here, let me say that the Oklahoma topline results (figures at bottom) are facially bizarre. I might have believed that 43% of OK students know the names of the two major parties, but do I believe that the only two responses they offered were "Democrat and Republican" and "Communist and Republican"? No, I do not. For that matter, I don't believe that almost twice as many students got that right as the question about who was the first president.

____________________

AlanK:

You can fake it much more easily than that; just lie about your sample size...THEN make up your data if need be. You can run a sample of 20 from your home phone, especially if you're not planning on cross-tabs. They you either enter the data (into Excel or something specialized) and you're done. Or use one of the automated toys that enter the data automatically...they're not expensive. Voila...you have a complete and hard to check (unless you do trailing number analysis--who would have thought of that outside of a stat course?) result and you can probably bluff or threaten to sue any Nate-like person who makes trouble.

Oddly enough, as everyone here probably knows, a sample of 20 is unlikely to be all that different from one of 200, and the difference will be irrelevant unless you're polling a tight political race. If you only make up your data when you "have" to your real results will swamp your phony ones and Nate's analysis won't catch you.

How much did you day he charges per survey...?

____________________

AZ David:

SV LLC did a survey of Arizona's high school students comprised of public school, charter school and private school students. It used the same questions used in the poll of Oklahoma students (in fact, the AZ poll was done several months prior).

The poll was both sponsored and published by the Goldwater Institute. As the name would imply it's a right-leaning anti-tax think tank. It showed abysmal failure of students' knowledge of civics, including how long the term of a U S Senator is, who America's first president was, the constituent houses of Congress, the title of Executive branch's office holder, and even a lack of awareness of which ocean lies off the eastern seaboard (that was answered accurately by the highest percentage), etc.

Pretty basic stuff, right?

Over 2,600 AZ students were polled, with more than half from private schools.

According to the poll only 3.5% of the students passed (passing = 6 or more correct out of 10).
NO ONE HAS QUESTIONED THE VALIDITY OF THE POLL RESULTS HERE IN ARIZONA.

The same test questions were asked of the Oklahoma students and their results were extremely similar in terms of percentages answered correctly.

The report for the OK survey, commissioned by Oklahoma Council for Public Affairs, was written by the same person, often using the same wording (acknowledged) as in the AZ report. The author, Matthew Ladner, holds a PhD and is VP for Research at the Goldwater Institute.

It is clear from the reports' wording that an agenda underlies what is being written by Dr. Ladner. Someone with the requisite skills should compare the results of both surveys to get a grasp of whether it's conceivable that the 2 states should have such similar results. If there are legitimate reasons to suspect the results then the same kinds of questions ought to be asked of Strategic Vision LLC as well as the Goldwater Institute.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR