Pollster.com

Articles and Analysis

 

Ezra Klein Reviews Microtrends

Topics: Pollsters

Ezra Klein really hates the new book by Clinton pollster Mark Penn:

[Penn's] new book Microtrends is so bad that the question--in a fair world--isn't whether it will destroy his own reputation, but whether it is so epically awful as to take the entire polling industry down with it.

That certainly grabbed my attention. Klein expands on that idea in his last few paragraphs:

Pollsters occupy a uniquely powerful space in American political discourse: They bring science to elections. Armed with heaps of raw data, they elevate their opinions into something altogether weightier: Conclusions. When an organization sends out a press release saying the organization is right, it's ignored. When a pollster sends out a poll showing the electorate agrees, ears in Washington perk up.

The enterprise has always been dodgy. Populist pollsters reliably discover that the electorate thirsts for more populism. Conservative pollsters routinely discover a small government consensus pulsing at the heart of the body politic. When the libertarian Cato Institute commissioned a poll of the electorate, they found--shockingly--that the essential swing vote was made of libertarians. Remarkably, whenever a politician or self-interested institution releases a poll, the results show a symmetry between the attitudes of the pollster's employer and those of the voters. But Penn's book shines light on this phenomenon: If he is the pinnacle of his profession, then the profession uses numbers as a ruse--a superficial empiricism that obscures garden-variety hackery. And that's a trend worth worrying about.

I have not yet read Penn's book, and like Klein, have never met him. But my own sense is that Penn is an unusual case as political pollsters go, both in terms of his paycheck and methods (for example, as Klein notes, "unlike most pollsters, Penn never releases his raw numbers, only his analysis").

But set Penn aside for a moment. Klein raises a fair point about the many publicly released surveys sponsored by partisans and interest groups. Consumers should always approach such data with skepticism because -- surprise, surprise -- interest groups and their pollsters tend to cherry pick results that make the most compelling case for their side. Educated consumers confronted with such releases should always wonder, "what results am I not seeing?"

Still, I think Klein goes a bit too far here. His conditional rhetoric -- "If [Penn] is the pinnacle of his profession, then the profession uses numbers as a ruse" -- strikes me as the same sort of speculative leap (based on a sample size of n=1) that Klein finds so troubling in Penn's book.

I am a pollster, of course, so some bias on this issue is inevitable. Readers, what do you think? Is Klein's criticism (of Penn and pollsters generally) fair? Is Penn "the pinnacle" of the polling profession?

Update - Mark Penn emails:

Given all the reviews of Microtrends I am rather surprised the only one you mention is Ezra Klein.

Would appreciate your mentioning or linking to some of the reviews in USA Today, Business Week, Economist, Bloomberg, Politco, Newsweek among others that had a very different and very high opinion of the book.

It is unfortunate but not surprising that an American Prospect writer put out a review like this given their past articles. Of course that would be a correlation, not necessarily causation. Or is it?

It is not at all a political book and I hope you will read it and enjoy it.

Fair enough. I added the links above. For what it's worth, I did link to the Politico review a week ago.

 

Comments
Midwestprof:

One thing I've been noticing with my students is that they are severily skeptical of any statistical arguments these days. "You can take any numbers you want, and have them 'prove' any conclusion you'd like."

I think abuse of dishonest statistical arguments such as the one described by Klein has weakened the power of persuasion of statistics. That, coupled with the inherent unintuitiveness (beware: not a word) of many statistical ideas makes up for a very unprepared citizenry.

Stats should be a bigger part of the math/social studies curriculum in high school. And there should be less hackery among hired statistical guns.

____________________

hwc:

Reading Klein's blog, one suspects he would say that Mother Theresa threatens the institution of sainthood if he thought Theresa were associated with the Clinton campaign. Note that his "review" starts out with two shopworn bromides about Penn the unionbuster and Penn the centrist.

I'll leave it to the polling professionals to sort through Penn's stature in the industry. I figure that, if he's good enough for succesful marketers like Bill Gates, Bill Clinton, and Hillary Clinton, he probably is a pretty talented guy.

____________________

Allan and Sasha:

Mark Penn would simply not have such a prominent role in Hillary's campaign if he weren't brilliant and effective. It's really that simple. She (Hillary) could have any pollster/advisor she wanted. Based on her continued success in the primary campaigns (is there even one state where she isn't leading?), I'd speculate that she has the best, or one of the best pollsters in the country.

____________________

hwc:

Likewise, I suspect that Bill Gates could probably attract most any pollster in the business.

Penn's strength is that he is able to identify mainstream rank 'n file Democrats and understand the issues that are important to them without allowing the visibility of the cultural elites and netroots to cloud the picture.

Clinton has now run two issue-oriented advertisements that hit on the key real-world issues that real-world middle class Democratic voters face in their real-world lives. Health care. Childcare for working women. Veterans services. Paying for college. And, so forth. These voters don't care about "hope" or "bringing corporations to their knees". It's really just apply polling technology to the age old marketing challenge of understanding your customer.

____________________

Mark Lindeman:

That's certainly a funny review, whatever else it is. (I haven't even seen the book, much less read it.)

But if Klein is going to rant about Penn's inability to handle numbers, he really, really can't afford to complain that Penn cites a survey proportion of 1% with "a margin of error of plus-or-minus 4 percent -- four being a larger number than one." That's just ugly.

No, I don't think Penn is the pinnacle of the polling profession (and now my lips are sore from all those plosives). What he does combines survey research, strategic analysis, and blarney. Observers differ about the relative proportions in that mix.

____________________

flyingkimura:

As someone who has worked both with and against Mr. Penn, I can emphatically say he is not "the pinnacle of his profession." While micro-blather makes a tasty pitch, it's of little practical value to the people charged with creating campaign advertising.

What an ad team needs from a pollster is a brief crytallization of the decisive thematic difference (yes, singular) between the candidates. A strategy statement.

Instead, Penn provides a Chinese menu of the issues most attractive to whichever sliver of the electorate he is obsessed with that week. Rarely is there a thematic thread to hold them together. A battery of issue positions is no substitute for a theme that can support ongoing creative executions and elicit a visceral response from multiple groups.

The result of Mr. Penn's smallthink is that his campaigns are meandering and messageless, a game of targeting whack-a-mole where the tagline behind the podium changes every three days (see HRC).

Does it work? Only when Penn's driving the train for a prohibitive frontrunner. If you're nursing a 10-point lead, Penn's "analysis" can limp you to the finish line. But if you're on the short side of the trial heat, you'd better find a new pollster.

____________________

hwc:

Since Penn is such an inept pollster/strategist, I guess he's just lucky that his political clients are such big frontrunners!

With a 20 point lead in the polls despite having such an inept strategist, imagine how well Hillary would be doing with one of the "big theme" whiz-kids like Axelrod, Trippi, or Schrum at the helm of her campaign. It boggles the mind!

____________________

hwc:

Of course, Penn's clients winning elections is just correlation, not causation. Or is it?

____________________

Andrew:

Klein wonders if this book will damage Penn's reputation. But the Penn & Schoen poll taken before the Venezuelan referendum to oust Hugo Chavez was probably the worst poll in the history of polls.
Penn & Schoen accurately predicted the result, but there was one problem. They got the winner wrong and the margin of error on the candidate was about 40.

PEnn worked for the opposition and his reputation is already on the floor.

____________________

flyingkimura:

hwc seems quite eager to credit Mr. Penn for double-digit leads NOT blown. Is that all it takes to reach the "pinnacle" of the polling profession these days?

When evaluating the relative merits of consultants, might it not be a better idea to consider degree of difficulty? How did Pollster X perform when clients faced a significant deficit? Faced an unfavorable environment? Got out-spent by a significant margin?

By that standard, it's hard to place Penn/PSB in the top half of the class.

Nor would I be in a hurry to make assumptions about Penn's performance based purely on his client list. Brilliance in a pitch and brilliance on a campaign are very different things. Shrum once brandished a platinum client roster as well, and we know how that story ends.

The "big theme" you disdainfully dismiss, hwc, is the currency of the marketer's profession. Just because we rarely see it in political campaigns doesn't mean it isn't desirable.

In my experience, some public opinion researchers recognize better than others that a strong brand (based on a big, lasting idea) is the ideal result of the marketing process.

Then there is Penn. Had Nike been his client, the "Just Do it" campaign surely would've been scrapped in favor of a direct mail/robocall blitz to time-pressured, coastal suburban dads 25-44 that said, "Titanium Grommet Reinforced Eyelets for A Snug Fit."

____________________

I have too distant a perspective to offer anything useful on Ezra's particular piece, but the idea that polling is dubious arises every four years (two, if you're really into polling and follow THOSE microtrends). Since I really started paying attention in 2000, I have observed two things:

1) Polls done before elections tend to be more accurate than people imagine. If you, ahem, took a poll of the electorate and asked them about polling accuracy, my guess is that you'd find a large chunk who were skeptical. Those whose team trails during election years tend to be more skeptical--as I was in 2004, when I thought cell phones or voter fatigue contributed to bias against youth and liberals. I couldn't imagine how the polls were failing to capture Kerry's obvious strength. I think this may be one part of the phenomenon.

2) Where polling has a more malign affect is when survey findings actually become the cause of election momentum. Weeks before an election, when the voters are aware of the candidates and their opinions, polls have less influence over elections. But months out, people may have never heard of candidates outside the context of polls, and therefore what they know is that some candidates are huge front runners. The self-reinforcing cycle means that name recognition becomes a proxy for popularity--and when people ignorant of the candidates are polled, they reflect name recognition. In this case, pollsters aren't measuring opinion so much as creating it. And in this way, pollsters on pols' payrolls can manipulate ignorance to give the appearance of strength.

____________________

TomT:

Klein may go too far in his criticism of pollsters in general, but it's hard to quarrel with his take on Penn. The fact is Penn doesn't release the internals of his polls so it isn't clear if he has evidence for what he's claiming or if he's simply claiming it to gain some advantage.

One thing that is not remarked upon enough is this: that polls are now used a club with which to bludgeon the other side. Why bother refuting another's arguments on merit, when you can just say "you're out-of-the-mainstream, an extremist, that's what the polls show." When Bush had a high approval rating you were called a nut if you opposed him. Now you're called a nut if you support him. But his policies haven't changed, of course.

Which brings me to the point made by Allan and Sasha:

Mark Penn would simply not have such a prominent role in Hillary's campaign if he weren't brilliant and effective.

He may be brilliant and effective, but that doesn't make him a good pollster per se. Maybe what he's brilliant at is providing bogus data that bolsters some argument he's trying to make. Maybe Hillary knows he can use bogus polling data that makes her look more popular, which (given the power of polls) will itself make her more popular.

____________________

Tony V:

Please please, read the book yourself Pollster, and let us know whether Ezra's a bitter critic, or Penn's actually a hack.

____________________

bob:

A lawsuit filed in New York by a former employee of Penn's polling firm, Penn, Schoen & Berland, alleges that when the employee left the firm and started a rival consulting business, workers at PSB hacked into his BlackBerry and illegally monitored his email. The lawsuit, filed in mid-June and reported by the AP on Wednesday, claims that Penn approved of the surveillance.

____________________

flyingkimura:

hwc: Can you hear me now?

____________________

szr:

It has been a really long time since this blog post was, uh, posted, but I did read Microtrends. Oh....My....Gawd....

It truly is a huge pile of hippo dung. I couldn't even read it in more tha 15 minute spurts before laughing out loud at one logical fallacy after another. I found it endlessly entertaing that anyone would find this book insightful. Man is it funny!

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR