Pollster.com

Articles and Analysis

 

Somebody's Gonna Be Wrong

Topics: 2008 , Divergent Polls

We certainly have a deluge of recent polls to consider from the states now voting (or that will caucus later tonight, but the highly divergent results on the final round of poll on the Democratic race in California caught my eye first today (as well as many readers who have left comments). I want to start there and, since time is too short this afternoon to delve into the minutia of the California polls, make a more general point about disclosure the bottom line of all this variation.

02-05 final CA Dems.png

The final California poll from SurveyUSA (and five local television stations in California), conducted Sunday and Monday, has Hillary Clinton leading Barack Obama by 10 percentage points (52% to 42%), while the final poll from Zogby/Reuters/C-Span shows Obama leading Clinton by 13 points (49% to 36%). Four other surveys released over the weekend show results somewhere in between. [Elsewhere on Pollster, Charles Franklin charts compares the differences among pollster in California for both the Republican and Democratic candidates].

Needless to say, random sampling error alone (the so-called "margin of error") does not explain these discrepancies. They result from some difference in the methodologies of the two polls -- who they interviewed, how they sampled and interviewed them, the questions they asked, how the selected "likely voters" and how they weighted the results. If we knew more about how these polls differed, we might not be able to predict a winner, but we would at least have a better understanding of the factors that will determine the outcome of the race.

The analysts at the Field poll, in their report on their final survey (conducted over seven days, from 1/25 to 2/1) had a helpful explanation of the variables that will affect the outcome of the election:

[Consider] that 12% of the Democratic voters and 6% of the Republicans either already voted for or expressed a preference for a candidate other than those who remain in the running. According to the poll, over half of these voters were early mail ballot voters, and have already sent them in. Yet, for those voters who haven’t yet cast their ballot, there is some uncertainty as to whether they will follow through and support a candidate who has withdrawn from the race. They may not vote at all or switch to another candidate.

In both major party races, there are also unusually large proportions of voters – 18% in the Democratic primary and 15% in the Republican – who were undecided in the final days of the campaign. These voters had indicated in prior questions that they were highly likely to vote. Considering the many months of campaigning, extensive free media coverage, advertising, and recent televised debates, these voters have been exposed to considerable information about the candidates and appear to be in some real conflict as to whom to support. How these voters come to judgment will have a big bearing on the election outcome of both sides.

There is another aspect to the Democratic primary findings that is unique and where there is not much precedence in previous presidential elections. It is the group of non-partisans who say they will vote in the Democratic primary. These voters have candidate preferences that counter those of registered Democrats. The relative size of each eventual voting bloc, therefore, will have a major impact on the outcome.

Further, there are unusually large differences in candidate preferences among some of the standard voter sub-groups of each party. To pose just one example from each of the Democratic and Republican contests: (1) the large divisions in support between men and women voters in the Democratic primary, and (2) the big split in preferences between strong conservatives and Republicans who are not strong conservatives. In each of these examples, it would not take large changes in voter turnout proportions from those shown in this report to produce a very different election outcome.

All of these factors mean that methodologies matter: The extent to which an interviewer or interviewing method "pushes" voters to make a decision and the mix of demographic groups the pollsters sample will both directly affect where the candidates stand.

I started to write this entry to focus on the two most divergent California polls, but then realized (with the help of a alert reader) that polls in other states are showing similar variation. For example,

  • In California, the margins reported by various pollsters for the Democrats over the last week vary by 23 points.
  • In Alabama, for the Republicans, the margins (between McCain and Huckabee) vary by 24 points
  • In Illinois, for the Democrats, the margins vary by 25 points
  • In Massachusetts, the margins for the Democrats vary by 19 points
  • In Connecticut, the margins for the Democrats vary by 15 points

I could walk through what we know about the methodologies of the SurveyUSA and Zogby polls in California, but it would not resolve the temporary mystery of which will be judged most accurate when the votes are counted. The reason, to focus for a moment on just these two polls, is that while SurveyUSA discloses a wealth of information in their interactive crosstabs and their online report -- how they sampled, what rough percentage of adults their sample represents, what the composition of their survey is in terms of gender, age, race and region -- we know next to nothing comparable about the Zogby poll. And unfortunately, while the Field Poll in California also produces a highly informative report, and while McClatchy released a filled-in questionnaire from Mason-Dixon that includes demographic composition data, the Zogby poll is more indicative of the general lack of disclosure elsewhere.

If ever there was a case for better methdological disclosure by pollsters, this is it. If one poll, conducted entirely over the last 48 hours, shows a candidate leading by 10 points, while another conducted in the same state over the same time period, shows a another candidate leading by 13, and we cannot see enough of the details of how the polls were done to at least explain why they differ, why should we trust what any of these polls tells us?

Typo corrected (thank you, Kevin).

 

Comments

Typo: "while the final poll from Zogby/Reuters/C-Span shows Clinton leading Obama by 13 points (49% to 36%)" should be 'Obama leading Clinton'

____________________

Daniel T:

I voted for Edwards. I realized that in a sense my vote doesn't count because he's dropped out but I voted for the person I wanted to win, with the hope (vain, I know) that if enough people still voted for him he might reconsider his decision.

I can also confirm that the polling places were packed; I have never see so many people at the polls in decades, maybe ever. I think the large turnouts might have an impact on which way these races go.

As for disclosure, perhaps a column is necessary to expalin why more polls don't disclose. I can understand why the campaigns themselves don't but I am not sure why the CNN poll, for example, doesn't.

____________________

Rick L:

I am a Hillary supporter. The other day, I was having dinner with family and friends and the election came up. There were some young people there. Three of the younger people said they were voting for Obama. The asked who I was voting for. I said I would either vote for Obama and Clinton. After the dinner party, I wondered why I said what I said. I don't support Obama, and I'm voting for Hillary, and yet, I said I might vote for him. Why? I think it may be because I didn't want to dampen the conversation. Or perhaps I didn't want to seem 'out of it'. Or I didn't want to be thought of as being racist.

I am now wondering if this is why there is a 'false Obama' effect in polls, and why Obama does better in caucuses, where things are done out in the open. If there are people around when a telephone interview is being done, the person being polled may be saying he or she is for Obama, to not offend others in the room, because Obama supporters are so passionate, calling Obama a movement and all.

I know that my public statement about supporting 'either Obama or Clinton' might be insincere, but at least I didn't have to sit through a debate about Obama with Obama supporters, but all this may explain the 'false Obama' effect.

____________________

Amen, Mark.

____________________

nimh:

Well, I think Survey USA deserves some congratulations :)

____________________

Mike:

lol, I told you that ZOGBY IS FULL OF ****, lol

his brother is an avid supporter of Obama so he fudged the numbers.

There is a story about in legal times or google it.

____________________

KM:

Survey USA rocks!

____________________

Tony:

Survey USA -- nailed it in Calif and Mass -- two places where the Obama-ites were claiming late-surges and victory. Wow. They totally missed Missouri, however.

____________________

BDM:

Zogby blew it for CA but nailed it for MO
CA Obama 49 Clinton 36

susa nailed it for CA and NJ

bUT BLEW IT FOR mo, al AND ct

Their final poll released 2/4 for MO
cLINTON 54 oBAMA 41

Bottom line both are good pollster's but their are times when they are not very accurate

____________________

Anonymous:

The same thing happens to me all the time Rick. I believe there is a "Hillary Effect" or whatever you want to call it.

Obama does better in caucuses CONSISTENTLY. While his ground game in tradiotional primary states is less than overwhelming.

None of the "pundits" have mentioned this.

____________________

Bryan :

At any rate, that's probably what happened in NH, MASS, and CA in some of the polling last night and in weeks past.

____________________

Amy:

I agree that the big distinction here is between caucus states and primary states. Caucuses are easier to attend for people who don't have to work all day, meaning that they are disproportionately attended by college kids. College kids all too often vote for who their friends are voting for (just having gotten over voting for who their parents are voting for), which explains the enthusiasm that dogged Rick L. The problem is that caucuses, with their relatively low turnout and disproportionate representation of certain subgroups, do not provide as clear a picture of statewide support. I don't think there's any question that MN would have voted differently than it caucused. There's just nothing to suggest the electorate would have voted that way, and how many working people can spend all day at a caucus site? I think that this caucus effect makes many polls look to be further off than they are.

I also think that the Democratic party needs to seriously reconsider its support for caucus states. They just are not representative of the voting preference of the majority of likely voters. Come November, voters must actually show up to vote, not caucus, and it would be much more beneficial for the party if it knew who the population was actually willing to show up to vote for.

We all know that college kids are vigorous and outspoken, doing such things as showing up at college rallies (why Obama holds so many there) and crashing blog servers that are not overtly pro-Obama, thereby squelching the opposition. These things make the Obama crowd seem so much more enthusiastic when the polls uniformly agree that Clinton supporters are comprised of a higher percentage of enthusiastic and committed voters. The problem is, though they may show up to caucus more, they still only get one vote in the general election. College students are always, every single election, big on promise, then show up in lower numbers than seniors. Should we really let such a small subgroup have so large an influence on picking a nominee?

____________________

Tom:

Experience matters!!! I don't understand what is with Obama supporters. I know Obama has a lot of young supporters and that's understandable, but it's the older supporters that I just don't get. When is the last time you went into an interview and try to convince the company that experience doesn't matter and that they should give you a very high position job in the company. But apparently, the Presidency is the only job that doesn't require experience.

Obama supporters don't be a hypocrite. Practice what you preach and next time you go to an interview, convince your interviewer that experience doesn't matter. Let's see how that turns out.

____________________

Tom:

Caucus may be over representing the young, but with the same arguement it would be over representing the retirees as well. Traditionally the retirees has a much better history of voter turn out and they favor Clinton much more over Obama. Just the fact that there are a lot more youth participating in this election is a wonderful thing in my opinion but I do agree that caucuses should be done away with since I do not think it addequately represent public opinion.

On the topic of experience of Clinton vs Obama goes, I think often people subconsciously mixed in name recognition with true experience and come up with a "perceived experience" that tends to favor Clinton. If you look at it, Obama has served in public office for 4 more years than Hillary Clinton. You would have to count Clinton's 8 years as first lady to beat out Obama in terms of years served, which is more or less a figure head position. In my opinion, however you count the number of years of experience they have, they are close enough that one should look at more factors than just experience to determine who they think is a better candidate.

____________________

Rick L:

What really amazes me is the bias that the media pundits at MSNBC and CNN have for Obama, and the complete lack of bias at FOX. I can't stand FOX, but last night, I found myself watching them, because they are not going GA-GA for OBAMA. Hillary is now 13 points ahead in the Gallup tracking poll. You can bet you won't be hearing "HILLARY SURGE" on Hardball. They will concentrate on the Obama "MOVEMENT".

I did notice some talk on MSNBC and CNN about 'the narrative' - which is, apparently, their own term for their own bias,. They talked about 'how the narrative will play out.' In other words - 'how are we going to portray what took place.' Yet, they don't seem to be bothered by their own bias.

At any rate, I'm concerned that the split in the democratic race will end up costing us the national election, which is ours to lose.

____________________

Tom, the argument that Barack has more experience is entirely specious. Why should only years in elected office count? Who made up that rule? Romney regularly gets credit for his efforts to save the Olympics, when he wasn't holding elected office. Does that experience not "count"? Please. No one has even suggested that.

Even Barack likes to talk about his years of experience as a community organizer (achieving what, no one knows). Are you ready to say that his experience is meaningless because he wasn't an elected official? What about his time teaching Con Law? Is that meaningless too?

Hillary has real accomplishments from the years before she was elected. She was not "just a wife", ornamental or a figurehead as so many like to imply.

I wonder if you ever thought about the "experience rule" applying to the guys. That's why they call it "the double standard."

I think the "Hillary effect" is real, though I only have my experience to go by. I saw it up close in New Hampshire, when I went to watch the debate at an event set up for bloggers. The Obama kids and middle aged men cheered Obama and hissed at Hillary like they were the only ones in the room. The women (not counting the kids) just looked at each other and rolled their (our) eyes. And last night, my friends and I purposely sought out a bar where Hillary supporters would be, because none of us wanted to deal with the watching the returns with obnoxious Obama fans. They act as though everyone either agrees with them or is an evil moron.

If you watch the news channels, read the major news sites or even watch the Daily Show (esp. since Jon Stewart did a really sexist piece mocking Hillary for appearing on an event sponsored by the Hallmark channel) you get the impression that NO ONE supports Hillary.

To sum up, I think there absolutely is a lot of social pressure to support Obama and oppose Hillary.

What this really reflects, in my opinion, is the divide between the elites and working class / poor voters in this primary. Obama's base of support is white upper income males - just like the older ones who dominate television and print news, and the younger ones who dominate the blogs. The women who have very publicly come out in favor of Obama, including certain erstwhile feminist leaders, are also part of that affluent white cohort.

The people who have the least representation in the media - latinos, lower income women - are almost invisible. But every time there's an election, we see them voting in ever increasing numbers, much to the surprise of the pundits.

By contrast, reading www.pollster.com is a real pleasure because of the intelligence, quality and integrity shown by Mark, the other Mark, the guest bloggers and the regular commenters.

____________________

pj:

ciccina, thank you for the post.
obama keeps saying he will get all
of hillary's voters in november,
but she won't get his.
he is so wrong.
i constantly feel bullied and
insulted by his supporters, and try
not to say anything. their
viciousness is so deep. i can't
imagine voting for him.
i am even considering, for the first
time in my life, voting for the
republican, mccain. (unless of
course he picks huckabee, in which
case i will write in a protest vote)

____________________

Will McLane:

I came to pollster.com looking for polling and statistical information but came away with much, much more. Mr. Blumenthal's article and the readers' posts are some the most thoughtful, original, and provocative ideas and reactions I have seen anywhere. And when I say anywhere, I mean, I scour the entire spectrum, running from Big Media, Big Web Media, Republican blogs, Democrat blogs, etc. And the polling.com conversation seems civil and courteous, to boot.

What have I learned today? Lots:

1. Mr. Blumenthal's compelling argument for full methodology disclosure by polling organizations.
2. Rick L's and Ciccina's enlightening comments about social pressure, social conformity, other socio-cultural dynamics, including the possibility of the "False Obama Effect", "Hillary Effect", and so forth.
3. Amy's and Tom's superb micro-essays/tutorials on the dynamics of caucuses vs. primaries, pointing out the possibilities of sub-groups (college students, elders/retirees, etc.) having disproportionate and misleading effects.
4. Rick L's stunning revelation that Fox---Fox!--- was, indeed, more objective about Obama-Clinton than MSNBC, CNN (and, I might add, ABC, CBS, etc.).
5. Finally, Rick L's genius revelation about "The Narrative". I noticed the use of that term by Chris Matthews and Keith Olbermann but it never clicked with me as being an admission of prior scripting at MSNBC. Thanks, Rick, for being the only writer I've seen to expose "The Narrative" for the danger that it is.

Thanks to Mr. Blumenthal and all of you good people for your original, rational, and unique commentary.


____________________

st paul sage:

Tom, I'm interested to know if you've practiced what you preached and went into a job interview and claimed your wife's work experience as your own?:)

Meanness and discounting of experience on either side is unconvincing and off-putting.

A few things:
1. The young are not only promising involvement. They are showing up in unprecedented numbers. I think we want them to show up in November.

2. I appreciate Ciccina's comments, but it seems to leave out a rather significant chunk of Obama's support: African Americans. If Obama was only supported by a white male elite, he would never have won 15 states. And African Americans are as "invisible" and "under represented" as latinas.

____________________

Check out my post on http://peoplepowergranny.blogspot.com. I write about how our voting machines are so screwed up and much more. Also folks can vote in my poll about voting machines. Stop by and add your comment.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR