Pollster.com

Articles and Analysis

 

The NYT Reader's Guide to Polls

Topics: Divergent Polls , Measurement

Jack Rosenthal, a former senior editor of the New York Times, filled in as the guest Public Editor" this past Sunday and devoted the column to a remarkable "Reader's Guide to Polls." The column (which also includes a kind reference to MP's coverage of the AMA's online Spring Break study) provides a helpful sampler of the various sources of imprecision in public opinion polls.  It is a worthy general primer, but as with any attempt to condense a complex subject into a very small space, a few items he covered would probably benefit from a more complete explanation. 

One example involves the reference by Andrew Kohut of the Pew Research Center to some findings from their polls on gay marriage:

The order of questions is another source of potential error. That's illustrated by questions asked by the Pew Research Center. Andrew Kohut, its president, says: "If you first ask people what they think about gay marriage, they are opposed. They vent. And if you then ask what they think about civil unions, a majority support that."

Those intrigued by that particular finding should definitely download the Pew report from November 2003 that documented the experimental findings.  Here's the key passage:

Granting some legal rights to gay couples is somewhat more acceptable than gay marriage, though most Americans (51%) oppose that idea. Public views on giving legal rights to gay and lesbian couples depend a good deal on the context in which the question is asked.  On the survey, half of respondents were asked their views on civil unions after being asked about gay marriage, and half were asked the questions in the reverse order.  When respondents have already had the opportunity to express their opposition to gay marriage on the survey, more feel comfortable with allowing some legal rights as an alternative.  But when respondents are asked about legal rights without this context, they draw a firmer line.

Pew_civil_union_19722


This context difference has little effect on core support and opposition to gay marriage itself, which is opposed by nearly two-to-one regardless of how the questions are sequenced.  But opponents of gay marriage are much more willing to accept the idea of some legal rights after they have had the opportunity to express their opposition to gay marriage. The percent favoring legal rights rises to 45% in this context, while just 37% favor the idea alone.  Put in other words, opponents of gay marriage are much more likely to accept allowing some legal rights when they have already had the opportunity to express their opposition to gay marriage itself.

Note also that the Pew surveys have shown a modest increase in support for both gay marriage and civil unions in surveys conducted since 2003.

Another topic, brought to my attention by a very alert reader, concerned this passage from the Rosenthal piece: 

Respondents also want to appear to be good citizens. When the Times/CBS News Poll asks voters if they voted in the 2004 presidential election, 73 percent say yes. Shortly after the election, however, the Census Bureau reported that only 64 percent of the eligible voters actually voted.

Ironically, as the reader points out, the Census Bureau's estimate of turnout is itself based on a survey, in this case the Current Population Study, which is prone to the same sort of measurement.  Professor Michael McDonald of George Mason University produces his own turnout statistics based on aggregate population and vote statistics.  His estimate of turnout among eligible voters in 2004 was closer to 61%, not 64%.  The Census Bureau explains the imprecision of this particular estimate in a footnote (#2):   

The estimates in this report (which may be shown in text, figures, and tables) are based on responses from a sample of the population and may differ from actual values because of sampling variability or other factors.

The "other factor" in this case is the same phenomenon that results in the over-reporting of voting in the NYT/CBS polling -- respondents wanting to be good citizens.  I suppose this example teaches that vetting survey results for publication is not as easy as we might imagine. 

Speaking of which, the Rosenthal piece also included this news:

The Times recently issued a seven-page paper on polling standards for editors and reporters. "Keeping poorly done survey research out of the paper is just as important as getting good survey research into the paper," the document said.

True enough.  But what standards will the Times now apply?  If that seven page document has been released into the public domain, I haven't seen it.  ABC News puts its Methodology and Standards guide online.  Why not America's "newspaper of record?"

 

Comments

"Respondents also want to appear to be good citizens. When the Times/CBS News Poll asks voters if they voted in the 2004 presidential election, 73 percent say yes. Shortly after the election, however, the Census Bureau reported that only 64 percent of the eligible voters actually voted."

While I have no problem believing that some claim to have voted when they didn't there is a possible additional explanation. In my case I did vote in 2004- but not for a presidential candidate. I didn't feel that either deserved my vote so I left it blank but I did vote on a number of state measures as well as for congresscritters, etc.

If asked the question above I would have answered that I did indeed vote. However when counting up how many people voted were they counting total ballots or total presidential votes? If the latter then there would be a discrepency between my statement and the "actual" measurement.

Of course that's probably a very minor case, most people are probably more likely to vote on presidential elections and leave other portions blank.

____________________

Mark Blumenthal:

Tlaloc,

I admittedly glossed over the issue you raise. The Census CPS turnout estimate is based on the reported vote cast for president.

If you look at Prof. McDonald's estimates, you'll see that he calculates turnout among the "voter eligible population" two ways: Turnout for the highest office in 2004 (in this case president)was 60.32%. Total turnout was 60.93%. So the difference is 0.61% -- obviously not big enough to explain the gap with the CPS estimate.

____________________

Aaron:

The difference between votes cast (tabulated) and the percentage of people who say they voted could also reflect that some percentage of people who thought they voted didn't have their votes counted. Whether that's due to election fraud, or voter error or equipment error is open to debate, but we know from Florida 2000 that not everyone who attempted to vote has their votes counted.

____________________


The census reported that 125.7 million voted, as compared to the 122.3 recorded count, a 3.4mm difference.

In EVERY presidential election, we know that about 3% of ballots are spoiled and NOT counted. The spoiled ballots occur mostly in minority precincts - and approximately 75% of them democratic votes.

The 60,000 sample Census margin of error (gender demographic) is 0.30%, according to the Census Bureau.

Now 3% (3.66mm) of the 122.3mm recorded 2004 votes were spoiled ballots, quite close to the above 3.4mm discrepancy.

The discrepancy between the census and (recorded vote +3%) is 0.20%, well within the 0.30% MoE.

0.20% = (125.96-125.7)/125.7 = .26/125.7

____________________

Mark Lindeman:

"In EVERY presidential election, we know that about 3% of ballots are spoiled and NOT counted."

We do?

Charles Stewart took a pretty close look, summarized at http://web.mit.edu/cstewart/www/election2004.html . Even allowing for missing data and adding over half a million uncounted provisional ballots, I don't think you will get very near 3% in 2004.

Of course, Stewart could be wrong. Some evidence to that effect would be useful.

____________________

I refer you to Florida 2000.
We know the spoilage rates from the recount.
Can we extrapolate the rates as a first approximation to nationwide spoilage?
Until I can find another link?

Undervotes: 70,000
Overvotes: 110,000
Butterfly: 5,000

Total: 185,000 minimum spoiled votes.
Total votes: 5,962,000
Spoilage rate: 3.10%

____________________

How They Stole Ohio and the GOP 4-step Recipe to 'Blackwell' the USA in 2008

Abracadabra: Three million votes vanish

http://www.gregpalast.com/detail.cfm?artid=510

Friday, June 2, 2006
By Greg Palast

This is a fact: On November 2, 2004, in the State of Ohio, 239,127 votes for President of the United States were dumped, rejected, blocked, lost and left to rot uncounted.

And not just anyone's vote. Dive into the electoral dumpster and these "spoiled" votes have a very dark color indeed.

In another life, I taught statistics. And these statistics stank: the raw data tells us that if you are a Black voter, the chance of you losing your vote to technical errors in voting machinery is 900% higher than if you were a white voter.

Any guesses as to whom those African-Americans chose for president on those junked ballots? Check Ohio's racial demographics, do the numbers, and there it is: Kerry won Ohio. And that, too, is a fact. A fact that could not get reported in the USA.

But the shoplifting of those votes in Ohio was just the tip of the theft-berg. November 2, 2004 was a national ballot-box bonfire. In total, over three million votes (3,600,380 to be exact) were cast -- marked, punched, pulled -- YET NEVER COUNTED. I'm not talking about the Ukraine or Uganda. I'm talking about the United States of America "with liberty and justice for all."

Well, not "all." The nine-to-one Black-to-White ballot spoilage rate is a national statistic -- not just an Ohio trick. Last year, I flew to New Mexico to investigate the 33,981 cast but not counted ballots of that state in the 2004 race. George Bush "won" New Mexico by 5,988 votes. Or did he? I calculated that, of the all the ballots rejected and "spoiled," 89% were cast by voters of color. Who won New Mexico? Kerry won --or he would have, if they had counted the ballots.

But they didn't count them. And that was deliberate. It's in the plan. It's the program. And the program for 2008 is simple. Two million ballots were cast but not counted in the 2000 race. (Over half, 54%, were cast by African-American.) In 2004, the GOP kicked it up to THREE million. Get ready, these guys aim high: "four in '06" and "five in '08" looks to be their game plan.

How will they pile up five million un-voters in 2008? Let's start with the three million "disappeared" of 2004...

____________________

Rosenthal is wrong when he claims that poll percentages should be rounded off to the nearest ONE percent..

He writes:
----------------------
False Precision

Beware of decimal places. When a polling story presents data down to tenths of a percentage point, what the pollster almost always demonstrates is not precision but pretension. A recent Zogby Interactive poll, for instance, showed that the candidates for the Senate in Missouri were separated by 3.8 percentage points. Yet the stated margin of sampling error meant the difference between the candidates could be seven points. The survey would have to interview unimaginably many thousands for that zero point eight to be useful.

Experienced researchers offer a rule of thumb: rather than trust improbably precise numbers, round them off. Even better, look for whole fractions.
------------------------------------------
Since when is Rosenthal a polling expert?

Zogby knows precisely what he is doing by rounding to 0.1%. He wants to provide the actual polling results, so that the numbers can be reconciled. There is nothing wrong with that.

On the other hand, Mitofsky knew exactly what he was doing when he rounded the National Exit Poll weights and vote shares to ONE percent, even though the National Vote share Margin of Error was ONE percent. He wanted to make it difficult for researchers who use weights and vote shares to compute probabilities.

There is a vast difference between a probability of 97.51% and 98.49%, although they both round to 98%.

There is a vast difference between a .51% MoE and a 1.49% MoE even though they both round to ONE percent.

Which is more meaningful: a 50.49%-49.51% vote split or a 50%-50% roundoff? A 0.98% difference in a 3.0% MoE is very significant.

But apparently it's OK when exit poll naysayers add a 50% increment to the MoE to account for a "design/cluster" effect. Such hypocrisy. All the more reason to avoid 1% roundoff.

Now, it is true that a 51.27% polling result dcan be rounded to 51.3% with little loss in accuracy. But to round to 51% is misleading.

Roundoff is a fudge. It obscures, rather than clarify. Research analysts need all the accuracy they can get. To camouflage survey results using 1% roundoff is counterproductive.

____________________

Mark Lindeman:

"I refer you to Florida 2000. We know the spoilage rates from the recount. Can we extrapolate the rates as a first approximation to nationwide spoilage? Until I can find another link?"

Well, no. You put "EVERY" in capital letters as if you had something to back it up. That will not fly here. And if it continues, I would not expect much response beyond ThereHeGoesAgain -- if you are lucky.

Research your claims before you make them, and things will probably go better. Believe it or not, a lot of us would like to know as much as we can about what actually happened. But unsupported claims waste everybody's time.

I have no clue where Palast came up with his figures for uncounted provisional and absentee ballots. Go do some work, please.

____________________

I gave you the Palast link. He did the research and provided the numbers. His reputation as an investigative reporter is beyond reproach. He was the first to uncover the disenfranchisement of 90,000 blacks in Florida 2000.

Yes, I am using his numbers. They are entirely consistent with what we know about the history of spoiled ballots. Do you object to that?

Can you refute HIS numbers? Or will you try to confirm them yourself?

I did the work on Florida 2000 in the prior post.
Three percent of the votes were spoiled. This is a documented FACT.

Now, you go do some work.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR