Pollster.com

Articles and Analysis

 

A "Flaw" in the Iowa Poll?

Topics: 2008 , Barack Obama , John McCain , Pollsters , Sampling Error , SurveyUSA

Earlier today, commenter "axt113" noticed what he or she described as a "flaw" in the just released Iowa poll from SurveyUSA: "It has McCain winning the AA vote 55% to 45." I was distracted and nearly let it pass, but then our friend Ben Smith blogged a similar though subsequently hedged comment:

I'm sure it's a very small sample -- this is a poll of Iowa -- but it does raise the red flag when a survey shows Obama losing African-Americans to McCain.

UPDATE: I should have been clearer. It raises a red flag about the poll. Though really, it's mostly just another reason not to read the cross-tabs when they involve tiny, perhaps single-digit, samples.

The problem, which Ben alludes to, is that the weighted value of the African-American subgroup in the Iowa poll is just 2%. If we assume (for the moment) that the black respondents had a weight of 1.0, then those African-American results are based on a sample of 8-12 respondents. Pollsters typically have to weight up the African-American percentage in national surveys, since the black population is typically clustered in urban centers where response rates are lower (causing a non-response bias that needs to be corrected with weighting). I have no idea if such an adjustment would be necessary in Iowa, but if so, it would make that tiny subgroup even tinier.

I won't even bother to try to calculate the "margin of error" for 10 respondents. Some statisticians believe that the assumptions of the formulas break down at that level, rendering the calculations largely meaningless. For that reason, Many pollsters -- including every firm I've ever worked for -- have a policy of never releasing crosstabs to a client with a crosstab of less than 100 or less than 50 interviews (or whatever number they feel comfortable with).

Does the fact that 10 interviews produced a screwy result indicate a flawed poll? Not at all. That's the point of random sampling. The more interviews you do, the less error you get. Pull out any subgroup of 10 and you're bound to see very screwy and utterly "random" results. The larger the sample gets, the more those screwy (and offsetting) results cancel out.

If anything is flawed, it is arguably the practice of releasing cross-tab results based on such a small subgroup, though in fairness there are trade-offs here. SurveyUSA, has a mostly standard set of crosstabs that are sometimes very tiny. They do this (I assume) partly because it simplifies their programming tasks and partly because their format includes "row percents" that tell us about the weighted value of their standard demographics measures (race, age, gender, party). In other words, people like me badger pollsters to tell us the demographic composition of their samples. By including smaller subgroups in their standard table, SurveyUSA provides an answer as standard procedure.

Having said all that, Mike_in_CA raises a different but very good question regarding this Iowa poll:

[W]hy would SUSA poll Iowa right now, in the midst of catastrophic flooding? One has to wonder how many people have been forced out of their homes, away from their telephones. Probably not the best time to poll a state.

I do not know the answer, though I would be curious about the likely political skew that might result from those not available to be surveyed? Is it more urban, more rural? The floods appear to be affecting the eastern portions of the state. Are those typically more Democratic or Republican? Readers with knowledge of Iowa are encourage to comment.

 

Comments
Mike_in_CA:

thanks for the shoutout at the end Mark! :)

____________________

Scott M:

The flooding was worst in Cedar Rapids and Iowa City. Cedar Rapids is the second largest city and heavily Democratic, especially in the downtown areas that were under water. Iowa City is home of the University of Iowa and is extremely Democratic. The two metropolitan areas account for more than 10% of total population and probably 15-20% of Democrats in Iowa. This seems likely to have made a difference.

____________________

My mother's family is from northeastern IA--Decorah and Cedar Rapids and they have not been doing anything but sandbagging, worrying, and wondering about how to recover from the devastation. They are Democrats and I'm sure would have hung up on anyone trying to poll them now. The phone lines were for relatives through lsat week.

____________________

Nick Panagakis:


I checked percent African-American in Iowa and found that the Nov. 2000 exit poll showed 2 percent and 1 percent in 2004, the same as in our Iowa poll, one of four battleground state polls. So African-Americans appear properly represented by SurveyUSA. As for revealing small sub-sample data in standardized cross-tabs, most readers should know the caveats, however, some cautionary note might be appropriate given such a large audience for their polls.

The field dates were June 13-16. That�s when Black Hawk county (Waterloo and Cedar Falls, my hometown many, many years ago) was partly flooded and when devastating flooding in Linn County (Cedar Rapids) had begun. At the time, students in Johnson Country (Iowa City) were filling sandbags that were ultimately breached. These are all Democratic counties, blue collar or university.

SurveyUSA cross-tabs show that 40% of the sample came from the NE region, one of four state regions, that would include Black Hawk and Linn but probably not Johnson. So it seems that coverage of the flood area was good, if not too good, but I don�t think this affected the overall results.

Nick

____________________

Scott M:

Iowa becomes steadily less Democratic as one moves from east (think Illinois) to west (think Nebraska). The severe flooding was entirely in the central and eastern portion of the state. In addition to Cedar Rapids and Iowa City, there was major flooding in Cedar Falls (university town), Waterloo (union stronghold), and Des Moines (largest city). If SUSA got 40% of responses from the NE, it is a safe bet that democrats were under-represented.

____________________

FYI, there were 9 AA respondents in this poll. SUSA includes a "Show Counts (Frquencies)" option in the left sidebar. I find this useful.

That said, there is a pattern of seemingly anomalous results in SUSA polling re AA support for Obama, especially in the VP-testing editions (which tend ot be outliers on general Obama support as well).

Are mischievous teens or Republicans self-identifying as "Black" when SUSA calls? Unclear.

____________________

Jacob S:

I do not know a lot about Iowa, but I do know that eastern Iowa generally is more Democratic (the same is true in my home state of Minnesota). This fact may have biased the poll by a few percentage points. Obama also did best in eastern Iowa during the caucuses (Clinton did best in the west and I think that Edwards did best in south).

____________________

DCCyclone:

Eastern Iowa is heavily Democratic, and western Iowa is even more heavily Republican. So if the flooding affected eastern Iowa more, then it was Democratic areas that were most affected.

That said, does it really make that much difference? It's not like half the state was without phone service and unavailable to poll. How much does something like this really skew a statewide poll? And yes, I am an Iowa native and know the state well; I have close family there even though I personally haven't lived there in many years.

____________________

Tom:

For my analysis, I am looking at the hardest hit areas, Waterloo, Cedar Rapids, Iowa City, and Des Moines. In 2004, Bush won Iowa by 0.48%, but Kerry carried Black Hawk County (Waterloo) by 11.49%, Johnson County (Iowa City) by 29.26%, Linn County (Cedar Rapids) by 9.93%, and Polk County (Des Moines) by 4.64%. It should be worth noting that more votes were cast in Johnson County than in Black Hawk County, but the former is actually smaller. I suspect the Kerry campaign put a huge GOTV effort at the University of Iowa. These four counties total 28.41% of Iowa's population, and 29.37% of the total votes cast in 2004, with Kerry receiving 32.74% of his votes in Iowa from these counties while Bush received 26.07% of his total vote in Iowa from these counties.

In general, the further east you go in Iowa, the more Democratic you get. At the time of this survey, I believe the University of Iowa was not in session, and a portion of Des Moines was evacuated in anticipation of the approaching flood.

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR