Pollster.com

Articles and Analysis

 

More on that USA Today/Gallup Poll

Topics: 2006 , George Bush , Likely Voters , Measurement , Slate Scorecard , The 2006 Race

Our update to the Slate Election Scorecard yesterday tries to put the results for the generic congressional vote from the USA Today/Gallup survey into some perspective. It also reintroduces the controversy over likely voter models in general with specfic attention to the Gallup likely voter model. More on that below. Given the obviously high interest in this particular survey, as reflected in the sometimes heated debate in the comments section yesterday, I want to first share some of my own reactions.

First, remember it's just one poll. One of the inherent weaknesses in political polls is that they come with a lot of built in variation. Some comes from interviewing a sample rather than the whole population. Some comes from other methodological differences across surveys. As such, it is always better to look at more surveys than few. We like to average results across polls, despite some theoretical shortcomings, for just that reason.

In hindsight, the single discordant poll my be just a random statistical outlier, but not always. Sometimes it can be the proverbial "canary in the coal mine" that warns us of some new and emerging trend. So we pay attention to polls like yesterday's Gallup Poll, even if we typically recommend caution in interpreting them. 

Second, let's put aside the likely voter conundrum and focus on the larger sample of adults and compare the Bush job approval rating among all adults to trends on other surveys. I have updated the table from Monday's post below, and I averaged the two Gallup polls conducted in September to try to make the data as comparable across pollsters as possible.

The pattern is now strong and obvious: While the precise level of approval shows the usual variation across pollsters, eight of the nine pollsters show some small increase in the Bush job rating between August and September. That is a highly improbable result by chance alone, analogous to flipping a coin and having it come up heads eight of nine times (roughly 2% according to my favorite binomial calculator).

Third, consider one issue that everyone overlooked except one very alert MP reader: On previous Gallup polls, the Bush job rating came first on the questionnaire, or at least before questions about congressional vote preference. This is the first pre-election poll in which Gallup switched the order, asking the congressional ballot question first and then the Bush job rating.

This practice is not unusual. Media pollsters frequently juggle the order of questions with events, especially those that conduct surveys on a wide variety of topics year-round. They will generally try to position the most important (or newsworthy) questions first to reduce the chance of bias. The problem is that in moving questions around, they sometimes introduce some unforeseen new bias that unintentionally skews a time series trend.

We have no way to know whether that happened on the latest Gallup poll (absent a controlled experiment**), but it is certainly possible the change in question increased the Bush approval rating by a few points.

Fourth, as many comments on yesterday's post have noted, the 48% to 48% tie in the generic Congressional ballot question was based on the sometimes controversial Gallup likely voter model. Our Slate update yesterday provided a quick and dirty summary:

Ideally, pollsters and pundits prefer to watch likely voters because, well, they're more likely to vote than those who are simply registered. But identifying the likely electorate is much more difficult when an election is still months away, because respondents are less able to honestly assess whether they're really going to vote. (Getting a large enough sample of likely voters also costs more money, so media pollsters usually wait until closer to the election.)

The problem is that once pollsters start screening for likely voters, their methodologies vary widely. This produces the scattershot results we've seen recently. An AP-IPSOS poll conducted last week showed likely voters preferring the Democrats by a 14-point margin (53 percent to 39 percent). Other surveys conducted over the last two weeks by Zogby, Harris, and Fox show results that were more encouraging for Republicans but not the even split that Gallup shows. Further, the Gallup poll's likely voter model has been criticized for producing volatile results, especially when used a month or more before the election.

I have written extensively about the way pollsters choose likely voters. For those without time to read it all, the key point is that while evidence shows the Gallup likely voter model typically provides a better estimate of the vote on the final poll than looking at all registed voters, it can produce a lot of volatility before October.   As Mickey Kaus put it yesterday, the model may tell us more about:

'who would vote if the election were held today' as opposed to what we really want to know, which is 'whom would the people who are going to vote on November 7 vote for if the election were held today.'

Finally, consider that we are obsessing over measures with very limited utility to predict election outcomes -- the generic congressional ballot and presidential approval rating. In the races for Senate, on the other hand, we have direct measures of the actual contests and far more surveys to compare.  Our Slate Senate Scoreboard has logged 36 new state-level Senate polls released in September alone. And despite the modest increase in the Bush job rating, as of today, these continue to indicate momentum toward the Democrats.  Of course, the Scorecard also shows 49 seats currently held or at least leaning Republican, with 46 held or leaning Democrat.

With seven weeks remaining until the election, the data above should leave no one feeling too confident about the outcome. All of these trends can and probably will change. We'll be watching, so stay tuned...

 

Comments
DemFromCT:

Bravo, Mystery Pollster!

Looking at the presidential year of 2004, for those that want some detail, check here and here (follow links to specific polls).

A 2-3 point change is exactly that.

____________________

DemFromCT:

Another data point:

The Times/CBS News poll also found that President Bush did not improve his own or his party�s standing through the intense campaign of speeches he made and events he attended surrounding the fifth anniversary of the Sept. 11, 2001, terrorist attacks. The speeches were at the heart of a Republican strategy to thrust national security to the forefront in the fall elections.

Mr. Bush's job approval rating was 37 percent, virtually unchanged from the last Times/CBS News poll, which was conducted in August. On the issue that has been a bulwark for Mr. Bush, 54 percent said they approve of the way he is managing the effort to combat terrorists, again unchanged from last month, though up from earlier this spring.

But voters are not very happy with congress, going along with MP's analysis in this post.

____________________

DemFromCT:

sorry... link to poll

____________________

Ike:

Great Post Mark,

Your Aug-Sept comparison really makes me want to ask Charles the date of the upward inflection point in his blue line of approval.

DemFromCT is on to something with that first link to the CNN/Gallup/USAT approval polls from 2004. Note the way that the Republican terrorism (and anti-Kerry) blitz worked in Sept 04. It drifted back somewhat in October (during the debates when Kerry fought back hard), but the effect of the September blitz kept Bush at a higher approval level than where he started in mid-summer. Without that Blitz (and a bad month from Kerry)...maybe he loses?

____________________

Gary Kilbride:

Well, the Los Angeles Times released a new poll tonight (http://www.latimes.com/news/nationworld/nation/la-na-poll21sep21,0,2149217.story?coll=la-home-headlines), mirroring the Gallup finding of a 44% approval rating. I suppose I could highlight that in bold, or would that be considered an agenda?

Regardless, the more I sample these polls it's obvious the unique dynamic this year features these factors:

* the Republican-controlled congress is considered so ineffective it's assumed a rare separate entity status, one that may not be correspondingly impacted by Bush's rising approval ratings. A full 68% in the NYT poll listed "Nothing" when asked, "Is there any one thing that Congress has done in the past year that stands out in your mind?" The relevant congressional numbers generally aren't as bad as this point in '94, but they resemble '94 much more than any other cycle. I'm not sure we shouldn't be scrutinizing the congressional questions as much as the presidential approval figures, or in conjunction with them. When one party has control of the presidency and both chambers, anti-incumbency aims at that party much more than a literal definition.

* a signfiicant percentage, mid-30s, have apparently determined their midterm congressional vote will be anti-Bush. I doubt that will change markedly even if his approval squirms upward. If you look at the PEW poll from last week, only Bush in this cycle has a midterm vote-against number in the 30s. The question was asked beginning with Reagan in '82. Clinton peaked at 23% in 1994 and that was the high number until Bush '06. In 2002, Bush had a positive 29-16 in that category, "will your congressional vote be for the president, against the president, or the president not a factor?" Bush is now at 20-36, according to PEW. CBS/NYT, which asked the same question, had Bush at 31-19 in 2002 and 16-35 now.

With that in mind, I'm wondering if it's a high percentage of women who have shifted away from Bush and are set to vote against him this time. I'd like to see the approval ratings and trends broken down by gender. I know some of the polling companies provide that. IMO it may not benefit Bush and the GOP a great deal if his approval rises overall, but still lags among women, specifically white women. That block put Bush and the Republicans over the top in 2002 and 2004 due to 9/11 and national security concerns. Current polling generally reveals a re-established gender gap.

____________________

DemFromCT:

Gary Kilbride, good points about gender.

Also, rising but still low approval (and most of stories gloss over how low 44 is historically) in predominantly GOP areas doesn't change the congrssional dynamics. If Alabama becomes a tad more red, that won't help Shays-R in CT-5, e.g. SUSA's 50 state poll is interesting for that data, and does have state-by-state gender (CT has a huge gender gap).

With ARG at 36, it seems clear that the small bounce Bush has gotten is either fading or isn't an onward and upward trend. Right now, the media narrative is more positive for Bush than the Bush numbers. Stu Rothgenberg agrees it's an anti-Bush election more than anti-incumbent.

Rasmussen doesn't have gender, but the "strongly disapprove" is still twice the "strongly approve". And today's number will not show an increase.

____________________

Brent:

The one thing you have to remember about polls is the nature of the questions being asked and the agenda of the person asking the question. I could ask a hundred questions all slanted to give me the greatest possible chance of getting the results I want. Polls should always be taken with a grain of salt. I am a conservative and a Bush supporter and am glad that his numbers are improving but at the same time I know that truthfully they could be lower or could be a lot higher. I don't get too excited about poll numbers because they are almost always slanted....take exit polling, that really works!

____________________

Additional analysis of the average monthly Bush rating deviations between these pollster groups: 7 corporates vs. 5 independents.

It is extremely powerful evidence that relative to independents, corporate pollsters have consistently been biased in favor of Bush.

http://www.geocities.com/electionmodel/BushRating_17902_image001.png

Bush Rating Deviations
1) Current month: 3.05%
2) Feb. 2001 to current: 3.76%
3) Latest 12 months: 1.63%

[code]
Sept. 2006
CORP 40.80;INDEP 37.75

Nwk na; Pew 37
Fox 40; Harris 38
CNN 41; AP 39
CBS 39; Zogby 37
ABC 42; ARG na
Time na
NBC 42

Feb 2001-present
CORP 55.98;INDEP 52.24

Nwk 55.46; Pew 51.83
Fox 56.01; Harris 54.12
CNN 56.58; AP 51.00
CBS 54.97; Zogby 52.00
ABC 57.95; ARG na
Time 56.58
NBC 54.29

Latest 12 months
CORP 38.50; INDEP 36.87

Nwk 36.88; Pew 36.85
Fox 38.69; Harris 36.15
CNN 39.17; AP 37.50
CBS 36.31; Zogby 37.58
ABC 40.09; ARG 36.25
Time 39.78
NBC 38.60
[/code]

____________________

Here is barchart of the monthly deviations as
per the prior post.

http://www.geocities.com/electionmodel/BushRating_24813_image001.png

____________________

Richard Hooker:

C'mon Mark,
You and all the other commenters here know that all of those upticks except perhaps one or two are within the confidence intervals of the first proportion. In other words, for those of you reading this comment who don't know statistics, for all by the AP-Ipsos poll, the numbers between August and September are statistically identical, which has a monumentally greater probability (not odds) than that of a coin landing heads eight times out of nine. When you factor in all other forms of error besides sampling error, such as historical error (it matters WHAT is happening when the polls are taken), there is no significance in these numbers.
For those of you who don't know statistics, the author is claiming that the uptick in numbers is statistically significant because he calculates an alpha of .02 for the event. But that is meaningless, since the difference in the numbers in individual polls (with the exception of AP-Ipsos) ARE not statistically significant (with an alpha of .05 or even higher). If you looked at all the differences as a whole, they would also not be significant. In other words, the author is taking one isolated fact (all the polls but one or two show an increase within their confidence interval) and assigning that one fact statistical significance (like tossing a coin nine times and getting heads) in order to argue that a proportion series that is not statistically significant using real tools of statistical proportion analysis is actually now statistically significant. And the author knows better.
Again, for those who don't know statistics, these differences are statistically insignificant given normal sampling error. When you factor in all kinds of other errors, such as historical error, then these are totally meaningless. When the difference in poll numbers goes up in most or all of these polls above and beyond the confidence intervals calculated for each proportion, then you can be flippant about flipping a coin nine times, though it still has no meaning relative to normal tests of statistical significance.

____________________

Richard Hooker:

Upon reading that last post, it sounds like I'm insulting the people on this board and I'm a little bit abashed. I'm not really addressing people on this board so much as people who have been blogging this post around the Internet. The people on this board are good statisticians all. I use the phrase, "for all you who don't know statistics" because this site is being quoted across several blogs by people who don't understand stats. The phrase is not meant to insult anyone here on this board. I am addressing my comments almost entirely to outside bloggers who are using this information incorrectly. If you don't understand statistics, then you should read my post above.

____________________

Richard,

Let me comment. First, thanks for the second comment. As we've all found from time to time, it is all too easy to sound harsher than intended in online exchanges. I know I've been guilty of that before. Hopefully not this time.

Richard raises a worthwhile point about how to assess change in polls of presidential approval. I don't think we reach the same conclusions, but it is a worthwhile topic to discuss.

I've discussed this issue at length at Political Arithmetik. The most extensive disussion is at http://politicalarithmetik.blogspot.com/2006/03/unchanged-since-yesterday-detecting.html

The problem, as I see it, is that "real change" in approval is small from week to week. My estimates of approval trend rarely find approval changing at a rate greater than 2 points per month, and sometimes as little as 1 point per month. If the typical poll has a 3 point margin of error then this rate of "real" change, will rarely be statistically significant between any pair of polls taken within a two or three week period.

In the linked post above I follow approval from "1/1/2002 through 3/19/2003. During this period approval declined from 84% to 58%, a rate of 0.061% per day, or one percent every 16.3 days. The downward trend is both statistically significant and undeniable based on visual inspection. Yet of 56 Gallup polls during these 15 months, only 5 (8.9%) found a significant change in opinion from the previous poll."

The rate of change is small compared to the precision of the survey and the interval between polls. Thus the vast majority of polls show no "significant change" since last time.

But check out the graph. There is overwhelming evidence that aggregate support for President Bush has varied. So the problem is how to detect this change?

If individual polls are not precise enough, then some strategy of aggregation across polls is required. But how to do this?

At Political Arithmetik, I'd adopted an approach that (oddly for me!) ignores statistical significance, and just fits a non-parametric local regression trend. I'm cautious about interpreting trends when the estimate is "too dependent" on new polls, but once a dozen or so polls are used in the fit, I've found the local trend to be quite stable. For comparison, I sometimes fit a segmented linear regression, which gives a test of the significance of the trend parameter during a segment (as well as an instantaneous shock effect.) These have agreed well enough that I'm not too concerned that the local fit is misleading, once a decent number of polls is in hand. And so on this basis, I'm very comfortable claiming that there is strong evidence for a sharp increase in Bush approval since August 15.

A different approach to pooling is the one that Mark adopts in this post. Each survey is an independent sample of the population. The change from poll to poll under the null hypothesis should be positive with probability .5. That change does NOT have to be statistically significant (as you seem to argue above) because we care only about the .5 probability of an "up" move, not how large it is. A binomial model of the number of "up" changes out of N polls, which is what Mark computes above, seems to me to be a pretty reasonable approach to pooling across polls. By focusing on changes WITHIN polling organization, he avoids contamination from house effects which could otherwise confound the analysis.

One might argue that other factors might induce some positive correlation between the polls, despite the independent samples, in which case a negative binomial distribution might be a more conservative test, but I doubt the evidence would be strong.

Another approach would be to pool the polls, get a really large "N", and conduct a difference of means test between the "early" and "late" pools. That would be complicated because of house effects and the unbalanced mix of polls. The uneven spacing between polls complicates this (and Mark's) test if we wanted to be picky.

How to best pool over these studies has a variety of possible answers. We could make this a good thesis for someone. We could also adopt a simple but not overly simple approach such as Mark's that accounts for House effects but with an independence assumption that may not be wholly warranted. Or we can look at my trend line and not worry about significance tests. Each has virtues and vices.

The basic point I want to make here (with the evidence from the link above) is that it is easy to deny a trend when using pairs of polls based on lack of statistically significant change. But in retrospect, such an approach has been clearly "wrong" over much of the Bush presidency (both when moving up and when moving down.) Some form of pooling gives answers that are much more reasonable. The technical details of those tests may be debatable, but the broad conclusions are, I think, pretty well supported.

To quote George Box: "All models are wrong. Some are useful." I've chosen a useful model that ignores some technical issues. So has Mark. I think both have proven pretty clearly useful over the last several years of data. I'd welcome new work by others that offers improved methods of pooling these data. We are working on some of our own. But in the meantime, I don't think we are far off in our conclusions.

But I will conclude by agreeing with one of your final points: There is a lot of interpretation in the blogoshere and in the news media that reaches unsupportable conclusions based on single pairs of polls and/or trivial changes in results. Some analysis that brings a bit of discipline to the task is, I think, quite useful.

Charles

____________________



Post a comment




Please be patient while your comment posts - sometimes it takes a minute or two. To check your comment, please wait 60 seconds and click your browser's refresh button. Note that comments with three or more hyperlinks will be held for approval.

MAP - US, AL, AK, AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OK, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY, PR