December 23, 2007 - December 29, 2007
The main impetus for our "Poll of Pollsters" (from which I posted the first results on Thursday) is the desire to find better objective criteria to help sort out pollsters. As both Charles and I review today in complementary posts (here and here), the handful of past Iowa polls do not lend themselves to easy assessments of accuracy. The lack of transparency by many public pollsters makes it difficult to fairly assess differences in their methods, although I would propose the degree of their disclosure as a surrogate measure of quality. A third possibility is to measure pollster reputation, especially among their peers. We decided to start with a survey of pollsters about the public polls now in the spotlight in Iowa and New Hampshire.
I included all the details in the previous post, but here are the highlights: We sent out invitations to just over a hundred pollsters and had 46 complete the entire survey, although a few more (49) completed the questions about the reliability of the polls in Iowa. Of those, 22 are media pollsters and and 27 campaign pollsters (16 Democrats and 11 Republicans). There is no margin of error because the results represent nothing more or less than the views of the pollsters that participated. Like any survey respondents, we promised to keep their identities confidential.
We started with a simple question asked about each of the 16 pollsters that have released public polls in Iowa: "How reliable do you consider surveys of IOWA CAUCUS goers done by each of the following organizations, very reliable, somewhat reliable, not very reliable or not reliable at all?" We also provided an option to say they "do not know enough to rate" each organization.
We left "reliability" in the eye of the beholder, but it is fair to assume that few are in a position to evaluate the performance of each organization in past Iowa caucuses. As one pollster put it (in a space provided for comments the end of the survey) that there is "no way we can know who's most reliable until we can compare their final estimates with actual vote." Instead, it is safe to assume that most based their judgements on the reputation of each organization and its methods. As you will see, the pollsters had little trouble making such judgements.
As the following table shows, the Des Moines Register "Iowa Poll" conducted by Selzer and Company easily earns the highest marks, with virtually all rating it either very (36%) or somewhat (50%) reliable. The other pollsters with the highest scores are nationally known media surveys: ABC/Washington Post, the Pew Research Center and CBS/New York Times.
The pollsters receiving the lowest scores are Zogby International, the American Research Group and Rasmussen Reports. In the case of Zogby, four out of five pollsters rated their surveys as not very (28%) or not at all reliable (52%).
Not surprisingly, the media pollsters are generally more positive than their campaign consultant colleagues (keep in mind that we invited principals at all 13 organizations that are polling in Iowa to participate). Their rankings are generally similar, although the University of Iowa and Strategic Vision rank higher among the campaign than the media pollsters.
We also asked our pollster-respondents to select from the same list the "pollsters you consider MOST and LEAST reliable in Iowa." As the table below shows, the Des Moines Register/Selzer survey easily stands out as the favorite, especially among the campaign pollsters:
As for the least reliable pollster in Iowa, once choice easily led the pack. One third (33%) of the campaign pollsters and just less than half (45%) of the media pollsters picked Zogby International:
Again, reputation cannot tell us everything we need to know about the quality of the numbers a pollster produces. Pollsters with poor reputations may conduct quality polls and even the best pollsters are fallible. However, when a pollster earns the respect of their colleagues, it should tell us something.
Up next, similar ratings of the pollsters in New Hampshire. Meanwhile, Pollster.com readers, what do these ratings tell you?
Note: I accidentally omitted InsiderAdvantage which released their Iowa poll as we were in the process of drafting this survey. I apologize for the oversight.
Here is an update comparing the standard "blue line" estimate of poll trends with the more sensitive "red line", and a simple 5 poll moving average for comparison.
I talked at length about these estimators and issues related to them in an earlier post here, so won't comment further. See the earlier post for details.
These data include polls released through December 29, with ARG's 12/26-28, Research2000's 12/26-27, Strategic Visions 12/26-27 and LATimes/Bloomberg's 12/20-23,26 polling as the latest from Iowa.
Cross-posted at Political Arithmetik.
Update: We have added regularly updating versions of these charts on our Iowa, New Hampshire, South Carolina, and National primary pages.
As we pour over the latest data, it is worth taking a look back at a previous year in Iowa.
The Democratic contest in 2004 was strikingly dynamic, with a sudden surge for Kerry and Edwards as Gephardt and Dean slumped. (And note the gap in polling. Those are the holidays we are in right now when polling wasn't done.) By the last polls, Kerry had established a clear lead, but Dean, Edwards and Gephardt were within four points of each other. The polling got the trends pretty close to right. Edwards was clearly on his way up through the last 12 days of the race, as was Kerry. Dean may or may not have reversed his fall in the last week, and the polls said Gephardt was coming down. So far, so good.
But the Democratic caucus process does a lot to change the outcomes, with supporters of non-viable candidates joining forces with their second choices. That process is likely to boost the well off candidate, while robbing the struggling campaigns. And of course there is always the issue of which candidate's supporters actually turn out on caucus night.
The entrance poll is the best measure we have of whose supporters actually show up on caucus night, even with all the appropriate cautions about the entrance poll itself. Comparing the entrance poll to that last estimated trend value for candidate support from pre-caucus polling, we can see how the two rising campaigns did even better than the polling predicted. Kerry's last poll trend estimate was 25.9%, good for first place. But his entrance poll support was 34.8%. For Edwards, his final trend estimate was 21.4%, but the entrance poll found 26.2% supporting him.
For Gephardt the story was the opposite. His supporters stayed home on caucus night. The pre-caucus trend had Gephardt falling but at 17.8%. But the entrance poll found only 10.3% to be Gephardt supporters.
Dean was the only top candidate whose preelection and entrance poll numbers match closely. His trend estimate was 20.3% and the entrance poll put his support at 20.5%.
Caucus night had the effect of stretching out the differences between the candidates, advantaging the top two while damaging the fourth place candidate. (Dennis Kucinich was the exception in the back of the pack, with a trend estimate of 1.7 but an entrance support of 4%.)
Then they vote. And form coalitions with non-viable supporters. And weight the delegates in a complex formula and finally there is an allocation of delegates to the state convention. That allocation is the best we can do to consider a "final" outcome of this process.
The results there further favored Kerry and Edwards. Kerry moved up to 38% of delegates, from 34.8% in the entrance poll and 25.9% in the pre-caucus poll trend. Edwards got 32% of delegates, up from his 26.2% in the entrance poll and 21.4% in the poll trend.
Dean ended up with 18% of delegates, down a bit from the 20.5% in the entrance and 20.3% in the poll trend. And Gephardt's delegates just about matched his entrance poll, 11% of delegates and 10.3% in the entrance poll. A disappointment from his 17.8% final trend estimate.
So let's take one important lesson away from this Caucus Past. The pre-election poll trend got the order of finish right, if only by a point separating Edwards and Dean. But the process of caucus night that makes it tough to come out means that enthusiastic supporters are more likely to turn out than those who are discouraged by recent slippage. That is probably true for both parties. On the Democratic side, the complex voting and coalition formation further exaggerates the lead of the top candidates and diminishes the showing of marginal ones. And that process is seen even in comparison with the entrance polls, let alone the pre-election polling.
In Iowa the "outcome" is quite a few steps removed from the simple balance of preferences among the population. The mechanisms themselves intervene to affect the final delegate counts. So don't expect to see the pre-election polls hit the final delegate percentages very closely. If the polls get the order right, that will be good enough.
For a fair test of how good the polling is this year, wait for New Hampshire where pollsters poll, voters vote, and they just count up the results.
Cross-posted at Political Arithmetik.
Voters in the early primary states are making up their minds, or at least picking a candidate when pollsters call. For all that has been written about how unsettled the races in both parties have been, we are now seeing a clear decline in the rate of "don't know" answers to the vote question. Interestingly, that trend is clear in the earliest primary/caucus states, but only faintly visible if at all for the nation as a whole. This is further evidence that early state voters really do pay more attention and move to a choice sooner than the nation as a whole. Looming election days concentrate the mind apparently.
In both parties, the undecided rate has turned down the most in Iowa, to about 5% or a shade less. This is down from about 14% early in the year for Democrats and over 15% for Republicans. Even as recently as November 1, over 14% of Republicans were unable to give a candidate choice. Democrats were at about 11% undecided at that time. In both parties the undecided rate has fallen rapidly in November and December.
New Hampshire also shows some fast movement to a decision. NH Dems have been steadily decreasing their don't know rate all year from 17% in January, but it still stands around 10% across the most recent polls. NH Republicans were a bit slower to choose until the first of December. Since then we've seen a rapid decline to about 8% now.
And in South Carolina where there are fewer polls and the undecided rate has been all over the place, we are seeing evidence of more decision making since early November. That pattern has held for both parties, with Dems now at about 9% and Reps at around 10%.
Nationally there is no trend at all in Democratic undecided rates, which have held at 10% all year. For Republicans there has been a little movement nationally, down from about 15% to about 11% since late October.
Pollsters allow voters a variety of ways to say they haven't decided, so there are various ways we can measure the crystallization of preferences. One is to just use the percent who say they are "undecided", which is simple enough. But some voters, especially early on, pick options like "someone else" or "won't vote" and pollsters vary in how the report the no-preference alternatives. So I've calculated the percent who fail to choose any of the candidates the poll asks about. This is the blue "no preference" line. As it happens, these alternative measures track together pretty well, and recently any gap between them has largely vanished.
You can also see large differences across pollsters in how large an undecided rate they produce. A few national polls have zero percent undecided, while the highs at the same time are over 25% for Republicans and over 20% for Democrats. This is one of the sources of house effects in surveys.
There is one methodological issue that we can't address with these data. Near the end of the race some pollsters push voters harder to get a response to the vote question. That would, of course, artificially lower the "don't know" rate, and to some extent may be what we are seeing in these data. Such practices are not normally disclosed so there is no way to statistically adjust for them here.
Cross-posted at Political Arithmetik.
Given the way we are scrutinizing the final Iowa Caucus polls, this seems like a good time to take a look a the final pre-caucus polls from 2004 and 2000. One of the questions I get most frequently is which pollsters were "most accurate" in previous years, and as the old data will show, that is a far more difficult question to answer than most people assume.
Consider the final polls for the Democratic Caucuses in 2004. Only five organizations released public polls conducted in the final week before the Caucuses, which were held on Monday, January 19 that year. Since both John Kerry and John Edwards experienced late surges in support, polls conducted before that would show considerably more "error," since they obviously missed the late surge. Also, those who continued to call through Sunday night might have some advantage in catching the late breaking trend (or, as more cynical pollsters will point out, those releasing late polls also had the benefit of seeing the results of the other earlier surveys).
The table below shows the results of the final week's polls, plus the results of the network "entrance poll," which asked participants their first preference as they entered their caucus location.
[Click on table to pop-up full size version]
The first issue, unique to the Democratic contest, is that even a perfectly accurate survey can tell you only about the initial preference of the caucus-goers. The Iowa Democratic Party does not report the head-count of initial preferences. Instead, the official "results" they report on caucus night will be based on the state delegates for each candidate chosen to the state convention. The delegate selections are based on a second round of voting after supporters of "non-viable" candidates (those who receive less than 15% on the first round) realign to their second choice.
So as should be obvious, the second round voting means that initial vote preference -- even as measured by an entrance poll -- does not directly measure the final results. So there is an important element of inaccuracy built into any Democratic preference poll. In 2004, both Kerry and Edwards did better in the reported results than the entrance poll. Most observers attribute much of the six-point gain for Edwards to a deal struck on caucus morning between the Kucinich and Edwards campaigns that sent most Kucinich supporters into the Edwards camp on the second round. Exit pollster Joe Lenski reports that most Kucinich supporters chose Edwards as their second choice in the entrance poll.
Putting aside the viability issue, which poll was "most accurate?" The answer depends on the yardstick applied, which is a tough call in a multiple candidate primary or caucus. The Des Moines Register poll has received much credit for being the only one to correctly "predict" the order of the top candidates, but notice that Edwards "led" Dean on their larger "likely caucus goer" sample by a not statistically significant three percentage points (23% to 20%). Among their narrower "definite voter" subgroup, the Edwards-Dean order was reversed (Dean had 21%, Edwards 19%). So the getting the order right may have been partly a matter of good fortune.
I will spare readers the minutiae of the various error scores, but if we measure accuracy in terms of how well the polls predicted Kerry's percentage the Des Moines Register's narrower "definite voter" subgroup does slightly better. The same Register sample also does best in terms of the average of the errors for all the candidates. Ironically, the smaller Register "definite voter" sample, the one that had Dean nominally (though not significantly) "ahead" of Edwards, was the most accurate on these criteria even though the larger Register sample has been credited with "predicting" the order of finish.
If, on the other hand, we focus on the Kerry-Edwards margin, the final Zogby poll comes slightly closer to the actual result. In any case, the differences between the pollsters are small enough on all of these criteria that random chance was certainly a factor in determining which did best. And notice that everyone was way off on the final margin between Edwards and Dean, whether we compare to the entrance poll head count (Edwards +6), or the post-realignment actual results (+14).
What about the 2000 Caucuses? The number of final week polls was again fairly limited. On the Republican side -- where the actual results are a simple head-count -- the LA Times and the Des Moines Register came closest to George Bush's ultimate share of the vote, and the Times had the narrowest (and thus most accurate) Bush-Forbes margin. But all of the polls underestimated the support received by Forbes and Keyes.
On the Democratic side, the Des Moines Register had the Gore-Bradley margin exactly right, but a University of Iowa poll, which overstated Gore's margin, had Gore's percentage of the vote exactly right.
So what's the point of these comparisons? Trying to score such a small number of polls solely on the basis of accuracy is a confusing, contentious and ultimately futile exercise. The 15% viability threshold on the Democratic side undermines the accuracy of all polls. The timing of the final poll is critical and the differences among the various pollsters in the final week have been relatively small. Moreover, different measures of accuracy can lead to different conclusions. So if you take away nothing else, remember that none of the Iowa polls have been a perfect crystal ball. All have missed significant aspects of the final results in 2004 and 2000.
Sources: I obtained the results above from the subscriber only archives of The Hotline and the Polling Report. The SurveyUSA results are still available online and the InsiderAdvantage survey from 2004 was released at the time via PRNewswires.
A new American Research Group statewide survey of likely caucus goers in Iowa (conducted 12/26 through 12/28) finds:
- Among 600 likely Democratic caucus goers, Sen. Hillary Clinton (at 31%) leads former Sen. John Edwards and Sen. Barack Obama (both at 24%) in a statewide caucus; Sen. Joe Biden and Gov. Bill Richardson both trail at 5%.
- Among 600 likely Republican caucus goers, former Gov. Mitt Romney leads former Gov. Mike Huckabee (32% to 23%) in a statewide caucus; Sen. John McCain trails at 11%, former Sen. Fred Thompson at 7%, former Mayor Rudy Giuliani and Rep. Ron Paul both at 6%.
- All other candidates receive less than five percent each. The margin of sampling error is 4% for both subgroups.
A brief break from polls to comment on graphics and politics. Today's New York Times has an op-ed by NBC's political director Chuck Todd and a graphic designed by Nicholas Felton. The text and graphic are here. The text describes the data (quite completely-- an unusual but welcome touch!) noting that candidates are stratified by time in rough line with their poll standing and that debates played a part in both the rise of Mike Huckabee and the slippage of Hillary Clinton.
The graphic is a variation on a pie chart, showing the total number of minutes each candidate spoke in the 21 debates held in 2007. It is an odd fact that statisticians and analysts of statistical graphics universally hate pie charts, while graphic designers for mass media love them. The latter seem to equate statistical graphics with pie charts, while the former have ranted for years about their defects.
The appealing metaphor of a pie chart is its division of a whole into parts. Whatever the slices represent, they have to add up to a "whole" pie. The trouble here is that the text and data are about amounts of time, and only implicitly about the share of a total that each candidate receives. Further, the pie mixes the two parties into one whole pie, but there really should be two pies, each divided within party and assuming we care about shares of the pies, since Republicans can't eat any of the Democratic debate pie, nor vice versa.
When what we want to compare are magnitudes, rather than shares of a whole, the data are more clearly presented as distances rather than areas. It is easy to compare which distances are longer than others, and relatively difficult to see differences between the areas of pie slices, especially when the slices are not adjacent to each other.
So let's look at the same data in a different format and see what we can see.
The chart above reproduces the information given in the original pie chart. It plots the number of minutes each candidate spoke during all debates, and distinguishes party by the use of color (here red and blue, in the pie chart by shading.)
The main points made in the op-ed are also evident here. There is clearly a great deal of stratification across the candidates, with front runners getting more time than the "back of the pack" candidates. But there are a few comparisons I think stand out more in this chart. The advantage of Obama and Clinton over Edwards and Richardson is clear here. Richardson got only about 2/3 of the time of Obama, with Edwards a bit better but still well short of Clinton. The next cluster of Democrats-- Dodd, Biden and Kucinich-- got only about half the time of Obama, with Gravel far behind even that.
On the Republican side, Giuliani and Romney were closely matched with McCain a little bit behind. As with the Democrats, there is then a large gap until we reach Huckabee at about 3/4 of Giuliani's time. A smaller but still clear gap separates Huckabee from the cluster of Paul, Hunter, Fred Thompson and Tancredo. Another gap separates Brownback and then one more puts the short-lived candidacies of Tommy Thompson and Gilmore and the single appearance by Keyes together.
To my eye, these differences are easier to perceive and compare when the number of minutes is simply the location of the dot in the chart above than when it is the area of a pie slice.
There is actually more data presented in the text of the op-ed than is present in the graphic. Todd's text notes differences in number of debates by party (11 for Dems, 10 for Reps), which means Dems should have about 10% more total time available than Reps (if, that is, the debates were equal length, something we don't know from the text but see below). He also gives the data on how many debates each candidate participated in, an obviously important point since we are comparing Keyes' minutes in a single debate with Obama's time in 11 debates. For some purposes we might care only for total time, but for others we might want to adjust for number of debates. I do that in the chart below, which shows the average number of minutes per debate for debates in which the candidate participated.
One immediately clear point in this chart is that Obama and Clinton retain their advantage over Giuliani and Romney even when we adjust for the extra Dem debate. Giuliani and Romney got the same time per debate that Edwards and Richardson received, but that leaves them well back of Obama and Clinton.
The most important shift in the chart is the movement of Fred Thompson to the midst of the top 4 Republicans. Thompson only participated in 5 of the 10 debates, so his total time in the first chart (and the original pie chart) dramatically misrepresents the attention he received after entering the race. Thompson received substantially more time per debate than did Huckabee, though in 10 debates Huckabee had 73 total minutes to the 49 Thompson got in 5 debates, as shown in the first chart.
In his one debate, Alan Keyes got more time than any other second tier Republican averaged over more debates.
But why do Giuliani and Romney continue to get less time per debate than do Obama and Clinton? Todd's text points out that there are 12 Republican candidates but only 8 Democrats who have to divide the time. That seems reasonable, but the data are a bit more complicated. While there are 12 Republicans in the charts, three of them participated in four or fewer debates while all but 1 of the 8 Democrats participated in at least 10 debates. When we count total candidate debate appearances, the Republicans had 88 and the Democrats 81, less than a 10% difference due to number of candidates and appearances, not the 12 to 8 ratio of candidates.
What is quite different is the total number of minutes the candidates of each party spoke. Democrats totaled 794 minutes over 11 debates, a total time per debate of 72.2 minutes of candidates actually speaking. For Republicans, the total in 10 debates was 666 minutes, or 66.6 minutes per debate. So the Democratic advantage in the original pie chart and my first chart above has built into it a longer total for Democrats regardless of the number of debates.
Further, if we divide by number of candidate appearances, we get how many minutes each candidate would have gotten if the total speaking time had been divided exactly equally for each debate appearance. The result for Democrats is 794/81=9.8 minutes per candidate per debate, while Republicans had 666/88=7.6 minutes per candidate per debate. For whatever reasons of debate format and schedule, Democrats enjoyed more time to talk even adjusting for number of candidates and number of debates. (Had Reps had the same 81 appearances as Dems, they would still have only had 8.2 minutes each.)
So the disadvantage in minutes per debate even for Republican frontrunners compared to Democratic leaders is not just an artifact of number of debates or of candidates. It is a real difference and it might be of political interest to know what accounts for it. Did Republican debates run shorter on average? Did questions to Republicans run longer on average, leaving less time for answers? The data don't answer these questions. But over even an equal number of 10 debates and equal number of candidates, Democrats would have enjoyed almost an hour longer to speak (721 minutes vs 666.)
Let's adjust the speaking time to show which candidates got more than their fair share relative to the time available per candidate per debate. In this case, 100% means the candidate got exactly the "fair share", or 7.6 minutes per debate for Republicans and 9.8 minutes per debate for Democrats. On this scale, leading candidates got up to 140% of their party's fair share, while the lowest share was 63%.
This new scale now removes the differences in total time between parties, and lets us compare relative advantage or disadvantage between parties and candidates. The data are plotted below.
Now that we are no longer confounding differences in total time between parties, new perspectives emerge from the data. The top two candidates in both parties were equally advantaged-- all four got about 140% of the time that an equal time rule would have given them. The previous comparisons masked this due to the shorter Republican times. But relative to a fair division of time, the top two were treated almost identically in both parties.
But the 3rd and 4th places were treated rather differently. In the Republican debates, McCain and Fred Thompson received about 120% of a fair share, while on the Democratic side Edwards and Richardson got only slightly more than a fair share would entitle them to, Richardson at 101% and Edwards at 109%. Based on this share of time comparison, then, the debates treated the Republican race as more of a 4 person contest, while the Democratic debates divided a top 2 from an "average" third and fourth.
The extra shares for leaders must come, of course, from the rest of the pack. Huckabee who now threatens to win Iowa has received only a 96% share of a fair time allocation. The bottom 5 among Republicans all got less than 80% shares. Among Democrats the gap between Richardson and the rest is from 101% to 80.7% and below.
So total time favored the Democrats, but did so even after adjusting for number of debates and debate participants. The separation into top 2 vs 3rd and 4th was especially clear for Democrats. The Republican race gave relatively more time to 3rd and 4th place candidates.
The text of the op-ed only contains four sentences that give interpretations of the data:
The front-running Democrats, thanks mostly to a smaller field (but also to one additional debate), got a lot more time to speak than the front-running Republicans.
Not surprisingly, the times for each candidate seem to follow the polls, with the leading contenders getting more minutes. As Mr. Huckabee’s poll numbers rose, his speaking time increased.
The debates had effects on both voters and candidates. Mr. Huckabee’s performances helped him emerge from the pack, and a few tough moments for Mrs. Clinton set the stage for her eventual fall in the polls.
The first point is right that Dems got more time, as the charts all show. But that advantage wasn't entirely due to fewer candidates and one more debate. The advantage was more real than that: Democrats got more time to speak per debate and per candidate. Changing the graph allows us to see this in a way the pie chart did not.
Also revealed by the charts here are systematic differences between candidates that illuminate the nature of the stratification within and between parties. Front runners are advantaged, but the Democratic race was treated as having 2 clear leaders while the Republican race had 4 apparent contenders, based on speaking time. That too is masked by the pie chart.
Finally, two of Todd's three points above are not addressed by the graphic or the data given. While it is clear that the order of speaking times roughly follows support in the polls, this is not entirely the case. For example Biden has more poll support than Dodd, yet Dodd got slightly more speaking time. (For that matter, and more powerfully, Clinton held a large lead in the national polls during almost all of the debates, yet trails Obama in total time.) Moreover, the dynamic element Todd mentions is not illustrated by the graphic at all. If Huckabee's speaking time rose with his polls we can't see it here. And did Thompson's time drop with his polls? Alas, the op-ed doesn't list these data (which would be quite lengthy.) But a graphic could have illustrated this dynamic aspect of the data in no more space than the pie chart.
Nor does the pie chart provide any evidence of the role of debates in the rise of Huckabee or the decline of Clinton. Did the polls move up or down noticeably following any of the debates? Did candidate time in one debate precede a rise in polls, or did a rise in polls precede more speaking time? We could see these things in a graph, but it would take well more than a thousand words to describe them. The better the graph the more words it is worth.
Cross-posted at Political Arithmetik.
A new Lee Enterprises/Research 2000 statewide survey of likely caucus goers in Iowa (conducted 12/26 through 12/27) finds:
- Among 500 likely Democratic caucus goers, former Sen. John Edwards and Sen. Barack Obama run even at 29%, Sen. Hillary Clinton runs at 28% in a statewide caucus; Gov. Bill Richardson trails at 7%.
- Among 500 likely Republican caucus goers, former Gov. Mike Huckabee leads former Gov. Mitt Romney (34% to 27%) in a statewide caucus, former Sen. Fred Thompson trails at 11%, former Mayor Rudy Giuliani, Sen. John McCain and Rep. Ron Paul all trail at 8%.
- All other candidates receive less than five percent each. The margin of sampling error is 4.5% for each subgourp.
Here are some additional details on the new Los Angeles Times/Bloomberg poll in Iowa. The last Times/Bloomberg poll in September drew a sample of "caucus voters" that represented a much larger slice of the Iowa population than other polls. The Democratic sample represented 39% of Iowa adults, while the Republican sample represented 29% of adults. While this statistic varied greatly among pollsters, most have reported "likely caucus goer" samples representing a range of 9-17% of Iowa adults for the Democrats and 6-11% for the Republicans (see the second table in my Disclosure Project post).
For this most recent survey, the Times release did not report the percentage of adults represented by each sample, but they did provide the unweighted sample sizes for the four different Iowa subgroups they released. All four are considerably closer to the low-incidence samples reported by most of the other pollsters that have disclosed these methodological details, although even the smaller Democratic "likely caucus goer" sample (17% of adults, unweighted) appears to be on the high side of what other pollsters reported to our Disclosure Project.
I put "appears to be" in italics above because the more accurate weighted values may be different. The methodology blurb in the Times release implies that the weighted size of each sample may be slightly smaller. Though unclear on the details, they say they "designed" their sample to " yield greater numbers of voters and thus a larger pool of likely caucus goers for analysis." That design may mean that the weighted value of the caucus voter and likely caucus-goer samples may be slightly smaller. I emailed a request for the weighted values and, as of this writing, have not received a response.
Update: Just received a response and added the weighted values to the table above. The weighting does bring down the size of the two "likely caucus goer" subgroups slightly, to 15% for the Democrats and 7% for the Republicans.
Update 2: "So what does this mean?" Two commenters ask that question, so I obviously neglected to explain. For those interested in all the details, the complete context can be found in this section of my Disclosure Project results post. The key issue is that the previous historical highs for caucus turnout are 5.5% of adults for the Democrats in 2004 and 5.3% of adults for the Republicans in 1988. Pollsters are generally not trying to screen all the way down to a combined 11% of adults, since (a) no one knows what turnout will be next week, (b) low incidence screens cannot select truly "likely" caucus goers with precision and (c) all political surveys presumably have some non-response bias toward voters (on the theory that non-voters are less interested and are more likely to hang up).
On the other hand, I consider it highly questionable to report results representing 68% of adults as representative of "caucus voters" as the Times/Bloomberg survey did in September.
So the results above mean two things. First, the latest Times/Bloomberg surveys are a vast improvement in terms of the portion of Iowa adults they represent. Second, at least in theory, the "likely caucus goers" are the more appropriate subgroups to watch. Of course, the percentage of adults sampled is just one aspect of accurately modeling the likely electorate. The kinds of voters selected are just as important, and can vary widely across polls that screen to the same percentage of adults. See the full Disclosure project post for more details.
Last week, I had the chance to briefly interview Joe Lenski, the executive vice-president of Edison Research, who will conduct exit polling in 2008 on behalf of the network consortium known as the National Election Pool. The following is a transcript of the interview.
Q. I want start by asking about the entrance poll that your company will be providing for the National Election Pool in the Iowa caucuses January third. What is an entrance poll and how does it differ from the exit polls that we've heard so much about?
Lenski: An entrance poll is similar to an exit poll in that we have interviewers that go to a random sample of precincts throughout the state and ask attendees to fill out these questionnaires. The only difference is we're asking caucus attendees to fill out these questionnaires as they are entering the caucus as opposed to a normal primary election or general election where our interviewers approach the voters and ask them to fill out the questionnaires after they have voted.
Q. What do you provide the networks with?
Lenski: We provide the networks with an estimate which updates very rapidly during an entrance poll because we're doing all of the interviews in a 60 to 90 minute period of time, as opposed to an exit poll where you're interviewing throughout the election day and in most states that's somewhere between 12 and 15 hours of voting. We only get the chance, in a caucus situation, to interview people as they're entering and that's usually 60 to 90 minutes before the caucus begins. So, these results get updated very quickly throughout the evening as interviews must be completed before the caucus begins. What we provide for an entrance poll is similar to what we provide our clients and subscribers for an exit poll on an election night and that is an estimate of how the voting is going, demographic detail of who's voting for whom, what voting groups are there, what their sizes are and what factors seem to be driving the vote that evening.
Q. Do you do anything in the questionnaire or in the analysis to try to anticipate the reallocation that occurs among Democrats when a candidate does not meet the 15% threshold within any given caucus location?
Lenski: Yes, there will be a question on the Democratic Iowa entrance poll asking "If your first choice does not gather enough supporters, who will be your second choice?" We can use that and we did use that in 2004. We did see that indeed a majority of the Kucinich voters chose Edwards as their second choice, matching the deal that the Edwards and Kucinich campaigns came up with the morning of the Iowa caucus in 2004 to have Kucinich voters switch to Edwards in precincts where the Kucinich group wasn't viable.
Q. Do you do that analysis as the aggregate for the whole state, or do you try to reallocate within each precinct?
Lenski: No, we only have sample sizes to do it for the whole state. Remember there are 1,781 distinct precinct caucuses. Actually, as an aside, some of the blogs say 3,562 -- that's combined Democrat and Republican caucus sites. Each party has 1,781 precinct caucus sites, and the dynamics within each one of those is very different and just because a candidate is receiving 15% statewide doesn't mean they are meeting that viability threshold on the Democratic side in each precinct. In fact even though John Kerry won in 2004, he didn't even reach viability in about ten percent of the precinct caucus locations. So, even the leading candidate may not reach viability in every single precinct.
Q. How many precincts are you sampling for each party?
Lenski: For each party we will be covering 40 precincts with the entrance poll. And at a larger sample, we will also have reporters there recording the initial preference so we have a statewide estimate of initial preference as well.
Q. I assume this a question you get a lot, but I've seen some commentary about it in the blogosphere recently. Why is it okay to sample only 40 precincts out of about 1,000?
Lenski: It is about the same sample size we use in most statewide exit polls. We will be interviewing over 1,000 actual caucus voters on each side, Democratic and Republican, and that's much larger than any of the pre-election likely caucus samples in the pre-election polls. We'll have over 1,000 actual Democratic caucus goers and over 1,000 actual Republican caucus goers in our entrance poll samples.
Q. I assume that in past years you've done the exercise where you've gone back and looked at the actual results from those sample precincts and asked the questions "What would you have done differently? Did the actual results predict the statewide results to assess whether that sample is appropriate?"
Lenski: That's easier to do on the Republican side because it is a straight straw poll and you get an official result: one person one vote. On the Democratic side it's a little more difficult because the final official result is the state delegate equivalence number that the Iowa state Democratic Party calculates on election night. So, it's not really an apples-to-apples comparison on the Democratic side. But we do look within our sample precincts at what responses we get on the entrance poll and what the initial preference breakdown was in those particular precincts because we have our vote reporters stay for the initial preference and record the initial preference breakdown in those sample precincts.
Q. And that's part of what gets reported?
Lenski: That's used in our estimate on election night for our statewide estimate of initial preference on the Democratic side.
Q. Last question, I've written a bit about this issue of whether there's a non-response correction possible with an entrance poll analogous to what you do for the exit poll? Could you explain that?
Lenski: In all exit polling, we do a non-response adjustment in which our interviewers record the gender, the race and the approximate age of those who refuse to fill out the questionnaire. We do find generally that younger people are a little more likely to fill out questionnaires than older people so this non-response adjustment corrects for that tendency. In past entrance polls -- and I've been involved in every Iowa entrance poll since 1988. I believe the first one was conducted by CBS in 1984 -- we have not done a non-response adjustment mainly because of the crunch of time. It's an added task for the interviewers to do, and in the crunch of time we try to get as many questionnaires completed as possible. But this year, pre-caucus telephone surveys show a difference in how, at least on the Democratic side, people are planning to vote by age. Younger people are more likely to support Obama, older Democrats more likely to support Clinton. We have decided this time to add a non-response adjustment to the entrance poll as well.
Q. So is that going to require additional people looking over the shoulder of the interviewer, or are they just going to have a little more to do?
Lenski: They're just going to have a little more to do. We do have two interviewers at each precinct for the entrance polls, again because of the crush of the number of interviews to be conducted in a short period of time. For our typical exit poll, there's only one interviewer all day. But for the entrance polls we always have two, so this will just be another task for the two of them to divide.
[Editor's Note: Andrew Kohut of the Pew Research Center did a longer interview with Lenski recently that went into more detail about the exit polls planned for the February 5 primaries and the rest of the year].
As we expected poll releases to be a little slower than usual this week, we decided to conduct one of our own. Last week, we invited about a hundred professional pollsters who work for the news media and political campaigns to participate in a brief survey about the public polls in Iowa and New Hampshire. We will be releasing results from our "Poll of Pollsters" over the next few days. Today I want to discuss the very last question we asked, which concerns the same topic I blogged on yesterday: The reliability of polls conducted this week, between Christmas and New Year's Day.
First, some details about the survey. We invited two categories of pollsters to participate, those that work for political campaigns and those that conduct surveys for news media outlets. We selected campaign pollsters that had been listed in a scorecard published by the National Journal's Hotline that had worked on behalf of candidates for Governor, Senate or U.S. Congress in 2006. The media pollsters included those who conduct the well-known national polls that we regularly track and the organizations that had released surveys in Iowa and New Hampshire.
We sent out email invitations to just over 100 individuals, and as of yesterday a total of 46 had completed the entire survey (21 media pollsters and 25 campaign pollsters; of the latter category 14 were Democrats and 11 Republicans). As with any other survey, we pledged to keep respondent identities confidential and report their answers only as aggregated data. The pollsters responded online, using an online survey form prepared by our sponsor/owner, Polimetrix.
We also want to be very clear on one point: This survey is not a "scientific" sampling of any population beyond the 46 pollsters that replied. Thus, there is no "margin of error." The results represent no more and no less than the opinions of 46 individuals that were willing to respond. Since we made it clear that we would ask about the public polls in Iowa and New Hampshire, I assume that many of the non-participants are simply less interested in the topic. As one prominent campaign pollster who emailed his regrets put it, "in this case I feel like an ill-informed respondent." So the 46 we heard from are probably among the most opinionated on this subject.
So let's start with the topic of polling between Christmas and the New Year. Inspired by the quotations from pollsters in this recent Associated Press article, we decided to ask pollsters choose between two statements about the challenge of polling this week. But rather than forcing them to agree entirely with one statement or another, we asked the pollsters to pick a point along a scale to indicate which statement they agreed with more. The scale they used is reproduced below. They could use their mouse to click on any point along the scale (you can click on it to see a full size version):
There has been a lot of talk about the difficulty of polling during final week or so before the Iowa caucuses that includes both Christmas and New Years Day. Some believe that polling in this period will be challenging but can be done reliably. Others believe that polls done in this period will be so unreliable that they should be ignored. What about you?
Please click or drag on the ruler below to indicate which statement you agree with more.
The pollsters' responses spread across the scale as shown in the chart below. Not one signaled total agreement with either statement by moving the slider all the way to the left or all the way to the right of the ruler. They divide almost evenly in terms of which statement they lean to, with 48% placing themselves closer to the notion that polls "can be done reliably" this week, and 50% placing themselves closer to the statement that polls this week are "so unreliable they should be ignored" (note: one pollster/respondent clicked "not sure," three skipped the question completely).
However, the skeptics hold stronger opinions than the believers. Compare the two bars at the far left of the chart above (representing 17% who have the most doubts about the reliability of Christmas week polling) to the two bars at the right (5% who have the last doubts). For purposes of tabulation, we have translated the hash-marks along the ruler into a 100-point-scale, where 1 is "should be ignored" and 100 is "can be done reliably." The mean score for the full sample is 48 and the median 45, indicating a modest skew toward "should be ignored" (for the truly wonky, each of the bars in the chart represents a decile, 1-10, 11-20, 21-30 and so on).
As the second chart (below) indicates, the media pollsters are more skeptical than the campaign pollsters. The average score for media pollsters is a 43 (with a median of 29). In other words, the media pollsters cluster near the "so unreliable they should be ignored" end of the scale, while the campaign pollsters divide more evenly, with an average score of 50 and a median of 48. [For those interested in more details, click here for a spreadsheet with the number of responses for each increment clicked on the 1-100 scale].
Think of it this way: Even professional pollsters are not sure what to make of the surveys done this week. While few advise ignoring these surveys altogether, most recommend treating the results with a lot more caution than usual. Their uncertainty probably reflects the overall lack of experience our profession has with surveys conducted during the latter half of December. Just about everyone sampled recommends treating the results from this week with a lot more caution than those available at other times, although few would advise that we ignore these results completely.
We will have more results from our "Poll of Pollsters" over the next few days.
A new Strategic Vision (R) statewide survey of likely voters in Iowa (conducted 12/26 through 12/27) finds:
- Among 600 likely Republican caucus goers in Iowa, former Gov. Mike Huckabee runs at 29%, former Gov. Mitt Romney at 27% in a statewide caucus; former Sen. Fred Thompson trails at 15%, Sen. John McCain at 14%.
- Among 600 likely Democratic caucus goers, Sen. Barack Obama runs at 30%, Sen. Hillary Clinton at 29%, former Sen. John Edwards at 28% in a statewide caucus; Sen. Joe Biden trails at 5%.
- All other candidates receive less than five percent each. The margin of sampling error for both subgroups is 4%.
Two new Los Angeles Times/Bloomberg statewide surveys (Times story, results; Bloomberg GOP story, Dem story) of 2,312 adults in Iowa and 1,459 adults in New Hampshire (conducted 12/20 through 12/23 and 12/26) finds:
- Among 580 Democratic caucus goers in Iowa, Sen. Hillary Clinton runs at 29%, Sen. Barack Obama at 26%, and former Sen. John Edwards at 25% in a statewide caucus; Gov. Bill Richardson trails at 6%. Among 389 likely Democratic caucus goers, Clinton runs at 31%, Edwards at 25%, Obama at 22%, Richardson at 7%, and Sen. Joe Biden at 6%.
- Among 310 Republican caucus goers in Iowa, former Gov. Mike Huckabee leads former Gov. Mitt Romney (37% to 23%) in a statewide caucus; Sen. John McCain and former Sen. Fred Thompson trail at 11%, former Mayor Rudy Giuliani trails at 6%. Among 174 likely Republican caucus goers, Huckabee runs at 36%, Romney at 28%, Thompson at 10%, McCain and Giuliani both at 8%.
- Among 519 Democratic primary voters in New Hampshire, Obama runs at 32%, Clinton at 30%, and Edwards at 18% in a statewide primary; Richardson trails at 6%. Among 361 likely Democratic primary voters, Obama runs at 32%, Clinton at 30%, and Edwards at 20%.
- Among 442 Republican primary voters in New Hampshire, Romney leads McCain (34% to 21%) in a statewide primary; Giuliani trails at 14%, Huckabee at 9%, Paul at 6%. Among 318 likely Republican primary voters, Romney leads McCain (34% to 20%); Giuliani trails at 17%, Huckabee at 12%.
All other candidates receive less than five percent each. The margin of sampling error is 5% for Democratic caucus goers and for likely Democratic caucus goers in Iowa, 6% for Republican caucus goers and 7% for likely Republican caucus goers in Iowa, 4% for Democratic primary voters and 5% for likely Democratic primary voters in New Hampshire, 5% for Republican primary voters and 6% for likely Republican primary voters in New Hampshire.
Additional analysis from the two most recent USA Today/Gallup national surveys (Dem, GOP) with a combined total sample of 1,007 Democrats and those who lean Democratic and 824 Republicans and those who lean Republican (conducted 11/30 through 12/2 and 12/14 through 12/16) compares vote preference among various sub-groups separated by sex, race, age, geography, education, and income.
A new AP-Yahoo panel survey (story, results) of 1,523 registered voters nationwide (conducted 12/14 through 12/20 by Knowledge Networks) finds:
- Among 847 registered Democrats, Sen. Hillary Clinton leads Sen. Barack Obama (47% to 25%) in a national primary; former Sen. John Edwards trails at 13%.
- Among 655 registered Republicans, former Gov. Mike Huckabee runs at 22% and former Mayor Rudy Giuliani at 21% in a national primary; Sen. John McCain trails at 14%, former Gov. Mitt Romney at 13%, former Sen. Fred Thompson at 11%.
- All other candidates receive less than five percent each. The margin of sampling error is 3.4 for Democrats and 3.8 for Republicans.
- "The poll was conducted over the Internet by Knowledge Networks, which initially contacted people using traditional telephone polling methods and followed with online interviews. People chosen for the study who had no Internet access were given it for free."
(LARGE graphs-- you probably should click once or twice to see them at full resolution in order to see the details.)
A new American Research Group (ARG) poll of Iowa has caused quite a debate in the comments at Pollster.com. Today Mark Blumenthal takes a close look at polling the "Dark Side of the Moon" in Iowa during the holiday season and the unknowns involved, including the unknowns of the ARG poll. So while the pollsters are busy trying to get one more Iowa poll in this week, let's look at the track records of the pollsters in Iowa.
The chart above shows the polling of the Democratic nomination race in Iowa since January. In 2007, twenty four different polling organizations conducted Democratic caucus polls. Of these, only 7 have conducted three polls or more. That means that for most pollsters, we have no way to separate random error from systematic "house" effects. To do that, we need multiple polls, and to do it reliably we need a number of polls from each organization. But we can at least look at the seven organizations with three or more polls, and see how they compare to the trend estimates. The trend, of course, reflects the best estimate across all pollsters, but until we see the actual vote (itself a very slippery concept in Iowa) we can't say if the trend was better than individual polls or not. Still, this reveals when pollsters seem to follow the trend and when they systematically seem to miss it.
The clear result of the comparison above is that ARG generally showed a much better performance for Clinton than the trend in the first half of the year. From January through June, ARG usually had Clinton some 6-10 points above the trend estimate, with one exception in which ARG agreed almost exactly with the Clinton trend. During this period, ARG was the most discrepant of all pollsters from the Clinton trend.
In the second half of the year, ARG's polling has generally been much closer to the trend estimates, usually less than 4 points away from the Clinton trend. During this time ARG has mixed in with other polling pretty consistently.
That is, until the most recent December 20-23 poll, which again shows a large Clinton deviation of about +6 points from the trend. (I'm using the standard blue trend line. The upturn in the red "sensitive" estimate is interesting, but it is also sensitive to the ARG poll, an example of why you might not entirely trust the red estimator.) This is the first large ARG deviation recently, and quite a change from ARG's previous poll of Dec 16-19 which was close to trend.
In an inversion of the Clinton effect, ARG consistently underestimated Obama's support, compared to the trend, in the first half of the year, but their polling fell much closer to the trend during the second half. At least until the latest poll which has Obama some 10 points below the standard trend. (Again, note the downturn of the sensitive estimator, which would be turning down even without ARG, but which is turning down somewhat more because of ARG.)
ARG's polling for Edwards has been more variable. Early 2007 polling fluctuated quite a bit, sometimes below and sometimes above trend. More recent ARG results for Edwards has generally been a few points below trend, with the latest result about as much below trend as has been "normal" for ARG recently.
ARG's results this year have been more heterogeneous than some oversimplifications claim. There was a substantial overestimate of Clinton early in the year, along with an underestimate of Obama's support. That was important during this period because there were relatively few other polls available for Iowa in this period. But in the second half of the year, the ARG results for Clinton and Obama have been much closer to trend estimates, though still with a small average advantage for Clinton and small disadvantage for Obama.
In light of this, the ARG poll for December 20-23 does look out of line with their own previous polling, certainly their polling of the last 4-5 months.
Let's look at their Republican results for Iowa.
On the Republican side, ARG's results have been especially favorable to John McCain, and to a lesser but still substantial degree to Giuliani. Unlike the Democratic results, these effects have not diminished much in the second half of the year. McCain has often been as much as 10 points above trend, with only one poll below trend and one more right on the trend. No other pollster has been so consistently far from the trend for McCain.
For Giulinai, the results are less far from trend, but still quite consistently above trend. Only 2 of the last 10 ARG polls have Giuliani below trend.
Romney has fared close to trend in the ARG surveys, though on average a bit below trend. The discrepancies for Romney are much less dramatic than for McCain or Giuliani.
The last two ARG polls show shifts of -3 and +4 points for McCain and Romney respectively, and a single point difference for Giuliani. (And a -5 and +6 for Huckabee and Paul.) For the Dems the shifts were +5, -6 and +2 for Clinton, Obama and Edwards respecively.
As Mark Blumenthal notes, the reasons for these discrepancies are largely matters of speculation. But the consistency of the ARG house effects are pretty clear in these data. The ARG results currently stand on the same side as their long term house effects: above trend for Clinton, Giuliani and McCain, and below trend for Obama, Edwards and Romney. Compared to other pollsters, these house effects for ARG appear to be the largest of any polling firm in Iowa.
Cross-posted at Political Arithmetik.
Those of us with memories of the Apollo moon missions will remember those moments when the orbiting command module went around the so-called "dark side of the moon."** At that point the moon itself physically blocked broadcast signals, and despite all the dazzling "space age" telecommunications on display, live transmissions from the astronauts came to a total halt. While Walter Cronkite stalled for time, we watched and waited for the astronauts to regain contact.
Right now, with interest peaking in Iowa and New Hampshire polls, we have entered into our own dark side of the moon period. Unlike the moon missions, however, we are not in a total blackout. At least one pollster has released an Iowa survey conducted over the weekend, and others will surely follow later this week. However, with so many Americans traveling away from home for holiday travel, those surveys will be of unknown reliability, at best. The worst-case scenario (for the pollsters) will be if these surveys indicate a false trend, a shift that is less an indicator of real change than an artifact of respondents missing due to holiday travel.
Unfortunately, the survey research profession has relatively little experience with surveys conducted in the week between Christmas and the New Year (and the weekends surrounding that week). All of the polling firms I worked with rarely fielded surveys in the second half of December and typically shut down altogether between Christmas and New Years. CBS News Polling Director Kathy Frankovic reports in her column this week that the Roper Center archives include no public polls conducted between Christmas and New Years for either 2006 or in 2003 (other than one continuously running financial monitoring survey).
The reason that pollsters typically avoid polling around the holidays is the assumption that a big chunk of the population is away from home and unavailable for survey calls. As with any sort of "non-response" problem, we risk getting skewed or biased results if the missing respondents are both numerous and different in terms of their political views from those at home when we call.
What kinds of voters might be missing right now? Three years ago, in a survey concluded a full week before the holidays (12/15-17/2004), the Gallup Organization asked a national sample of 1,002 adults whether they planned "to travel more than fifty miles from home this holiday season." Twenty-eight percent (28%) said "yes." More important, as the table below indicates (based on data drawn from the Roper Archives), those planning holiday travel had a very distinctive demographic profile. Holiday travelers were much more likely to be younger and better educated. Notice also that holiday travelers were not just college students. Adults between the ages of 30 and 44 are twice as likely to travel for the holidays than those over 65. (Also, while I do not show it here, the pattern in these results by age, education and income was nearly identical for Democratic and Republican identifiers).
So pollsters fielding surveys this week are going to have a harder time finding younger, better educated respondents. Why is that important? Because in the Democratic race, at least, there are huge differences in vote preference by age and education: In virtually every survey, including those in Iowa and New Hampshire, Barack Obama does best among younger, better educated voters while Hillary Clinton's base of support is older and less well educated. Consider the data from the Boston Globe/University of New Hampshire survey conducted last week: Obama wins the support of 47% of voters age 18 to 34, but only 22% of those over 65; 40% of those with graduate degrees but only 25% of those without a college degree.
Given the age and education pattern, a sample with fewer voters under 45 or fewer with a college education will skew in Clinton's favor. Pollsters who see a demographic shift may choose to "weight" their samples to match the demographics of pre-holiday polls. That approach may lessen the bias but not correct it entirely. Weighting by age and education in this instance essentially replaces the younger, college educated voters who are away from home with other younger, college educated voters who are available to do the survey. If travelers and non-travelers are still different politically, regardless of age or education, then some bias will remain. And even if not, an extreme "weighting up" always increases the statistical error, in effect reducing the sample size.
Do we have any evidence that holiday travel bias might affect vote preference? Maybe. Consider the data below reported by Rasmussen Reports just before and after the Thanksgiving holiday.
Rasmussen runs a daily, rolling-average national survey that tracks presidential primary vote preference. Rasmussen uses an automated interactive-voice-response (IVR) methodology in which respondents answer recorded questions by pressing keys on their touch-tone phones. Each night, they call roughly 175-200 likely Democratic primary voters and roughly 150-160 likely Republican voters, then roll together and report a rolling average of the last four nights of interviews. Thus, each of the points on the chart below represents 750-800 likely Democratic voters with a reported margin of sampling error of +/-4%. The trend line is a regression estimate that Charles Franklin created for me using the Rasmussen data.
Notice what happens to the Obama trend line just before and just after Rasmussen took a five-day break from interviewing, Wednesday through Sunday, over the Thanksgiving weekend (Thanksgiving was November 22). The regression trend line essentially splits the difference between the 17% low points immediately before and after the Thanksgiving break and the much higher 26% results that came just before and just after that. What one makes of the variation may be in the eye of the beholder. Either the Obama trend got unusually erratic in both directions during the last two weeks of November, or there was a very unusual and precipitous plunge from what should have been a plateau around 24-26 to 17% centered on the period of Sunday, Monday and Tuesday before Thanksgiving and the Monday just after. Either way, the Obama variation around Thanksgiving was highly statistically significant. It did not occur by chance alone. Either there was a see-saw in Obama's real world support that week, or something changed in the kinds of voters Rasmussen sampled.
Given what we know about the demographics of holiday travelers and Obama's supporters, I'd bet on the latter.
Now I should point out that the Rasmussen national tracking may be a special case. According to Scott Rasmussen, each daily sample is essentially "fresh." Unlike many other pollsters, they do not attempt to call back unavailable respondents on successive nights. If they sample your phone number on Monday and you are not home, they will not call you back again on Tuesday. As such, their surveys may be more prone to a holiday effect than others that do more callbacks. And while I believe that Rasmussen weights their samples by gender, age and race to force consistency for each four-day report, they may not weight by education.
All of this brings us to the survey that the American Research Group released on Monday fielded between Thursday December 20 through Sunday December 23, a survey that shows Clinton gaining and Obama falling. Some will read this post as an attempt to debunk that result, and the findings above certainly argue for considerable caution in reading results from any survey this week. But the problem in trying to assess the ARG poll is that we know so little about it. Does ARG make call-backs to unavailable respondents? What was the sample composition on any ARG Iowa survey this year in terms of age and education level, and was this one suddenly different? Did ARG weight the results by age or education this time, and if so, by how much? We are in the dark on all of these questions.
It is also worth remembering, as some commenters noted yesterday, that real changes may be occurring in vote preference this week even if surveys may be severely challenged in their ability to measure it. Clinton may be gaining and Obama falling. So it is quite a leap for anyone to say they know conclusively that the ARG result is either right or wrong.
The hard truth is that we are behind the dark side of the moon this week, and we may not know much with certainty until next Wednesday night.
**Technically, the moon does not have a "dark side." Although the more appropriate term is "far side" of the moon, I still prefer the Pink Floyd version.
On behalf of Eric, Charles and our families, we wish you a Merry Christmas, Happy Holidays and a season of Peace.
We will be back on Wednesday, obsessing over the latest results as we usually do. See you then.
A new American Research Group statewide survey of likely caucus goers in Iowa (conducted 12/20 through 12/23) finds:
- Among 600 likely Democratic caucus goers, Sen. Hillary Clinton leads former Sen. John Edwards (34% to 20%) in a statewide survey; former Sen. Barack Obama trails at 19%, Sen. Joe Biden at 8%, Gov. Bill Richardson at 5%.
- Among 600 likely Republican caucus goers, former Gov. Mike Huckabee runs at 23%, former Gov. Mitt Romney 21%, and Sen. John McCain 17% in a statewide primary; former Mayor Rudy Giuliani trails at 14%, Rep. Ron Paul at 10%.
- All other candidates receive less than five percent each. The margin of error is four percent for both subgroups.
Everyone is chewing over the results of the new Boston Globe/UNH survey this morning. One sentence in the Globe write-up caught the eye of both Noam Schieber and Ben Smith:
One aspect of the healthcare debate that has divided Democratic candidates is whether individuals should be required to purchase coverage - Clinton and Edwards favor a mandate, while Obama does not. A slight majority of Democratic voters who were polled - including pluralities of Clinton and Edwards supporters - opposed such a requirement.
Scheiber took it a step further:
I wondered a few weeks ago why the Clinton campaign was going negative on Obama's character instead of hitting the healthcare issue, which seemed less fraught and likely to pay dividends on both a policy level and a preparedness level. This poll hints at an explanation: Maybe it just wasn't working.
Or maybe it risks a backlash with Clinton's base. Buried in the UNH crosstabs is a pattern I had not seen before. Opposition to the notion of an individual health insurance mandate -- "should individuals be required to buy health insurance" -- is greatest among the less well-educated and downscale voters that are the core of Clinton's base in New Hampshire and elsewhere:
The Associated Press reported a few weeks ago that the Clinton campaign "is preparing television ads here criticizing Barack Obama's health care plan" to air in New Hampshire in the event the Obama campaign "ignites in Iowa." Most surveys show that Democrats perceive Clinton as better able to handle the issue of healthcare, and the argument made by the Clinton campaign, that Obama's plan "could leave as many as 15 million people uninsured" no doubt scores well in message testing surveys. The data above, however, suggest that such an attack risks a backlash if the debate engages on the issue of "requiring individuals to buy health insurance."
A new Boston Globe/UNH statewide survey (Globe story, results; UNH results) of likely primary voters in New Hampshire (conducted 12/16 through 12/20) finds:
- Among 420 likely Democratic primary voters, Sen. Barack Obama runs at 30% and Sen. Hillary Clinton at 28% in a statewide primary; former Sen. John Edwards trails at 14%, Gov. Bill Richardson at 7%.
- Among 405 likely Republican primary voters, former Gov. Mitt Romney runs at 28%, Sen. John McCain at 25% in a statewide primary; former Mayor Rudy Giuliani trails at 14%, former Gov. Mike Huckabee at 10%, Rep. Ron Paul at 8%.
- All other candidates receive less than five percent each. The margin of sampling error is 4.9% for both subgroups.
Correction: The original version of this post mislabeled the results for Clinton and Obama