Mark Blumenthal: December 23, 2007 - December 29, 2007
December 29, 2007
By Mark Blumenthal
The main impetus for our "Poll of Pollsters" (from which I posted the first results on Thursday) is the desire to find better objective criteria to help sort out pollsters. As both Charles and I review today in complementary posts (here and here), the handful of past Iowa polls do not lend themselves to easy assessments of accuracy. The lack of transparency by many public pollsters makes it difficult to fairly assess differences in their methods, although I would propose the degree of their disclosure as a surrogate measure of quality. A third possibility is to measure pollster reputation, especially among their peers. We decided to start with a survey of pollsters about the public polls now in the spotlight in Iowa and New Hampshire.
I included all the details in the previous post, but here are the highlights: We sent out invitations to just over a hundred pollsters and had 46 complete the entire survey, although a few more (49) completed the questions about the reliability of the polls in Iowa. Of those, 22 are media pollsters and and 27 campaign pollsters (16 Democrats and 11 Republicans). There is no margin of error because the results represent nothing more or less than the views of the pollsters that participated. Like any survey respondents, we promised to keep their identities confidential.
We started with a simple question asked about each of the 16 pollsters that have released public polls in Iowa: "How reliable do you consider surveys of IOWA CAUCUS goers done by each of the following organizations, very reliable, somewhat reliable, not very reliable or not reliable at all?" We also provided an option to say they "do not know enough to rate" each organization.
We left "reliability" in the eye of the beholder, but it is fair to assume that few are in a position to evaluate the performance of each organization in past Iowa caucuses. As one pollster put it (in a space provided for comments the end of the survey) that there is "no way we can know who's most reliable until we can compare their final estimates with actual vote." Instead, it is safe to assume that most based their judgements on the reputation of each organization and its methods. As you will see, the pollsters had little trouble making such judgements.
As the following table shows, the Des Moines Register "Iowa Poll" conducted by Selzer and Company easily earns the highest marks, with virtually all rating it either very (36%) or somewhat (50%) reliable. The other pollsters with the highest scores are nationally known media surveys: ABC/Washington Post, the Pew Research Center and CBS/New York Times.
The pollsters receiving the lowest scores are Zogby International, the American Research Group and Rasmussen Reports. In the case of Zogby, four out of five pollsters rated their surveys as not very (28%) or not at all reliable (52%).
Not surprisingly, the media pollsters are generally more positive than their campaign consultant colleagues (keep in mind that we invited principals at all 13 organizations that are polling in Iowa to participate). Their rankings are generally similar, although the University of Iowa and Strategic Vision rank higher among the campaign than the media pollsters.
We also asked our pollster-respondents to select from the same list the "pollsters you consider MOST and LEAST reliable in Iowa." As the table below shows, the Des Moines Register/Selzer survey easily stands out as the favorite, especially among the campaign pollsters:
As for the least reliable pollster in Iowa, once choice easily led the pack. One third (33%) of the campaign pollsters and just less than half (45%) of the media pollsters picked Zogby International:
Again, reputation cannot tell us everything we need to know about the quality of the numbers a pollster produces. Pollsters with poor reputations may conduct quality polls and even the best pollsters are fallible. However, when a pollster earns the respect of their colleagues, it should tell us something.
Up next, similar ratings of the pollsters in New Hampshire. Meanwhile, Pollster.com readers, what do these ratings tell you?
Note: I accidentally omitted InsiderAdvantage which released their Iowa poll as we were in the process of drafting this survey. I apologize for the oversight.
By Mark Blumenthal on December 29, 2007 11:06 PM
| Permalink
| Comments (6)
By Mark Blumenthal
Given the way we are scrutinizing the final Iowa Caucus polls, this seems like a good time to take a look a the final pre-caucus polls from 2004 and 2000. One of the questions I get most frequently is which pollsters were "most accurate" in previous years, and as the old data will show, that is a far more difficult question to answer than most people assume.
Consider the final polls for the Democratic Caucuses in 2004. Only five organizations released public polls conducted in the final week before the Caucuses, which were held on Monday, January 19 that year. Since both John Kerry and John Edwards experienced late surges in support, polls conducted before that would show considerably more "error," since they obviously missed the late surge. Also, those who continued to call through Sunday night might have some advantage in catching the late breaking trend (or, as more cynical pollsters will point out, those releasing late polls also had the benefit of seeing the results of the other earlier surveys).
The table below shows the results of the final week's polls, plus the results of the network "entrance poll," which asked participants their first preference as they entered their caucus location.
[Click on table to pop-up full size version]
The first issue, unique to the Democratic contest, is that even a perfectly accurate survey can tell you only about the initial preference of the caucus-goers. The Iowa Democratic Party does not report the head-count of initial preferences. Instead, the official "results" they report on caucus night will be based on the state delegates for each candidate chosen to the state convention. The delegate selections are based on a second round of voting after supporters of "non-viable" candidates (those who receive less than 15% on the first round) realign to their second choice.
So as should be obvious, the second round voting means that initial vote preference -- even as measured by an entrance poll -- does not directly measure the final results. So there is an important element of inaccuracy built into any Democratic preference poll. In 2004, both Kerry and Edwards did better in the reported results than the entrance poll. Most observers attribute much of the six-point gain for Edwards to a deal struck on caucus morning between the Kucinich and Edwards campaigns that sent most Kucinich supporters into the Edwards camp on the second round. Exit pollster Joe Lenski reports that most Kucinich supporters chose Edwards as their second choice in the entrance poll.
Putting aside the viability issue, which poll was "most accurate?" The answer depends on the yardstick applied, which is a tough call in a multiple candidate primary or caucus. The Des Moines Register poll has received much credit for being the only one to correctly "predict" the order of the top candidates, but notice that Edwards "led" Dean on their larger "likely caucus goer" sample by a not statistically significant three percentage points (23% to 20%). Among their narrower "definite voter" subgroup, the Edwards-Dean order was reversed (Dean had 21%, Edwards 19%). So the getting the order right may have been partly a matter of good fortune.
I will spare readers the minutiae of the various error scores, but if we measure accuracy in terms of how well the polls predicted Kerry's percentage the Des Moines Register's narrower "definite voter" subgroup does slightly better. The same Register sample also does best in terms of the average of the errors for all the candidates. Ironically, the smaller Register "definite voter" sample, the one that had Dean nominally (though not significantly) "ahead" of Edwards, was the most accurate on these criteria even though the larger Register sample has been credited with "predicting" the order of finish.
If, on the other hand, we focus on the Kerry-Edwards margin, the final Zogby poll comes slightly closer to the actual result. In any case, the differences between the pollsters are small enough on all of these criteria that random chance was certainly a factor in determining which did best. And notice that everyone was way off on the final margin between Edwards and Dean, whether we compare to the entrance poll head count (Edwards +6), or the post-realignment actual results (+14).
What about the 2000 Caucuses? The number of final week polls was again fairly limited. On the Republican side -- where the actual results are a simple head-count -- the LA Times and the Des Moines Register came closest to George Bush's ultimate share of the vote, and the Times had the narrowest (and thus most accurate) Bush-Forbes margin. But all of the polls underestimated the support received by Forbes and Keyes.
On the Democratic side, the Des Moines Register had the Gore-Bradley margin exactly right, but a University of Iowa poll, which overstated Gore's margin, had Gore's percentage of the vote exactly right.
So what's the point of these comparisons? Trying to score such a small number of polls solely on the basis of accuracy is a confusing, contentious and ultimately futile exercise. The 15% viability threshold on the Democratic side undermines the accuracy of all polls. The timing of the final poll is critical and the differences among the various pollsters in the final week have been relatively small. Moreover, different measures of accuracy can lead to different conclusions. So if you take away nothing else, remember that none of the Iowa polls have been a perfect crystal ball. All have missed significant aspects of the final results in 2004 and 2000.
Sources: I obtained the results above from the subscriber only archives of The Hotline and the Polling Report. The SurveyUSA results are still available online and the InsiderAdvantage survey from 2004 was released at the time via PRNewswires.
By Mark Blumenthal on December 29, 2007 2:52 PM
| Permalink
| Comments (14)
December 28, 2007
By Mark Blumenthal
Here are some additional details on the new Los Angeles Times/Bloomberg poll in Iowa. The last Times/Bloomberg poll in September drew a sample of "caucus voters" that represented a much larger slice of the Iowa population than other polls. The Democratic sample represented 39% of Iowa adults, while the Republican sample represented 29% of adults. While this statistic varied greatly among pollsters, most have reported "likely caucus goer" samples representing a range of 9-17% of Iowa adults for the Democrats and 6-11% for the Republicans (see the second table in my Disclosure Project post).
For this most recent survey, the Times release did not report the percentage of adults represented by each sample, but they did provide the unweighted sample sizes for the four different Iowa subgroups they released. All four are considerably closer to the low-incidence samples reported by most of the other pollsters that have disclosed these methodological details, although even the smaller Democratic "likely caucus goer" sample (17% of adults, unweighted) appears to be on the high side of what other pollsters reported to our Disclosure Project.
I put "appears to be" in italics above because the more accurate weighted values may be different. The methodology blurb in the Times release implies that the weighted size of each sample may be slightly smaller. Though unclear on the details, they say they "designed" their sample to " yield greater numbers of voters and thus a larger pool of likely caucus goers for analysis." That design may mean that the weighted value of the caucus voter and likely caucus-goer samples may be slightly smaller. I emailed a request for the weighted values and, as of this writing, have not received a response.
Update: Just received a response and added the weighted values to the table above. The weighting does bring down the size of the two "likely caucus goer" subgroups slightly, to 15% for the Democrats and 7% for the Republicans.
Update 2: "So what does this mean?" Two commenters ask that question, so I obviously neglected to explain. For those interested in all the details, the complete context can be found in this section of my Disclosure Project results post. The key issue is that the previous historical highs for caucus turnout are 5.5% of adults for the Democrats in 2004 and 5.3% of adults for the Republicans in 1988. Pollsters are generally not trying to screen all the way down to a combined 11% of adults, since (a) no one knows what turnout will be next week, (b) low incidence screens cannot select truly "likely" caucus goers with precision and (c) all political surveys presumably have some non-response bias toward voters (on the theory that non-voters are less interested and are more likely to hang up).
On the other hand, I consider it highly questionable to report results representing 68% of adults as representative of "caucus voters" as the Times/Bloomberg survey did in September.
So the results above mean two things. First, the latest Times/Bloomberg surveys are a vast improvement in terms of the portion of Iowa adults they represent. Second, at least in theory, the "likely caucus goers" are the more appropriate subgroups to watch. Of course, the percentage of adults sampled is just one aspect of accurately modeling the likely electorate. The kinds of voters selected are just as important, and can vary widely across polls that screen to the same percentage of adults. See the full Disclosure project post for more details.
By Mark Blumenthal on December 28, 2007 2:22 PM
| Permalink
| Comments (2)
By Mark Blumenthal
Last week, I had the chance to briefly interview Joe Lenski, the executive vice-president of Edison Research, who will conduct exit polling in 2008 on behalf of the network consortium known as the National Election Pool. The following is a transcript of the interview.
Q. I want start by asking about the entrance poll that your company will be providing for the National Election Pool in the Iowa caucuses January third. What is an entrance poll and how does it differ from the exit polls that we've heard so much about?
Lenski: An entrance poll is similar to an exit poll in that we have interviewers that go to a random sample of precincts throughout the state and ask attendees to fill out these questionnaires. The only difference is we're asking caucus attendees to fill out these questionnaires as they are entering the caucus as opposed to a normal primary election or general election where our interviewers approach the voters and ask them to fill out the questionnaires after they have voted.
Q. What do you provide the networks with?
Lenski: We provide the networks with an estimate which updates very rapidly during an entrance poll because we're doing all of the interviews in a 60 to 90 minute period of time, as opposed to an exit poll where you're interviewing throughout the election day and in most states that's somewhere between 12 and 15 hours of voting. We only get the chance, in a caucus situation, to interview people as they're entering and that's usually 60 to 90 minutes before the caucus begins. So, these results get updated very quickly throughout the evening as interviews must be completed before the caucus begins. What we provide for an entrance poll is similar to what we provide our clients and subscribers for an exit poll on an election night and that is an estimate of how the voting is going, demographic detail of who's voting for whom, what voting groups are there, what their sizes are and what factors seem to be driving the vote that evening.
Q. Do you do anything in the questionnaire or in the analysis to try to anticipate the reallocation that occurs among Democrats when a candidate does not meet the 15% threshold within any given caucus location?
Lenski: Yes, there will be a question on the Democratic Iowa entrance poll asking "If your first choice does not gather enough supporters, who will be your second choice?" We can use that and we did use that in 2004. We did see that indeed a majority of the Kucinich voters chose Edwards as their second choice, matching the deal that the Edwards and Kucinich campaigns came up with the morning of the Iowa caucus in 2004 to have Kucinich voters switch to Edwards in precincts where the Kucinich group wasn't viable.
Q. Do you do that analysis as the aggregate for the whole state, or do you try to reallocate within each precinct?
Lenski: No, we only have sample sizes to do it for the whole state. Remember there are 1,781 distinct precinct caucuses. Actually, as an aside, some of the blogs say 3,562 -- that's combined Democrat and Republican caucus sites. Each party has 1,781 precinct caucus sites, and the dynamics within each one of those is very different and just because a candidate is receiving 15% statewide doesn't mean they are meeting that viability threshold on the Democratic side in each precinct. In fact even though John Kerry won in 2004, he didn't even reach viability in about ten percent of the precinct caucus locations. So, even the leading candidate may not reach viability in every single precinct.
Q. How many precincts are you sampling for each party?
Lenski: For each party we will be covering 40 precincts with the entrance poll. And at a larger sample, we will also have reporters there recording the initial preference so we have a statewide estimate of initial preference as well.
Q. I assume this a question you get a lot, but I've seen some commentary about it in the blogosphere recently. Why is it okay to sample only 40 precincts out of about 1,000?
Lenski: It is about the same sample size we use in most statewide exit polls. We will be interviewing over 1,000 actual caucus voters on each side, Democratic and Republican, and that's much larger than any of the pre-election likely caucus samples in the pre-election polls. We'll have over 1,000 actual Democratic caucus goers and over 1,000 actual Republican caucus goers in our entrance poll samples.
Q. I assume that in past years you've done the exercise where you've gone back and looked at the actual results from those sample precincts and asked the questions "What would you have done differently? Did the actual results predict the statewide results to assess whether that sample is appropriate?"
Lenski: That's easier to do on the Republican side because it is a straight straw poll and you get an official result: one person one vote. On the Democratic side it's a little more difficult because the final official result is the state delegate equivalence number that the Iowa state Democratic Party calculates on election night. So, it's not really an apples-to-apples comparison on the Democratic side. But we do look within our sample precincts at what responses we get on the entrance poll and what the initial preference breakdown was in those particular precincts because we have our vote reporters stay for the initial preference and record the initial preference breakdown in those sample precincts.
Q. And that's part of what gets reported?
Lenski: That's used in our estimate on election night for our statewide estimate of initial preference on the Democratic side.
Q. Last question, I've written a bit about this issue of whether there's a non-response correction possible with an entrance poll analogous to what you do for the exit poll? Could you explain that?
Lenski: In all exit polling, we do a non-response adjustment in which our interviewers record the gender, the race and the approximate age of those who refuse to fill out the questionnaire. We do find generally that younger people are a little more likely to fill out questionnaires than older people so this non-response adjustment corrects for that tendency. In past entrance polls -- and I've been involved in every Iowa entrance poll since 1988. I believe the first one was conducted by CBS in 1984 -- we have not done a non-response adjustment mainly because of the crunch of time. It's an added task for the interviewers to do, and in the crunch of time we try to get as many questionnaires completed as possible. But this year, pre-caucus telephone surveys show a difference in how, at least on the Democratic side, people are planning to vote by age. Younger people are more likely to support Obama, older Democrats more likely to support Clinton. We have decided this time to add a non-response adjustment to the entrance poll as well.
Q. So is that going to require additional people looking over the shoulder of the interviewer, or are they just going to have a little more to do?
Lenski: They're just going to have a little more to do. We do have two interviewers at each precinct for the entrance polls, again because of the crush of the number of interviews to be conducted in a short period of time. For our typical exit poll, there's only one interviewer all day. But for the entrance polls we always have two, so this will just be another task for the two of them to divide.
[Editor's Note: Andrew Kohut of the Pew Research Center did a longer interview with Lenski recently that went into more detail about the exit polls planned for the February 5 primaries and the rest of the year].
By Mark Blumenthal on December 28, 2007 8:51 AM
| Permalink
| Comments (6)
December 27, 2007
By Mark Blumenthal
As we expected poll releases to be a little slower than usual this week, we decided to conduct one of our own. Last week, we invited about a hundred professional pollsters who work for the news media and political campaigns to participate in a brief survey about the public polls in Iowa and New Hampshire. We will be releasing results from our "Poll of Pollsters" over the next few days. Today I want to discuss the very last question we asked, which concerns the same topic I blogged on yesterday: The reliability of polls conducted this week, between Christmas and New Year's Day.
First, some details about the survey. We invited two categories of pollsters to participate, those that work for political campaigns and those that conduct surveys for news media outlets. We selected campaign pollsters that had been listed in a scorecard published by the National Journal's Hotline that had worked on behalf of candidates for Governor, Senate or U.S. Congress in 2006. The media pollsters included those who conduct the well-known national polls that we regularly track and the organizations that had released surveys in Iowa and New Hampshire.
We sent out email invitations to just over 100 individuals, and as of yesterday a total of 46 had completed the entire survey (21 media pollsters and 25 campaign pollsters; of the latter category 14 were Democrats and 11 Republicans). As with any other survey, we pledged to keep respondent identities confidential and report their answers only as aggregated data. The pollsters responded online, using an online survey form prepared by our sponsor/owner, Polimetrix.
We also want to be very clear on one point: This survey is not a "scientific" sampling of any population beyond the 46 pollsters that replied. Thus, there is no "margin of error." The results represent no more and no less than the opinions of 46 individuals that were willing to respond. Since we made it clear that we would ask about the public polls in Iowa and New Hampshire, I assume that many of the non-participants are simply less interested in the topic. As one prominent campaign pollster who emailed his regrets put it, "in this case I feel like an ill-informed respondent." So the 46 we heard from are probably among the most opinionated on this subject.
So let's start with the topic of polling between Christmas and the New Year. Inspired by the quotations from pollsters in this recent Associated Press article, we decided to ask pollsters choose between two statements about the challenge of polling this week. But rather than forcing them to agree entirely with one statement or another, we asked the pollsters to pick a point along a scale to indicate which statement they agreed with more. The scale they used is reproduced below. They could use their mouse to click on any point along the scale (you can click on it to see a full size version):
There has been a lot of talk about the difficulty of polling during final week or so before the Iowa caucuses that includes both Christmas and New Years Day. Some believe that polling in this period will be challenging but can be done reliably. Others believe that polls done in this period will be so unreliable that they should be ignored. What about you?
Please click or drag on the ruler below to indicate which statement you agree with more.
The pollsters' responses spread across the scale as shown in the chart below. Not one signaled total agreement with either statement by moving the slider all the way to the left or all the way to the right of the ruler. They divide almost evenly in terms of which statement they lean to, with 48% placing themselves closer to the notion that polls "can be done reliably" this week, and 50% placing themselves closer to the statement that polls this week are "so unreliable they should be ignored" (note: one pollster/respondent clicked "not sure," three skipped the question completely).
However, the skeptics hold stronger opinions than the believers. Compare the two bars at the far left of the chart above (representing 17% who have the most doubts about the reliability of Christmas week polling) to the two bars at the right (5% who have the last doubts). For purposes of tabulation, we have translated the hash-marks along the ruler into a 100-point-scale, where 1 is "should be ignored" and 100 is "can be done reliably." The mean score for the full sample is 48 and the median 45, indicating a modest skew toward "should be ignored" (for the truly wonky, each of the bars in the chart represents a decile, 1-10, 11-20, 21-30 and so on).
As the second chart (below) indicates, the media pollsters are more skeptical than the campaign pollsters. The average score for media pollsters is a 43 (with a median of 29). In other words, the media pollsters cluster near the "so unreliable they should be ignored" end of the scale, while the campaign pollsters divide more evenly, with an average score of 50 and a median of 48. [For those interested in more details, click here for a spreadsheet with the number of responses for each increment clicked on the 1-100 scale].
Think of it this way: Even professional pollsters are not sure what to make of the surveys done this week. While few advise ignoring these surveys altogether, most recommend treating the results with a lot more caution than usual. Their uncertainty probably reflects the overall lack of experience our profession has with surveys conducted during the latter half of December. Just about everyone sampled recommends treating the results from this week with a lot more caution than those available at other times, although few would advise that we ignore these results completely.
We will have more results from our "Poll of Pollsters" over the next few days.
By Mark Blumenthal on December 27, 2007 11:15 PM
| Permalink
| Comments (3)
December 26, 2007
By Mark Blumenthal
Those of us with memories of the Apollo moon missions will remember those moments when the orbiting command module went around the so-called "dark side of the moon."** At that point the moon itself physically blocked broadcast signals, and despite all the dazzling "space age" telecommunications on display, live transmissions from the astronauts came to a total halt. While Walter Cronkite stalled for time, we watched and waited for the astronauts to regain contact.
Right now, with interest peaking in Iowa and New Hampshire polls, we have entered into our own dark side of the moon period. Unlike the moon missions, however, we are not in a total blackout. At least one pollster has released an Iowa survey conducted over the weekend, and others will surely follow later this week. However, with so many Americans traveling away from home for holiday travel, those surveys will be of unknown reliability, at best. The worst-case scenario (for the pollsters) will be if these surveys indicate a false trend, a shift that is less an indicator of real change than an artifact of respondents missing due to holiday travel.
Unfortunately, the survey research profession has relatively little experience with surveys conducted in the week between Christmas and the New Year (and the weekends surrounding that week). All of the polling firms I worked with rarely fielded surveys in the second half of December and typically shut down altogether between Christmas and New Years. CBS News Polling Director Kathy Frankovic reports in her column this week that the Roper Center archives include no public polls conducted between Christmas and New Years for either 2006 or in 2003 (other than one continuously running financial monitoring survey).
The reason that pollsters typically avoid polling around the holidays is the assumption that a big chunk of the population is away from home and unavailable for survey calls. As with any sort of "non-response" problem, we risk getting skewed or biased results if the missing respondents are both numerous and different in terms of their political views from those at home when we call.
What kinds of voters might be missing right now? Three years ago, in a survey concluded a full week before the holidays (12/15-17/2004), the Gallup Organization asked a national sample of 1,002 adults whether they planned "to travel more than fifty miles from home this holiday season." Twenty-eight percent (28%) said "yes." More important, as the table below indicates (based on data drawn from the Roper Archives), those planning holiday travel had a very distinctive demographic profile. Holiday travelers were much more likely to be younger and better educated. Notice also that holiday travelers were not just college students. Adults between the ages of 30 and 44 are twice as likely to travel for the holidays than those over 65. (Also, while I do not show it here, the pattern in these results by age, education and income was nearly identical for Democratic and Republican identifiers).
So pollsters fielding surveys this week are going to have a harder time finding younger, better educated respondents. Why is that important? Because in the Democratic race, at least, there are huge differences in vote preference by age and education: In virtually every survey, including those in Iowa and New Hampshire, Barack Obama does best among younger, better educated voters while Hillary Clinton's base of support is older and less well educated. Consider the data from the Boston Globe/University of New Hampshire survey conducted last week: Obama wins the support of 47% of voters age 18 to 34, but only 22% of those over 65; 40% of those with graduate degrees but only 25% of those without a college degree.
Given the age and education pattern, a sample with fewer voters under 45 or fewer with a college education will skew in Clinton's favor. Pollsters who see a demographic shift may choose to "weight" their samples to match the demographics of pre-holiday polls. That approach may lessen the bias but not correct it entirely. Weighting by age and education in this instance essentially replaces the younger, college educated voters who are away from home with other younger, college educated voters who are available to do the survey. If travelers and non-travelers are still different politically, regardless of age or education, then some bias will remain. And even if not, an extreme "weighting up" always increases the statistical error, in effect reducing the sample size.
Do we have any evidence that holiday travel bias might affect vote preference? Maybe. Consider the data below reported by Rasmussen Reports just before and after the Thanksgiving holiday.
Rasmussen runs a daily, rolling-average national survey that tracks presidential primary vote preference. Rasmussen uses an automated interactive-voice-response (IVR) methodology in which respondents answer recorded questions by pressing keys on their touch-tone phones. Each night, they call roughly 175-200 likely Democratic primary voters and roughly 150-160 likely Republican voters, then roll together and report a rolling average of the last four nights of interviews. Thus, each of the points on the chart below represents 750-800 likely Democratic voters with a reported margin of sampling error of +/-4%. The trend line is a regression estimate that Charles Franklin created for me using the Rasmussen data.
Notice what happens to the Obama trend line just before and just after Rasmussen took a five-day break from interviewing, Wednesday through Sunday, over the Thanksgiving weekend (Thanksgiving was November 22). The regression trend line essentially splits the difference between the 17% low points immediately before and after the Thanksgiving break and the much higher 26% results that came just before and just after that. What one makes of the variation may be in the eye of the beholder. Either the Obama trend got unusually erratic in both directions during the last two weeks of November, or there was a very unusual and precipitous plunge from what should have been a plateau around 24-26 to 17% centered on the period of Sunday, Monday and Tuesday before Thanksgiving and the Monday just after. Either way, the Obama variation around Thanksgiving was highly statistically significant. It did not occur by chance alone. Either there was a see-saw in Obama's real world support that week, or something changed in the kinds of voters Rasmussen sampled.
Given what we know about the demographics of holiday travelers and Obama's supporters, I'd bet on the latter.
Now I should point out that the Rasmussen national tracking may be a special case. According to Scott Rasmussen, each daily sample is essentially "fresh." Unlike many other pollsters, they do not attempt to call back unavailable respondents on successive nights. If they sample your phone number on Monday and you are not home, they will not call you back again on Tuesday. As such, their surveys may be more prone to a holiday effect than others that do more callbacks. And while I believe that Rasmussen weights their samples by gender, age and race to force consistency for each four-day report, they may not weight by education.
All of this brings us to the survey that the American Research Group released on Monday fielded between Thursday December 20 through Sunday December 23, a survey that shows Clinton gaining and Obama falling. Some will read this post as an attempt to debunk that result, and the findings above certainly argue for considerable caution in reading results from any survey this week. But the problem in trying to assess the ARG poll is that we know so little about it. Does ARG make call-backs to unavailable respondents? What was the sample composition on any ARG Iowa survey this year in terms of age and education level, and was this one suddenly different? Did ARG weight the results by age or education this time, and if so, by how much? We are in the dark on all of these questions.
It is also worth remembering, as some commenters noted yesterday, that real changes may be occurring in vote preference this week even if surveys may be severely challenged in their ability to measure it. Clinton may be gaining and Obama falling. So it is quite a leap for anyone to say they know conclusively that the ARG result is either right or wrong.
The hard truth is that we are behind the dark side of the moon this week, and we may not know much with certainty until next Wednesday night.
**Technically, the moon does not have a "dark side." Although the more appropriate term is "far side" of the moon, I still prefer the Pink Floyd version.
By Mark Blumenthal on December 26, 2007 3:34 PM
| Permalink
| Comments (9)
December 24, 2007
By Mark Blumenthal
On behalf of Eric, Charles and our families, we wish you a Merry Christmas, Happy Holidays and a season of Peace.
We will be back on Wednesday, obsessing over the latest results as we usually do. See you then.
By Mark Blumenthal on December 24, 2007 9:49 PM
| Permalink
| Comments (2)
December 23, 2007
By Mark Blumenthal
Everyone is chewing over the results of the new Boston Globe/UNH survey this morning. One sentence in the Globe write-up caught the eye of both Noam Schieber and Ben Smith:
One aspect of the healthcare debate that has divided Democratic candidates is whether individuals should be required to purchase coverage - Clinton and Edwards favor a mandate, while Obama does not. A slight majority of Democratic voters who were polled - including pluralities of Clinton and Edwards supporters - opposed such a requirement.
Scheiber took it a step further:
I wondered a few weeks ago why the Clinton campaign was going negative on Obama's character instead of hitting the healthcare issue, which seemed less fraught and likely to pay dividends on both a policy level and a preparedness level. This poll hints at an explanation: Maybe it just wasn't working.
Or maybe it risks a backlash with Clinton's base. Buried in the UNH crosstabs is a pattern I had not seen before. Opposition to the notion of an individual health insurance mandate -- "should individuals be required to buy health insurance" -- is greatest among the less well-educated and downscale voters that are the core of Clinton's base in New Hampshire and elsewhere:
The Associated Press reported a few weeks ago that the Clinton campaign "is preparing television ads here criticizing Barack Obama's health care plan" to air in New Hampshire in the event the Obama campaign "ignites in Iowa." Most surveys show that Democrats perceive Clinton as better able to handle the issue of healthcare, and the argument made by the Clinton campaign, that Obama's plan "could leave as many as 15 million people uninsured" no doubt scores well in message testing surveys. The data above, however, suggest that such an attack risks a backlash if the debate engages on the issue of "requiring individuals to buy health insurance."
By Mark Blumenthal on December 23, 2007 11:57 AM
| Permalink
| Comments (10)
