Claim: Gallup Tracking Poll May Be Flawed
A lot of pundits rely on the Gallup poll as the standard national tracking poll — but Mark Blumenthal has noticed an interesting quirk in its results. It turns out that the poll has a definite "day-of-the-week" effect: Hillary Clinton has consistently done better in samples from the middle of the week, while Barack Obama has benefitted from samples late in the week or on weekends.
The quirk, says Blumenthal, suggests that the poll may not be a reliable indicator of day-to-day movement — even though pundits have been spinning whole theories based on these volatile micro-shifts.
We're going to check in with Gallup about this, and keep you posted.
Advertisement















Cool. Maybe now Obama isn't winning in delegates and popular vote after all!
March 31, 2008 6:18 PM | Reply | Permalink
March 31, 2008 8:11 PM | Reply | Permalink
Amidst all this Galling news (no great thanks to Eric, the self-proclaimed poll analyzer of TPM), Obama nets one more delegate in MS thanks to the diligent work of a math wizard and a prodigious blogger called thevoiceofreason aka Jeff in democraticunderground.com. It is official now. Obama is 20-13 in MS instead of 19-14.
Read and weep, Eric.
http://marcambinder.theatlantic.com/archives/2008/03/another_pledged_delegate_for_o.php#comments
(You also have relevant inner links to democraticunderground.com in the replies by Jeff in Ambinder blog).
Go Jeff aka thevoiceofreason.
April 1, 2008 4:07 AM | Reply | Permalink
It could be that Clinton attacks during the week and Obama tends to release good news later in the week.
Curious none the less.
March 31, 2008 6:19 PM | Reply | Permalink
Proof of its accuracy is in the pudding - Obama was tanking in Gallup numbers during the Wright flap and surged ahead after his race speech and the Tuzla flap. if that was just quirks, that's pretty darn coincidental.
March 31, 2008 6:19 PM | Reply | Permalink
It is not a bug, it is a feature!
March 31, 2008 6:23 PM | Reply | Permalink
From one gamer to another, high five.
March 31, 2008 7:48 PM | Reply | Permalink
One more reason for folks to stop waving this tracking poll around. 40 states have voted. Let it go....
March 31, 2008 6:24 PM | Reply | Permalink
Funny.
This made sense:
The results also suggest that we would get a less "volatile" sense of the race by looking at a seven-day rolling average that eliminates the apparent day-of-week effect.
I guess the next step is to look at coverage of the candidates, and whether there are systematic trends in coverage: better for Hillary over the weekend, better for Obama during the week?
I'd start there before conjecturing about the work habits, lifestyle habits, etc, of the supporters.
March 31, 2008 6:27 PM | Reply | Permalink
Scan today's TPM headlines.
TPM = Clinton surrogate.
Mention Kentucky polls when its 2 months out? Check
Mention any polls in which Obama is ahead? nope
Accuse polls with Obama being ahead? Check
Too bad they haven't had any Wright today.
March 31, 2008 6:32 PM | Reply | Permalink
Hmmm...
I'm thinking the folks over at Gallup have been using the same basic polling model for the entire campaign season.
I'm thinking I haven't heard anything about them changing the way they poll, especially for their daily tracking.
So I'm thinking that it's interesting that now that Obama is in the lead, the poll is flawed. But it wasn't flawed when Hillary was leading earlier this year.
That's what I'm thinking...
March 31, 2008 6:36 PM | Reply | Permalink
Survey USA is credible.
Rendell is on Hardball saying Gallup is not credible.
March 31, 2008 7:05 PM | Reply | Permalink
Wow, things are really changing. Watched Lou Dobbs tonight and he was actually saying how biased the media has been against Hillary. 73% of the viewers supported him in a poll asking is it true that the Media is Biased against Hillary.
Lou made the Obama pundit look like a complete fool when he was calling on Hillary to quit. Lou was correctly saying that the Obama Campaign was trying to disenfranchise Florida and Michigan voters along with the remaining ten states still to vote. The Obama guy was whining away like a typical Obama Whiny Whiner and Lou said "do you mind if I finish?" It was classic. The race is turning. I can feel it. It was a HUGE mistake to try to slime Hillary into dropping out.
March 31, 2008 8:14 PM | Reply | Permalink
I feel it too.
It backfired and now they do not want her to drop out.
Superdelegates are waiting because she will have the momentum after Penn.
Folks are rethinking Obama and McCain.
March 31, 2008 10:01 PM | Reply | Permalink
Pensylvania is the next Texas. Her lead in the polls shrinks as the primary aproaches. She wins the popular vote by a slim margin. And she loses the delegate count.
April 1, 2008 9:32 AM | Reply | Permalink
I'm sure all 500 people in PA who watch Lou Dobbs see it the same way you do. Hillary really is getting beat up by all those horrible men, isn't she? Where is Lorena Bobbitt when you need her, anyway?
I definitely thnk the race is turning on this issue of Pat Leahy asking Hillary to get out of the race last Friday. I know most people hang on Pat's every word, and when he pisses them off, well, millions are moved, that's all I can say. I mean, people are really steamed. So much so, that;s all you hear anybody in the Clinton campaign say" "stop bullying us, stop trying to get us out of the race" -- even though her opponent isn't doing it, it's an attack that really resonates . . . . with the same people who want to her to stay in through Denver even if she is 200 pledged delegates behind in June.
It should be interesting to see how well the Clinton campaign operates without money, since they seem to be out of it. That's the problem when you depend on a smaller number of $2,300 donors -- eventually, it's not that they don't want to give you more, it's that they can't. But hey, details details -- why worry about that when you can attack people telling her to do the party a favor and give up her quixotic quest. The nerve!
March 31, 2008 10:20 PM | Reply | Permalink
This is EXCELLENT TYPING by DEMBILIC!!!!!
March 31, 2008 11:36 PM | Reply | Permalink
Rendell also said Fox is most unbiased news network.
Rendell's opinion is no better than Rove, O'Reilly or Hannity as far as I'm concerned.
...unless he ends up Obama's running mate. Then I can be convinced that he was misled.
March 31, 2008 8:19 PM | Reply | Permalink
Ummm... Haven't many of the reader comments right here on TPM said this same thing for quite some time now? Evidently Blumenthal is now the go-to guy. TPM - you could have just read the comments on your own website to find this out.
March 31, 2008 7:10 PM | Reply | Permalink
Now I know that TPM is not to be trusted anymore than you can trust the Butcher that keeps his thumb on th scale.
Here is what Eric Omitted from the report;
UPDATE: Hold the, er, phone. I sent Hickman and Margolis a spreadsheet with the Rasmussen Reports daily tracking and they generated comparable charts. Oddly enough, the pattern is almost the mirror opposite of Gallup. On the Rasmussen automated tracking (which reports a four-day rolling averge), Obama does consistently better in the middle of the week, Clinton better on the weekends. Here is the chart of each week:
And here are the average values for each daily release:
I will have a chance to add more thoughts later, but I think the bottom line is that the conflict of the patterns shown by the two pollsters suggests that the apparent day-of-week effects are about differences in poll methodology, not a real variation in voter preferences over the course of the week.
-- Mark Blumenthal
March 31, 2008 7:12 PM | Reply | Permalink
Blumenthal doesn't say the polls are "flawed" - in fact he offers no reason for the variation. In a follow-on update, he notes that the Rasmussen tracking poll shows the opposite effect, with Clinton up on weekends and Obama doing better in the middle of the week. Blumenthal hypothesizes that this demonstrates that the variance reflects methodology differences rather than a real change in voter preference over the course of the week.
None of this though is to say that day to day results are not accurate, rather that there appears to be some noise in the statistics that might be eliminated if Gallup used a seven day average. And to be fair to Greg, he isn't saying anything about whether Obama "really" has a lead. It's an interesting quirk pointed out by the leading poll blogger. Jeesh.
March 31, 2008 7:24 PM | Reply | Permalink
This thread is not by Greg.
March 31, 2008 7:36 PM | Reply | Permalink
And I am not a close reader.
Thanks.
March 31, 2008 7:54 PM | Reply | Permalink
yeah, i knew that long ago.. hee hee hee. You just found out??
March 31, 2008 7:32 PM | Reply | Permalink
RaeKKK in the Hood.
March 31, 2008 7:37 PM | Reply | Permalink
I wish this race would just Gallup away..........
March 31, 2008 7:39 PM | Reply | Permalink
Jeez. When she's ahead in the tracking poll by a few points, TPM makes it the front-page headline. When Obama is up by unprecedented double digits, there is no front-page headline, and a backstory about how the poll is flawed.
I'm not arguing the poll is good--it's probably not. However, it's not changed it's methodology since Clinton was in the lead and TPM was front-page headlining it. So what gives?
Josh? Greg? Eric?
March 31, 2008 7:53 PM | Reply | Permalink
Strangely enough I have often thought I detected the opposite effect in the Rassmussen tracking poll. It seemed that the lowest poll numbers of the week for Obama would almost always fall on Saturday or Sunday. Haven't done a statistical study or anything but that was my impression. Thought maybe it was caused by all the young Obama supporters not being at home on Friday night to answer the phone.
March 31, 2008 7:56 PM | Reply | Permalink
What i hate is that they use land line phones...
Its been years since ive owned a land line phone.
March 31, 2008 8:10 PM | Reply | Permalink
I've wondered about this, too. Lots of people without landlines, which means the polls are missing lots of younger people and urbanites. And some middle-aged suburbanites like me. Neither I nor my partner have had a landline in years. His three adult daughters have never gotten landlines, and I don't think my adult nephews have either.
Does anyone know how or if pollsters compensate for this?
March 31, 2008 8:42 PM | Reply | Permalink
No idea, but i can honestly say that the only people i know who still have a land line are my grand parents. Who are republicans. (they Hate Hillary and dislike McCain, so ill be working on getting their vote for Obama)
March 31, 2008 10:05 PM | Reply | Permalink
word
April 1, 2008 3:27 AM | Reply | Permalink
We definitely established this a while ago...
Clinton performs well in polls released on Tuesday and Wednesday because they are collected over the weekend. Clinton voters (the so called beer-track blue collar workers) are likely to be home on Friday and Saturday nights in order to answer the phone. Whereas Obama voters are out at the Opera or at the ball or whatever on the weekends and are therefore unavailable to pollsters.
I'm pretty sure you posted on this around Texas and Ohio, because the polls fluctuated then too.
March 31, 2008 7:59 PM | Reply | Permalink
Come on did you really think this was not coming.
Small states that Obama wins does not count
Big states Obama wins does not count
Caucus states that Obama wins does not count
Primary states that Obama wins does not count
People who vote for Obama don’t need president so they don’t count
Supper delegate that endorse Obama are insignificant
Why would you think that Gallup (once it showed Obama leading) would count?
Expect Hillary’s camp to come out against realty because reality DOES NOT COUN.
Gallup has made itself irrelevant even though Mark Penn was pointing to it when Obama was down during the crazy pastor flap
March 31, 2008 8:56 PM | Reply | Permalink
Wait, obsessing over every slight fluctuation in the polls every day might be stupid and misleading? That's crazy talk.
March 31, 2008 9:27 PM | Reply | Permalink
Eric's managed to spin his entire young career out of "micro-shifts"
8-10 points isn't a "micro-shift"...Slept through Charles Franklin's lectures??
March 31, 2008 9:32 PM | Reply | Permalink
Lets see, Obama bowled a howling 37 and blamed it on drinking beer with Casey.
Clinton talks to outraged truck drivers about high gas prices and attack's Paulson's regulation plan that deregulates.
Al-Sadr shows who controls Iraq but mcbushie is on a biograghy tour.
Looks like Clinton is the only one working hard for change as usual.
Issues people, I know you like to read the skewed polls and listen to the biased pundits but it is about the issues.
March 31, 2008 9:34 PM | Reply | Permalink
you should definitely point to one day on the trail and take it as indicative of the entire campaign.
How about the fact that Clinton's campaign manager worked for a sub-prime lender? Or the fact that she hasn't paid her employees health insurance? Or the fact that she, while striking a populist note in hopes of gaining an Edwards endorsement, has failed to pay back small businesses which put on events for her ($25,000 dollars is a lot of money for a small business).
And perhaps the most outrageous debt of all, thousands of dollars owed to a charity. A charity!!
Clinton may be talking a lot about fixing people's problems, but her campaign is only exacerbating them.
So yeah, Obama went bowling, but at least he paid for the shoes up front.
March 31, 2008 10:03 PM | Reply | Permalink
These are the issues.
April 1, 2008 1:13 AM | Reply | Permalink
Finally, talk about the issues...
April 1, 2008 1:13 AM | Reply | Permalink
Maybe hillary should pay the health care premiums for her own employees, oh wait she is out of money do to overspending and mismanagement, my bad.
April 1, 2008 3:30 AM | Reply | Permalink
I can feel it too Dembillc!
Oh wait, that was just my lunch coming up listening to spin, spin, spin.
March 31, 2008 9:38 PM | Reply | Permalink
When we take over this blog after Clinton wins, they will be the trolls.
March 31, 2008 10:03 PM | Reply | Permalink
Who's drinkin' the Kool-Aid™?
April 1, 2008 1:16 AM | Reply | Permalink
looking forward to it.
April 1, 2008 10:18 AM | Reply | Permalink
Off topic:
Mississippi recalculated and Obama gains another delegate and Hillary loses one. According to Ambinder
Texas: +5 Obama.
+7 delegates without contests. Not a bad week for Obama.
March 31, 2008 10:04 PM | Reply | Permalink
I think its pretty simple, really. Since Hillary's numbers do better between Tuesday - Thursday, rather than between Friday - Monday (when Obama does better), what that shows us is that Hillary is more electable, since Election Day takes place on a Tuesday. I can't believe you people were so ignorant.
March 31, 2008 10:11 PM | Reply | Permalink
Comparing the day-of-the-week change is functionally identical to a seven-day rolling average, and involves much simpler math.
I mean Tuesday this week to Tuesday last, weds to weds, etc., for whatever the latest day is. If Obama is up two points from the week before, that's what the rolling average will also say.
March 31, 2008 10:11 PM | Reply | Permalink
"Obama's Alliance With McCain, And How It All Fell Apart";
http://www.washingtonpost.com/wp-dyn/content/article/2008/03/30/AR2008033002401_pf.html
Ah, the unity lasted a week.
When Clinton wins, you will be the trolls.
March 31, 2008 10:19 PM | Reply | Permalink
"When Clinton wins..."
It's kind of sweet. You're like the little engine that could.
April 1, 2008 9:03 AM | Reply | Permalink
I see you just changed your picture. Much better!
April 1, 2008 9:09 AM | Reply | Permalink
The discussion here assumes that the poll "has a definite 'day-of-the-week' effect", and it's based on graphs that Blumenthal says show an "impressively consistent pattern". I've looked at those graphs and I just don't see the pattern through the noise. Sometimes the change day to day is up for Clinton; other times it's up for Obama.
Why is it appropriate to read anything into this, aside from statistical noise or all-too-human propensity to see pattern?
March 31, 2008 10:32 PM | Reply | Permalink
Wow, I am so blown away at the bigots spewing forth their venom, under the guise of no experience (like that matters, last Republican to have a balanced budget was Ike), Obama's pastor, like he can control the guy, and Obama supposedly bullying poor HRC, who spews hateful comments towards him every day. Today he is disenfranchising voters per HRC (and yes, I am a white woman) so, go ahead, vote for McCain, who has had melanoma, which quite often reappears down the road; he is having senior moments, all but 14 percent of people his age show some dementia, that is a medical fact. His wife broke federal drug laws and walked away with a swat. He has changed his positions more often the weather, he will give us no health care relief, no budget balance as he will continue with tax cuts and war, and he was thick in the Keating thing, barely survived. He cheated on his first wife several times, his words, married his biggest affair who is now his wife, has violated FEC rules, and on and on. No one with any conscience and love for this country could vote for him, and yes, I lived in Arizona, I do know the man. Yet patriots say if Obama wins the nomination, they will vote for McCain. That is just a lie, they were never voting for a Democrat if that is the case.
March 31, 2008 11:50 PM | Reply | Permalink
Relax, everyone. The key word is "apparent" day-of-week effect.
At their best, polls are fuzzy pictures taken of moving objects in dim lighting. Sampling size and methodology give a theoretical "margin of error," but you also have real humans asking other humans in a variety of locations their likes and dislikes about another batch of humans, based on a constantly changing set of facts and impressions.
So the resulting data are approximations, more or less credible "19 times out of 20." You must assume every 20th poll to be outside the margin of error; you just don't know if the current one is it.
All this to say it's no surprise polls are often "wrong," or that different ones give widely different results. That's the reason rolling-average tracking polls were devised -- to smoothe out the inevitable anomalies.
What's being tried here is the exact opposite -- to tease out a statistically significant variation from data that isn't up to the job. For one thing, we're looking only at polls from February and March: eight full weeks to compare with each other. Too small a sample to establish any real day-of-the-week "pattern."
In both Gallup and Rasmussen results, the difference between the averages for Wednesdays and for Saturdays is about four percentage points -- in one case favoring Clinton, in the other Obama. But if you give the two companies equal weight, add their daily stats together and average them (imagine the pollsters merging), virtually all daily differences disappear.
No need to seek an explanation in media coverage; there is simply no phenomenon to explain.
Nothing to see here, folks; move along.
March 31, 2008 11:57 PM | Reply | Permalink
Hey did anyone catch Obama Bowling?
He is really comically bad.
I actually give him points for being so bad.
April 1, 2008 12:28 AM | Reply | Permalink
We find your posts endearing for exactly the same reasons.
April 1, 2008 10:20 AM | Reply | Permalink
It is looking like Obama really should pack it up if he loses Kentucky.
April 1, 2008 12:29 AM | Reply | Permalink
Didn't you just say that Obama's call for Hillary to quit backfired???
Now you're asking him to quit?
Is delusion a learned trait or were you born with it?
April 1, 2008 9:58 AM | Reply | Permalink
"Barack Obama has benefitted from samples late in the week or on weekends"
When when everyone is already drinking and could give a hoot. Sober people Vote for Hillary.
April 1, 2008 12:47 AM | Reply | Permalink
Media, media, and media.
Study the stories coming out of CNN, etc., and you just might notice an Obama's turn to squirm, now Clinton's, now Obama's, quality to the reporting. I've noticed it for weeks now, and based on last night's reporting, this is a pro-Clinton week, barring any huge and newsworthy surprises. So, figure that up until Friday, you'll hear the word "comeback" a great deal.
Totally unscientific and silly, I know, but Gallup may simply be more accurately reporting the emotional roller coaster ride the media has had us on since mid February.
April 1, 2008 3:15 AM | Reply | Permalink
Any chance of incorporating the full article you link to in your analysis, instead of just the half you feel proves some sort of point.
April 1, 2008 3:26 AM | Reply | Permalink
Hey "dembillc", I say, I say, boy, lay off the tokes.
April 1, 2008 4:20 AM | Reply | Permalink
Thanks to a comment in a Kos thread, I went to Drudge for the first time ever just now to see if the heads up was true.
Fwiw, he's saying that Rasmussan later today for PA will come out and have it 47 Clinton 42 Obama.
I would guess even TPM election central would be forced to note that poll. Maybe. I am still stunned they gave that kind of run to a KY poll and ignored the NC poll at the same time.
April 1, 2008 7:57 AM | Reply | Permalink
Thanks to a comment in a Kos thread, I went to Drudge for the first time ever just now to see if the heads up was true.
Fwiw, he's saying that Rasmussan later today for PA will come out and have it 47 Clinton 42 Obama.
I would guess even TPM election central would be forced to note that poll. Maybe. I am still stunned they gave that kind of run to a KY poll and ignored the NC poll at the same time.
April 1, 2008 8:03 AM | Reply | Permalink
Sorry for the double post, I can't stand the delay in this system, where sometimes it eventually posts, and sometimes it doesn't.
April 1, 2008 8:22 AM | Reply | Permalink
Since Eric has opted to cherry pick what Mr. Blumenthal actually said, you need to read the entire article. Here it is:
March 31, 2008
Day-of-Week Effect in Gallup Daily?
I received some interesting charts this afternoon from Harrison Hickman and Ben Margolis, both of the Democratic firm Global Strategy Group that polled for John Edwards until he withdrew from the race (full disclosure: Hickman was my employer longer ago than either of us wants to admit). When Hickman and Margolis plotted the Gallup Daily results for the Obama-Clinton race, they noticed an impressively consistent pattern by day of the week. In data released during February and March, Obama typically does best on three-day samples that end on Saturday (combining interviews from Thursday, Friday and Saturday), while Clinton typically does best on samples ending on Tuesday or Wednesday night (covering Sunday through Wednesday).
The chart below shows the pattern in the Clinton lead (Clinton minus Obama) for each of the last nine weeks:
Hickman and Margolis sent a second chart which shows the average for each daily release across the nine-week period. On average, the results have shown a more than three point shift from Sunday-Monday-Tuesday (Clinton ahead by 1.3) to Thursday-Friday-Saturday (Obama ahead by 2.6).
03-31dayofweek_avg.png
So, given this pattern, do not be surprised if the ten-point Obama lead reported yesterday (based on a interviews from Thursday through Saturday) narrows a bit more by mid-week from the eight -point lead reported today. The results also suggest that we would get a less "volatile" sense of the race by looking at a seven-day rolling average that eliminates the apparent day-of-week effect.
The more interesting question is, why are we seeing this pattern? It seems counterintuitive, at least at first blush. I would have expected Obama's supporters to be harder to reach on weekends, given that they tend to be younger than Clinton supporters. Back in December, Obama's supporters seemed to be harder to reach around Thanksgiving and the Christmas holiday. Unless I am missing something obvious, this pattern is different.
Any thoughts? Theories?
UPDATE: Hold the, er, phone. I sent Hickman and Margolis a spreadsheet with the Rasmussen Reports daily tracking and they generated comparable charts. Oddly enough, the pattern is almost the mirror opposite of Gallup. On the Rasmussen automated tracking (which reports a four-day rolling averge), Obama does consistently better in the middle of the week, Clinton better on the weekends. Here is the chart of each week:
And here are the average values for each daily release:
I will have a chance to add more thoughts later, but I think the bottom line is that the conflict of the patterns shown by the two pollsters suggests that the apparent day-of-week effects are about differences in poll methodology, not a real variation in voter preferences over the course of the week.
UPDATE 2: Both in comments and in email, readers have questioned the statistical significance of the patterns in the charts above. I am persuaded that they have a point.
One argument many are making, with good cause, is that the values for adjacent days are not statistically independent. In other words, the value plotted for each day for Gallup represents a rolling average of three days of data (and for Rasmussen, four days). So it is not surprising to see smooth day-to-day trends. That's the point of the rolling average, something I've observed previously.
We can, however, isolate two independent measurements each week for Gallup from the available data. For example, there is no overlap between the data collected on Sunday, Monday and Tuesday (labeled "Tuesday" on the chart above) and on Thursday, Friday and Saturday (labeled "Saturday). Looking at the first chart above, we see what certainly looks like a "consistent" pattern: In seven out of eight weeks, Clinton does at least a point better in the Sunday-Monday-Tuesday sample than than the Thursday-Friday-Saturday sample.
If random chance were the only factor at work, then the odds of seeing a difference one way or another between the two independent samples in any given week should be like flipping a coin. The odds of flipping a coin and having it come up the same way seven times (either heads or tails) is about 7% -- highly improbable but still slightly higher than 5%, the level of confidence we usually require to call something "statistically significant."
However, as my friend Mark Lindeman tells me, my test is sketchy in several ways. Tuesday was primary day in many of these weeks. The biggest drop on the chart, for example, occurs during the first week of February following Super Tuesday. If we throw out just that one week, then odds of the late week drop occurring by chance alone on the Gallup series increases to about 13%.
Moreover, if we look at all eight weeks, but compare the Clinton margin each Sunday-Monday-Tuesday sample to the preceding Thursday-Friday-Saturday, the consistency seen in the chart disappears: Four ups, two downs and two comparisons that show no change. The odds of getting that pattern by chance alone are not much better than 50-50.
Things are a bit more complicated with the Rasmussen data, since the four-day rolling average makes it impossible for us to isolate two independent measurements each week, but the pattern illustrated in the chart is less consistent by week than for Gallup.
Others may suggest better approaches to the statistics and, of course, patterns in future weeks will add further evidence one way or the other. However, I am convinced that these patterns owe more to random chance than I had first assumed. Hopefully, the analysts at Gallup and Rasmussen Reports, who can run tests using the independent daily samples, will help clarify.
-- Mark Blumenthal
March 31, 2008
April 1, 2008 10:46 AM | Reply | Permalink