Subscribe Subscribe   Find us on Twitter Follow POL on Twitter  



One more note on the election


A followup to my earlier post:

  • Nate Silver's column today has two throwaway sentences that reminds me of one more reason I'm cautiously skeptical of the polls:

    Some of the consistency in these results may reflect a tendency of polls to converge or "herd" around the polling average close to Election Day. This may occur because some polling firms alter their turnout models or other aspects of the polls so as not to produce outliers -- a dubious practice if the goal is to provide an objective take on the race.

    In other words, it's not the case that there are two dozen pollsters that are each deciding independently that Obama is ahead in Ohio. What we're seeing is the most self-confident handful of pollsters standing by their decisions, and less confident pollsters succumbing to peer pressure rather than produce outliers. To have faith in averaging (or even weightedly averaging) these results, one has to believe that the most self-confident pollsters who don't shift their results are also the most accurate and statistically unbiased pollsters. The Dunning-Kruger effect suggests that there is a good chance this might not be so. Thus, all it would take is a handful of obstinate pollsters with statistically-biased likely-voter models to produce a systemic statistical bias in the polls. It's not necessarily the case: the polls that fudge at the end of the cycle might be reducing, rather than introducing, error. But we're not seeing the wisdom of crowds here; the Bayesian probability that there's a systemic bias in the polls is substantially higher than if the pollsters were producing results independent of one another.

  • Josh Marshall responds to me and correctly points out that we're working with very very few data points. But that should just reduce our confidence that we know what's going on. Do the 1972 polls tells us anything about the 2012 polls? What about 1992 or 2000?

  • Silver looks at recent instances of state polls and finds they got the winner right the overwhelming majority of the time. Of course, as Silver recognizes quietly, that's not the correct question. How often were the polls 3 points off? One has to do the counting oneself, but it was nine times out of 29 in 2008 (albeit less often in closer state races).

  • Colby Cosh has the best short piece on the Nate Silver phemonenon I've seen yet.

  • Separately, thanks to Nate Silver for a generous tweet. He has a much larger platform than me, and is facing much more prominent critics than me with much stupider arguments; it would have been very easy for him to pretend I didn't exist and simply shoot fish in a barrel, and it's to his credit that he lent his twitter feed to promote my thoughts.


If we believe Nate, then he has discovered what no one else ever has. Namely that statistical analysis has reached such an advanced state that it can predict the future. That unto itself begs credibility.

I was reading some old articles on line. Nate was trashing Rasmussen as a totally worthless polling outfit back in January, 2011. About a week before the Wisconsin gubernatorial recall in June, 2012, Nate wrote there was a 95% chance Walker would survive the recall. Apparently he based this opinion on a small number of polls which utilized Rasmussen's methodology. He also didn't post an opinion until fairly late in the game. At least I can't find where he did until late May, 2012.

The present election boils down to this: If the Democrats get D +7 or D + 8 as they did in 2008, Obama wins. If it is D +3 of less, Romney wins. No one can predict what the turnout will be.

I am amused by all the posters on the NYT and other Democrat leaning sites. They hail Nate as being "objective" while decrying conservatives as "not understanding science and math." They repeat this meme ad nauseum.

I spent quite a while in grad school studying quantum mechanics. The two areas I worked in were computer modeling of protein folding and solving the Schrodinger equation. You would think these are straight forward problems. Put the charges, spaces, bond length, molecular size, geometry, etc into the computer and get an accurate answer. But it doesn't work out like that. Computers can't solve these problems because the math is too difficult. So one has to continuously adjust the programs, make guesses, assumptions and approximations to get a more accurate answer that agrees with observation in the lab.

As one of my sponsors used to say, "No one has ever proved Schrodinger to be wrong."

To which I would reply, "No one has ever proved him to be correct, either."

All this led to the obvious question: If these theories are so great, why do we have to keep tweaking the computer programs to get an answer that is 1% to 5% within empirical observation?

So I am skeptical of Nate's predictive value. Even if he gets it right this time, I am skeptical. I read on one blog you can get the same thing Nate is predicting by using a Yahoo spread sheet. At some point, he will fail.

If you read Scarne's big book on Dice, he makes a statement. "It is almost statistically impossible for some one to throw Boxcars six times in a row. But it happens every night somewhere on the Vegas Strip." I am quoting from memory. It was something like that.

He also stated there are some unlucky crap tables that lose money for weeks defying the statistics. It doesn't have to do with cheating, or stealing. It is just the way it works out in the real world.

You say: "Silver looks at recent instances of state polls and finds they got the winner right the overwhelming majority of the time. Of course, as Silver recognizes quietly, that's not the correct question. How often were the polls 3 points off? One has to do the counting oneself, but it was nine times out of 29 in 2008 (albeit less often in closer state races)."

Of those 9 instances, in six cases the actual results were more Republican than the polls, and in three cases the actual results were more Democratic than the polls. If the same proportions hold true for 2012, then the chance that polling in Ohio is 3% off in Obama's favor, so that Romney actually wins Ohio, would be 6/29, or 21%. 538.com says that Romney's chance of winning Ohio is 13%. Different, but not wildly so.

Alternatively, you also admit the poll error rate was smaller in closer races. In fact, in 2008 there were 8 races where the polls said the margin would be under 5% (AZ, FL, GA, IN, MO, MT, NC, OH), and in only one of those were the polls off by more than 1.1% - AZ with a polling error of 4.3% in favor of the Democrats. That would imply, if the same error rate occurred in 2012, only a 12.5% chance that the poll consensus is wrong about Ohio, all but identical to 538.com's 13% odds for Romney in Ohio. (For 2004, the comparable error rate was 2 close states that voted more than 3% more Republican than they polled, out of 13 states that pollsters thought would be within 5%, for a 2/13 = 15% chance of a pro-Romney outcome in Ohio; for 2000 there was one such error out of 10 states).

So if you're saying Romney has a 21% chance of winning Ohio, and with it the election, I might buy that, although the data you cite supports a 12.5% chance as well, and earlier-election data support a 10-15% chance. But anything stronger than a 21% chance for Romney is closer to wishful thinking, and not supported by data on polling errors in 2008.

Silver explicitly accepts a likelihood that the polls are wrong enough for Romney to still win. As he says -

"All of this leaves Mr. Romney drawing to an inside straight. I hope you’ll excuse the cliche, but it’s appropriate here: in poker, making an inside straight requires you to catch one of 4 cards out of 48 remaining in the deck, the chances of which are about 8 percent. Those are now about Mr. Romney’s chances of winning the Electoral College, according to the FiveThirtyEight forecast

As any poker player knows, those 8 percent chances do come up once in a while. If it happens this year, then a lot of polling firms will have to re-examine their assumptions — and we will have to re-examine ours about how trustworthy the polls are. But the odds are that Mr. Obama will win another term."

I am not sure that is a materially different view than yours just you are a bit more of a sceptic about the polls than he is.

And I think the amount of abuse the poor guy is getting is unwarranted. He's not the only one saying this, just the best and most rigorous one, which suggests the motive for attacking him more than the others are not exactly pure of heart.

Leave a comment

Once submitted, the comment will first be reviewed by our editors and is not guaranteed to be published. Point of Law editors reserve the right to edit, delete, move, or mark as spam any and all comments. They also have the right to block access to any one or group from commenting or from the entire blog. A comment which does not add to the conversation, runs of on an inappropriate tangent, or kills the conversation may be edited, moved, or deleted.

The views and opinions of those providing comments are those of the author of the comment alone, and even if allowed onto the site do not reflect the opinions of Point of Law bloggers or the Manhattan Institute for Policy Research or any employee thereof. Comments submitted to Point of Law are the sole responsibility of their authors, and the author will take full responsibility for the comment, including any asserted liability for defamation or any other cause of action, and neither the Manhattan Institute nor its insurance carriers will assume responsibility for the comment merely because the Institute has provided the forum for its posting.

Related Entries:



Rafael Mangual
Project Manager,
Legal Policy

Manhattan Institute


Published by the Manhattan Institute

The Manhattan Insitute's Center for Legal Policy.