Most people seem to love 538 and I do too. In the past 2 elections they have been correct with 99 out of 100 state predictions and have been very close in their numbers. However, I have felt my love for them diminish over the course of this election, starting with their primary predictions, which although correct in the end, were probably their weakest showing. Let me explain why.

The Assumptions - http://fivethirtyeight.com/features/election-update-why-our-model-is-more-bullish-than-others-on-trump/

In this article, Nate lines out the assumptions that may or may not change the predictions in the race. In the first, he claims that more independent and undecided voters are creating more uncertainty. However, studies and polls and his own research have indicated that there are actually very few truly undecided voters and independents almost always vote the same for one party or the other. So why not make that assumption?

He also says that his model is based on the 1972 election and lines out why. His reasoning makes sense on the surface, but why would it be more accurate to go with an election where state polls start to pop up, than the most recent election where you can make very clear correlations between polls and results?

He states that state polls are not individual in their errors, it is more of a country-wide trend that he is using. However, when you look at the polls he is using and how he weighs them, against the same outcomes from 2012, that doesn’t seem to be the way he is using them right now. You’ll see more of that below.

In his final assumption he flatly states that if they used those more recent polls, it would overwhelmingly make Clinton the prediction. He doesn’t really seem to indicate why they didn’t make this assumption, which is the part where things seem to fall apart, but I’ll go into that later.

The “Secret Sauce”

Nate has a proprietary secret sauce that he cooks into the formula to hopefully make it more accurate, and indeed over these two elections, it has been very accurate. But has it been MORE accurate than other polls? Not particularly. These past two elections were some of the most clear outcomes in the past 30 years (besides Bush/Gore), and most reputable polling sites (emphasis on reputable) had essentially the same outcome across the board. It would almost seem that based on the assumptions from above, that Nate is trying to cook in more uncertainty. But we don’t really know. It’s his blend and he’s right to keep it a secret. But let’s put these two issues together; the secret sauce and the assumptions.

It’s great that 538 has gotten so much right. But how ‘right’ were they? I broke down the 2012 predictions vs the actuals and applied those to 2016. And they are displayed here.

National 2012 Predicted/Actual =

50.8/48.3 D

51.0/47.2 D +3.9 D

Difference = -.2/-1.1

Applied to Current Prediction for 2016 =

48.7/45.0 D

48.5/43.9 D +4.6 D

Battleground States 2012

State: Predicted: Actual: Difference

CO :

50.8/48.3 D

51.5/46.1 D

+.7/-2.2

FL :

49.8/49.8

50.0/49.1 D

+.2/-.8

IA :

51.1/47.9 D

52.0/46.1 D

+.9/-1.8

NB 2nd :

47.5/51.8 R

45.8/53.0 R

-1.7/+1.2

NH :

51.4/47.9 D

52.0/46.4 D

+.6/-1.5

NC :

48.9/50.6 R

48.3/50.4 R

-.6/-.2

VA :

50.7/48.7 D

51.6/47.3 D

+.9/-1.4

MI :

53.0/45.9 D

54.2/44.7 D

+1.2/-1.2

PA :

52.5/46.6 D

52.0/46.6 D

-.5/+0

WI :

52.4/46.9 D

52.8/45.9 D

+.4/-1

OH :

51.1/47.7 R

50.7/47.7 D

-.4/+0

Battleground States 2016

State : Predicted : 2012 Difference : Potential Actual

CO :

47.4/43.6 D

+.7/-2.2

48.1/41.4 D

FL :

47.9/47.8 D

+.2/-.8

48.1/47.0 D

IA :

45.6/47.7 R

+.9/-1.8

46.5/45.9 D

NB 2nd (Strongly R)

NH :

48.0/43.6 D

+.6/-1.5

48.6/42.1 D

NC :

47.9/47.7 D

-.6/-.2

47.2/47.4 R

VA :

49.9/44.0 D

+.9/-1.4

51.8/42.6 D

MI :

48.7/44.2 D

+1.2/-1.2

49.9/43.0 D

PA :

49.3/45.1 D

-.5/+0

48.8/45.1 D

WI :

49.2/44.5 D

+.4/-1

49.6/43.5 D

OH :

47.8/46.2 R

-.4/+0

47.8/45.8 R

As you can see, in some cases they are really right on. But in other cases they are off by a fair margin. Now, most of these differences wouldn’t really flip a state, although it has happened. But it would make many states seem much more comfortable than predictions would indicate and that move is almost always in the net positive for Democrats. So why is Nate not using his own findings in his predictions? I don’t really know. The problem here is that Nate uses the polls a lot, it’s obviously the main predictor of every race, but even he admits that they are tons of new polls that have no reputation (and he is including those at a lower strength) and even tracking polls that he admits are really only useful for tracking (but he includes those at a lower strength also).

So here’s the final reason I have fallen out of love with 538 and Nate’s predictions over the election. He doesn’t include anything but these three parts into his mix; polls, assumptions and ‘secret sauce’. His assumptions, at the very least, are questionable as they go against his own findings. We have no clue on the secret sauce, and when we depend on the polls so much, in an election where even he has admitted that the polls are probably their most unreliable, I remember this quote from a GOP pollster.

“The thing about polling is: polling assumes both campaigns are equal. You look at quality of surrogates, money raised, ground game — she has an advantage in all that. So you basically have to look at the polling and say it reflects the worst-case scenario for her.”

Clinton is still a 70% probability of winning, she was as high as 88%, what’s the big deal? Mainly because Nate wants you to believe that his site is the best, his model is the right one and his (now personally owned) site is the one to go to. But you have to question a site that is this popular, that can sway voters and thoughts, especially in an election that is so stressful and full of volatility. I don’t think its hard to paint the picture that 538 paints potentially the most difficult position for Clinton and includes the most R-leaning uncertainty. That’s all fine. But people should be aware of that before they go out and treat it as gospel, either way. You should ask yourself why would he build in such uncertainty, such a lean. He’s an admitted liberal-leaning number cruncher, so why? I have my own opinions. The top one being that he has bought into the horse race narrative as much as anyone. The closer those red and blue lines are, the more clicks he gets and he knows that. Maybe it’s also to drive people to the polls. The more voters the better, it’s just good for America. But whatever the reason, think about it before you treat this as a foregone conclusion because 538 said so.

I, for one, will be closely monitoring his Monday night predictions with the actuals after the election. For me, being right on 49 of 50 states isn’t enough for me at this point, when 40 of those states are locks in every election. That being said, I love the effort he puts in, the graphs, the charts, the constant updates. I just hope to see his predictions becoming more right than wrong this election.