Overview

FiveThirtyEight’s election model is an outlier compared to other polls-based models, giving Trump a 1/3 shot of winning. Following up on the recent controversy between Nate Silver and HuffPo editor Ryan Grim, I re-read 538’s modeling assumptions. Today, I did some basic data analysis using a really simple poll aggregation method (median of all the polls), but incorporating the trendline adjustment similar to the one that 538 uses. It turns out that, depending on how you implement trendline adjustment, you get very different results. I’m going to show the sensitivity here.

By keeping the model simple and varying only the way of computing trend lines, we can look at its effect more directly. Please don’t come bug me that these predictions were wrong. Code is here.

What is trend line adjustment?

In a nutshell, the trend line adjustment works like this: let’s say we conduct a poll in North Carolina in August. It’s now September and we want to incorporate it into our model, but the national polls have changed since the NC poll was taken. We borrow information from the national polling change from August to September and re-weight (change the outcome of) the state poll. To be concrete, say that the NC poll shows Trump +1 in August, and we are making a forecast in September. Since the NC poll was taken, assume Trump has gained +3 nationally. Then we add some of that +3 (according to a weight) to the August NC poll. If our weight is 1, we would add +3 to the poll, and enter a +4 August NC poll into our model.

This methodology works if we have an accurate estimate of the weight, and an accurate estimate of the trendline. If our weight or trendline estimate are off, then our results might be wrong. Estimating the trendline is done by LOESS regression in 538’s model, so I’ll repeat that here.

Data

I used the Huffington Post Pollster API to get all national polls (N = 200) since January 2016, and state polls from commonly-cited battlegrounds: VA, PA, CO, NH, FL, NV, OH, IA, NC, AZ, GA (N = 437). These are likely voter polls as identified by the API. Because this is a simple analysis, I’m not worrying about pollster quality, house-effects, etc., and one could undoubtedly incorporate more polls (e.g. registered voters).

If I say NV +3, it means Clinton leads by 3, whereas NV -3 indicates Trump leads by 3.

Model

Our model of outcomes is simple: take the median of all polls over the whole period and call that our estimate. PEC likes medians, so that seems fine to repeat. I don’t do any time-decay because I’d really just like to do the most straightforward thing that seems reasonable.

At the national level, this produces Clinton +4. Compare to RCP’s 4 way average of +2.2, and HuffPo’s +4.9.

Doing the same thing at the state level gives: AZ -2, CO +5, FL +2.5, GA -4, IA -1, NC +1, NH +6, NV +0.5, OH +2, PA +6, VA +7.5, WI +5.0. These are not terrible estimates. The worst one is NH +6, when The Upshot gives +3. Compared to The Upshot, we get WI, IA, GA correct; VA within 0.5; AZ, CO, FL, NC, PA within 1 point; NV within 1.5; OH within 2.

#LiterallyNotTerrible

National trend

Using LOESS regression requires deciding how big we should make our smoothing window. Think of this as how much information we want to incorporate on either side of the point we are considering. Lower smoothing values are more local (jumpier line), while higher ones are more global (smoother line). In R, the default parameter is 0.75.

Here’s a picture of the national trendline using smoothing of 0.65 (left) and 0.85 (right). The outcomes are qualitatively different for election day. In the former case, a Trump victory is in the margin of error (dashed line is even). In the latter case, Clinton’s lead appears quite safe.