7. Model evaluation

So now that we have a final fit where we can’t confidently add or remove any interactions variables and other transformations it’s time to evaluate if our model actually fits our data and if there’s even a statistically significant difference between our final fit and the first “naive” fit.

Lets start by taking a look at our fitted values vs. the Pearson residuals:

par(mfrow = c(1, 1)) # 2 row(s), 2 column(s)

plot(p.resid ~ fit.val)

lines(ksmooth(y = p.resid, x = fit.val, bandwidth=.1, kernel=’normal’), col=’blue’)

abline(h=0)

This is a pretty decent fit, lets take a look at the fit for the major explanatory variables as well:

With the exception of ‘gov.support’ everything looks quite nice, also the reason for the bend in ‘gov.support’ seems to be a single outlier in which someone was granted a substantially lower amount of support compared to all our other points of data.

Checking for potential overfitting

Overfitting is the bane of all statistical modelling, how can you make sure your model isn’t just fitting to the exact data you fed it? The goal is to make a model which generalizes and doesn’t just cater to the current data at hand. So how do we test if our model is overfitting on our data?

A popular metric to test is the delta value generated through Cross-Validation, we can calculated these by using the cv.glm function from the ‘boot’ package and compare our final fit to our first!

cv.glm(dat[-c(416),], final.fit, K = 13)$delta

cv.glm(dat[-c(416),], update(fit.1, data = dat[-c(416),]), K = 13)$delta

In the code above we’re using k-fold cross-validation with k = 13 (since 13 is a factor of 871 which is the length of our data after removing the outlier) this means we’re splitting our data in 13 ‘chunks’.

The delta values is a vector wherein the first component is the raw cross-validation estimate of prediction error and the second component is the adjusted cross-validation estimate (designed to compensate for the bias introduced by not using an exhaustive testing methods such as leave-one-out)

Running the code above yields the following delta-values, note that these are subject to some random variance so you might not get the exact same values:

The prediction error is lower for the final fit, even when testing with cross-validation. Thus we can assume that our model hasn’t been overfitting on our data!

The final piece of the puzzle

So now we’ve concluded that our model is actually a pretty decent fit for our data, but is it a statistically significant difference from the “naive” model without any transformations and variable interactions? We can use an ANOVA test for this, we just have to remove the same point of data in both fits:

There’s definitely a significant difference between the two fits, we can happily conclude that our hard work has paid off!