ON NOVEMBER 6th 2018 the United States will hold elections for all 435 seats in the House of Representatives, the lower chamber of Congress. These are contested every two years, both alongside the presidential race and in mid-term elections. (There will also be 35 seats up for grabs in the Senate, the upper chamber, which holds elections every two years for about a third of its six-year positions.) The Economist has developed a statistical model for the House, to predict the results of every individual race and the battle for a majority. We will publish updates every day until the election at economist.com/midterms. For readers curious to understand how we produce these estimates, our methodology is outlined below.

The Economist’s model of the contest for the House of Representatives proceeds from two basic premises. The first is that, when facing an uncertain future, virtually every scenario has some chance of occurring, and highly unlikely events can be particularly important. So accurately modelling the shape of all points on a probability distribution—particularly the tails that represent long-shot scenarios—can often be more important than getting the average right.

Our model’s other axiom is that legislative races are determined by both local and national factors. Voters decide whether to show up to the polls and whom to support based both on which party they want to control Congress and on which of the individual candidates on the ballot they prefer as their representative. Forecasting methods that see every district in terms of a fixed departure from the national trend—the sort of approach that simply labels somewhere “D+6”, meaning it is expected to favour the Democrats six percentage points more than is the case nationwide—fail to capture the complexity of voters’ choices. Doing justice to them requires modelling how a district’s preferences vary in response to the nationwide political environment.

Putting this principle into effect means building not one model but two. The first model uses available national-level variables to forecast what the popularity of the two major parties will be on election day. The second uses local data to predict how voters in each district will behave, given that national climate.

In this two-stage approach, the first task is forecasting the relative nationwide popularity of the Democratic and Republican parties in November. This might seem simple: look at the national vote for all the elections since regular polling became available in 1942, and then look for variables that would, in those past years, have allowed you to predict those vote shares.

Surprisingly, it is not quite that easy. The total votes tallied for the two parties do not really reflect the situation on election day, because a surprising share of races—around 15%, since 1942—are not contested by both major parties. In these districts the party not in the race will receive no votes—even if it is supported by, say, 33% of the population. The votes for the party that is represented will likely be on the low side, too.

If both parties fielded a similar number of unchallenged incumbents in each cycle, these effects might cancel each other out. But the figures are highly variable (see chart 1). In 1958 there were 96 uncontested seats won by Democrats and one by a Republican. In 2010, Republicans outnumbered Democrats in uncontested districts 24 to five. To remedy these imbalances, the very first step in our model is to estimate how many people would have voted for each party if every district had been contested in every election. We do this by looking at how the same district voted in an earlier or later election, or how similar districts in the same state voted. This gives us a “true” target to aim for. The best single predictor of the overall result of a House election is “generic ballot” polling, which simply asks respondents to say which party’s Congressional candidates they plan to support. Since 1942, generic-ballot polls taken on the eve of the election account for fully 83% of what we see in our adjusted national vote figures. Earlier polling is less useful, but in ways that can be accounted for (see below).

However, the generic ballot is not perfect. Most importantly, it tends to overshoot: a big lead in generic-ballot polls usually translates to a somewhat smaller lead in the actual vote (see chart 2). For example, in 1964, in the wake of John F. Kennedy’s assassination, Democrats received 65% of the two-party support in polling, but collected less than 57% of the adjusted national vote. To correct for this tendency, generic-ballot figures need to be nudged back towards the equilibrium state, by an amount that depends on how far away from it they started. Moreover, the generic ballot is far from the only useful source of information about nationwide political preferences (see chart 3). The popularity of the parties’ presidential candidates, or of the White House incumbent, also plays a role. Adding consideration of presidential polling data allows better predictions than relying on generic ballots alone. And between House elections, there are dozens of special elections held for vacant legislative seats across the country. Adjusting the results of these contests for the partisan lean of the districts where they are held also improves predictions for the next House election. Finally, there are “fundamental” factors, such as whether an election is a mid-term or in a presidential year; whether the president can or cannot stand for re-election; and economic performance (as measured by the unemployment rate). All of these add some accuracy beyond the polling and voting data.

The final element in this stage of our model takes into account the fact that over the past fifty years America has become significantly more partisan. According to data compiled by Corwin Smidt, a political scientist at Michigan State University, as recently as 1980 11% of Americans who had voted in two consecutive presidential elections had backed candidates from one party first and the other next (see chart 4). By 2016, that figure was just 4.7%. As a result, a margin of victory of five percentage points, which was unremarkable from the 1950s through the 1970s, counts as a landslide today (see chart 5). By incorporating the degree of polarisation of the electorate into our model, we can make use of data from elections held in very different political climates.

With just 38 House elections in the dataset, there are not enough historical examples available to tease out the individual impact of each of these variables with all the others held constant—particularly since many of them tend to point in the same direction in any given year. However, for the purposes of prediction, we did not need to know exactly how much each type of information matters when taken in isolation. Instead, we can be satisfied simply by determining which composite blend of all the ingredients yielded the most accurate forecasts when presented with data about an election that had not been used to train our model. To achieve this, we used a statistical technique called “elastic-net regularisation”. This method works by sequentially chopping up a dataset into pieces. For example, at one stage, it might segregate out the 1946 election, pretending that it doesn’t know the results from that year. Using only the remaining elections, it then fits a series of different models, reducing the impact of certain predictors by varying amounts and dropping others entirely. Among these candidate models, it then identifies which one comes closest to predicting the withheld 1946 election accurately. By repeating this cycle until every election has been held out once, we were able to identify the permutation of variables expected to perform the best in the future.

Our model of the adjusted national popular vote uses predictors as measured on the morning of Election Day. In order to produce a forecast now, we need to estimate what all of those variables are likely to be months ahead of time. Fortunately, in the past they have tended to follow reliable patterns. For example, under normal conditions, undecided voters tend to break towards the opposition party in the generic ballot during mid-term years—by a large amount if the president’s party is highly popular, and much less if his party is already extremely disliked. Similarly, in mid-term cycles the president’s party has historically fared far better in special elections held over a year before Election Day than it has in those occurring much closer to the next national election. By taking today’s data and extrapolating into the future using these historical trends, we can make educated guesses about the final values of all of our predictors.

Such patterns are only a guide. Most election cycles will wind up deviating from them to some degree. As a result, the model is less accurate when predicting results far in advance than it is at the end of October. We take account of this increased error when producing probabilities of victory. Thus if the model’s best guess eight months out is that the leading party will receive 51.5% of the two-party vote, we say that there is an 80% chance of that party winning the popular vote. If it says the same on election day, the chances are 89%.

With the national political climate taken care of, we move on to the district level. Here we use two types of data to measure the characteristics of each district. First, there is its voting history: how it has voted in the past in elections for Congress and the presidency, how its state has voted in the same elections, and how variable its voting patterns have been from year to year. Separately, there is information about the candidates: whether an incumbent is running for re-election; how many elections the most recent office-holder had previously won; which candidate has raised more money; and a commonly used measure of left-to-right ideology for sitting legislators called the DW-NOMINATE score.

To fit our district-level model we added to these factors two nationwide numbers: the share of swing voters in the electorate, and the actual, observed adjusted national popular vote seen in that year. We then deployed the same elastic-net regularisation method for this model that we did for the previous one, aiming to produce the equation best-suited to predicting unseen future data, rather than the one that best fits the historical information used to train it.

With this second model, and the prediction of the most likely figure for the adjusted national popular vote from the first model, we could make a prediction. But that would simply produce a “point estimate” of the single most likely vote share in each district. And the first of our two starting axioms was that we were interested not in point estimates of the most likely outcome, but in understanding the distribution of possibilities—including unlikely ones.

To derive a probability of victory consistent with that aim, we need to know the odds of every possible scenario, even the extreme ones. And those chances can vary widely between races with the same expected vote share. The range of outcomes is wider in districts with open seats than it is in those where an incumbent is seeking re-election, and narrower in years with few swing voters than it is in eras where they were abundant. For example, take Connecticut’s fifth district in 2014, and North Carolina’s 11th in 1978 (see chart 6). In both cases, given how the country voted as a whole, our model predicted the president’s party to win about 53% of the vote in these seats. But because the North Carolina election occurred in a low-polarisation year and the Connecticut one when partisanship was higher, the model gives the opposition an 18.2% chance of victory in the 1978 contest and a slightly lower 16.7% chance in the 2014 one. Moreover, district-level votes do not follow the “bell curve” of a normal distribution, in which a result a bit higher than the mean is just as likely as a result the same amount lower than the mean. The favoured party tends to win close House races more often than such a normal distribution would suggest. And district votes also tend to yield big surprises more frequently than a normal distribution would indicate—the distributions have “long tails”. As a result, we have used the same variables to fit an extremely flexible distribution, called a “skew-T”, to this dataset. With the freedom to lean to the left or right and to display unusually fat or skinny tails, the skew-T can take on whichever shape is necessary to accommodate the idiosyncrasies of each pairing of a given district with a given national political climate. The next step in the model is to put the pieces together. Starting with our distribution of outcomes for the adjusted national popular vote—centred around the most likely result, but reaching far away from it at the tails early in the election cycle—we choose 10,000 different random values for the nationwide political climate. Each one represents one plausible path for the election to take. Some will represent good years for the Democrats, others for the Republicans. Most will cluster close to the average, but a few will reflect scenarios where one party does far better than the data currently available would tend to imply. Next, for all 10,000 of these simulated elections, we use our district-level equations to draw customised skew-T distributions of the vote share for all 435 districts. These distributions do not simply maintain the same shape while moving to the left or right in response to a strong national performance for one party or the other. Instead, their width, lean, and tail height—formally, their “standard deviation”, “skewness”, and “kurtosis”—all change as well depending on the simulated national context.

For example, consider Ohio’s 12th district in 1974. The seat appeared firmly Republican. However, the 1974 midterm was held amidst the Watergate scandal, and the Democrats were poised for a large nationwide victory. Whether the seat would be in play or not depended on whether the opposition to Richard Nixon enjoyed a merely comfortable win or an unprecedented landslide. If the Democrats secured a healthy 53.3% of the national two-party vote that year, our model would still have assigned them only a 6.2% chance to win the district. In contrast, if they had achieved an overwhelming 62.3% of the nationwide vote, their odds of victory in Ohio’s 12th would go up to 45.6% (see chart 7). (As it happened, they won 57.8% nationally, and fell just short in a close race in the district.) After repeating this procedure for every district in every simulation, we pick one two-party vote share at random from each skew-T distribution. The probability of each party securing a given seat simply equals the share of the simulated elections in that district that it wins. Similarly, each party’s chances of a majority are the percentage of simulations in which it holds at least 218 seats.

Having put together our model, we had it scrutinized by experts elsewhere in The Economist Group, and sought reviews from electoral experts in academia. Aware of the risks of unconscious bias, we attempted to dot all the i’s and cross all the t’s before we gave the model data from this year, in order to make our first prediction. Unfortunately, that first prediction contained some district-level results we thought highly unlikely. Working back from them, we discovered that a coding error meant that the model was not dealing with seats that lacked an incumbent in the way that it was meant to. This error was fixed—which is to say the coding was changed so that the model actually behaved as we intended it to. This reduced the likelihood of a Democratic victory. If in the coming months we discover other such errors, or make any other changes for any reason, we will document them here.