Despite Kerryn Phelps appearing the clear winner of the Wentworth by-election on election night, victory looked less certain twelve hours later after a dramatic narrowing of her lead.

On election night I called a victory for Kerryn Phelps at 7:18pm. Before 9pm Phelps herself claimed victory, and Prime Minister Scott Morrison and Liberal candidate Dave Sharma conceded defeat.

The predicted swing at 7:18 was little changed by 10:30pm. The result for every polling place had been reported and the predicted win for Phelps looked secure. By all past postal and pre-poll vote trends in Wentworth, Phelps' lead was unassailable.

Yet by lunchtime on Sunday Phelps' lead was only 884 votes and her two-candidate preferred (2CP) result only 50.6%. Liberal Party hopes of victory rose.

By 6pm on Sunday Phelps was clearly back on track to win as the cause of the narrowing became apparent.

So what happened? Had I called the result to early. Had the Liberal Party with its own data made the same mistake? Had we all forgotten about the "Jewish vote" in Wentworth?

In this post I'll explore what caused the post-election day narrowing.

In short, everyone underestimated how the Phelps campaign peaked on polling day and that it was less organised than the Liberal Party when postal voting began. The government's late campaign wobbles may have boosted Phelps' polling day performance.

There are also oddities with the postal and pre-poll votes, and the result is a warning on how to deal with absent votes, a missing category of votes at by-elections.

The Progress of the Count

The graph below charts two measures of two-candidate preferred percentage for Kerryn Phelps against the percentage vote counted.

The blue line is the actual 2CP% and includes counting on subsequent days.

The black line is a projected 2CP% for Phelps calculated from the progressive count on election night. The projection has been calculated by the matched swing method used by most election analysts and parties to accurately predict results.

The gold line is the final result, Phelps winning with 51.2% after preferences.

Dashed vertical lines in the graph highlight three important turning points in the 15 hours after the last polling place reported.

The first line marks when the result for the large and strongly Liberal voting Rose Bay pre-poll voting centre was entered. The second marks the entry of the first batch of strongly Liberal postal votes. The third dashed line marks when election preference count errors from the night before were corrected during the check count on Sunday.

The check count correction doubled Phelps' lead and removed the doubt over her victory that had emerged on Sunday morning.

The matched swing prediction method works by first calculating a swing comparing percentage votes by party at the current election with the percentage vote by party at the previous election in the same polling places. This matched swing is then applied to the overall result of the last election to produce a relatively stable predicted result.

The reason for using swing rather than 2CP% is that the swing is less variable by polling place than the 2CP%. In Wentworth the polling place Liberal 2CP% rose from a low of 27.2% to a high 68.6%, a range of 41% and a standard deviation of 11.3. The swing was between -15.0 and -30.9 per polling place, a much smaller range of 15.9% and standard deviation of 3.9.

As the swing has a much smaller standard deviation, the progressive swing figure used to make the prediction stabilises more quickly than the 2CP%. That is shown clearly in the above graph, the blue line of 2CP% bouncing around much more than the black prediction line based on swing.

With all polling places counted, representing 50.5% of enrolled votes from a likely final turnout of under 80%, Phelp's 2CP% was 54.4%. Using the matched swing that takes account of the trend in postal, pre-poll and absent votes at the last election, Phelps' predicted 2CP% was 53.3%.

With 54.4% of the two candidate preferred vote with all polling places reporting and half the vote counted, and with an expected turnout of 80%, the Liberal Party needed to poll 57.7% on pre-polls and postals to overcome Phelps' lead.

The predicted Phelps figure of 53.3% assumed Phelps would poll 51.1% of the votes to come. In fact she polled only 45.5%, which is why her victory margin was a narrower 51.2%.

When you call an election result, you are saying that the candidate has a large enough two-candidate preferred lead to be certain they cannot lose from that point. The call of victory for Phelps on election night proved correct as her lead was large enough to withstand even the unexpectedly large narrowing that occurred.

Every electorate has a particular trend in postal, pre-poll and absent votes that you factor in when making a prediction. It is true that the high "Jewish" vote in Wentworth gives a particular trend to postal and pre-poll trends. That particular trend in Wentworth was factored into the by-election night prediction.

As I will describe below, the post-election night counting trend at the Wentworth by-election was very different to the trend seen in Wentworth at elections over the past two decades. Compared to all previous elections in Wentworth, the surge in support for the Liberal Party in pre-poll and postal voting was much greater than seen before.

Previous Wentworth Results by Vote Type

The graph below plots the percentage point difference in Liberal 2CP% for Absent, Postal and Pre-poll votes compared to votes cast on polling day in Wentworth at elections since 1996. The dashed line shows the difference between polling day votes and the overall result for Wentworth.

The graph shows clearly that Liberal support is much higher for Postal votes than for votes cast on polling day. Liberal 2CP% support has consistently been 10-13 percentage points higher than polling day votes. At the by-election the final postal vote difference was 14.8%, though the figure was 19% compared to the first batches of postal votes reported.

Pre-Poll votes have also been stronger for the Liberal Party, usually 2-5 percentage points higher, but at the Wentworth by-election double this figure at 10 percentage points higher.

The Absent vote trend has been very different, the Liberal 2CP% consistently 6-8 percentage points lower than on polling day. There were no Absent votes at the by-election.

An Absent vote is one cast outside of a voter's home division on election day. Absent votes measure the propensity for voters to be outside of their electorate on polling day. Absent voters are more likely to live nearer the edge of the electorate.

In the case of Wentworth, geography means that a voter who lives in Elizabeth Bay, Clovelly or Randwick, is more likely to vote Absent in a neighbouring electorate than a voter from Liberal heartland in of Vaucluse and Rose Bay. For that reason, Absent voting in Wentworth tends to be less favourable for the Liberal Party than polling day votes.

The inclusion of Absent Votes usually knocks around 0.5 percentage points off the Liberal Party 2CP% in Wentworth. Had Absent votes been excluded from the last election totals at the Wentworth by-election, the predicted result for Phelps on election night would have been reduced from 53.3% to 52.8%. Something similar happened at the 2012 by-election in the Victorian state seat of Melbourne, when including Absent votes from the previous election resulted in an over-prediction of the Green vote. It may be best to exclude Absent votes from an electorate's history when analysing by-elections.

Focussing on the dashed line comparing polling day results with final results, between 1996 and 2016, the largest post-election night shift in Liberal 2CP% seen previously was 1.5% in 2013 and 1.2% in 2016.

At the by-election the gap more than doubled to 4%. The narrowing from Phelps' by-election night margin wasn't because I hadn't factored in the Postal and Pre-poll votes. It was because I hadn't predicted that the difference between votes cast on polling day and votes cast would be so much greater than seen previously in Wentworth.

That the gap was larger is a reflection of the swing being larger on polling day compared to votes cast before polling day.

The two-candidate preferred swing against the Liberal Party was 21.8% on polling day, 15.2% in Pre-poll votes, and 17.0% with Postal votes. If the swing on polling day had been the same as that recorded amongst votes cast before polling day, the Liberal Party would have won the Wentworth by-election.

Explaining the Postal and Pre-Poll Difference

More than 12,800 postal vote applications were received, processed and ballot papers dispatched. 9,003 postal votes were returned and admitted to the count. Once you include postal votes returned but rejected from the count for a variety of reasons, the return rate is around 75%, normal for a by-election.

Of the postal vote applications received, 3,713 were via the AEC's on-line postal vote application. 5,913 were received via paper application forms, almost all of which were prompted by a Liberal Party mail-out.

Another 3,234 were for registered general postal voters. Voters who are old, ill or otherwise infirm, are registered as silent voters, or who have religious objections to voting on a Saturday, can choose to be registered as general postal voters. General postal voters are sent postal vote packs as soon as nominations are closed without the need for an application. (You can find out who is entitled to be a general postal voter at the AEC website.)

Voters can apply for a postal vote as soon as the date of the election is known. Postal votes packs are dispatched on the Monday after the close of nominations, and Pre-poll voting starts on the Tuesday after the close of nominations. Applications for Postal votes close the Wednesday before polling day.

The graph below plots the total Postal vote applications by day and the total Pre-poll votes cast.

The huge increase in postal vote applications a few days before the close of nominations corresponds to general postal voters being added to the mail-out list. The graph shows that by the close of nominations, around 7,000 postal vote packs were ready for mail-out. These votes were overwhelmingly general postal voters and early responders to the Liberal Party's postal vote application.

Of Pre-poll voters, roughly 8,000 were cast in the two weeks after the close of nominations, and 10,000 were cast in the last week of the campaign, the numbers peaking on the Thursday and Friday before polling day.

Presumably the early postal votes were a large proportion of the roughly 5,000 postal votes received, processed and counted on election night for inclusion on the Sunday morning. These votes were overwhelmingly in favour of the Liberal Party.

The graph below plots the Liberal 2CP% in the Postal vote count with each day's counting. As the plot shows, the Liberal 2CP% slipped every day.

On first preferences, the first batch of postal votes split Liberal; 57.3%, Phelps 17.2%, Labor 12.4% and Greens 4.8%. By the last batch, the first preferences were Liberal 35.4%, Phelps 31.8%, Labor 14.4% and the Greens 11.9%.

Assuming the postal vote receipts by the AEC reflect when votes were completed and returned, there are two possible explanations for the postal vote trend.

The first is logistical, that the Phelps campaign was disorganised at the start of the campaign and did not have a well organised postal vote strategy. The Liberal advantage in the general postal votes and early applications was reflected in the early returned postal votes being stronger for the Liberal Party.

A second may reflect the campaign itself, with the Phelps campaign peaking on election day. The Liberal Party spent the last 10 days of the campaign either trying to stamp out political bushfires, or trying to contain their own political backburns that had got out of control.

But that doesn't explain why the Pre-polls were so solid for the Liberal Party. If it was the last 10 days of the campaign that lost Wentworth for the government, why wasn't this reflected in the Pre-poll votes given how many were cast in the last days?

The answer may be in the positioning of the Pre-poll Voting Centres.

At the 2016 Federal election there were four Pre-Poll Voting centres, Haymarket (720 votes, Liberal 2CP 58.9%), Waverley (10,509, 73.2%), Randwick (1,820, 62.1%) and Sydney Town Hall (2,174, 66.1%). There were also 3,830 pre-polls taken outside of the electorate with Liberal 2CP of 62.9%. Only the Waverley Pre-Poll Voting Centres was within the boundaries of Wentworth.

At the by-election, two new centres within the electorate were used. The four pre-poll centres were Haymarket (1,142 votes, Liberal 2CP 39.4%), Paddington (3,424, 43.1%), Rose Bay (6,781, 69.7%) and Waverley (7,106, 49.5%). There were only 306 external pre-polls recording a Liberal 2CP of 37.2%.

It may be that the smaller swing on pre-polls reflects a change in where pre-poll voting centres were located. In 2016 7,800 pre-poll votes were recorded at pre-poll centres outside of the electorate where the Liberal vote was lower than within the electorate. At the by-election, 6,781 pre-polls taken in the Liberal heart of the electorate at Rose Bay where the Liberal Party vote was stronger than recorded in any polling place on election day. There was a drop in votes cast at nearby polling places on the day.

The Rose Bay Pre-Poll centre was on New South Head Road in the Rose Bay shopping centre, the main transport route through to the strongest Liberal voting parts of the electorate. The Rose Bay Pre-Poll centre may have re-arranged when and where people voted in Liberal parts of the electorates, which explains why the Pre-Poll trend was so different.

A final note to make about Postal and Pre-Poll votes is the rise in their use. In 1996 82.7% of all votes were cast in polling places in Wentworth on polling day. Postal Votes comprised 3.9% of votes and Pre-Poll Votes 6.0%.

At the 2016 election polling day votes were 61.6% of all votes, Postal Votes 11.2% and Pre-Poll Votess 23.4%. The numbers for the by-election were not significantly different, 65.3% on the day votes, Postals 10.8% and Pre-Polls 23.4%

While the numbers of Postal and Pre-poll votes has increased over the last two decades, the change between the 2016 election and the by-election was not significant enough to have an impact on election night predictions.

Conclusions

There are several trends that resulted in the overnight narrowing of the election result.

0.7% of the narrowing next morning was caused by the preference count errors that were corrected in the afternoon.

0.7% of the narrowing next morning was caused by the preference count errors that were corrected in the afternoon. 0.5% of narrowing came about through factoring in Absent Votes that didn't exist at the by-election.

0.5% of narrowing came about through factoring in Absent Votes that didn't exist at the by-election. The first batch of postal votes was much worse for Phelps than the final postal vote totals.

The first batch of postal votes was much worse for Phelps than the final postal vote totals. The Liberal Party gained another 0.2% on its lead through winning the Special Hospital vote 266-54.

The Liberal Party gained another 0.2% on its lead through winning the Special Hospital vote 266-54. Phelps did much better on polling day than among votes cast before polling day, though it is not possible to be certain if the difference is due to campaign issues of the last week or organisational issues for Phelps earlier in the campaign.

Phelps did much better on polling day than among votes cast before polling day, though it is not possible to be certain if the difference is due to campaign issues of the last week or organisational issues for Phelps earlier in the campaign. The much smaller swing in Pre-Poll votes may be due to changes in the location of Pre-Poll Voting Centres.

There may also be an issue with the composition of the final two candidates. If the by-election had been a contest between Labor and Liberal, I suspect the swing would have been more uniform. With Kerryn Phelps being the Liberal Party's opponent, there were cross cutting issues that allowed Liberal voters to switch to Phelps that were different for Labor or Green voters switching to Phelps.

There was always the chance that the swing across the electorate would vary more than normal. That didn't happen as the polling day swing was very uniform. Instead, the difference in swing between polling day votes and votes cast before polling day was much greater than expected and much larger than previously recorded in Wentworth.