Labour has a mountain to climb to win a majority in 2020. On Saturday, at the Fabian New Year Conference, delegates will discuss research which estimates that the party will need to gain 106 seats and earn 40 per cent of the vote in order to triumph. Irrespective of our choice of leader, it is a daunting challenge.

Perhaps that helps explain why the party has swung to the left, and turned in on itself? When the route to victory is so uncertain, maybe Labour folk have concluded there is little point in facing outwards to the electorate; or in contemplating the complexity and compromise that majoritarian politics and the realities of government will always require?

If that’s true, it’s not good enough. Even the toughest journey is achievable, when you go, one step at a time, in the right direction. The whole route, from here to Downing Street, may be shrouded in mist, but the first few steps are not. Labour needs to put the last nine months behind it and focus on getting to ‘first base’.

This year, that means the party’s goal should be to regain the trust of current and recent supporters; to remake the connections and re-earn the votes of the 9.3 million people who backed Labour last May. It sounds modest. It certainly isn’t enough to win in 2020. But it is first base. And it’s a first base we cannot take for granted.

We know that, because we’ve been here before. After the defeat of 2010, almost everyone in the party unconsciously ‘banked’ the support of the diminished ranks who had backed Gordon Brown. We were proved wrong. Even outside Scotland, Labour was deserted by well over a million of its 2010 voters, according to estimates by Electoral Calculus. If we’d held on to them, Ed Miliband would be prime minister.

Unless Labour acts now, history will repeat itself. Last week YouGov published a poll which included data on the views of people who voted Labour last May. It was a small-ish sample of 447 voters, but the results set alarm bells ringing. First, the shocking headline: only three quarters of Labour’s 2015 supporters say they would vote for the party in a general election held today. Beneath it the sobering detail: just half say that Jeremy Corbyn is doing well as leader; and half would prefer him to David Cameron as prime minister.

It goes on. When asked which party is best placed to handle challenges facing the country, only 53 per cent back Labour with respect to unemployment, 50 per cent on tax, 49 per cent on the economy, 44 per cent on law and order and 36 per cent on immigration. And these are people who voted for Ed Miliband less than 12 months ago.

Getting to first base means turning these numbers around. We must stop talking to ‘ourselves’, in a deafening conversation among the 550,000 people with a vote in the party leadership election. And start talking and, more importantly, listening to the people who voted for us in May. For although Labour’s selectorate is large in number, and represents a fantastic campaigning resource for the future, it is not reflective of Labour voters in all their diversity.

So if 2015 was about the half million, 2016 must be the year of the nine million. That means fighting to retain past supporters in tough elections this May: putting in the hours on the doorstep, but adding real conversations to the voter ID script. It means never having a policy debate within the party, without giving as much credence to the views of Labour voters as members and supporters. And it means running a Europe campaign that is authentically Labour and meets people where they are.

When it comes to policy making, I’m all for digital surveys and polls, as long as they are robust and fair. But every time members are asked for their views, the same question should be put to a representative sample of Labour voters. So, on trident, the party should hear from activists, NGOs and think tanks, but it should give equal weight to polls and citizens’ juries of ordinary Labour voters. And if the half million and the nine million disagree, the leadership must pause for thought.

Making 2016 the year of the nine million also means running a super-charged, super-Labour campaign for Europe. As things stand, around a quarter of Labour’s 2015 voters support Brexit. If that number grows, or if working-class Labour voters stay at home, the referendum will be lost. Plus, if we speak to EU sceptics and agnostics in the wrong way, a victory in the referendum could come at the cost of losing more former Labour voters for good.

Labour can win the EU argument and rebuild fraying relationship with its 2015 voters by making a practical ‘kitchen table’ case for Europe. It must be about livelihoods and jobs, and about who we are as a country. And it must be fronted by Labour ambassadors who can together reach every part of our broad constituency, including Jeremy Corbyn, Alan Johnson and Gordon Brown.

Just worrying about first base – our own recent voters – may sound timid and unambitious. But get it right and it is a step forward. Then, success can build on success, and we may find we start reaching out to new supporters too. But if Labour fails with its own 2015 vote, then the prospects for the party are truly bleak.

Andrew Harrop is general secretary of the Fabian Society. For more information about Fabian New Year Conference and to book your ticket, visit the Fabian Society website