This election was no ordinary defeat for Labour. It marks a moment in history. The choice for Labour is to renew itself as the serious, progressive, non-Conservative competitor for power in British politics; or retreat from such an ambition, in which case over time it will be replaced.

The election can be analysed in conventional ways – and here it does not take political genius to work out what happened.

I feel deeply for those good Labour MPs and candidates who lost through no fault of their own and the thousands of party workers and volunteers who, as I know well, are the backbone of the party.

Of course, Brexit was an issue. It was a Brexit general election – which was why it was a cardinal error for Labour ever to agree it. But an already difficult situation was made impossible by the failure to take a clear position and stick to it.

I take very seriously the argument that we “deserted” or “disrespected” our working-class voters by reopening the referendum result.

But the problem with this position, is that there was no way of uniting the country over Brexit. Britain is deeply divided over it. Now that Brexit will happen, we must make the best of it and the country must come together.

But until the election settled the debate, as unfortunately it has, if Labour had gone for Leave it would simply have alienated the half of the nation that opposed Brexit; as well as the vast bulk of party members.

Post-election polling shows that between 2017 and 2019, we lost only a small number of voters who were Leave and all the way through we had more than double the number of Remain voters. The biggest percentage fall in Labour voters between 2017 and 2019 was amongst young people, probably dismayed by the ambiguity over a Brexit they detested.

What we should have done, following June 2016, is accepted the result, said it was for the government to negotiate an agreement but reserved our right to critique that agreement and should it fail to be a good deal for the country, advocate the final decision should rest with the people. Ultimately, we might have lost the most ardent Brexit support, but I believe, with different leadership, we would have kept much of our vote in traditional Labour areas, whilst benefiting from the fact that even in those areas, the majority of those voting Labour, were Remain.

Instead we pursued a path of almost comic indecision, alienated both sides of the debate, leaving our voters without guidance or leadership.

The absence of leadership on what was obviously the biggest question facing the country, then reinforced all the other doubts about Jeremy Corbyn.

What is important is to understand why his leadership was so decisively rejected.

This is not about Jeremy Corbyn as a person. I have no doubt he is someone of deeply-held and sincere beliefs, who stayed true to them under harsh attack.

But politically, people saw him as fundamentally opposing what Britain and Western societies stand for. He personified an idea, a brand of quasi-revolutionary socialism, mixing far-left economic policy with deep hostility to Western foreign policy, which never has appealed to traditional Labour voters, never will appeal and represented for them a combination of misguided ideology and terminal ineptitude that they found insulting.

No sentient political party goes into an election with a leader who has a net approval rating of – 40 per cent.

The takeover of the Labour Party by the far left turned it into a glorified protest movement, with cult trimmings, utterly incapable of being a credible government.

The result has brought shame on us. We let our country down. To go into an election at any time with such a divergence between people and party is unacceptable. To do it at a time of national crisis when a credible opposition was so essential to our national interest, is unforgiveable.

Anti-Semitism is a stain. The failure to deal with it, a matter of disgust that left some of us who voted Labour feeling, for the first time in our lives, conflicted about doing it.

So, at one level, sure let's have a period of “reflection”; but any attempt to whitewash this defeat, pretend it is something other than it is, or the consequence of something other than the obvious, will cause irreparable damage to our relationship with the electorate.

Let us demolish this delusion that “the manifesto was popular”. The sentiment behind some of the policy reflected public anxieties, but in combination, it was one hundred pages of “wish list”. Any fool can promise everything for free. But the people weren't fooled. They know life isn't like that. And the loading in of “free broadband” run by government was the final confirmation of incredibility.

So, messrs Johnson and Cummings had a strategy for victory, and we had one for defeat. And I noted the cockiness of the Johnson visit to Sedgefield to rub salt in the wound! But I would like to see their strategic brilliance measured against a team other than one whose striker was directionally oblivious, its midfield comatose, the defence absent in the stand chatting to a small portion of the fans and its goalkeeper behind the net retweeting a clip of his one save in a 9-0 thrashing.

For the Labour Party the choices are stark, starker than it realises.

It is gearing up to fight an “ultra-Thatcherite” Tory party.

But Boris Johnson also understood that the country can't be united over Brexit. So, his strategy is to do it and then treat it as an uncomfortable fact of life but not one defining the Conservative Party. He will adopt centre ground rhetoric on everything other than Brexit and possibly even on that; expect to see some former rebels back in the fold and the Lords; and having turned Brexit from a Tory problem into the nation’s problem, expect the tenor of the debate around Brexit to change.

His challenge will be formidable not least on the new trade deal and the threat to the Union, quite apart from delivering on all those promises to northern former Labour voters.

But most people would not bet against 10 years of Tory government.

The first rule of politics, however, is there are no rules of inevitability.

Labour can keep with the programme and positions of Corbyn with a new leader. In which case it is finished.

Or it can understand that it must recapture the party from the far left, make radical changes and begin the march back.

But the biggest necessity is understanding the challenge didn't begin in 2015. It is rather the culmination of political and socio-economic changes over the last half-century and the circumstances of Labour’s birth more than a century ago.

This is a moment where either we use the lessons of defeat to build a progressive, modern political coalition capable of competing for, winning and retaining power; or we accept that the Labour Party has exhausted its original mission and is unable to fulfil the purpose for which it was created.

As the 19th-century industrial revolution gathered pace, the Whig Party became the Liberal Party and the effective alternative to the Conservative Party. The Liberal Party suffered the stresses of Home Rule for Ireland and the cleavage between the radical elements represented by Chamberlain and the more conservative remnants of Whiggery and then later still had to cope with Chamberlain’s departure from the party and shift to a populism combining support for the working class and Imperialism; but it was the main instrument of social reform and could still win the election of 1906 and govern up to and through 1914 and the outbreak of the First World War.

A competitor appeared: the newly formed Labour Party born out of the Labour Representation Committee, a trade union-based organisation designed to bring true representatives of the working class into parliament; and socialist.

In time, the Labour Party took over as the main alternative to the Conservative Party and the Liberal Party retreated to minority status.

So, Lloyd George a great Liberal reformer and Clement Attlee a great Labour reformer ended up in different parties.

But the division in progressive politics had long-term deleterious consequences. In the last century with the Labour Party and Liberals separated, Tories have been in power much longer than the opposition, including winning eight out of the last 11 elections, whereas in the years of Tory/Liberal Party competition, the Liberals were ahead.

The Labour Party became reliant on traditional working-class organisations and constantly pulled towards a socialism which blunted at crucial moments its appeal to the aspirant working class. It had its Liberal wing represented by the likes of Roy Jenkins, but it was always viewed with some suspicion.

The traditional left and right of the party – Bevin and Bevan - were themselves often uncomfortable bedfellows, but they united around: Labour as a party of government, parliamentary not revolutionary politics, pro-Nato and the Transatlantic alliance, and within the mainstream of European socialist and social democratic politics.

Then there was a third strand of leftist politics which derived from Marxist/Leninism and was an early thorn in the side of the Labour leadership. From the beginning, the leadership shoved this strand to the fringe of Labour politics. It mounted an assault on the party’s commanding heights through Tony Benn in the 1970s and 1980s, but it was repulsed under Michael Foot who supported Denis Healey against Benn and moved to expel the Militant Tendency.

Throughout this time, something else was happening. The economy and society were changing. The middle class grew and those instruments of collective power like unions lost their industrial base.

And as the state also grew, in size and authority, it became clear that though it was a means of social progress, it could also be a vested interest, and the limitations of the state in an era of individual choice and increased income became more apparent.

The trade union and industrial base became hollowed out. The Labour Party structures had shown themselves vulnerable to infiltration. The values remained strong; but the offer to the people weak and outdated.

New Labour was an attempt to reunite the Liberal and Labour traditions of progressive politics. Both traditional left and right of the Labour tradition were expressly included, symbolised by me and John Prescott; but the far left was back to the fringe.

A study of Labour history showed that in the 20th-century it had governed only intermittently. The longest unbroken period of power was six years. Never had the Labour Party won two full successive terms.

So, the party programme was reshaped around an appeal to aspiration as well as social justice, to business as well as unions; culturally it was strong on defence and law and order but also socially liberal.

We won three successive full terms and governed for more than twice the length of any previous Labour government.

The party reached out beyond tribe but didn’t neglect its traditional voters.

It placed itself firmly on the side of the victim, not the criminal.

On the side of the working class who believe you must earn what you get.

On the side of the patient and pupil interest, not the producer interest.

It championed investment in public services but matched it with reform to ensure this money was spent wisely.

It explicitly rejected the anti-Western worldview of the far left and was on the side of people who were patriotic about their country.

It was not – for all the caricature since we left office – a project of the metropolitan liberal elite. It assembled a new coalition of traditional working class voters, aspirational voters who had previously turned to Margaret Thatcher, and it fused together the progressive vote that had been split in the previous century.

The extraordinary thing is the Labour Party’s desire to rewrite its only period of majority government in half a century in negative terms.

We did not “neglect” those traditional Labour heartlands in some appeasement of the middle class. We made the largest ever investment in schools and hospitals in those communities; redistributed wealth through tax changes and tax credits; cut pensioner and child poverty; took the homeless off the streets; and through Sure Start, the minimum wage and a host of other programmes helped those who needed help most.

And we kept their support. In 2005, in Sedgefield, my majority was almost 20,000. In Bolsover, it was 18,000. In Scotland we had 41 of the 56 seats, including two with increased majorities over 2001. The support we lost was mainly amongst the middle class especially over tuition fees and Iraq.

The point is not to go back to the policies of New Labour, but to understand New Labour’s place in Labour’s history, so we better understand how to forge Labour’s future; and to set it within the history of progressive politics in Britain where Labour sits, but not in sole occupation.

This defeat is seminal. We cannot afford to repeat 1983, moving crab-like towards reality. You know the narrative. Among the far left, “we won the argument” it’s just for some inexplicable reason the British people having accepted we were right, decided to vote for the other guys.

Among others, it is: many of the policies were really popular, but too many of them, our leadership was a problem but did inspire a lot of people, it wasn't that we were too extreme but we allowed ourselves to be portrayed that way, we now have to stand with our communities in the assault which will be mounted upon them by the Tories etc. etc. plus a bit of we need to be with working class communities against the “liberal London elites” sort of populism.

If we go down this line, it will be 15 years more of Tory government.

The country won't tolerate this. There are people disenfranchised in our politics today, angry at the way the country has been let down by its non Conservative opposition, and feeling hopeless. And for the country, there is a generation of smart, capable, politically conscious people who will never be Tories but have no place in Parliament because of the state of the Labour Party and whose talent is therefore shut out.

Two things must happen.

First, there should be a parallel debate in and out of the Labour Party about the future of progressive politics, how it is reconstructed and reshaped into a winning coalition. This should include Labour, traditional left and right, the Lib Dems, those disenchanted with both main parties and those not at present engaged in any party. It must be a big tent debate, open and frank.

Second, we need urgently a new policy agenda for progressive politics. At the heart of it will be understanding and mobilising the technological revolution, the 21st-century equivalent of the 19th-century industrial revolution. It will mean a complete re-ordering of the way state and government is conceived and organised; huge focus on education and infrastructure; new ways of dealing with generational poverty; a recasting of corporate governance and responsibility; a stimulus nationally and internationally of the science and technology for environmental change; and very specific measures to connect communities and people left behind by the changes driven by globalisation.

We need policy for the future. Radical but modern. The agenda of the far left is not progressive; it’s a form of regression to an old statist, tax and spend programme of the Sixties and Seventies.

I understand why for some it has real attractions. It speaks to the intense feelings of marginalisation and desire for radical change.

It is a cry of rage against “the system”.

But it isn't a programme for government.

To win power, we need self-discipline not self-indulgence; listening to what people are truly saying, not hearing only the parts we want to hear; understanding that you can't play with passion alone, but require strategy, preparation and professionalism; winning the intellectual as well as political battle.

In 1983, after my first election, having been out on the doorstep for several weeks listening to Labour voters telling me that they were voting Labour despite the state of the party rather than because of it, I attended a meeting in my constituency organised by the far left, still strong after the Bennite surge, entitled “Learning the Lessons of Defeat” or some such.

Dennis Skinner was the main speaker. At the outset, the chairman urged us to be honest. Naively, I took this instruction literally. I did speak honestly. I said we were way out of date in our thinking, were far too left, seemed like we were living in the era of black and white TV, in an age of colour, and so on.

I was heard in silence. Right after me, came Dennis, who tore me limb from political limb.

I came out of the meeting in shock. My very wise agent John said to me: “You were the only person talking sense, but in future learn to say it better.”

By 1994, when standing for the leadership, I had learnt to say it better. I chose my ground carefully. I didn't unnecessarily offend.

But, no one doubted where I stood.

The Labour Party is presently marooned on fantasy island. I understand would-be leaders will want to go there and speak the native language in the hope of persuading enough eventually to migrate to the mainland of reality.

But there is a risk that the only people speaking the language of reality to the party are those who don't aspire to lead it.

Unfortunately, 2019 is much worse than 1983.

Then was our second defeat; now is our fourth. The country is different. Politics is different. The country is less fixed in political affiliation. Politics moves at speed accelerated by social media.

We don't have the luxury of the slow march back.

We can correct our historical and contemporary weaknesses; or be consumed by them.

But that choice is unmerciful.

And before us, NOW.