Britain’s 40-year effort to find a way to either influence or dislodge Robert Mugabe is one of the country’s great post-war foreign policy failures. It is a story spanning six UK prime ministers, nearly £1bn in aid and every conceivable strategy.

Whether the cause of that failure lies at the door of a colonial mindset in the Foreign Office, a failed land transfer policy, the collective weakness of the Commonwealth, a cowardly African political elite or simply the corrupt thuggery of Mugabe himself will be a matter of dispute for generations.

At the heart of the story is misunderstanding and apprehension, neatly illustrated by Lord Howard de Walden’s description of Christopher Soames on being told the latter was to be made Britain’s interim governor in Rhodesia during the colonial handover in 1979: “A cloud passed over his face, as if he could see himself being plucked of his plumed hat and being eaten by savages. There was no alternative to agree. Before leaving the building to face the press, he asked for another drink.”

Fears for Soames’s personal safety in Zimbabwe proved misplaced, but from the outset the Foreign Office had doubts about Mugabe, and how to respond to the manipulation of elections by his Zanu-PF party. Diplomats told Lord Carrington, the then foreign secretary, that the ballot rigging was blatant.

Carrington – who had been reluctant even to send outside election observers – demurred on calling for Zanu-PF to be banned from participating, in part believing, according to his biographer Christopher Lee, that Mugabe would not be elected.

When he was indeed victorious, and by a landslide, Carrington was reassured by the tone of Mugabe’s inauguration speech as prime minister, which preached reconciliation and forgiveness worthy of any 20th century liberation hero.

A policy of benign tolerance, and hoping for the best, ensued for nearly a decade. As late as 1988 Carrington wrote he had been fortified by Mugabe’s magnanimity and intelligence, seemingly unaware of the slaughters Mugabe instigated in his political rival’s power base, the two Matabeleland provinces of western Zimbabwe, from 1982 to 1987.

Timeline Robert Mugabe Show Mugabe is born in what was then British-ruled Southern Rhodesia. He is educated at Catholic schools and attends South Africa's University of Fort Hare. During the 1940s and 50s he teaches in Zambia and Ghana, where he is influenced by African independence movement leaders. After campaigning for Zimbabwe's independence, he is imprisoned for political agitation. While incarcerated, he earns two law degrees from the University of London. Released from prison, he escapes to Mozambique where Zimbabwe African National Union guerrilla fighters elect him to lead their struggle against white minority rule. Mugabe's Zanu-PF party wins independent Zimbabwe's first election and he takes office as prime minister. Mugabe deploys North Korean-trained troops to crush an insurgency by former guerrillas loyal to his liberation war rival Joshua Nkomo. Government forces are accused of involvement in the killing of 20,000 civilians, which Mugabe denies. Mugabe becomes president with sweeping executive powers after changes to the constitution and signs a unity pact with Nkomo, who becomes one of his two deputies. In 1990, Zanu-PF and Mugabe win further parliamentary and presidential elections. An economic crisis marked by high interest rates and inflation sparks riots. Zimbabweans reject a new constitution in a referendum, Mugabe's first defeat at the ballot box. Thousands of independence war veterans and their allies, backed by the government, seize white-owned farms, saying the land was illegally appropriated by white settlers. The US puts a financial freeze on Mugabe's government in response to land seizures, beginning a wave of sanctions against his regime. Mugabe wins a disputed presidential vote, which observers condemn as flawed. Zimbabwe is suspended from the British Commonwealth over accusations of human rights abuses and economic mismanagement. Hyperinflation reaches 500bn per cent, forcing millions of people to leave the country. Mugabe loses a presidential vote but wins the run-off after opponent Morgan Tsvangirai withdraws, citing violence against his supporters. A power-sharing agreement is signed. Despite reports saying Mugabe has been seriously ill with cancer, he wins another disputed presidential vote. Western observers cite multiple accounts of electoral fraud. After protesters stage the biggest show of defiance against Mugabe in a decade, he is forced to resign following an army coup and is replaced by Emmerson Mnangagwa. Mugabe dies aged 95.

Above all, Carrington was reassured that Mugabe did not seem willing to tamper with the special protection sunset clauses in Lancaster House given to white Zimbabwean landowners at least for the first 10 years of independence, including provisions that the government of Zimbabwe would not engage in compulsory land acquisition but would instead proceed on a “willing buyer, willing seller” principle.

The warning signs for the British became unmistakable in 1992 when, with the economy stagnating, a compulsory land acquisition programme slowly started. But in many ways 1997 was the decisive year both for Zimbabwe, and its relationship with Britain, now led by a Labour government with an ethically-driven foreign policy.

Although Mugabe had won the 1995 elections easily, the arrival of a pressure group of war veterans demanding land soon made itself felt. In November 1997 Mugabe offered large one-off payments to each of the 70,000 “war veterans”, in addition to a monthly payment.

Tony Blair and then foreign secretary Robin Cook initially responded by convening a Land Donors conference to address the issue, but then pulled out of talks. Mugabe’s hatred of Blair started, as he accused the British of trying to engineer a coup by funding his political opponents.

As the Zimbabwean economy rapidly deteriorated, tensions over land reform reached boiling point in February 2000, when the government’s new draft constitution, which contained a clause providing for land acquisition without compensation unless paid for by the British government, was defeated in a referendum. Shortly thereafter, so-called “war veterans” invaded white-owned farms across the country.

As late as 2000 Cook offered an extra £40m to fund land reform. But he set conditions including “a fair price to the farmer” and reducing poverty among the working poor.

Labour believed Mugabe’s programme was not only unlawful, but corrupt. Half of all the farms redistributed since 1997 had gone to employees or members of the Zimbabwean government.

The shutters went down. Britain imposed an arms embargo against Zimbabwe on 3 May 2000. It halted the provision of 450 Land Rovers to its police force, withdrew the British military advisory training team and cut aid to Zimbabwe by one-third. An EU-wide travel ban was imposed on Mugabe and 19 members of his inner circle.

Blair did not just try isolation. He used the Commonwealth to urge Mugabe to sign a declaration at Abuja in 2001 and elsewhere later committing to democracy. But such words proved largely worthless when it came to actual elections won by Mugabe. Blair led the call to suspend Zimbabwe from the Commonwealth in March 2002. Mugabe never forgave him.

Since then all the weapons of diplomacy – quiet megaphone, economic, political, multilateral, unilateral – were used to weaken Mugabe, or strengthen his opponents in the stolen elections that have succeeded.

Short of using force, the British alone had the ability to wound, but never to end the Mugabe regime. Mugabe was repeatedly protected by other African leaders, notably the ANC in South Africa, that declined to criticise him on the grounds of non-interference and national sovereignty.

“Mugabe was one of those people the British empire created who specialised in knowing how to twist the British government’s tail,” Lord Hurd, another British foreign secretary, said: “He was well-trained in the art of annoying the British if he needed to. He knew our ways.”

• This article was amended on 9 September 2019 because Matabeleland is in the west of Zimbabwe, not the east as an earlier version said. This has been corrected.