When Lord North, prime minister at the time of the American revolution, received the news that British forces had lost the war, and with it the American colonies, he was reported to have been physically struck by the magnitude of the news, as if hit by a musket ball. “Oh God, it’s all over,” he wrote in his diary.

The loss of the American colonies was the first time the process of British empire building had been put significantly into reverse, and became the starting point for a nostalgic yearning for lost colonies – and the wealth and global influence that came with them – that has become part of our national psyche. Even the building of a second British empire in the 19th century never fully healed the wound of losing America, and the end of Britain’s imperial prestige after the second world war has cut deeper.

Yet there has long been a vaguely defined notion that, somehow, Britain might reassemble chosen fragments of the empire. Winston Churchill talked of a global alliance of the English-speaking peoples, which included America, and generations of politicians and historians of all political stripes have from time to time nursed similar ideas, latterly calling the concept the Anglosphere.

Part of the problem with such schemes today is that they emerge out of a national conversation about the lost empire that is all too often an internal monologue, largely focused on what “we” did or did not do, and whether or not “we” should be proud of it. What “they” – the peoples of the former empire and the current Commonwealth – felt about it then, and think about it now, rarely enters into this solipsistic debate.

But since last week – when it was reported that Whitehall officials had described plans for Britain’s post-Brexit trading relationship with the Commonwealth as “Empire 2.0” – “they” have had a lot to say. African and Indian journalists and bloggers have had a lot of fun at our expense, ridiculing the idea. Worse still, the phrase – which might have been used ironically – emerged when the formidable Indian diplomat turned historian Shashi Tharoor was in the UK, promoting his damning new book on Britain’s colonial exploitation of India. Strident as ever, Tharoor was in no doubt that the idea of Empire 2.0 would go down in India “like a lead balloon”.

This newfound focus on the Commonwealth feels uncomfortably akin to recent divorcees looking up their former partners on Facebook; and being shocked to discover that they have got married, had kids and moved on. They might have fond memories, they might even want to be on good terms, but don’t really miss us. Former colonies, like old flames, build new relationships, based on their own needs and ambitions. Some may well be up for a better trade deal or more freedom of movement, but they don’t want to be part of Empire 2.0, any more than most of them wanted to be part of Empire 1.0.

But this is not the only neo-colonial fantasy to have been recently taken out of the deep freeze and hastily reheated. Another is the Canzuk concept, the dream of a free trade and free movement zone between the UK, Canada, Australia and New Zealand – three nations from what used to be called the “white dominions”. The theory suggests that because these countries have similar institutions and cultures it is natural for them to be brought closer together.

We’ve been dreaming these same dreams for more than half a century. In 1962, the Labour leader Hugh Gaitskell gave a party conference speech on the Commonwealth, in which he talked of the bonds that had been forged between Britain, Canada, Australia and New Zealand during the first world war battles of Vimy Ridge and Gallipoli. As we mark the passing of a century since those battles, it’s impossible not to recognise that in Ottawa, Canberra and Wellington they are remembered as milestones in the journey towards independence and nationhood; moments when those former dominions began to move away from Britain rather than closer to her. Once again, “they” have their own memories, and they do not always tally with ours.

Ultimately, what makes Empire 2.0 a fantasy are the forces of geography and history. While Canada, for example, has undoubted cultural links to the UK, she also has a 5,500-mile border with the US. This, and her membership of Nafta, means that her key trading partner is, inevitably, the US, not the UK. Australia and New Zealand have perhaps even stronger emotional and cultural links to Britain, but as the economies of Asia have risen, the antipodean nations have found themselves on the doorstep of the greatest manufacturing region on earth. Australia has just been through a mining boom akin to the gold rushes of the 19th century, and it is to Beijing, not London, that their importers and exporters look.

In anglophone Africa, the game is already up. The motorbikes on the freeways of Accra and Lagos are Chinese, assembled by local mechanics from kits shipped direct from Shandong. West Africa’s new convenience food is Chinese instant noodles, not fish and chips, and the supermarkets that sell them are South African-owned. Many anglophone Africans still have deep emotional, economic and often familial links to Britain, but those with money are now as keen to holiday in Dubai as London.

Yet the most jagged rock upon which the Empire 2.0 fantasy flounders is history itself. Britain in the 19th century was two things simultaneously; the hub of the largest empire on earth and the greatest manufacturing and trading nation the world had ever seen. Yet the formal empire and the trading empire were not the same thing. While the empire, especially India, undoubtedly helped make Britain rich, even at the height of our imperial power we traded more with Europe and the United States than with the colonies. It was to the booming cities of America, and to the slave-driven cotton economy of the deep south, that British capital surged in the 19th century. And while much of Africa was painted imperial red on the maps that famously hung on every classroom wall, Britain did more trade with tiny Denmark than with Nigeria, one of her biggest west African colonies.

The empire, even at its height, never came close to absorbing the majority of our exports or providing the bulk of our imports, and neither will the Commonwealth, no matter how good a trade deal we win. Empire 2.0 is a fanciful vision of the future based on a distorted misremembering of the past. It’s a delusion and, like all delusions, has the potential to lure us into a false sense of security and lead us to make bad decisions.