It has all the ingredients of a John le Carré novel. For decades there are allegations of terrible abuse during the Mau Mau rebellion; historians are baffled by missing documentation. A court case finally prompts the Foreign Office to discover hundreds of boxes of previously hidden papers stored in a house, Hanslope Park, in Buckinghamshire. They reveal not just the brutality – which historians had already unearthed – but official recognition of the illegal violence and dogged determination to cover it up.

Illustration by Andrzej Krause Photograph: Guardian

The Foreign Office attributed the forgotten boxes to "an earlier misunderstanding about contents" and stated that there needed to be an "improvement in archive management". In a superbly smooth statement, the Foreign Office commented that "it was the general practice for the colonial administrations to transfer to the UK ... selected documents held by the governor which were not appropriate to hand on to the successor government". I'd cast Bill Nighy for that bit of the script.

But the Mau Mau boxes buried in Buckinghamshire are only a small part of a hoard of 2,000 detailing the end of empire in 37 British colonies. Without skipping a beat, William Hague announced that their release was "essential to upholding our moral authority as a nation". An odd comment to make while a court case was revealing detailed and graphic descriptions of horrific violence perpetrated by the British on thousands of Kenyans. Hague even had the chutzpah to go on to declare that our willingness "to shine a light on our faults and to learn from mistakes of the past is an enduring strength of British democracy".

So just to be clear: cover-ups are problems in "archive management", records of illegal violence are "inappropriate", and in case we are in any doubt, Britain's moral authority as a nation continues, regardless of the inconvenient truth. For anyone interested in how narratives of national identity are maintained through all manner of contrary evidence, this is a textbook case. The former detainees have now flown back to Kenya to await a verdict on their compensation claim against Britain.

Meanwhile, does Hague realise quite what he has promised with his clear-out of the attics of Hanslope Park? Myths about the British empire abound, and one of the most cherished is that its end was orderly. As Peter Oborne put it recently, "compared with the French, the Belgians or the Italians, we handed over our colonial empires with good-natured and civilised ease". This is the orchard myth; Disraeli once talked of colonies as fruit that dropped off the tree as they ripened. How very English to apply such a bucolic metaphor to strategies of global power.

The reality – as historians have been doggedly trying to point out in recent decades – was violent, often chaotic, and marked by a desperate struggle to maintain British prestige and influence. It is the end chapters of the empire that reveal most starkly the naked pursuit of self-interest and the willingness to use all available means to achieve it. On the subcontinent there was horrific ethnic cleansing with the deaths of about 500,000 and a bloody history continued for most of the middle decades of the 20th century: from Palestine and Malaya to Cyprus and Kenya.

Britain's strategy was to repress any challenge to the friendly regimes it wanted to institute as it withdrew from formal political control. It was an era dominated by the linked challenges of frenzied nation-building and counterinsurgency. A book published next month charts this era of realpolitik with chilling detail; Benjamin Grob-Fitzgibbon's conclusion in Imperial Endgame is that "liberal imperialism can only be sustained by illiberal dirty wars. Britain's imperial endgame demonstrates that it is possible to achieve success in each. Whether moral or not is a question best left to philosophers and kings."

But there is an even bigger myth about the end of empire that could be dismantled by the contents of Hanslope Park's attics. Namely, that it was the end at all. Retreat in some areas, such as India, contrasted with continuity in others, such as the Middle East. In many places the mechanisms of projecting British power simply reverted to those used in building the empire in the first place: avoid formal political control and use trade, finance and military power to build alliances with client states who could secure British influence. Use military intervention when necessary (for instance, SAS involvement in the Dhofar rebellion 1962-75). The only postwar innovations were first, to incorporate the US as imperial successor, and second, to add aid as another strategy for British influence. But the bigger picture is of continuity; that's why the notion of apologising for empire is so odd – it's not really over yet, we just became junior partners to our successors, the US.

There are two reasons why all this is of interest – or should be – to more than historians: first, much of British decolonisation policy is with us still – similar aims, methods, language and justifications. The continuities are unnerving; politicians were talking of protecting "our way of life" half a century before Blair did. When counterinsurgency stalled in Afghanistan, Malaya was the model examined most closely.

Second, this imperial endgame explains so much about today: for instance, the growing crisis in Bahrain, where new arrests over the weekend appear to herald a fresh bout of violent repression, and why we are not currently bombing this Gulf state with as much enthusiasm as we are Libya.

It has been one of the most successful chapters in British imperial domination; the Al Khalifa dynasty signed its first treaty with the British in 1820 and they finally "left" in 1971. The British have backed a repressive regime in a very cosy, mutually advantageous relationship of finance, military training, arms deals and royal ceremony (one of the less edifying aspects of the imperial endgame has been the use of the royal family to flatter and seduce client regimes, however unpalatable). In the last few months the Bahrain government has beaten, killed, tortured the Shia protest movement. On Saturday, the Guardian reported that Bahraini students who had protested against this repression in Britain now feared violent reprisals. The west has done little but mumble incoherently; too many interests are at stake to live up to the grand moral rhetoric now being lavished on Libya.

In Asia, Bahrain is characterised as evidence of the west's endemic hypocrisy: it promotes democracy and human rights only when it suits its self-interest. It's a sobering reminder that the day will come when we are no longer the ones who decide how our history is told.

Pleading ignorance or forgetfulness of the imperial record will hardly wash. Indeed, one of the most striking continuities of Britain's quest for power has been a studied forgetfulness. It was often said Britain acquired an empire "in a fit of absence of mind", and much the same appearance of distracted pragmatism – a sort of "we do what has to be done with no great masterplan" – applied to the imperial endgame. Very quickly, empire became an obscure subject reserved for a few historians to worry about; everyone else was instructed to "move on". Has any empire been so quickly forgotten by its imperialists as Britain's?

Forgetfulness proved a clever way to mask ruthless ambition. And, as we have seen in the last week, it's also a deft manoeuvre to conceal official complicity in brutal violence. Muddle, confusion. "Oops, dear me, we have a problem with archival management." It's been the Miss Marple model of empire, but who are we kidding?