It feels like I live in the middle of a culture war. On one side is a kind of state-sponsored amnesia. It’s pervasive. It’s an Oscar-winning movie perpetuating the idea that Winston Churchill stood alone, at the Darkest Hour, as Nazi fascism encroached, with Britain a small and vulnerable nation isolated in the north Atlantic. In reality the United Kingdom was at that moment an imperial power with the collective might of Indian, African, Canadian and Australian manpower, resources and wealth at its disposal.

It’s also Poland passing a law so that errant historians, survivors or Auschwitz guides who raise the inconvenient fact of Polish complicity in atrocities now risk up to three years’ imprisonment. It’s Tennessee in the US legislating against the removal of Confederate statues when, as the former New Orleans mayor Mitch Landrieu puts it, they “purposefully celebrate a fictional, sanitised Confederacy; ignoring the death, ignoring the enslavement, and the terror that it actually stood for”.

On the other side are those who understand that historical narratives, monuments and statues are not some pristine record of history, but projects – often created long after the event they remember – that have weaponised history against specific groups. This is why South Africans question statues that glorified apartheid, why Native Americans protest against Thanksgiving, why indigenous Australians required a correction to the ludicrous ideas that Captain Cook “discovered” their continent or that they should celebrate the intrepid explorers who massacred their ancestors.

The bronze statue of British colonialist Cecil John Rhodes is removed from the Cape Town University campus, South Africa. Photograph: Schalk van Zuydam/AP

There are the New Yorkers who removed the statue of J Marion Sims, the gynaecologist who experimented on enslaved black women without anaesthetic; the Ghanaians who removed a Gandhi statue; and the Bristolians who successfully campaigned to change the name of the vicious slave trader Edward Colston from the city’s famous concert hall. There are people in the Caribbean asking why they have churches dedicated to the slaveowners who tortured their ancestors; or in the case of Barbados, a statue of Horatio Nelson – friend to slaveowners and plantation interests – right in the middle of their capital.

My entry into this war zone happened by accident a year ago, when I suggested on these pages that we take another look at Nelson’s legacy. I wasn’t actually waiting in a bulldozer, ready to storm Trafalgar Square, as some people seemed to believe, but simply calling for an appraisal of the hitherto obscured facts.

Our great, white male media grandees meet simple statements of fact about the historical record with hysteria

Those facts remain of minor consequence to the vocal and influential parts of British society that regard the act of raising them as heretical. The instinct is to shoot the messenger. Our great white male media grandees – it is a remarkably consistent demographic – meet simple statements of fact about the historical record with hysterical, table-thumping personal attacks. It’s the kind of lashing out that happens when you try to wean someone off an addiction. In this case, Britain is addicted to glory.

It’s understandable: it must feel nice. But, like all addictions, it brings out the worst in our psyche. Making a TV documentary on this subject that will air on Tuesday night, I was told British people should be proud of our empire because our colonies were better run than those of the Belgians or those under Nazi rule. As the rapper and activist Akala says memorably in his new book, Natives: “It’s true, but it’s a shit boast.”

When the supply of glory is threatened, the gloves are off. Racial slurs become acceptable; the threats begin. One respected academic told me he was advised that if he pursued the study of Churchill’s responsibility for the number of deaths in the Bengal famine, his academic career would be compromised. This is the level of censorship to which we are willing to stoop.

Britishness – at least this patriotic, defensive, glory-addicted version of it – seems to be in a highly fragile place. It cannot withstand being problematised or critiqued. Many of those I’ve raised this with in recent weeks feel especially aggrieved that someone who was apparently “allowed” into Britain’s educational and professional establishments should feel anything other than gratitude. Being privileged, and black, and then having the audacity to use my intellect and education to challenge the narrative those establishments handed me down, seems to prove particularly irksome.

But we can’t detoxify Britishness and build it into a more robust, less fragile identity, until we do this work. We need to assess the true legacy of empire and the impact of its loss. We need to unpick the idea of glorious Britannia that defeated Napoleon and Hitler to find out what wrongs are still lingering in our midst.

These wrongs do linger, because people on the other side of the imperial experience – descended from the enslaved, from the victims of colonial wars or from ravaged and exploited nations of empire, as hundreds of thousands of British people now are – are unwilling to accept the idea that the lives of their ancestors are worth less than those of the glorious British perpetrators we celebrate.

I’m one of them. And when I am accused of “coming here” and “attacking our culture”, it becomes abundantly clear that the fact that someone like me is British, and that Britain is also the country made possible by the labour, wealth and culture of my antecedents, still hasn’t actually sunk in.

That’s why this is not actually about statues, monuments or history, but about culture and narrative, and who gets to feel the “high” of glory. People who had that luxury stripped away from them several centuries ago already know it’s not that simple. But in Britain, there are so many still looking for their next hit.

• The Battle for Britain’s Heroes will air on Channel 4 at 9pm on 29 May