Usually, when a new campaign to rectify our national failure to acknowledge black ingenuity in building modern Britain comes along, I support it enthusiastically. I’ve backed the critique of our statues, for example, which promote white supremacists as heroes and ignore people of colour; and of our curriculum, which promotes ignorance about the racialised nature of British history. I’ve seen the need for a museum of empire, and the importance of books such as Washington Black, which told the fictional story of a black inventor erased from the historical record.

But when I was asked to add my name to the petition calling for a black person to be the face of the new £50 banknote – a decision expected by the Bank of England this summer – I was hesitant. Not because I have any doubt about the strength of the argument. There have been 24 banknotes featuring a notable person since the first was issued in 1970: all have been white, and only three have been women – the last of these, Jane Austen, was chosen only after an extensive social media campaign and the threat of litigation.

The campaign for banknotes of colour makes an excellent case. Black and ethnic minority people have been in Britain for millennia and yet their contributions have been systematically overlooked. Born into slavery, Oluadah Equiano’s phenomenal campaigning for abolition has been buried under the preferred memory of William Wilberforce. Efforts to introduce British people to Mary Seacole, the black nurse of the Crimean war, are still smeared as a “PC myth” in favour of her contemporary Florence Nightingale, whose own controversies are rarely mentioned. In fact, during centuries when Britain regarded black people as an inferior species, we can assume that those who gained enduring recognition had to be all the more remarkable to overcome the odds stacked against them.

The question for me is not whether black people deserve to be on the money, but whether the money deserves us.

In the US, when the giant of the Underground Railroad, Harriet Tubman, was announced as the face of the new $20 note – the first woman and African American to be commemorated on the currency – some voiced unease about the image of a woman who was enslaved being placed next to the price of a bill. That unease stems from a rational distrust of American capitalism, which is yet to fully acknowledge, let alone begin to atone for, the depth of its original and ongoing sins against its black citizens. Is it really progressive to cast those who rebelled against the US’s enslavement of Africans – a device to industrialise the black body in a depraved pursuit of profit – on the currency that symbolises that very system?

The inherent link between racism and capitalism has been well documented. Hegel and Nietzsche saw how essential the exploitation of so-called inferior people would be to European growth – the former said a “civilised nation” understands that “the rights of the barbarians are unequal to its own and treats their autonomy as only a formality”. In the UK, the Bank of England both emerged from and presided over a system that commoditised black people, enabling Britain’s historic economic growth. Four Bank of England governors and directors owned slaves, slave ships or plantations, or were chairmen of the Society of West India Merchants – which represented those interests. After slavery, which stole black bodies, empire stole their land.

As long as we still use that system, we must relentlessly acknowledge that history. In the end, I joined 150,000 others who added their names to the banknotes of colour campaign: and the importance of this movement was driven home by the response of the Bank of England itself. In contrast to other nations such as Canada and New Zealand, which have ethnic minority figures on their currency, the Bank barely even confronted the request over the new £50 note – instead deflecting questions of race on to the need to recognise “military service” and “science”, both of which have already been reflected on our banknotes for decades.

Like many other British institutions, the Bank of England talks a good game when it comes to diversity. I have been to speak there myself, while the governor, Mark Carney, and other senior figures nodded earnestly at my words about history, and how it affects all of our racial identities. Yet instead of taking this campaign seriously, it seems to be taking a stance closer to President Trump’s, whose administration has delayed the date of the new Harriet Tubman bills – initially planned for 2020 under President Obama – to 2028, which looks a lot like kicking it into the long grass.

Including black figures on banknotes can’t resolve our real problems of racial injustice. But the refusal to include them is a powerful reminder of how little our institutions seem to care.

• Afua Hirsch is a Guardian columnist