The world is wrong,” wrote the American poet Claudia Rankine. “You can’t put the past behind you. It’s buried in you; it’s turned your flesh into its own cupboard.” To be black, in a society that invented race for the specific purpose of dehumanising people who are black, and then invented an equally formidable system of denial, is to carry the burden of history that others would rather forget.

I found myself having to explain this reality last week, on the Sky News show The Pledge, in what I had hoped would be a debate about the utility of Trump’s “shithole countries” remark, and the racism of Jo Marney, girlfriend of the Ukip leader. Remarkably, given the premise, the argument became a race to the bottom. “Does racism exist any more?” my white co-panellists wanted to know. They thought not.

I don't claim to be an example of extreme disadvantage. Yet it’s fascinating how people attack me for my background

There are so many ways to prove the simple falsehood of this belief, it’s hard to know where to start. How about how people self-define? The British Attitudes Survey last year found that one quarter of British people acknowledged they were racist. A YouGov poll in 2014 found that most British people thought the British empire – whose ideology was one of the innate superiority of the white race – was something to be proud of. In 2015 a survey of more than 24,000 people found that 30% of employees in the UK had witnessed or experienced racial harassment in the workplace first-hand in the previous year.

These are the least sophisticated indicators of how race works in Britain. Outcomes influenced by structural, persistent inequality are only now beginning to attract the research they deserve, by, for example, the prime minister, in her major race disparity audit last year. This found that unemployment among black, Asian and minority ethnic people was nearly double that of white Britons. Though the number of ethnic minority graduates is growing, they still face penalties in the job market compared with their white peers. The Guardian’s own research found that just 3% of Britain’s most powerful elite are from ethnic minority groups.

In all this, race intersects with class disadvantage and deprivation – which I, personally, have not experienced. I never hesitate to point this out. When I raise, passionately, the question of how the system of race operates in modern Britain – as I have done recently ahead of the launch of my book on identity – I do not claim to be an example of extreme disadvantage. Yet it’s fascinating how so many people’s instinctive reaction is to attack me for my own background. My former Guardian colleague Michael White, for example, suggested my argument somehow lacked legitimacy because I have benefited from “white privilege”; David Goodhart, the founder of Prospect magazine, called me a “high priestess” of the “religion of antiracism”, which “encourages victim status among minorities”.

Afua Hirsch on The Pledge - video

It’s fascinating when white people, who invariably have no personal experience of the frequent othering and subtle prejudice that comes from being born or raised in a country that does not recognise your unconditional right to its identity, tell you what you have and have not experienced.

“Life’s moved on from race,” one of my fellow panellists told me on The Pledge. “If it’s well intentioned, it’s not racism,” said another. All of this was, very ironically, good evidence of my point: that white fragility operates powerfully against progress; that there are those in our society, including high-profile and influential people, who prefer defensiveness to a cold, hard analysis of the patterns of prejudice.

The ensuing debate makes uncomfortable watching. Taking part was worse. I usually have a rule not to talk about race in this setting – adversarial debates in which I’m in the extreme minority are not a good space for talking about things that are both personal and poorly understood. It’s hard to talk about the personal in a public setting at the best of times. When the content relates to the experience – since childhood – of white people delegitimising your voice, then having to defend that before a group of white people who attempt to delegitimise your voice, is doubly painful and draining.

I was not alone in my frustration. “I sometimes wish there was a rule that anyone publicly discussing racism, without at least a basic understanding of these two paragraphs, could be politely asked to go away and read them before continuing,” tweeted Matthew Ryder, the deputy mayor of London, highlighting findings of the Macpherson inquiry into the death of Stephen Lawrence back in 1999, that specifically deal with the role of “well-intentioned racism” in perpetuating historic problems, and the failures of a “colour-blind” approach.

While some white people are enjoying their colour-blindness, people of colour are getting on with having no choice but to live in a racialised reality. Take any random day. In a 24-hour period this week, Nadiya Hussain, who won the Great British Bake Off, wrote: “I’m Muslim, Brown, working class and a woman! I may as well have punching bag written on my torso.”

Michaela Coel, the writer and actor whose series Chewing Gum won two Baftas, tweeted: “I used to think the door to BAME working class writers had been left ajar by the UK industry … I’m now realising it was more like the keyhole of a firmly bolted door, that I painfully squeezed through. Open the fuckin’ door.”

Idris Elba, one of Britain’s best known actors, called out the tokenism that sees him constantly referred to not as the actor who could play James Bond, but as the person who could fulfil the need for the producers of Bond to say they cast someone black.

These are the nation’s black and brown national treasures. The fact that they have attained this status is testament to the extent to which Britain has changed; the fact they feel compelled to use their platforms to reveal the prejudice they face is testament to the extent it has not. To write them off as people desperately seeking some kind of victimhood is spectacularly offensive.

None of us want to have basic arguments about race. Speaking for myself, there are a million things I would rather do. Instead I find myself on TV arguing over the very fact of my existence, the reality of my daily life, while being shouted down by people who have neither experienced it nor had the humility to acknowledge they have not.

“I don’t see colour,” they say. And so I find James Baldwin echoing round my head: “I don’t really believe in race. I don’t really believe in colour. But I do know what I see.”

• Afua Hirsch is the writer of a fortnightly column for the Guardian