There’s a lot of talk these days about new political tribes, so allow me to introduce another one: the “anti-woke”. The first curious thing to note about members of this group is that they define themselves in opposition to an identity that doesn’t actually exist. They are anti-woke, even though there is no “woke”.

Woke – a term once used by African Americans to denote people who were alert to racism and social injustice – has been retired. As is often the case with black innovations, overuse by the white mainstream killed off its authenticity. Today, the person using the word is likely to be a rightwing culture warrior angry at a phenomenon that lives mainly in their imagination.

Take Douglas Murray, whose new book is almost entirely a rant against what he perceives as a kind of authoritarian “wokeness”. The real cause of conflict and polarisation in political and social discourse today, he argues, is this group (whose existence I think he’s imagined) pursuing its grievances (which he thinks we have imagined). Confusing, I know, but the simple version of his argument is this: the struggle for feminist, gay, trans, and black equality is over; the battles are won. And for some strange, irritating reason, instead of just enjoying the spoils of the victory, the woke keep on complaining that they want more.

Or take Camilla Long, whose recent Sunday Times column complained that when woke hero, the Canadian prime minister, Justin Trudeau,, was outed as having worn blackface, the woke let him off. Because they have double standards that police the right but reprieve the left.

In reality, blacking up has always been offensive to black people – it’s just that in the past, the mainstream ignored us when we said so. And the fact is that the woke were swift to call Trudeau out, as they would have with any other public figure who did something racist: CNN ran a story about how it would cost Trudeau his black support, and the Labour MP David Lammy pointed out that the affair showed how much racist thinking still prevails.

Which leads me to the second strange thing about this worldview. It’s the idea that the woke have laws and a police force to enforce their wishes.

I had my own run-in with Murray during an interview about the singer Sam Smith, who has come out as non-binary and asked to be referred to using the pronoun “they”. Murray – who, as a gay man, has little choice but to acknowledge that the progress towards LGBT rights has advantages – thinks non-binary rights, on the other hand, “beyond parody”. Murray’s thesis is that the woke now seek to oppress everyone by insisting on the use of pronouns like “they”, which he regards as “dementing”.

When I called Smith “he” – mainly because I was referring to “him” in the past tense, while at the same time defending their right to self define as non-binary in the present – Murray’s theory would dictate that I would be slaughtered by other woke people furious at my misgendering. On the contrary, there was silence.

Some might say that is because the woke regard me as an ally, and they therefore let me off the hook. In other words, I fall into the woke exemption. But the truth is, there are no woke police. Instead there are – and here is the real enemy of the reactionary right – a lot of people who recognise that the progress that has been made is all the result of struggle. Barring those who would still like to deny women reproductive rights, who use overtly racist language (among these I include our prime minister) or who dislike the idea that Black Lives Matter, mainstream opinion no longer advocates a return to segregation, violent racism or the formal exclusion of women from social, political and economic life.

But these gains are the result of constant, relentless struggle, as each era has battled against the conservatism of the time. A struggle in which people like Murray have always been on the wrong side, but whose gains end up benefiting us all.

Which brings me to the final, sweetest irony of the anti-woke. Perhaps their greatest grievance of all is the alleged “weaponisation” of identity. The idea that the woke – which tends to include brown and LGBT people, among many others – have the audacity to express and be proud of their identities, rather than keep them quiet, the better to fit in with conservative norms.

In reality, the only thing that unites the woke is an intellectual curiosity about identity and how complex, how nuanced, how rooted in disparate histories it can be. The real groupthink, the genuinely cohesive crowd, it’s increasingly clear, is that of the anti-woke, the most weaponised identity of all.

• Afua Hirsch is a Guardian columnist