The furore over Penguin’s wise and brave decision to “reflect the diversity of British society” in its publishing and hiring output seems to have awoken the usual knuckle-dragging, semi-blind suspects with their endlessly repeated terrors and fears. They appear to believe that what is called “diversity” or “positive action” will lead to a dilution of their culture. Their stupidity and the sound of their pathetic whining would be funny if it weren’t so tragic for Britain. You might even want to call it a form of self-loathing; it is certainly unpatriotic and lacking in generosity.

The industries I’ve worked in for most of my life – film, TV, theatre, publishing – have all been more or less entirely dominated by white Oxbridge men, and they still mostly are. These men and their lackeys have been the beneficiaries of positive discrimination, to say the least, for centuries. The world has always been theirs, and they now believe they own it.

Some of us have been fortunate enough to force a way through the maze and make a living as artists. It was a difficult and often humiliating trip, I can tell you. There was much patronisation and many insults on the way, and they are still going on.

We are still expected to be grateful, though those in charge – never having had to fight for anything – have always been the lucky ones. And these lucky ones, with their implicit privilege, wealth and power – indeed, so much of it they don’t even see it – are beginning to intuit that their day is done. Before, with their sense of superiority and lofty arrogance, they could intimidate everyone around them. No more.

It was never not a struggle to become an artist, with racism, prejudice and assumption all around, visible and invisible. I remember standing in a room with Salman Rushdie in the early 1990s, just after The Buddha of Suburbia had been published, discussing how it could be that we were the only people of colour there – indeed, the only people of colour in most of the rooms when it came to books. And that was the case with all of the culture industries. The first TV producer I ever met asked me why my characters had to be Asian. “If they were white, we’d make this,” he said to me.

It is not coincidental that at this Brexit moment, with its xenophobic, oafish and narrow perspective, the ruling class and its gatekeepers fear a multitude of democratic voices from elsewhere and wish to keep us silent. They can’t wait to tell us how undeserving of being heard we really are. But they should remember this: they might have tried to shut the door on Europe, refugees and people of colour, but it will be impossible for them to shut the door on British innovation. We are very insistent, noisy and talented.

When I was invited to join Faber, in 1984, the fiction editor was Robert McCrum. He was excitable then, and so was I. I couldn’t wait to be on his list of writers, since he was publishing Kazuo Ishiguro, Milan Kundera, Josef Škvorecký, Peter Carey, Mario Vargas Llosa, Caryl Phillips, Paul Auster, Lorrie Moore, Danilo Kiš, Marilynne Robinson and Vikram Seth. Not long before, Rushdie had won the Booker prize for Midnight’s Children and that masterpiece, with its echoes of Günter Grass and Gabriel García Márquez, suddenly seemed like a great opportunity. The world was coming in. What had been a narrow and sterile place was opening up. These books were successful; readers discovered that they wanted them. Today something similar can happen to Penguin.

This is not a gesture that can be made only once. It has to be repeated over and over again. British culture – the single reason for wanting to live in this country – has always thrived on rebellion, cussedness and nonconformity. From pop to punk, from Vivienne Westwood, Zadie Smith and Damien Hirst to Kate Tempest, from Alexander McQueen to Oscar-winning Steve McQueen, the voices of the young and excluded have made British culture alive and admired. There is no other country in Europe with the cultural capital of Britain, and no more exciting place for artists to live. This is where art and commerce meet. These artists’ work sells all over the world.

The British creativity I grew up with – in pop, fashion, poetry, the visual arts and the novel – has almost always come from outside the mainstream: from clubs, gay subcultures, the working class and from the street. Many of the instigators may have been white, but they were not from the middle class – a class that lacks, in my experience, the imagination, fearlessness and talent to be truly subversive.

The truth is, the conservative fear of other voices is not because of an anxiety that artists from outside the mainstream will be untalented, filling up galleries and bookshops with sludge: it’s that they will be outstanding and brilliant. Those conservatives will have to swallow the fact that, despite the success of British artists, real talent has been neglected and discouraged by those who dominate the culture, deliberately keeping schools, the media, universities and the cultural world closed to interesting people.

It is good news that the master race is becoming anxious about whom they might have to hear from. At this terrible Brexit moment, with its retreat into panic and nationalism, and with the same thing happening across Europe, it is time for all artists to speak up, particularly those whose voices have been neglected.

No one knows what a more democratic and inclusive culture would be like. It is fatuously omniscient to assume it would be worse than what we already have. The attempt of reactionaries to shut people down shows both fear and stupidity. But it’s too late: they will be hearing from us.