You can see the dilemma. On the one hand, there is a bestselling novel that has nailed a political moment, by a writer hailed as a prophet of our times. On the other, there is a verse novel capturing the hidden voices of black British women, by a writer who has honed her art for years outside the spotlight. To give the prize to Margaret Atwood’s The Testaments would put the Booker in the centre of the cultural tide. To give it to Girl, Woman, Other by Bernardine Evaristo would push a deserving but unsung book towards a much wider audience.

That, at least, is how this year’s Booker fudge looks from the outside. Inside the judging room on Monday, all kinds of arguments and rhetoric may have been employed; all kinds of views aired and argued. I have been on enough prize panels to realise how very odd these conversations can be, how you get the feeling that it all could have gone differently on another day, in another mood, let alone with another set of judges. Of course, there can be only one winner of the Booker prize. Until, that is, the very judges themselves choose to flout the regulations, as they did on Monday, and give it to two. (Confusingly, it used to be OK to split the prize, until the rules were changed after the 1992 prize was given to both Michael Ondaatje and Barry Unsworth.)

What I learned from my year of reading outside the box | Afua Hirsch Read more

Everyone agrees that competition is the enemy of art. And yet, on the whole, there is also an agreement to conspire in the notion that it isn’t. This paradox, this doublethink, usually works fine, since it opens up the space in which the extra-artistic functions of prizes can be fulfilled. By this I mean that the people who care most about the Booker are publishers and the book trade – its effects on sales can be transformative. The Turner prize, equally, exists not because its organisers strongly believe in the objective reality of the “best” artist making work in Britain in any given year, but because it brings a huge audience to contemporary art. A case of: come for the gladiatorial combat, stay for the two-hour video installation and knotty performance art.

The doublethink of prizes has worked in the past because it is, in fact, usually perfectly possible for a set of judges to arrive at a majority position that one artwork is better than a set of others. That’s what subjective, critical opinion is. The trick of a big, institutionally hefty prize is then for everyone to conspire in imagining that this does not represent some shaky provisional position, but means, objectively, that the “best” book has prevailed. A further paradox is then necessary, in which the culture on some level also admits to the impossibility of accurate measurement of artistic quality by arguing strenuously with the judges’ decision, thus making the prize a talking point among the media and wider public. These incompatible positions are part of the game, and it all makes perfect sense as long as you don’t poke the nest.

The refusal of this year’s Booker judges to follow the prize’s own rules has made a gash in this delicate fabric of unspoken conventions. As well as irritating a number of people – not least the previous judges who have painfully and dutifully followed the guidelines – it is bound to dent the prize’s traditional standing, since it pushes back the Wizard of Oz’s curtain, suggesting the imperfect, flawed humanity lurking behind the award’s supposedly authoritative facade.

And yet this year’s Booker decision is only a symptom of a wider phenomenon. Arts prizes are losing standing for other reasons too – the Nobel prize for literature, damaged by an infamous sexual harassment scandal and this week’s award to Peter Handke, who has denied the Srebrenica genocide, shows how quickly authority can leak away from once revered institutions.

But more fundamentally, artists are rebelling against the notion of winner-takes-all prizes. There is an invisible, but I suspect growing, number of people who refuse to put themselves up for awards at all. Then there are those who have defied the prize structure by splitting their winnings – Helen Marten, for example, shared the 2016 Turner prize and the Hepworth sculpture prize with her fellow nominees. The writer Olivia Laing did the same when she won this year’s James Tait Black award. She had been writing, she said, “against an era of walls and borders, winners and losers. Art doesn’t thrive like that and I don’t think people do either. We thrive on community, solidarity and mutual support”.

We will see more of this, I predict: shortlisted artists and writers getting together to repudiate the notion of a single winner. The question is whether the prizes themselves can adapt to the new mood – or whether their cultural dominance is slipping away.

• Charlotte Higgins is the Guardian’s chief culture writer. Her latest book is Red Thread: On Mazes and Labyrinths