Philip Pullman became cheerleader for a growing band of refuseniks last week when he resigned as president of the Oxford literary festival because it didn’t pay speakers. Thirty more writers immediately picked up the chant, with a letter to the trade journal the Bookseller calling for all authors and publishers to boycott festivals that expected writers to appear for free.

With a 2014 survey reporting that the median income for authors is now just £11,000, these are undoubtedly lean times, and it’s understandable that emotions are running high. But, according to recent figures, the earnings gap has also widened dramatically between the top 1% and the rest – which translates, in festival terms, into the sort of writers who audiences will queue up to hear, and those for whom they won’t.

In more than a decade of chairing festival events, I’ve fronted many in which the audience barely outnumbered the panellists. They usually feature novelists who desperately need the sort of exposure a festival can bring. These sessions are the outside bets of the festival circuit: not only do they attract small audiences, but they are usually bunched into themed panel events so as to offset the risk. They are effectively being subsidised by more famous authors speaking to bigger audiences in bigger tents.

Some might believe that if writers don’t attract audiences they shouldn’t be appearing. But the reality is that no festival director deliberately books writers who nobody wants to hear, and there is always an element of uncertainty over turnout. When the Nigerian writer Chigozie Obioma took to the festival circuit last year, he was a debut novelist who was yet to be endorsed by any prize jury, so there was no guarantee he would find the audience that flocked in once he had been listed for the Man Booker.

But there is another, more abstract reason for continuing to take risks: good festivals are more than the sum of their sellouts; they need to have depth and intellectual coherence. They are about education as well as entertainment. Part of their role in cultural life is to offer readers the possibility of discovering new writers and making fresh connections.

One memorable event did just this for me in Edinburgh in 2012. It involved three excellent writers of east Asian heritage – Krys Lee, Madeleine Thien and Kim Thúy – who were in town as part of the World Writers festival. Their books dealt vividly with the fallout from persecution and societal breakdown in South Korea, Cambodia and Vietnam.

It was a small session with huge resonance: I doubt anyone in the audience had read, or even heard of, all three before it, yet nearly everyone queued up afterwards to buy a book. In one of the bigger sessions the same week, an audience member asked why there was so much writing about the Holocaust and so little about other, more recent genocides – to which my answer at the time was: “More recent genocides have been discussed, but you needed to go to the smaller venues to find them.”

Edinburgh is one of the biggest festivals and an honourable exception to the no-pay rule, offering the same flat rate to all its contributors. But it’s not unusual to hear writers grumbling that this is tokenism, and no recompense for the hours (and expense) of travelling. So what is a reasonable return? Should it be calibrated to audience size, or offset against book sales? Or should it be a flat rate – only bigger than it currently is?

There are now more than 350 literary festivals in the UK, which adds up to a whole heap of calls on writers’ time and energy – and one argument is that if they can’t afford to pay contributors they should simply shut down. But small festivals do more than simply put writers on stage; they support local bookshops and create a buzz around books. They circulate flyers publicising authors and their work. They are part of the great reading group boom that has bolstered book sales by turning reading into a social activity.

The smaller, poorer festivals – often powered by the enthusiasm of volunteers – also bring books to areas of the country where there may no longer be bookshops or even libraries. At an average price of £8-£10 a ticket (and far less for children), they are an economic way of keeping a window open on intellectual and imaginative horizons that might otherwise disappear from view.

So while I have every sympathy for hard-pressed authors, I feel they need to be careful what they wish for. The logic of the marketplace – in book festivals as in every other arena – is that, were fees to become obligatory, the haves will end up having more, while the have-nots will find themselves banished to outer darkness. It would mean the end of a golden era of access to books and the people who write them. And that would be impoverishing for all of us.