Cutting-edge British literary fiction risks being undermined by its growing reliance on a handful of powerful book prizes, a leading literary agent has warned. But the associated costs of entering the biggest awards mean independent publishers willing to take risks on “difficult” works without obvious marketing potential are being shut out of contention.

The warnings, from Jonny Geller of Curtis Brown, come as the Costa book of the year judges prepare to announce the shortlist for the £30,000 award. He said: “Literary fiction is under threat in this country due to a combination of factors – reluctance by major houses to take risks; a bottleneck in the distribution chain [and] diverse voices being ignored by a predominantly white, middle-class industry.”

Literary prize culture had exacerbated the situation, added Geller, whose clients include Man Booker winner Howard Jacobson and William Boyd, who won the Costa novel of the year in 2006. “Every major literary prize is under the same pressures – the balance between picking books that break new ground, challenge readers and those books that will be popular,” he said.

Some small publishers say the cost of entering the three top awards – the Man Booker, the Baileys prize for women’s fiction and the Costa books of the year – alongside pressure on juries to choose winners that have sponsor-pleasing commercial impact, rather than “difficult” books less appealing to general readers, mean they are wary of entering experimental work.

“It’s such a huge cost that most small publishers can’t do it,” said Adele Ward of the independent firm Ward Wood. “I can only cover the fee, but some of the awards also charge a lot for the awards ceremony.” Though her authors have had success with prestigious smaller prizes, including LGBTQ awards, she said the three top book prizes were the only ones that really affected sales

The biggest three prizes, including the Costas, require a £5,000 fee from publishers if a book is shortlisted. This is a contribution towards marketing and should, the organisers claim, be offset by increases in sales. However, one independent publisher whose books have fared well with the three awards said hard calculations had to be made about which authors cold recoup such investment.

“If the shortlisted publisher also has to fly their author over from, say, Australia and put them up in a hotel for a few days to take part in the events in the run-up to the awards dinner, it is possible that the costs exceed the financial benefits,” the publisher said.

The value of book prizes in propelling sales in a sector that has struggled against commercial genres was illustrated before Christmas when The Essex Serpent by Sarah Perry topped the bestselling fiction charts after being shortlisted for the Costa novel of the year and being chosen by Waterstones as its book of the year.

According to publisher Profile Books, its sales more than doubled, taking it ahead of established genre bestsellers by Ian Rankin and Lee Child. Perry acknowledged shortlisting was crucial for literary authors. “Prizes bring to readers’ attention books that might otherwise have passed them by, including books by writers from communities often marginalised in the arts – which of course can then transform writers’ careers,” she said.

Dan Franklin, associate publisher at Jonathan Cape – who has worked with authors including Salman Rushdie and Martin Amis – admitted that though prize-winning potential was not “a large factor” in his editors’ decisions to publish novels “I’m guessing it’s always at the back of an editor’s mind”. Prizes, he added, had become more significant as it had become harder to publicise literary novels, due to the decline in review coverage by mass-media outlets.

Novelist Jane Harris said that when her debut novel The Observations was shortlisted for the Orange prize, this propelled her sales and subsequent career. “Prizes sell books,” she said. Her second novel, Gillespie and I, was longlisted for the same prize, which has since changed its name to that of its new sponsor Baileys. She added: “The main difference in getting a book on to a prize list is in terms of sales. The Orange was a tremendous boost for both my books.”

Alex Wheatle, winner of the Guardian children’s fiction prize, said the impact on his career had been significant. “Even the longlisting of Crongton Knights for the prize brought me to the attention of critics and readers who had never read or considered my work before.”

The Sellout, Paul Beatty’s Man Booker-winning novel, had sold only a few thousand copies before the prize, but sales topped 23,000 in the week before Christmas, according to his publisher Oneworld. Beatty’s publisher Juliet Mabey said prizes were now essential in order for independent publishers of literary fiction to be heard above the noise of books by well-known names backed by conglomerates.

“The attention prize judges pay to books which might otherwise go under the literary radar is critical,” she said. “For literary fiction, a prize nod can make all the difference. A novel that might only sell 2,000 copies could sell 500,000 if it wins the Baileys women’s prize for fiction or the Man Booker prize.”

The category winners of the Costas are announced on 3 January. The five finalists, covering children’s, poetry, debut fiction, novel and biography, will each receive £5,000 and go into contention for the overall award, which will be revealed on 31 January.