Increasingly fixated on the stars of today, such as Hilary Mantel and JK Rowling, publishers are neglecting the experimenters who could save their industry tomorrow: the mid-list writers

The tickets sold out months ago. Long before the admiring reviews of the stage adaptation of Hilary Mantel's novels Wolf Hall and Bring up the Bodies hit the press at the end of last week, theatre-goers were in no doubt they wanted to see six hours of blazing Tudor intrigue.

A £7m BBC adaptation beckons, with the actor Mark Rylance teaming up with Peter Kosminsky, director of The Government Inspector.

The runaway success of Mantel's story could be seen as a heartwarming tale for the book industry, but it comes at a time when many insiders worry such a tale will become increasingly rare as talented authors find it ever harder break through.

Authors with middling sales – like Mantel, before she led Thomas Cromwell up the bestseller list – are getting less care and attention from large publishers, with readers ever-more fixated on fantasy blockbusters, it is said.

For HarperCollins, the publishing giant behind 4th Estate, the imprint that publishes Mantel, bestselling authors have become more important than ever. Charlie Redmayne, chief executive of HarperCollins, recently described Mantel as one of several writers, along with Michael Morpurgo and JK Rowling, "who have transcended being an author and are brands in their own right".

He added: "In a digital world, they are going to create a huge amount of value."

Other industry insiders, and probably many readers, find it jarring to hear authors of magical sentences dressed in the language of business. It does not appear to describe an author such as Mantel, who before embarking on her Tudor trilogy wrote wildly different books, including a memoir, a black comedy and a historical epic set in revolutionary France: hardly a series of uniform Ford Focuses rolling off the production line.

"Brand" may be an ugly word when applied to an author, literary agent Jonny Geller acknowledged, but it is only a shorthand for a way in which publishers are attempting to hold on to the reading public at a time when sales of print books are flat and electronic gadgets vie for readers' attention.

And, from Byron to Barbara Cartland, the name on the spine has always mattered.

"There have always been big brands in publishing," said Philip Jones, editor of the Bookseller. "The difference now is that digital and globalisation gives them a much more diverse and larger playing field. Digital means they can be multimedia: books, films, video, apps and other enhancements; globalisation means they can extend the brand far and wide."

The world's appetite for stories is getting bigger all the time. The children's author Enid Blyton is believed to have sold 500m copies of her 200 titles and countless stories since her death, in 1968. But a mere 15 years after Harry Potter first boarded the train for Hogwarts, in 1997, JK Rowling is already closing in on that record, with 450m books sold in 74 languages. That figure does not include several million ebook sales.

But even Harry Potter is not immune to vampires. The Twilight teen vampire saga reached the 1m copies mark even faster than Harry Potter – although its overall sales are smaller at, 50m books worldwide, according to the industry analyst Nielsen BookScan. The Hunger Games, the bloodthirsty dystopia aimed at early teens, is the latest page-turning phenomenon, with 25m sales in English worldwide.

"The large bestselling authors are taking a bigger and bigger share of the market," said Andrew Franklin, founder of the independent publisher Profile. "Just as in every branch of late post-industrial capitalism, the rich are getting richer. New authors and struggling authors and mid-list authors are finding it harder."

This was bad news for the average writer, he said: they get paid less so that publishing houses can hold on to bestsellers with higher advances. But it is also bad news for readers: "When everything becomes more homogenised, and everyone is reading the same book, there really is a loss of diversity and choice."

Publishing used to exemplify the classic business model, where the top-selling 20% funded the rest, some of whom, hopefully, would become the bestsellers of the future. Jonny Geller, the agent who represents John le Carré, thinks the balance is now closer to 4% v 96%.

With so much riding on the success of top titles, there are risks for the whole industry. "If publishers focus too much on the obvious hooks or names, then the new or unsuspecting will disappear," he said. "All the major success[es] of the last few years, or the majority, have come from unexpected places."

Mantel, who wrote well-reviewed novels for years, perhaps typified what publishers call the mid-list: books destined to sell moderately without sparking TV tie-ins or T-shirts. In an essay in the New York Times this week, the publisher Colin Robinson warned that this space, where, he said, "the most interesting things happen in the book world", was in decline, warning: "The mid-list, publishing's experimental laboratory, is being abandoned."

This warning resonates for Nicola Solomon, chief executive at the Society of Authors. In a world where "the big brands have got bigger", she sees less interest from the five biggest publishing companies in promoting smaller titles and backlists.

"Publishers are not investing in authors in a way they would have once, to see if they will take off after their fourth or fifth book, if their first or second were steady, but didn't go through [to a huge readership]," she said.

While more books are being published every year – almost 150,000 annually in the UK, at the last count – the places to discover them are thinning out: bookshops have closed, cash-starved libraries have slashed their book-buying budgets and newspapers have pared down book reviews.

"Discoverability" has always been a problem, according to Solomon, but it is getting worse. "People are left with the brand, and it is very hard to get to know new things," she said .

No matter how difficult the industry is, however, people are still sidling up to her to confess they are writing a novel, she says. Those people with half-finished manuscripts in their drawers might take heart from the story of Nathan Filer, the mental health nurse who this week won a Costa award for his first novel. He had 11 publishers vying for the rights to publish The Shock of the Fall. The search for the new is not over yet.

Blockbuster tales: global sales for cult bestsellers

Harry Potter: 450m

Twilight 50m

The Hunger Games 24m

Sources: Bloomsbury, Nielsen BookScan

Top-selling fiction writers in the UK in 2013 (books sold)

James Patterson 1.5m

Lee Child 1.13m

George Martin 890,000

Source: Nielsen BookScan

Top-selling non-fiction writers in the UK 2013 (books sold)

Alex Ferguson 805,000

Jamie Oliver 528,000

Mimi Spencer and Michael Mosely 498,000

Source: Nielsen BookScan

Top-selling writers for children in the UK 2013 (books sold)

Julia Donaldson 2.7m

Jeff Kinney 1.54m

David Walliams 1.37m

Source: Nielsen BookScan