When it started, Waterstone's was a breath of fresh air. But as it got ever bigger, many say it lost its soul. What effect has that had on publishing?

In the Bloomsbury branch of Waterstone's, I am trying to find a quiet seat to read Tacitus's account of Seneca's suicide when I come across something more diverting. A customer is asking an assistant to explain the baffling price deal on Hilary Mantel's Booker-winning novel Wolf Hall.

"I'm confused," she says. "It says here that if I spend more than £10 I can have the book for £8.99." That would be a good deal: the recommended retail price (RRP) for the hardback is £18.99. But there is a problem. "I only want to buy this book and nothing else. Does that mean I'll have to pay the full price, £18.99?"

"I wish they wouldn't do that," the assistant says. "They shouldn't have deals that are so confusing it takes more than a minute to explain."

But Waterstone's does. The sticker on Wolf Hall's dustjacket offers a half-price discount only if you buy something else too. The assistant explains that if the customer only buys the Mantel today, she would get £5 off the recommended price (ie she would pay £13.99). "But there's nothing on the book to tell you that." "That's right," says the assistant, with a disarming I-only-work-here-and-the-bosses-need-shooting tone. "Oh," says the customer. "I don't know what to do now."

I tell the customer, a lecturer from London University's School of Oriental and African Studies, she could have bought Wolf Hall from Amazon or from Waterstone's website for £8.45. "Yes," she replies, "but two words: postal strike. And even if I had ordered it online, I'd have had to pay postage and I don't know how much that is." Exasperated, she decides to leave without the book.

Such – or so you might think – is the nature of late capitalism. It makes buying the most straightforward item such a nightmare that you leave the shop having saved yourself a tenner, but in the process a nice lunchtime excursion has become a frustrating fiasco.

Waterstone's has embraced capitalism's logic firmly. Even in this Gower Street branch, with its five miles of bookshelves at the heart of London's university quarter and in an area denser with literary heritage than perhaps any in the world, discounted piles of Leona Lewis biographies and Frankie Boyle's My Shit Life So Far sit on the tables with the latest JM Coetzee. This lunchtime, the three-for-two tables are ringed by shoppers clutching two books and wondering if they can find a freebie worth reading. Here on the ground floor, the discounting of book prices is so ferocious that if you leave having paid the RRP you feel a right mug.

"They simply treat books as a commodity," says Nicholas Spice, publisher of the London Review of Books, and one of the chain's sternest critics. "There's no sentiment to it. If it's celebrity biographies that are going to sell, then that's what they'll focus on. They're not looking at it from a cultural perspective."

Is that a problem? After all, I am the one who brings sentiment and culture to the book-buying experience. Spice's thought, though, is that Waterstone's has lost its literary soul in stooping to compete with supermarkets and stationery retailers WH Smith. "A big retail business will inevitably move to the lowest common denominator position. Their commitment to book quality has to wane." Why? "Because once companies get big they draw in business management that doesn't have any sensitivity to the product. That's certainly the case with Waterstone's: the books knowledge of the people who run it is relatively small. Staff aren't paid well, so turnover is high and knowledge of what they're selling falls."

"The emphasis given to the few is staggering," says Mark Le Fanu, general secretary of the Society of Authors. "It's our mid-list authors, who may not write the most commercial books but who often write the best, who are suffering. The big corporate publishers dominate the shelves and squeeze out smaller publishers."

Hilary Mantel's agent Bill Hamilton worries that books are being sold like shampoo. "In retail, if you are selling a new shampoo you would expect to pay Boots, for instance, for a promotion, to make sure your shampoo is more visible than other ones. That pattern has been copied by Smith's and Waterstone's to an extent that has never been seen before in bookselling: you pay for almost any presence in the stores, you pay a huge amount for special promotions in the front of the store, and you go on paying every week even if the books are selling strongly anyway.

"There seems to be a frantic scramble in the book retail world to rush downmarket in order to compete with the challenges of Amazon, the supermarkets and next the ebook. Publishers have to fight their corner, year after year, against ever more aggressive demands for higher discounts from the chains, but seem at a loss to know how to cope with the underlying problems they face. They fear speaking out about how their books are being sold."

Novelist Giles Foden argues: "I get a strong sense that publishers, generally speaking, are angry about the terms on which they do business with retailers. And they are also worried, in this recessionary period, about extending credit in the shape of stock. Everyone puts a brave face on it but the relationship between publishers and retailers is under a lot of stress."

I hear these arguments repeatedly from publishers who would rather cut their throats than go on the record. It's understandable: they can't afford to annoy Britain's biggest book chain. "One of our novels, which shall remain nameless, sold 60% of its run through Waterstone's," says one publisher. "So I'm not going to slag them off even though I hate what they're doing to bookselling in this country."

So the argument goes: in going big, Waterstone's lost its soul. It gains credence if you consider what is happening in the US. There, Amazon and Wal-Mart are fighting a discounting war. If you really must buy Sarah Palin's Going Rogue, you can get it from Wal-Mart or Amazon.com for 60% less than list prices, which means the two competitors are probably selling the titles at a loss.

How can they afford that? For Wal-Mart and Amazon, books can be loss leaders, luring customers in so that they might then buy other merchandise which does make a profit. The only sure-fire losers in this war are the booksellers who have no other merchandise. Bigger US booksellers such as Borders and Barnes & Noble have suffered alarming drops in share prices recently. Smaller ones face oblivion.

"Waterstone's has really already done to British bookselling just the kind of things that we're seeing in the US," says Spice. "By competing with supermarkets, they can't afford to care about the quality of what they're selling."

It didn't used to be this way. Waterstone's used to be good guys in the literary world. The chain was established in 1982 by Tim Waterstone. "Then they had amazing shops filled with unusual books," recalls Spice nostalgically.

Tim Coates, former Waterstone's MD, says that in that decade the chain was responsible for creating new demand for books in provincial British towns and cities. "Twenty-five years ago, in many towns where there's now a Waterstone's, there would have been just a WH Smith and, probably, a not very good independent.

"Waterstone's did Britain a huge favour," agrees Nicholas Clee, author and former editor of the Bookseller. He argues it still does Britain a huge favour: "The criticism they get, you would think they're just selling Leona Lewis. That's not true: go into any Waterstone's branch and there'll be 20,000-plus titles – more than a lifetime's reading."

But soon Waterstone's became the books retailer that the literati loved to loathe. Why? In 1991, Waterstone's became one of the first British booksellers to start discounting book prices. For 91 years previously, booksellers had adhered to the Net Book Agreement, whereby retailers sold books at prices recommended by publishers. By 1995, the NBA collapsed; two years later it was outlawed. Supermarkets moved into bookselling, offering titles at unprecedented discounts. One result was that nearly 500 independent bookshops closed. Another was that Waterstone's expanded massively.

But surely the NBA was a constraint on free trade that meant we had to pay artificially inflated prices for books? One reason for the NBA's existence given by the Restrictive Practices Court, when it analysed the agreement in 1962 was that it enabled publishers to subsidise the printing of the works of important but less popular authors by using money from bestsellers. Today, the worry is that the demise of the NBA has meant there is no new generation of British literary talent to follow the likes of Martin Amis, Julian Barnes and Ian McEwan.

"There's been a slow bonfire of literary authors in the last 18 months," says Hamilton. "Publishers are sending out to pasture established literary novelists because they realise they aren't going to be sold by the chains. The complaint now from publishers is that most of their quality books hardly get a look in at all. In the past, sales for many literary novels were never very high, but now publishers are cutting down on their lists in desperation."

Hamilton cites the example of the crime novelist Ian Rankin: "Rankin was selling nothing at all for the first few novels he wrote, but publishers knew he would take off and so they kept with him. The opportunity isn't there to do that any more because sales are so low that you lose too much money initially, even if you make money later. That old, very successful business model doesn't make sense any more. Thanks to the prevailing way in which books are sold there would be no new Rankin."

But didn't the abolition of the NBA make books available to new audiences? Hamilton concedes that book prices in Britain have barely gone up in the last 10 years. "We have had fantastic price deflation in books because of the end of the NBA and discounts," he says.

Isn't that great for our literary culture? "I don't think it is about price," argues Hamilton. "It's about shops. People who love books have fewer and fewer places they can walk into and know they will find things that will suit them. We have a thriving literary culture and a sophisticated readership, but retail doesn't help sustain either."

The demise of the NBA certainly revolutionised British bookselling. Its death made American firms look across the Atlantic and see rich pickings. Borders set up large bookstores with allied coffee shops. Amazon invaded and offered customers access to an online literary database – undreamed of even in Waterstone's philosophy.

"Tim Waterstone was out-couraged by Jeff Bezos [Amazon's CEO]," says Coates. "Tim had been courageous in having huge stocks of books but then he was faced with Bezos's idea, which was that any book in print can be got to the customer very quickly."

Amazon undercut Waterstone's USP (its vast stocks of books, unprecedented in Britain) at a stroke. Waterstone's had a competitive disadvantage: it had to pay to run shops often in prime sites in British city centres; Amazon didn't. "I'm not inclined to be harsh on Waterstone's," says Clee. "If you run 300 bookshops in the British Isles with very expensive rents, you have to be a mass book retailer. That's the only way you can respond to a challenge like Amazon's."

For Coates this was when the chain lost its way as a cultural institution. "They decided to take on the supermarkets and Smith's by discounting prices and celebrity biographies. It was a strategic error. What they should have done was take on Amazon by offering something Amazon can't – the lovely, serendipitous experience of being in a really good, big bookshop."

Where some saw commercial good sense in Waterstone's development, the literati just saw philistinism. Last year, for example, staff from Waterstone's flagship Piccadilly store met writers including biographer Michael Holroyd, poet Wendy Cope and novelist Deborah Moggach. Why, writers asked, had the biography department been moved from the ground floor and replaced by stationery? "The answer was that customers preferred stationery to biography," recalled Holroyd. "The blank page was in demand – and it was the job of the bookseller to meet that blank demand."

Again, it didn't used to be that way. Booksellers once thought that their job was to create demand for books worth reading. "Waterstone's daren't risk doing do that now," says Clee.

Waterstone's failed to understand what a bookshop has to be like to survive in a new era of online retailing, argues Coates. "They should have realised that they must make their stores pleasant. Coffee concessions don't really deliver that. If I was running Waterstone's now, I would increase what the marketing people call 'dwell time'. I'd make the shops like old-fashioned reading lounges lined with books, like you used to have in Edwardian times. They wouldn't be bookshops with a cafe, but bistros with books. The arithmetic doesn't look immediately obvious, but that's often the case when you try innovative things."

He has a point. I wander five floors of the Bloomsbury Waterstone's, fruitlessly looking for a corner to sit and read. There are, though, signs telling you that if you want to visit the Costa Coffee in the basement (which no sane person would: it's a dank, scruffy space), you can't take your book unless you've paid for it. That is understandable: no bookseller wants muffin crumbs in their Tacitus. But signs establish mood: for all the beauty of the Gower Street store's Franco-Flemish facade, this isn't a place to dally or have a literary experience. It's one where you're invited to buy as much as possible and then shove off.

In 1998, Waterstone's was bought by the HMV Media group and, in 2006, it swallowed up the Ottakar's chain, consolidating Waterstone's stranglehold on high-street book retailing. At the time, Ottakar's chairman, Philip Dunne, said: "Over the last year, the book market has undergone a significant change with new levels of competition from the supermarkets and online retailers impacting all specialist booksellers and in particular those with insufficient scale to compete on equal terms."

It was a depressing admission: market logic dictated that bookselling was now a big-scale business. The little guys were doomed to be impacted which, however you looked at it, didn't sound pleasant.

Some little guys, at least, think otherwise. The noughties have seen several culturally significant small independent bookshops open, including the six-year-old London Review of Books bookshop, which lies a stone's throw from the Bloomsbury Waterstone's. Literary agents Sarah Lutyens and Felicity Rubinstein opened their own shop a few weeks ago in west London. Isn't it mad to set up shop now? Apparently not. "We have wanted to expand into bookselling for a long time and now the moment seems right," they say. "We seem to be entering a golden age of independent bookselling as readers become increasingly disenchanted with the supermarket atmosphere of the chains, and there is a new appetite among book buyers for a more carefully curated experience."

A carefully curated experience – in Notting Hill, maybe. But I doubt it is a model that appeals to Waterstone's accountants.

Instead, Waterstone's is now intent on cost-cutting. Earlier this year it introduced a central distribution warehouse in Burton-on-Trent called the Hub. "[The] original goal," wrote Neill Denny, the Bookseller's editor, "had been the delivery of shelf-ready parcels of books to each store, removing the need for much of the unpackaging and instore admin, with attendant savings, plus a simplified returns structure through a central point." One aim was to free staff to spend more time with customers. It hasn't worked that way. Instead, Waterstone's is cutting 10% of its 4,500-strong workforce.

The Hub confirms for some that Waterstone's has lost interest in treating books as anything but product to be shifted. "The philosophy they used to have was very different," says Le Fanu. "It allowed each shop to buy from wholesalers and encouraged staff to be individualistic and select books that appeal to local customers. Perhaps that wasn't very efficient, but it showed they cared about the local needs."

Another problem: staff are reportedly worried that delays in distributing book orders thanks to problems at the Hub are alienating customers. If this is the cost-cutting logic of capitalism in action, it looks like one that serves customers ill. Until recently Waterstone's had a good reputation for service, if retailing awards are anything to go by: it won the 2008 the British Book Industry awards' High Street Retailer of the Year. In Which? Magazine's customer satisfaction survey this January, Waterstone's scored high.

The glitch with the Hub – if that is what it is – is a worry in the run-up to another recession-hit Christmas, normally a period in which one fifth of annual books are sold. Despite repeated emails and phone calls, no one at Waterstone's is prepared to discuss these issues with me.

These are, no doubt, tough times to sell books. Books Etc is poised to close. In June, Waterstone's MD Gerry Johnson said that he faced a "subdued book market". Sales fell by nearly 3% and profit by almost 40% in its last financial year.

Perhaps harder times are ahead. This, if you believe the hype, is going to be the Christmas of the e-reader. Last month saw the worldwide launch of the Kindle e-reader. What would that mean for Waterstone's? "A financial analyst would say, 'We have to sell e-readers because they make money," says Spice. "But they may destroy Waterstone's." Why? "Remember what happened to Tower Records or Zavvi? They were reduced to selling the MP3 players that were destroying their CD business. And then they closed."

Perhaps, similarly, Waterstone's is sowing the seeds of its own destruction by selling e-readers. Last year, it became the first British bookseller to sell the Sony Reader, a rival to the Kindle. "If e-books take off it might force Waterstone's to go into other products more than they already do," says Spice. "In five years, Waterstone's may not be selling books at all. It may not even exist."