Late on Thursday afternoon, would-be wizards across the UK dropped what they were doing to join the professors of Hogwarts for Harry Potter night. At 6pm in the basement of Waterstones’ six-storey London Piccadilly building, staff were scurrying around with bowls of jellybeans and bottles of raspberry lemonade, but the Harrys and Hermiones were nowhere to be seen.

A couple of elderly customers looked faintly disgruntled to find their favourite section shut for a private party. Two young sisters, Alex, 11, and eight-year-old Polly, fidgeted by the closed door clutching a box of quidditch balls, while Yang, a 22-year-old physics student from Korea, appeared baffled.

Five minutes later they started to arrive, threading their way through book-browsers on the hushed shop floor. Young women pulled Hufflepuff blazers and Hogwarts ties out of backpacks, a small girl produced an owl cage, a larger one donned scholar’s robes. Soon, the queue snaked up the stairs and across the ground floor. “I’m reading the fifth book again at the moment,” said 28-year-old Alex, jiggling her wand. “This is the third event I’ve been to and it’s quiet compared with the launch of Harry Potter and the Cursed Child last year, when they transformed the second floor into Diagon Alley.”

In many of of the chain’s 275 branches across the UK, similar scenes were being played out. “Our first wizards have arrived for #harrypotterbooknight” tweeted staff at the Bradford store, who had earlier professed themselves “totally giddy kippers” at the prospect of the night’s revels.

But Harry Potter night wasn’t the only cause for celebration for staff and customers of the 35-year-old company. A day earlier it had revealed that it had gone back into profit for the first time since the recession under the leadership of its very own wizard, banker turned career bookseller James Daunt, who was brought in to rescue the chain in 2011 after a buyout by the Russian billionaire Alexander Mamut.

Quidditch players at Waterstones Piccadilly in London during the launch of Harry Potter and the Cursed Child in July last year. Photograph: Waterstones/Twitter

“We came within a millisecond of losing everything. We were dead in a ditch,” said Daunt in 2014, as the company was easing its way back to break-even. From its origins as “a new breed of bookshop” dreamed up by founder Tim Waterstone in the early 1980s, the chain had lost its mission and its way. Successive takeovers by WHSmith and HMV had left it attempting to mimic the pile-em-high philosophy of the mass-market outlets, with centralised buying and Faustian pacts with publishers to showcase their books for large fees.

With a mixture of tough love and an unshakeable belief in the power of the physical book, which seemed quixotic in the era of e-readers and online discounting, Daunt began to turn things around. He closed underperforming stores and fired 200 booksellers, at the same time as declaring that his managers would be given back responsibility for their own stock, because what sold in Hampstead might not go down well in the Highlands. One of his boldest moves was to inform publishers that he would no longer do business through sales reps and they could no longer buy window space – which meant turning his back on £27m a year.

Instead, a small team of buyers – in close consultation with Daunt himself – would select titles to feature as books of the month across all the stores, while individual managers were free to tailor much of their stock to their customers’ tastes. One of the centrally chosen books of the month for 2016 was a historical novel, The Essex Serpent, from Serpent’s Tail, an inprint of indie publisher Profile Books. Its author, Sarah Perry, had written one previous novel, which had been respectfully received before sinking to the special limbo reserved for decent literary debuts.

The Essex Serpent by Sarah Perry , the success of which was driven in large part by Waterstones’s enthusiastic support

Daunt himself was among the first to recognise that The Essex Serpent was special, says Profile’s Andrew Franklin. It was featured as novel of the month in May 2016 and went on to be voted Waterstones book of the year by booksellers across the chain.

“Waterstones’ role in the success of The Essex Serpent is nothing less than extraordinary,” says Franklin. “They have – to date, and they haven’t finished yet – bought 100,740 copies of the hardback. And that gives them a 70.53% market share.”

For Perry, the experience was “more than a little surreal. Some of my most vivid memories from the early days of publishing The Essex Serpent are of waking in the morning, feeling fretful about how it might be received, to find the most extraordinary Waterstones window displays popping up all over the country.

“None of us knew it was happening, so each one was the most delightful shock – I especially recall being tagged on Instagram in the very early hours, as booksellers in Deansgate [Manchester] stayed up all night painting the most extraordinary mural. And I remember standing outside the Piccadilly branch, where a bookseller had created a display complete with oysters and moss and reels of blue thread, feeling that never, not even as a daydreaming 15-year-old, did I think I’d see something like that.”

When a book has become as successful as The Essex Serpent, one can forget how it so easily could not have happened. For a tiny outfit like Somerset-based children’s publisher Chicken House Books, selection by Waterstones can make the difference between many thousands of sales and virtually none at all. Chicken House has had four of its titles selected as children’s books of the month – the latest being Maz Evans’ debut novel Who Let the Gods Out, a “freewheeling fantasy” in which a 12-year-old boy finds his home near Stonehenge beset by bumbling Greek gods.

Barry Cunningham, who set up Chicken House in Frome in 2000, points out that the relationship with Waterstones doesn’t simply involve selling the book. “They’re experts in how to catch the fleeting attention of buyers, so they’ll advise on how a book looks, the words on the back, or even the title. In this particular case, there’s a lightning flash running around the edge. They said they really liked that, but could we bolster it.”

Part of the new strategy has been to customise orders, so as to drastically reduce the numbers of unsold copies that are returned to publishers. A report from the US in 2013 revealed an average return rate across the bookselling trade of 15%. When Daunt took over, the percentage returned by Waterstones was “far higher than that”, but has now been reduced to 2-3%.

At a recent conference of the Independent Publishers Guild, questions were raised about the downside of this policy in terms of “sleepers” by unknown authors that might never get the chance to become hits. Publisher Juliet Mabey, whose company Oneworld has published the last two winners of the Man Booker prize, paints this in stark terms. “Waterstones’ opening order for Man v Nature by Diane Cook, a dazzlingly inventive collection of short stories, was a single copy.” she says. “It was saved by being shortlisted for the Guardian First Book award.”

James Daunt, head of Waterstones, in 2012. Photograph: Geoff Pugh/REX

Even the big-hitters often start small. The initial order for Marlon James’s A Brief History of Seven Killings was 220 copies, rising to 57,534 to date, while Paul Beatty’s The Sellout soared from an opening order of 260 to sales of 65,048 copies so far. “You could argue that this shows their system does actually work. That said, without the prizes I’m not convinced such small advance orders would have really propelled these important books into the spotlight. Obviously we will never know,” says Mabey.

One reason for the turnaround in the chain’s fortunes has been the stagnation of the ebook market. It stopped selling Kindle e-readers in 2015 in a move that was regarded as a watershed moment in the battle between physical and digital books. Sales of children’s books have played a big part in its resurgence, and data from market researchers Nielsen Bookscan revealed that, far from embracing the digital revolution, young readers were among the most resistant, with 75% of children favouring physical books and 35% refusing to read digital copies at all.

“Publishers pushed a cheap-and-cheerful approach with ebooks, and it seems to have become more the associated domain of trashy, poundshop-type products,” said Simon Lowe, a former bookseller. “With the refits of Waterstones reflecting a more boutique, upmarket style, the physical and digital realms of bookselling seem to have separated and become distinct, rather than competing with each other.” Inevitably, five years of such radical change has left some feeling disgruntled. “There’s been a massive restructuring of staff and the way stores are run so the wage bill has dropped drastically over the years,” said the bookseller. “Most of the managers I know have left, taking redundancy. When I left my old store, there were six full-timers; now there are three.”

Daunt is undaunted, pointing out that Waterstones is no longer the minimum-wage company he took on in 2011. “Running companies does involve doing all sorts of things that are deeply unpleasant, but if you are very sure of the outcome you’re looking for, it makes things better in the long run. I’d hope to employ fewer booksellers and pay them much more.”

Nine days before the European referendum, Daunt made an unexpected political intervention, sending a stark email to employees warning that if the UK left Europe, there would be a “significant retail downturn”, which would “reverse much of the hard-won gain of the last few years”, forcing the firm to axe jobs.

So far, he admits, his fears seem unfounded. “The economy remains buoyant. We had a very good Christmas and the notion that there would be an immediate downturn has been proved wrong. But I remain very pessimistic about the longterm prospects.”

The day after announcing his company’s surge from a £4.5m loss in 2014-5 to a £9.8m pretax profit in the year to 30 April 2016, he caught a morning train to Edinburgh. He aims to spend two days a week at the London head office and the other three roving the country. “I want to visit them all, but I try to focus on the stores where I can do something. I think there is a science to it. It’s a slow old tanker to turn around, but it is turning.” No detail of the bookselling life is too small to catch his attention. Coffee shops and merchandising play their part in the creation of 21st-century stores, he says, but the most important thing of all is improved lighting.

Among his booksellers, there is a palpable sense of civic mission. Jane Skudder, of Yorkshire’s Bradford branch, which is atmospherically housed in the former Victorian Wool Exchange, says: “You have to look at the history of the last 10-12 years: we’ve had riots and by the late 2010s there was a big hole in the ground where the new shopping centre was going to be. I think Bradford has got its pride back. It spent a long time trying to be Leeds, and now it’s remembered how to be Bradford.”

In bookselling terms, being Bradford involves lots of poetry and maths books, washed down with Yorkshire tea and locally produced Yorkshire scallywags (“posh scones to you and me”). Jordan Ellenberg’s How Not to be Wrong: The Hidden Maths of Everyday Life has been a big seller, while the launch of a debut thriller from local pharmacist turned “Bradford noir” pioneer AA Dhand drew more than 100 people. Plans are now in hand to capitalise on the route of the Tour de Yorkshire cycle race in April by selling tickets for charity to watch the peloton streak past from the arched windows at the back of the store.

Eighty miles north, in the small market town of Yarm, the manager of one of Waterstones’ newest and smallest shops is catering for a very different sort of clientele. “We’re a rural community and we’re quite dog-friendly,” says its 27-year-old manager, Michael Howlett, who heads a team of three. A picture book called Oi Dog! is a current in-store favourite as is Horses, Heifers and Hairy Pigs, the second volume of memoirs from Yorkshire vet Julian Norton.

In common with several of Waterstones’ more recent outlets, the Yarm shop picked up where a previous bookseller had failed. Though it serves a small but prosperous community, “there’s a real commitment to keep shops open in places that are really quite difficult,” says Franklin. “They have branches in areas that have been economically blighted for a very long time. They have a real vision of the importance of books and bookselling.”