The other day, Belle & Sebastian’s singer Stuart Murdoch received an email from Carey Mulligan. The actress had sung on the band’s last album, Write About Love, but Murdoch hadn’t heard from her in years so he was pleased and intrigued that she had got in touch again. He opened the email: it was a generic, celebrity-authored charity message asking for donations. Murdoch laughs as he delivers the punchline. Like many Belle & Sebastian anecdotes, it portrays the Glaswegian septet as awkward interlopers on the fringes of real celebrity.

The thumbnail version of Belle & Sebastian is music made for, by and about sensitive misfits. Their miraculous 1996 debut Tigermilk and the subsequent If You’re Feeling Sinister had a delicate, hermetic quality so alien to Britpop it was tempting to see them as a Wes Anderson version of an indie band, a meticulous diorama brought to fragile life.

In fact, they are much more ambitious and robust. Their ninth album, Girls in Peacetime Like to Dance, is a confident, expansive record injected with disco and synthpop and recorded in Atlanta with Animal Collective and Cee-Lo Green producer Ben H Allen. Their last two were made in Los Angeles, where they headlined the Hollywood Bowl. “I think we’re exotic in America in a way that we’re just not here,” multi-instrumentalist Sarah Martin says, smiling. “We’re just average and boring here.”

A few weeks ago, Matthew Barzun, the US ambassador to the UK, invited them to play a gig at his official residence: introducing the group, Barzun credited them with helping him to woo his wife. The BBC journalist Robert Peston danced. Jimmy Carr and Ricky Wilson of Kaiser Chiefs played percussion on the final song. It was that kind of night. “It will come in handy when one of us is in jail,” guitarist Stevie Jackson says wryly.

Yet no matter what Belle & Sebastian achieve, the old stereotypes persist. When I first met Jackson in London a week earlier, he grumbled about another interviewer. “He was still talking about corduroy and twee. It was like, God, this is from the 90s! Are we still perceived like that? One of the things I like about the band is that people don’t even care about us, they care about the songs. I don’t know how much people have ever been into the band because of the force of our personalities or our good looks.” A sideways smile. “Although we are very good-looking.”

In Glasgow, I ask Murdoch which adjectives he never wants to hear again, but he’s not biting. “Ach, I think we’re everything everyone says we are and it’s probably my fault. You know, this is a pop band that sprang out of infirmity. There’s no doubting that.”

We’re drinking tea on the sofa in the West End house he shares with his American wife Marisa and one-year-old son Denny. Every band member still lives in Glasgow; Martin and drummer Richard Colburn are next-door neighbours. In a neat tableau of Murdoch’s dual life as a 46-year-old father and frontman, you have to walk past a playpen to get to the piano. The other day, Marisa sternly asked if he’d written a single line about Denny. Murdoch gave a tactful reply but maintains that his songwriting has always been escapism.

He’s a witty character with a streak of steely determination, but today he’s subdued by coughs and sniffles. He speaks softly and moves carefully, as if concerned that something might break. While starting work on the new album, after a long hiatus to direct his Kickstarter-funded debut movie God Help the Girl, his health was parlous. “I was a wreck,” he says. “They poured me into a plane and I trickled into the hotel.”

Murdoch was laid low by myalgic encephalomyelitis (ME), or chronic fatigue syndrome, which he first experienced after he left school. The relapse led him to write his most startlingly autobiographical song yet, the album-opening Nobody’s Empire. When Martin first heard it she was “dumbfounded. It’s the most open thing I’ve ever heard.”

“There’s a year of my life bottled into every line,” Murdoch says. “When I wrote it I was fighting the same demons. I wrote the song to cheer myself up. I wanted to write my own ‘glorious’, in inverted commas, history. I want to walk out of this room and be a normal person. I want to go on tour with the band. All of this is a day-to-day battle so it was absolutely real for me, writing that song.”

Murdoch’s first experience of ME was, as he sings, “a vision of hell”, but it turned him into a songwriter. He considers his younger self profoundly unremarkable until, aged 19, his life was, in effect, shut down and rebooted. He spent two years incapacitated and a further five slowly recovering, during which time he wrote his first song (about his best friend and fellow sufferer Ciara, who appears on the cover of If You’re Feeling Sinister) and found his religious faith. “It was absolutely at the same instant,” he says. “I’d got through my lowest ebb and woke up with a sense of otherness about the wider spiritual world and the ability to string a few notes together.”

When he returned to full health, he felt reckless, unstoppable, full of schemes. “I was probably quite an embarrassing person to be around. You’re like a baby bird breaking out of an egg. I had a dream band in my head for years and years, while I was cogitating in darkness.”

In 1996, Murdoch attended a music-production course at Stow College and somewhat exceeded the course work requirements by recruiting a full band and recording an entire album, Tigermilk. “Meeting Stuart was very refreshing because he just thought bigger than everyone else,” Jackson says. “He had a very specific vision.”

Tigermilk was a sublime debut, the cornerstone of a cult phenomenon, but Belle & Sebastian were a precarious coalition of virtual strangers, as far as eight years apart in age, made more volatile by Murdoch’s five-year relationship with cellist Isobel Campbell. “Most bands are a gang,” says Murdoch. “We were a loose assembly of men and women who fell together to make a record for a college project.” Martin remembers feeling that it could “implode because we didn’t know each other”.

Vulnerability bred a siege mentality, as if the slightest external pressure might trigger a total collapse. They barely played live and refused to do interviews, creating inadvertent mystique. The cracks began to show when they were finishing If You’re Feeling Sinister. “The plan all went to buggery the instant that record was mixed,” Murdoch says. “But that’s when things got interesting and ever since then we’ve been on this adventure.”

“He had a certain idea and when he hadn’t visualised further we did notice,” confirms Martin. “It did fall apart a little bit.” Belle & Sebastian were in flux for the next five years, learning to play live properly and share the songwriting burden. Martin and Jackson’s contributions grew stronger with every record but it was tough early on. “I hated it,” says Jackson, who is pathologically modest about his writing chops. “I was very nervous because those first two albums conveyed a self-made universe and now it was going to be screwed up.” Martin feels similarly awed by Murdoch’s talents. “I’m always left in the dust when it comes to lyrics. Stuart’s just so amazingly focussed and prolific and it’s such a wrench for me.”

By 2002, the band had lost bassist Stuart David (amicably) and Campbell (less so) and wreaked havoc on Murdoch’s health. Until the strain of trying to keep the band together triggered a relapse, Murdoch had kept his ME a secret. “I always hated telling people,” he says. “That’s one reason I lived on my own for so long, because I had this thing and I knew how to deal with it and I hated talking about it because it was such a waste of energy and time. After that the band’s been really supportive.” In Atlanta, Murdoch worked mornings and evenings and rested in the afternoons.

On one new song, The Everlasting Muse, Murdoch appeals for inspiration and receives the reply: “Be popular, play pop, and you will win my love.” He always wanted to be popular – early on, he was influenced as much by Carole King’s multi-platinum Tapestry as by unerachieving indie touchstones Felt – but he flinched from the spotlight for years. When enthusiastic fanbase vote-rigging snatched the Brit award for breakthrough act from Steps in 1999, the singer dispatched two bandmates to collect the trophy. By the time the band was fighting fit on 2003’s Trevor Horn-produced Dear Catastrophe Waitress, their top-20 moment had passed. Even new songs as catchy as The Party Line and Play For Today won’t penetrate daytime Radio 1 in 2015.

“I think Stuart wanted to be more successful,” says Jackson. “He didn’t understand the concept of the zeitgeist. You only get one chance and you have to grab it. It bugged me for ages that we didn’t take advantage of it at all. It seemed like we were always screwing things up. But in terms of longevity I think it’s good that we just quietly got on with it.”

Murdoch agrees. “Stevie wants to be in the Velvet Underground, Bob [Kildea, bass] wants to be in the Rolling Stones and I want to be in Abba but we’re probably more like the Grateful frigging Dead. We just trundle along. But we amuse ourselves.”

He is so self-critical that I feel obliged to say that Girls in Peacetime Want to Dance is a wonderful record, both adventurous and moving, full of compassion for the bruised and the broken, and gratitude for the consoling power of art. “I’m interested in seeing the stresses and strains,” says Murdoch. “People have this veneer of calm and it amazes me. I want to get under the surface. Don’t give me the small talk.”

We venture out into the waning afternoon to join the rest of Belle & Sebastian at their rehearsal room and resume preparations for their longest world tour yet. It’s a familiar scene of a band at work: six musicians relearning old songs amid a maze of instruments, amps and snaking cables. But in light of what Murdoch has said, it looks idyllic. Here he is, surrounded by his friends, singing beautiful songs about imperfect lives. For him, this isn’t the model band he cradled in his head for so long. It’s better.

• Girls in Peacetime Want to Dance is released on Matador on 19 January.