He made his name as a crooner with the Walker Brothers, then disappeared. Back with a new avant-garde album, Scott Walker says the darkness has always been there

Whatever happened to the great 20th-century crooner Scott Walker? Simple really. He disappeared, had any number of breakdowns, or one long one, and returned as a great 21st-century, avant garde composer.

Today David Bowie, Nick Cave, Jarvis Cocker, Brian Eno, Julian Cope, the great innovators in contemporary music, line up to proclaim the genius of Scott Walker. But back in the 1960s, his currency was romance, not complexity. His deep velvet baritone belied his youth. As the lead singer with the Walker Brothers, he enjoyed a number of melancholy hits with songs such as The Sun Ain't Gonna Shine Anymore, My Ship Is Coming In, No Regrets. Nobody did aching love quite like Scott Walker.

There was always more to the Walker Brothers than appeared at first sight – they weren't called Walker, weren't brothers, and however much they appealed as eye candy, their music contained a surprising depth and frame of reference. Even back in the mid-60s, Walker wrote haunting songs such as Archangel, hinting at what was to come.

Walker was born Noel Scott Engel, in Ohio in 1943. He briefly studied art at college in California before playing bass as a session musician, then joining the Walker Brothers. At 22, and by now Scott Walker, he and the band had their first UK No 1 with Make It Easy On Yourself. For the next three years they were huge, particularly in Britain, where he has lived pretty much ever since.

But Walker, a handsome young man who looked as if he should have been surfing the waves of California, was locked in his own agonies. He suffered depression, drank too much, took too many drugs and, like many of the great musicians of the 1960s, went missing in action.

He rarely talks to the media. When he does, he wears a cap over his eyes, and sometimes shades to boot. He has not performed live in 34 years, after apparently being enraged by an out-of-tune trumpet at a Birmingham cabaret. Walker has a forbidding reputation. In the few interviews he has given over the past 50 years, he has come across as sombre and taciturn.

He arrives at his manager's house in west London exactly on time, and walks into the room wearing the cap. He is 70 next year, snaggle-toothed where his bike recently smacked him in the face, youthful, with a surprisingly tough handshake. His talking voice is just like the old singing voice – deep, mellow, comforting.

Did he find fame hard to deal with? "I did. But it's OK now. You had to be there to understand what it was. There was a lot of pressure. I was coming up with all the material for the boys, and I was having to find songs and getting the sessions together. Everyone relied on me, and it just got on top of me. I think I just got irritated with it all."

After the Walker Brothers split, he went solo, sang gorgeous, angsty songs, many by Jacques Brel, and enjoyed great success with a series of albums: Scott, Scott 2 and Scott 3. The tunes weren't quite as easy and lush as they had been, and hints of dissonance crept in. The lyrics made up unsettling short stories, all the more creepy for their delicate orchestral backdrop. The public rejected his fourth album, Scott 4, as too weird and it didn't chart, though today it is seen as a classic. That was when Walker cracked up big time.

He drank himself into oblivion, withdrew from the world and thought about killing himself. "Well, I was an intense young guy," he says. Which is a classic Walker understatement. To be fair, I say, you're hardly unintense now. I can't see his eyes, but I can see a lovely boyish smile. He smiles and laughs a lot. It's one of the great surprises about meeting Walker. "Well, no. But this was hyperintense." Is he surprised he's still making music today? "Of course. Surprised to find myself alive. Hehehe!"

The Walker Brothers in 1967. Photograph: Rex Features

It's amazing how sane Walker sounds. Most people expect you to be mad, I say. He nods. "I think I did temporarily go crazy, because I don't remember the period at all very well. I can hardly remember it." How long did it last? "Probably from 1969 to 75." The funny thing is, I say, during that period, you began making your most commercial records – show tunes, movie themes, songs from a short-lived TV series.

Absolutely, he says. It all came about after Scott 4 failed to chart. "The record company called me in and carpeted me and said you've got to make a commercial record for us." So he did. And another, and another. "I was acting in bad faith for many years during that time." Why? "Well, I was trying to hang on. I should have stopped. I should have said, 'OK, forget it' and walked away. But I thought if I keep hanging on and making these bloody awful records..."

A lot of people like them, I say.

"I know they do." He looks embarrassed – whether it's at the albums themselves or his intolerance of them, I'm not sure. I tell him that nowadays many of his fans have revisited them and found darkness in them. He laughs loud and joyously. "Well, there's darkness in everything I do. But I did act in bad faith. I thought, this is going to turn round if I just hang in long enough, and it didn't. It went from bad to worse – until the 80s when I did the Hunter record." Walker has just added another decade to his lost weekend. It includes a period during which the Walker Brothers reformed, had a huge hit with a cover of No Regrets and recorded an experimental, electronic album, Nite Flights. Six years after Nite Flights came Climate Of Hunter – reputedly the smallest selling album ever on Virgin Records – and for the first time we met the remodelled avant garde Walker.

After Climate Of Hunter, another decade was lost to the world. When he returned in 1995, it was as a fully fledged modernist composer. On the surface, there couldn't have been a more unlikely transformation – imagine Andy Williams reinventing himself as Stockhausen. Yet in a way it was all of a piece. His latest album, Bish Bosch, is only his third in 17 years, all of them elaborate, epic and inaccessible. It is a post-apocalyptic opera of sorts, with blasts of rams' horn, dog barks, scraping swords, machetes. The music nods at Gregorian chant, doffs its cap to Shostakovich, gives a thumbs up to industrial metal, and is uniquely Scott Walker. The lyrics reference sexual disease, brown dwarf stars, court jesters and dictators, all delivered in a strangulated baritone, as if Walker's testicles were being squeezed. At times there's a terrible beauty to his poetry ("Earth's hoary/fontanelle/weeps softly/for a/thumb thrust") while at others there's a bloodthirstiness that could be straight out of Jacobean tragedy ("I've severed my reeking gonads, fed them to your shrunken face"). It's brilliant and bonkers. The opposite of a guilty pleasure: a guilty torture.

I ask where on Earth it came from. He says he's not sure, and wouldn't tell me if he could because he thinks other people interpret the work better than he does. But he can explain the title. "I started thinking about Bish Bosch, and it means sorted or job done. But bish also means bitch in urban slang. You'll hear a lot of guys say, 'This is my bish.' Then, of course, I used the Bosch 'cos of the artist, so I thought, well, this is a wonderful image of a universal woman artist." Whatever Bish Bosch is about, Hieronymus Bosch seems a suitable muse – for this is a vision of hell on Earth, with tiny pockets of hope and humour.

It's hard to believe this is the same Scott Walker who can be seen singing on the Frankie Howerd show, who had a solo show that the BBC hoped would turn him into the next Jack Jones, whose version of First Love Never Dies was the theme tune for Mike Read's mushy slot on Radio 1.

Walker says he suffers from terrible nerves. Is that why he hasn't performed live for so long? Partly, he says. "It was initially terror, but it was a combination of things. In those days the sound was so bad, and I couldn't stand going on stage. Now, of course, they have fabulous stuff; it's like being in the studio."

Great, I say, so we can expect to see you playing live again soon? He gulps. "See, what happens is every time I sit down to write a record, I start with this premise in mind. That I'm going to perform it. But then my imagination starts working overtime and I've got a cast of thousands before I know it, and there's no way you can make it pay."

But you have always said you're not interested in money.

"No, but the promoters are. And they would lose. There's no way you can do that."

I tell him I'm not sure I believe him; that I reckon making these epic records with live camels and dogs and slapped dead pigs are a way of getting out of touring. He smiles. "That's an interesting theory. Maybe you're right. Ha! I'd not thought about it that way." He is a famously tough task master in the studio. There is footage of him recording 2006's The Drift, making a percussionist punch a slab of pork again and again until he gets it right. On Bish Bosch, he says, he was just as hard on the man playing the ram's horn. "The guy we used is one of the great classical horn players. He had to play Brahms that evening, and we totally destroyed his lip."

Today, he says, he feels more at ease with himself. He has a long-term partner, is a grandfather (his granddaughter lives with her mother in Denmark), has plenty of friends, and is making the kind of music he wants to. I tell him it seems as if there's an inverse correlation between his sanity and the insanity of his music – the more loopy his records are, the more balanced he becomes. He laughs, delighted at the theory. Is music a way of leeching the madness out of you – only when your music is utterly broken do you feel whole? "You're playing my song," he says. "I am sane. I hope so."

Bish Bosch is wilfully obscure. I test him on one of the song titles. "OK, what's the full title of the song beginning SDSS?"

'I'm surprised to find myself alive,' says Walker. Photograph: Rex Features

He stops to think. "SDSS 1513," he says eventually. But it's more of a question than an answer. "No," I say, "it's SDSS1416+13B (Zercon, a Flagpole sitter). It's not exactly a catchy title, is it?"

"It's a hit single that," he says. "Hehehe! It's a 21-minute song, it's a hit."

Was there one moment when he can remember turning away from conventional melody? No, he says, but he remembers discovering classical music and how important that was to him. "A good friend in the 60s, mid-Walker Brothers period, was very into classical music and showed me how to appreciate it. He used to play it really loud, and make comparisons with different interpretations, so I listened to a lot of rock, jazz and classical music."

In the past Walker has suggested that the baritone can offer a fake emotion; a consolation too easily won. "Well, yeah, it can. It depends on how you use it. People do feel a warmth with baritone voices that they don't feel with others. It's like the sound you get from a cello, and people love that sound straight away."

Over the years he has stripped his voice of that warmth, pared it down to what he considers its essence. I have always wondered whether he thought his voice was too beautiful; whether he began to distrust it because it could cause such feeling in others while he felt so little himself? "Well, it's a beast all on its own. I think of it as another thing, another person. When it's working well I couldn't wish for anything better. But it's temperamental. Sometimes you get up and he's just not ready to go. The great thing about it is that I don't use him for ages then I can open the box and take him out, and there he is." He talks about his voice with real affection, but says he can't judge its beauty.

What he does believe is that harmony is heightened when you have to work for it. Beauty is best appreciated buried in the grotesque. And yes, he says, there is a pessimism to his work, but the chinks of life offer hope. "That's why I'm so puzzled when people say it's all dark, dark, dark, whereas I think there's a lot of beauty in it. Obvious beauty." And in the end, he says, that's what his music is about – the search for meaning and purpose in the wreckage. "I'm not a religious man, but it's a longing." For what? "For who knows. For existence itself. True existence. It's a longing for a calling. It's just a feeling that it might be there." Can we all find this purpose? "Oh yes, I believe so. We just need to find enough silence and stillness to experience it."

I tell him he reminds me of Samuel Beckett. Despite his existential nausea, there was something strangely life-affirming about the great absurdist playwright. The cap nods, approvingly. "Beckett would tell you the same: he was a very big pessimist, but he found it absurdly funny – like Kafka did." What? "Life. When you take everything away, then you're reliant on yourself and you gain strength that way," he says. "And a lot of it has to do with... contentment might be too strong a word, but it might be to do with the fact that I'm at last able to make the records I want to make. Because I went through a patch when I was a leper, when nobody wanted to get near me."

Doesn't he ever feel like knocking out a conventional chart-busting ballad? He looks at me as if I'm a couple of tracks short of the full playlist. Do you sing in the bath, I ask. "Oh yeah, sure." I start singing an appalling version of No Regrets. "Oh no. Hahaha! Nonono. I might sing a blues song or standard that comes into my head." Would you never do an album of commercial songs or blues? "No. I don't see the sense in matching your music up to old blues riffs. If you're gonna make a new lyric and new atmosphere, then the music should match it. You should create the music for it." For Walker, the lyrics always come first.

It amuses me that he calls his new material "songs" when they are so unsingable. Doesn't he think of himself as a composer? "I think of myself as a songwriter, but I agree they are maybe not traditional songs. I know what people mean, but what else can you call them?"

What does he listen to these days? "I'll listen to someone like Radiohead. I love them. Or Animal Collective, or someone like that."

Does he think pop is essentially fake? Not necessarily, he says. "There can be great purity in pop music. You hit certain moments when you can get a great single. The Sun Ain't Gonna Shine Anymore was for that time a great pop single, and it had a lot of beauty in it."

And that is some concession from Scott Walker. He knows that most people wish he'd stuck to the ballads. Occasionally, he comes across fans of his early material and, he says, they can be aggressive. "Some people are particularly protective of the early work, and they feel resentful. I had a guy on the tube one day, I was full of paint because I'd just been painting a friend's house, and I was sitting there with all this paint stuff, and this guy from across the way came and sat next to me and asked, 'Are you Scott Walker?' He says, 'I'm a big fan of yours, got all the early records, but the stuff you're doing now I fucking hate.' I went, 'Great.' I didn't know what to say to him."

Do you get a perverse pleasure when people tell you they hate your music? "No. That's totally wrong." He sounds hurt. "I realise it's not for everybody, but of course I don't want people to say, 'I hate this.'"

I tell him he still looks incredibly young. He blushes beneath the cap and giggles. "Oh bless you! Say that into here." He points to the tape recorder. "I've never had anybody say that. You're embarrassing me, Simon." Perhaps the music is your elixir? "That's it – I'm getting all the bad things out!"

Look, I admit, I do think Bish Bosch is amazing, but I'm not going to sit down and listen to it for hours on end. He looks relieved. "No! No! You'll end up dead if you do that."

Anyway, he says, he's going to surprise me. Now he's finished Bish Bosch, which he considers to be the final instalment in a trilogy, is he going to work on something completely different? "I've made three records in the same atmosphere, and I feel I have to do something really different next time. Just for my own stimulation, something lighter. Maybe a dance record." Brilliant, I say, next time you'll return as a proper pop star and outsell the world? He grins. "Yeah, Barry Manilow look out!" For the first time the tip of his cap is slightly raised, and I can see his eyes – they are smiling .

• Bish Bosch is released on 3 December.