In February 1965, a Californian trio called the Walker Brothers arrived in London, by all accounts without much of a clue exactly what they intended to do there. Their relocation didn’t make sense. After all, they were starting to make headway back in the US. A residency at a Sunset Strip club called Gazzari’s, where they played Beatles and Rolling Stones covers, had led to TV appearances and a record deal. They had just taped their second single, a Barry Mann and Cynthia Weill ballad called Love Her, that producer Jack Nitzsche had modelled on the Righteous Brothers’ version of another Mann/Weill hit, You’ve Lost That Loving Feeling: swathing everything in cavernous reverb and piling on the strings. And now they appeared to have walked away, more or less on a whim. “I still remember that incredible feeling when I got off that plane,” recalled John Maus, who, like his two bandmates, had changed his surname to Walker. “That this was going to be the ultimate adventure.”

He couldn’t have known precisely what an adventure it would turn into for Scott Engel, who was already emerging as the trio’s frontman: impressed by his baritone voice, Nitzsche had coaxed him into singing lead on Love Her. For Engel, the Walker Brothers’ arrival in the UK marked the start of one of the most extraordinary careers in pop history.

Timeline Scott Walker: his key albums Show Hide From smooth 60s lounge music to ragged doom metal, Scott Walker made one of the most extraordinary journeys in pop Take It Easy With the Walker Brothers Debut LP. Three unrelated musicians – Scott Engel, John Maus and Gary Leeds – each adopted the surname Walker. The chart-topping, Burt Bacharach-penned Make It Easy on Yourself remains one of Scott Walker's signatures, and one of the greatest breakup songs ever. Scott 4 After the Walker Brothers disbanded in 1968, Walker created a run of self-titled, numbered albums that took string-drenched easy listening into an increasingly strange, dark place, full of tragic storytelling. Scott 4 is the apotheosis, containing everything from ruminations on the 1968 invasion of Czechoslovakia to a rhapsodic, spectacularly beautiful retelling of Ingmar Bergman's film The Seventh Seal. Nite Flights The Walker Brothers reformed in the 1970s, and their brilliant final album together, Nite Flights, again foreshadows Walker's later, weirder work. The four songs that Walker contributed range from the strutting glam of the title track and Shutout, to the glacial soundscape of The Electrician. Climate of Hunter Walker's take on the slick soft rock of the 1980s – in his only album that decade – is predictably off-beam. Dire Straits' Mark Knopfler on guitar meets circular-breathing sax from Evan Parker; pristine production is made eerie by dissonant chords. Tilt Just as other artists might start to reach for easy nostalgia or see their creativity wane, Walker strikes out into avant-garde territory for this disturbing, dramatic record. He lets his voice off the leash into ever greater vibrato and wailing freedom, on the way to accidentally creating a parallel vision for post-rock. The Drift The experiments on Tilt become more mature a decade on, as the disquiet modulates into full-on agitation and terror. One song reflects on the death of Mussolini's mistress, another examines 9/11 through the prism of Elvis's dead twin brother. Wildly ambitious – and utterly magnificent. Soused A brilliant idea on paper – Walker collaborating with equally heavy, atmospheric doom metallers Sunn O))) – is brilliant in practice too: the band's titanic, glacial riffs prove the perfect bedrock for Walker's startled pronouncements. He said it was his best record yet.

It’s hard to think of many other pop stars who travelled the kind of artistic distance that Scott Walker did over the ensuing decades: from scream-inducing MOR idol to the auteur behind fraught, frequently terrifying avant garde masterpieces Tilt and The Drift, albums so challenging that a subsequent collaboration with experimental drone metal band Sunn O))), Soused, seemed like light relief by comparison. Perhaps unsurprisingly, it was not an easy journey, either for the artist – his career was marked by problems with alcohol and drugs, a troubled relationship with fame and lengthy periods of apparent inaction, during which, he claimed, he “sat in pubs watching guys play darts” – or indeed some of his fans. “I had a guy sit down next to me in the tube not so long ago,” he told the Guardian in 2006. “And he said, I have to tell you, I’m a big fan of yours. I bought your last two records and I’m never buying one again. He was really pissed off.”

Facebook Twitter Pinterest The Walker Brothers, with Scott, centre. Photograph: Dezo Hoffman/Rex/Shutterstock

At first, however, Scott Walker’s career seemed like the most straightforward success imaginable. Two months after they landed in the UK, the Walker Brothers were in the British charts. By September, they were at No 1, belting out Bacharach and David’s Make It Easy on Yourself, and the press had taken to calling Scott Walker the Boy With the Golden Voice or the Blond Beatle. They were such a perfect confection it was as if a canny music industry executive had invented them in order to sell a lot of records. Three handsome Americans who looked hip – they had grown their hair long when the Rolling Stones arrived in the US – but sang lushly-orchestrated ballads, they were perfect for an audience turned off by the left-field turn pop was taking. Tellingly, Make It Easy on Yourself lorded it over a chart that also contained Bob Dylan’s Like a Rolling Stone and the Indian-influenced drone of the Kinks’ See My Friends.

But if the Walker Brothers made middle-of-the-road music, it was middle-of-the-road music of an exceptionally high standard: blue-chip songwriters, beautifully arranged and performed. Nor could you really call it easy listening. Their hits almost invariably dealt with heartbreak in one form or another – The Sun Ain’t Gonna Shine Anymore, Another Tear Falls, Stay With Me Baby – and for all the richness of Scott Walker’s voice, it conveyed a real sense of angst and anguish. His vocals managed to sound perfectly controlled while completely inhabiting the desperation and loss in the lyrics. They had their limitations – their version of Land of 1000 Dances shows they were ill-suited to R&B – but it scarcely mattered when they were as good at conveying misery as they were on In My Room or After the Lights Go Out.

However, behind the scenes, things swiftly went awry. Scott Walker hated his new-found celebrity, and soon chafed at the artistic confines of what the Walker Brothers were expected to do to get hits: you can infer a great deal from the fact that in August 1966, he attempted to escape to a monastery on the Isle of Wight, ostensibly in order to study Gregorian chant.

A kind of eureka moment seemed to come when he heard the music of Belgian chansonnier Jacques Brel – he was fond of telling interviewers that he was first exposed to Brel’s raw emotion and existential angst by a Playboy bunny he was dating. If the Francophone world had chansonniers – whose huge popularity never precluded their songs from dealing with serious, intellectual topics – then why couldn’t the English-speaking world embrace a mainstream star whose music was similarly weighty?

Facebook Twitter Pinterest Scott Walker performing on the BBC’s Presenting Nana Mouskouri TV show, 1970. Photograph: Michael Putland/Getty Images

You could hear the idea forming on two Scott-penned songs the Walker Brothers released in December 1966 – the mini-kitchen-sink-drama Mrs Murphy and the remarkable B-side Archangel – but it came into full flower when the Walker Brothers split in 1967. On Walker’s subsequent solo albums, Brel songs and other well-chosen covers – by Tim Hardin and Bacharach and David – mixed with self-penned material that demonstrated his dramatic development as a songwriter, constructing unhappy and frequently unsettling little short stories set to music: Montague Terrace (In Blue), Plastic Palace People, 30 Century Man.

Initially at least, his gambit worked commercially, its profile bolstered by his BBC TV series Scott: his first three solo albums, Scott 1-3, all made the UK Top 3. But, by the time of 1969’s Scott 4 – entirely self-penned, credited to Noel Scott Engel, featuring songs pondering Ingmar Bergman films and the Soviet invasion of Czechoslovakia and a quote from Albert Camus on the cover – he finally found his audience’s limit. It was a brilliant piece of work, rich, dark and moving, but it failed to chart.

Walker seemed crushed. Its follow-up, 1970’s ’Til the Band Comes In, had its moments – not least the superb Thanks for Chicago, Mr James – but he spent the rest of the 70s making albums that cravenly attempted to re-establish him as an easy-listening crooner and appearing in cabaret, his reluctance to perform live partly ameliorated by an increasing dependence on alcohol and tranquillisers. In truth, his early-70s albums aren’t bad exactly – even if his heart wasn’t in it, and the material wasn’t up to much, he couldn’t stop his voice sounding beautiful – but Walker hated them, blocking all attempts to rerelease them decades later. A 1975 Walker Brothers reunion returned him to the charts – the hit single No Regrets subtly updated their old sound for a new decade – but Scott Walker had a tendency to talk about that as if it were an unmitigated disaster as well, describing it as “an abyss”. “I hated myself so much for all the years of bad faith,” he said. “I still do … I was making records to pay off bills.”

Had anyone been listening to the third Walker Brothers reunion album, Nite Flights (1978), they would have heard what you might call Scott Walker 3.0 being unveiled on the four songs he wrote for it. Shutout and the title track sounded not unlike the music David Bowie had made in Berlin and released the previous year on Low and “Heroes”, but Fat Mama Kick was wilfully discordant, strafed with free-blowing saxophone and electronic noise, while The Electrician was disturbing and elliptical. It could have been about sado-masochism, or equally torture.

Facebook Twitter Pinterest The Walker Brothers in 1976. Photograph: Michael Putland/Getty Images

But no one was listening: on release, the album vanished, as did Scott Walker, walking out of a nightclub appearance, never to return to the stage. In his absence, a myth around him grew: the doomy tenor of his early solo albums made sense in the post-punk world. His loudest cheerleader was the Teardrop Explodes’ frontman Julian Cope, who assembled a selection of his favourite Walker tracks into a 1981 compilation called Fire Escape in the Sky: The Godlike Genius of Scott Walker. Its sleeve was a “grey post-punk photograph-free design” that Cope believed would stop potential listeners feeling “they were buying into some dodgy 60s MOR icon”.

Walker offered no comment, but it’s hard not to think the renewed interest may have helped spur him into recording again, or at least getting a record deal. In 1984, he released Climate of Hunter – more obtuse lyrics, set to music that hovered somewhere between new wave rock and something more avant garde – to critical acclaim and negligible sales. Then he fell silent again, this time for nearly a decade. Bizarrely, his solitary public appearance was in a 60s-themed advert for orange juice, alongside Dusty Springfield, Sandie Shaw and the Tremeloes.

In Walker’s own telling of his mid-90s re-emergence, he was signed to a major label who believed they could capitalise on the ever-growing cult that surrounded his 60s solo work. By now, the albums Scott 1-4 had moved from the fringes of acceptance to become part of the canon of classic rock: their influence lurked around Britpop’s penchant for big orchestral ballads – Blur’s To the End, Pulp’s Something Changed – while Radiohead’s Thom Yorke apparently turned up to the recording sessions for OK Computer clutching a copy of Scott 4.

The label’s idea, Walker claimed, was to put him in a studio with a selection of young acolytes and recreate the sound of his early work. Instead, they got the nightmarish soundscape of Tilt, the album that defined Walker’s latterday style. It was discordant, abstract and forbidding, vaguely recalled both industrial music and modern classical and it frequently eschewed melody altogether in favour of what Walker termed “blocks of sound”. It bore very little relationship to the music on which his reputation rested, although its rapturous critical reception burnished his reputation further. Here was a completely unbiddable artist making music that sounded almost nothing like anything else.

Facebook Twitter Pinterest Scott Walker in 2018.

His subsequent albums, The Drift (2006) and Bish Bosch (2011), were, if anything, even more disturbing. Among The Drift’s more straightforward pieces was Clara, inspired by the public execution of Mussolini and his mistress Clara Petacci and featuring the sound of a percussionist repeatedly punching a side of pork. But the albums never sounded like a dry, intellectual exercise. There was occasionally a pitch-black sense of humour on display. Elsewhere, they had a genuine, gut-level emotional impact: that was one of the things that made them so hard to listen to.

Occasionally, as if to prove he could do it when he wanted, Walker would release a song that recalled his late-60s work – a cover of Bob Dylan’s I Threw It All Away, Only Myself to Blame from the soundtrack of the 1999 Bond film The World Is Not Enough. Then he would immediately return to avant garde mode. The latter produced music that felt as if it was written by someone in throes of catastrophic despair, and yet, in the handful of interviews he gave around the release of each album, Scott Walker seemed quietly contented. He struggled to find a niche that suited him, but found one in an entirely different musical universe to the one in which he began: the ultimate adventure, with an extraordinary soundtrack.