Wired, pale, exhausted and curdling in cynicism, Bob Dylan cut a sorry figure at the end of his 1966 tour of Europe. The near-hysterical polemics over whether he should be playing acoustic or electric guitar were being played out as if the cultural destiny of the world were at stake. Dylan had had enough. “I’m gonna get a new Bob Dylan,” he says in a bitter backstage moment captured in Martin Scorsese’s bio-documentary No Direction Home. “Use the new Bob Dylan – see how long he lasts.”

The old Bob Dylan seemed intent on further self-destruction on his arrival back in the US. In July he was seriously injured when the wheel of his Triumph Tiger motorcycle locked and sent him hurtling to the ground near his home in Woodstock, New York. It was a turning point in his life. He wasn’t to tour again for eight years. Rumours abounded that he was a spent creative force; or that he had been hideously disfigured, or killed.

What Dylan did next was much more interesting than that. In the beginning of 1967, he called the musicians with whom he had been touring, the Hawks, and asked them to come to his house to work on some new music. Within weeks, they moved to Big Pink, a rented house in nearby West Saugerties. In the next few months, they recorded more than 100 songs, many of them newly written by Dylan himself. Far from being spent, he was reborn.

The music made during that sojourn has become known as the Basement Tapes, after the cramped space in which the songs – nine tape reels’ worth – were recorded. The sessions acquired instant fame, not because they were released to an expectant public, but because they weren’t. It was never Dylan’s intention, nor that of his fellow band members, Robbie Robertson, Garth Hudson, Richard Manuel, Rick Danko and later Levon Helm, to expose this work to the crazed audiences which had nearly destroyed him a year earlier.

Instead, recording the Basement Tapes became an act of therapy. Dylan, in a phrase which has since become a rock-music cliché but wasn’t then, went back to his roots. This was the first time that a major rock star had wilfully stopped his world because he wanted to get off. The rest of the rock scene was, in 1967, entering its baroque phase. It was stretching itself to the limit: it celebrated psychedelia and hippiedom; it swirled self-indulgently around musical virtuosity and lyrical obtuseness (to which Dylan himself, in his 1966 album Blonde on Blonde, had contributed in no small measure).

Dylan and the Band at Carnegie Hall, New York, in 1968

At Big Pink, however, Dylan travelled in precisely the opposite direction. He and his sympathetic comrades, who would catch the newfound mood of stripped-back simplicity by renaming themselves the Band, dived into the back catalogue of American folk, country, blues and gospel music. Through their workout, they made new music, some of it among the very best that Dylan has ever written.

Some of that music was subsequently released, either in official form, notably on 1975’s double album The Basement Tapes, which brought together 16 Dylan-sung tracks with eight new recordings by the Band, or in the countless bootlegs that leaked from the sessions. One of those, Great White Wonder, became the most sought-after illegal recording in rock history. It was, if you were in the know, not so difficult to find. But as a young schoolboy, I wasn’t, and hankered after it throughout the early 1970s with a solemn desperation. Subsequent releases were sporadic and incomplete. Many of the songs recorded at Big Pink have not been heard to this day.

The access-all-areas culture of the 21st century does not much care for the unavailable, however, and Dylan’s record company, Columbia, has now, 47 years after the recording of the Basement Tapes, finally succumbed to frustrated demand by releasing The Basement Tapes Complete, a definitive release of every usable track from the legendary sessions. Here in six CDs is everything that happened in those fertile months of Dylan’s rehabilitation. It is a vital document of postwar musical archaeology.

…

The singer in London in 1966

Many of the novelties contained in the set, compiled under the supervision of the Band’s organist Garth Hudson and record producer and archivist Jan Haust, are, ironically, technological. Ironic, because quality of production was never a serious consideration while these songs were being recorded. While the music world was getting increasingly obsessed by the sophisticated sounds of the studio, Dylan and his fellow musicians were focused almost entirely on “feel” at Big Pink.

And yet the colourful history of the reels, told in detail in the accompanying booklet by Clinton Heylin, Dylan’s most capable biographer, took its toll. One reel spent several years in Neil Young’s ranch, just about the right vibe for it but perhaps not kept under ideal storage conditions. The tapes have been stripped and cleaned, any extraneous overdubs removed, and mono mixes rebalanced to reflect the original recordings. While they still sound like demos and tryouts, they sound like they matter. The Band’s playing, not least Hudson’s playful keyboard embellishments, is revealed in all its elegance.

There are more treats: we get one of Dylan’s greatest middle period songs, “I’m Not There”, previously released in 2007 for the eponymous Todd Haynes movie but sounding more splendid than ever here. There is the gospel-infused “Sign on the Cross”; alternative takes of the magnificent “Tears of Rage” and the cheerful “Open the Door, Homer”.

There is the simple, and sadly unfamiliar, sound of Dylan having fun. He bursts out laughing in mid-verse on more than one occasion. He plays with nonsense lyrics, as on an early version of “You Ain’t Goin’ Nowhere”:

Now look here dear Sue,

You best feed the cat,

The cat needs feeding,

You’re the one to do it,

Get your hat,

Feed the cat,

You ain’t goin’ nowhere.

The Band in Woodstock, 1969

No one is going to put that forward in support of Dylan’s award of the Nobel Prize for literature but you can’t help smiling. This is Dylan the anti-eloquent; the folk musician, as he put it in 2004 in the first part of his autobiography Chronicles, “who gazed into the grey mist with tear-blinded eyes and made up songs that floated in a luminous haze”. The spokesman for his generation, who wished for nothing more passionately than his dethroning.

In “Clothes Line Saga”, to a slothful backing, Dylan both advertises and satirises his newfound contentment with domestic minutiae:

After a while, we took in the clothes,

Nobody said very much.

Just some old wild shirts and a couple of

pairs of pants,

Which nobody really wanted to touch.

He sounds happy. Bored and happy.

Dylan’s reverence for America’s musical history is clear. He and his group try their hands at sea shanties, antebellum saloon songs, Mississippi work songs, blues and country standards. They performed covers of songs by Hank Williams, Curtis Mayfield, Johnny Cash, Woody Guthrie, Patsy Cline, John Lee Hooker, Fats Domino and Pete Seeger. Dylan takes on Cash’s “Folsom Prison Blues”, perhaps just to try out the song’s most chilling line for size: “But I shot a man in Reno, just to watch him die”.

…

Cover of the 1975 double album of The Basement Tapes

The Basement Tapes Complete is a box set for the already-converted. I would never recommend it to anyone unfamiliar with Dylan’s work, who is in the fortunate position of having much greater treasures to discover. And yet, if nothing else, the set stands as a testament to what has happened to rock music since its recording. As it becomes increasingly difficult to imagine how it happened in the first place, the story behind its making assumes a near-mythological status.

None of the circumstances that led to Dylan’s retreat makes any sense today: the pressures that prompted his near-breakdown, not the mere pressures of celebrity but a sense of cultural expectation that must have been unbearable; the subsequent hiding away with friends, and his engaging of a reverse gear that would take his own music into a new direction; his wilful refusal to commit any of this process to public scrutiny.

Today’s popular music landscape is the polar opposite of Dylan’s experimental sideshow in Woodstock. In the television talent shows that dominate the sales charts, there is an obsession with process, a scrutiny of the brittle architecture of pop stardom, a disregard for genuine innovation. Not only are the programme-makers unembarrassed by lack of originality, they make a virtue out of it: Lennon-McCartney week, Motown week and so on.

Hardly fair, you may think, to compare what Dylan was doing and the impulses behind The X Factor. And yet: at the end of 1967, Hudson made a reference copy of 15 of the best songs from the Basement sessions, which was sent around to assess if there was any interest from others to cover them. This, Dylan said in 1984, had been the intention all along. “They were just songs we had done for the publishing company . . . for other artists to record those songs. I wouldn’t have put them out.”

Much as the pop world remained confused by what Dylan was up to, it responded. And so it came to pass that two Basement Tapes songs, “Quinn the Eskimo” and “This Wheel’s on Fire” found their way to the top five of the UK singles charts in early 1968, covered by Manfred Mann and Julie Driscoll respectively. The latter song became the theme tune of the 1990s sitcom Absolutely Fabulous. As well as everything else, the Basement sessions were also the tentative beginnings of a hit factory.

Today that synergy between the relaxed musings of a genius-at-rest and the sharp edge of commercial hit-making has been lost, and it is hard not to lament its passing. But the Basement Tapes had a still wider impact on the rock scene. It encouraged many acts who had sprung to improbable levels of fame so quickly, to sit back and retrench. Hence The Beatles’ “Get Back” project; the Rolling Stones shacking up in the south of France to make Exile on Main Street; Led Zeppelin’s recordings at Bron-Yr-Aur in rural Wales.

Dylan’s musical investigations also prompted rock music to arrest its headlong rush and study its own history. From the Basement Tapes came the alt-country and Americana movements that continue to champion authenticity and musical integrity at the expense of commercial opportunism. That all this should be achieved without Facebook, or Twitter, or press releases; indeed without telling anybody anything, seems like a miracle to our information-soaked time. We thought that Dylan was counter-cultural back then. What does that make him now?

Having made the world a better place, Dylan re-emerged into the limelight a changed man. His next album would be John Wesley Harding, which was steeped in the spirituality he had encountered in his basement explorations. There would be some more failed experiments in the coming years, and then the triumph of 1975’s Blood on the Tracks. There was to be a startlingly rich Second Act to this American life, and here, in its messy, sprawling glory, is what happened backstage during the interval.

Bob Dylan’s ‘The Basement Tapes Complete: The Bootleg Series Vol 11’ (Sony) is released on November 4

Peter Aspden is the FT’s arts writer; he reviews ‘Let Me Be Frank with You’ by Richard Ford

Photographs: Elliott Landy/Magnum Photos; Getty/Hulton Archive