It is a winter's morning in Berlin in 1900. The maid has put an apple to bake in the little oven at eight-year-old Walter Benjamin's bedside. Perhaps you can imagine the fragrance, but even if you can, you won't be able savour it with the manifold associations that Benjamin experienced when he memorialised the scene 32 years later while living in precarious circumstances and fearing worse. He wrote in his autobiographical masterpiece, Berlin Childhood Around 1900, that the apple extracted from the oven's heat "the aromas of all the things the day had in store for me. So it was not surprising that, whenever I warmed my handson its shining cheeks, I would always hesitate to bite in. I sensed that the fugitive knowledge conveyed by its smell could all too easily escape me on the way to my tongue. That knowledge which sometimes was so heartening that it stayed to comfort me on my trek to school."

But comfort was quickly displaced: at school he was overtaken by "a desire to sleep … I must have made that wish a thousand times, and later it came true. But it was a long time before I recognised its fulfilment in the fact that all my cherished hopes for a position and proper livelihood had been in vain."

So much of Benjamin is in this vignette, starting with the apple, whose aroma prefigures his expulsion from childhood Eden, which prefigures his adult exile from Germany. There is the melancholic who gets what he wants (sleep) only when it is irredeemably associated with the frustration of other wishes. There is the jump-cut montage (from bed to school to disenchanted adulthood) prefiguring his championing, in his 1936 essay "The Work of Art in the Age of its Technological Reproducibility", the techniques of cinema and their revolutionary potential (a potential that Hollywood, as you may have noticed, nixed). There is the Proustian reverie of one for whom writing is remembrance. There is the character of memory that Benjamin described thus: "The stratum of the writer's present day, broken through and made transparent, becomes a window on to remembered experiences that … depend upon it for the realisation of their latent meaning."

In fact, there is everything for the Benjamaniac except the kabbalah-inflected Marxist tenor of his late writings that made him a hero to leftist critics including Giorgio Agamben, Terry Eagleton, Perry Anderson and Fredric Jameson – but you can't have everything.

We might forgive Benjamin those notes of self-pity, certainly after reading in Howard Eiland and Michael W Jennings's vast, heroically researched and yet frustrating critical biography about the circumstances in which the bittersweet memoir was written. In the summer of 1932, Benjamin was very nearly at the end of his rope. Professionally, his dreams of academic tenure had been crushed and he was struggling to make a living as a writer at the moment when opportunities for a Jew publishing in Germany were, thanks to Hitler's poisoning of intellectual life, about to dwindle almost to nothing.

His personal life, too, was in tatters. Acrimoniously divorced from his wife Dora, all but estranged from his only child Stefan, having recently proposed to and been rejected by Olga Parem, a German-Russian woman he'd fallen for, he was so close to suicide that he had recently written his will and letters of farewell in a Nice hotel room to some of his dearest friends ("And even now that I am about to die," he wrote to his former lover, the sculptor Jula Cohn, "my life has no greater gifts in its possession than those conferred on it by moments of suffering over you.")

In this mood, Benjamin arrived at a little seaside resort called Poveromo (Italian for "poor man") and started writing his remembrances of an upper-middle-class Jewish home in Berlin's West End at the turn of the century. Meanwhile, in the city of his birth, the chancellor Franz von Papen was revoking the ban on the Sturmabteilung, the Nazis' paramilitary wing, and thereby unleashing a torrent of political violence and terror that ultimately led to Hitler's assumption of power. Radio stations for which Benjamin had presented talks became only mouthpieces for rightwing propaganda, the Frankfurter Zeitung, so long a home for his work, began leaving Benjamin's letters and manuscript submissions unanswered. Publishers looked even more askance at his work than before.

Benjamin was clear-eyed about what this prefigured. His exile was imminent and that summoned up, as Eiland and Jennings put it, a "need for inoculation". It "became clear to me that I would have to bid a long, perhaps lasting, farewell to the city of my birth," he wrote in 1938. "In this situation … I deliberately called to mind those images which, in exile, are most apt to awaken homesickness: images of childhood. My assumption was that the feeling of longing would no more gain mastery over my spirit than a vaccine does over a healthy body."

Some assumption. One might expect that awakening unquenchable homesickness would crush the exile's spirit, but no: in Berlin for a year and thence during the Parisian exile that ended with his tragic flight from the Nazis and suicide in Spain in 1940, he wrote some of his greatest essays for the few publications, mostly run by Jews in exile, that would take them. That's not to say adversity was a catalyst, nor that the assumed inoculation worked, but, rather, that Benjamin – on the run and on the skids – still managed to write brilliantly. His essay on Franz Kafka was written for the Jüdische Rundschau; his essay on Brecht's epic theatre for Thomas Mann's Mass und Wert.

Most important, there was that melancholy band of fellow exiled German Jewish intellectuals of the Frankfurt school (Adorno, Horkheimer, Löwenthal, Marcuse, Fromm) who supported him with monthly stipends and published in the Zeitschrift für Sozialforschung not just his study of Baudelaire, but his "The Work of Art … ". And then, too, there was the incomplete, perhaps impossible to finish, The Arcades Project – his critique of bourgeois culture in 19th-century Paris and his grand, failed, attempt to answer the call for a history centred on the sufferings of the vanquished rather than on the achievements of the victors – on which he worked fitfully for the last 13 years of his life.

Why is Benjamin important in 2014? Why should we read him? Eiland and Jennings, two Harvard scholars who have translated many of his works into English (not least The Arcades Project), argue compellingly that as a critic he not only reshaped our understanding of many important writers, but he recognised the potentials and hazards of technological media that revolutionised culture during his lifetime. Others find in him a literary deconstructionist avant la lettre, a social theorist who envisioned a wholesale renovation of the human sensorium through the reform of modern media, a fire-breathing communist, a messianic Jewish mystic.

Eiland and Jennings are content that readers find their own Benjamins in his work and in their book. Until now, they argue, he has been an enigma as a man if not a thinker, but here he appears in a series of guises: faithless husband, unreliable father, compulsive gambler, frequenter of prostitutes, draft dodger (he and Gershom Scholem stayed up all night drinking black coffee to simulate weak hearts and thereby fail an army medical), serial participant in love triangles (he was erotically and perversely drawn, the authors convincingly suggest, to triangles from which he couldn't get what he wanted). They chronicle his experiences with mescaline and hashish. They wonder what was in the famous black attache case that mysteriously disappeared after his suicide in 1940 – did it contain a manuscript that would, had it been found, have rocked the world? Eiland and Jennings are doubtful.

All this would make for a superbly racy narrative, but Eiland and Jennings don't roll that way. Theirs is an impressive work of exegesis, but their book struggles under the weight of the detail they present. They write his life as a historical continuum – eschewing the montage techniques, the fracturing of chronology, and the gambles with form and style to which their subject was programmatically committed and temperamentally prone. "From first to last," the authors write of this Dostoevskian gambler, "Benjamin took chances in the subjects he addressed and on the form and style of this writing." They don't.

And that feels like an abnegation of responsibility: the point of writing a biography, as Ray Monk once said, is to present a life. Otherwise, he might have added, it's just one damned thing (what Benjamin read that month, who he met for coffee) after another. As I read, I kept thinking of the way in which Richard Holmes presented the life of Samuel Taylor Coleridge, like Benjamin an intellectual with drug issues, a wandering eye and a disgraceful disregard for his duties to wife and children.

Ah yes, but Eiland and Jennings have an answer to my objections. We must not enclose a writer, particularly one whose writings amount to a "mobile and contradictory whole" in one mere narrative; better to allow readers to project their own Benjamins on to the material. Near the end, to support this point, they quote what he wrote about great writers in One-Way Street: "For only the more feeble and distracted take an inimitable pleasure in closure, feeling that their lives have thereby been given back to them." I admit to being feeble and distracted, but not that closure is the same as presenting a life.

For all its outstanding scholarship, and for all that their biography is as a result indispensable, there is a risk that Benjamin drifts away from these pages like the fragrance of baked apple on a winter's morning. Or, indeed, as Benjamin himself slipped away from us prematurely in 1940, his suicide representing, as Scholem put it, with pardonable exaggeration, "the death of the European mind".

• To order Walter Benjamin: A Critical Life with free UK p&p call Guardian book service on 0330 333 6846 or go to guardianbookshop.co.uk.