In his later years – struggling to care for his ailing wife and smarting from another, losing bout with the Hollywood studios – Raymond Chandler wrote a bleak epitaph for his career. "In every generation there are incomplete writers, people who never seem to get much of themselves down on paper," he concluded. "I guess maybe I belong in there too."

Few novelists were so uneasy with their achievements as Chandler; the sour king of hard-boiled pulp fiction, forever nagged by the sense that he was built for better things. Few, for that matter, present so twitchy and evasive a target for the intrepid biographer. How to square the tough, self-confident American prose that spilled from the typewriter with the frail, repressed English-educated gent who picked at the keys? Here we have the crime and here we have the culprit. The problem lies in connecting the two.

Tom Williams's dogged, diligent biography trails Chandler from his Chicago birthplace through his upbringing in England and schooling at fee-paying Dulwich College. From here he worked as a reporter on Fleet Street before returning to the US to pick apricots and string tennis rackets. He fought in the trenches of Europe and worked as an executive for the Dabney oil syndicate. It was only at the age of 44 – having been fired for alcohol-related absenteeism – that he finally knuckled down to a writing career, rattling off a series of detective yarns for Black Mask magazine.

At least Williams clears up one of the key mysteries of Chandler's life – showing how the author's hide-bound English schooling actually proved instrumental in seeding a new strain of American fiction. "A classical education saves you from being fooled by pretentiousness," Chandler explained. In America "the mystery writer is looked down on as sub-literary merely because he is a mystery writer, rather than for instance a writer of social significance twaddle. To a classicist – even a very rusty one – such an attitude is merely a parvenu insecurity."

Having been "raised in Latin and Greek" (his words), Chandler parlayed his romantic notions of honour and nobility into hard-drinking, tough-talking Philip Marlowe – named after the author's student house at Dulwich but transplanted to the noirish wilds of southern California. Marlowe stood as Chandler's glorious alter-ego, his perfect ambassador through a series of eight novels that began with The Big Sleep and concluded with the unfinished Poodle Springs. The private eye was Chandler's twist on Lancelot or Gawain, tackling the rogue knights of Los Angeles' criminal underbelly or the robber barons up in Beverly Hills. Yet these adventures ditched traditional, high-flown prose in favour of stylised American slang, the pages juggling steel-trap dialogue with silken similes. Along the way, Chandler evoked a modern American loneliness of city streets and rustling palms, where the "jigsaw gothic mansions" have been repurposed as rooming houses, and the old men sit out on the porch, staring at the sun "with faces like lost battles".

According to Paul Auster, "Raymond Chandler invented a new way of talking about America, and America has never looked the same to us since." At the time, however, the author was regarded with a deep ambivalence, relegated to the pulp-fiction ghetto and forced to draw succour from more favourable reviews in his beloved UK. Yet it may be that Chandler fell prey to the very prejudices he railed against. At times he gave the impression of being blithely reconciled to his calling as a writer, on a mission to break the mould of detective fiction and have it recognised as art. At others, he appeared riddled by rage and self-doubt.

What should by rights have been his heyday was hobbled by a combination of heavy drinking and a garbled code of honour that manifested itself as thin-skinned irascibility. He fell out with director Billy Wilder during the production of Double Indemnity and threw the studio into crisis while struggling to complete The Blue Dahlia. Most notoriously, he clashed with Alfred Hitchcock when adapting the script for Strangers on a Train, referring to the film-maker as a "fat bastard". Hitchcock (just as sensitive in his way) could never quite forgive him.

Williams's book paints an intriguing portrait of a man in fragments, a series of sides that won't match up. In its pages we catch glimpses of a quiet, even colourless figure periodically galvanised by alcohol; a rumoured homosexual who married a woman old enough to be his mother; a compromised artist who half-despises the books he writes and yet bridles at the slightest whiff of criticism. Reading his agent's gentle comments on the proof for The Long Good-bye, for instance, was "like being slapped in the face".

In chasing his quarry to the final chapter, the biographer sometimes risks losing the scent, falling back on conjecture and equivocation. "He was probably a bit drunk," Williams writes in one later passage. "Possibly a lot drunk, in fact." In the meantime, the man himself is creeping fast towards the exit door. His wife is dead and his career has stalled. He is dogged by depression and drinking hard to keep the demons at bay. By the end, improbably, he looks at once more vivid and more incoherent than he did in his green, unformed youth.

Still, if Chandler effects a partial getaway, that may well be for the best. Detective fiction, he once argued, was only useful as a springboard, a set of rules to be played with and broken. Character was more important to him than plot, while clear-cut resolutions were almost always a scam. "The ideal mystery," he felt, "was the one you would read if the ending was missing." Incomplete to the last, Raymond Chandler goes out as he came in: contradictory and confounding, an abiding mystery, perhaps even to himself.