The inside story of the charming oddball who became a master spook and the inspiration for 007’s spy chief

The great British spy story, in fact or fiction, is a 20th-century genre that begins with The Riddle of the Sands and ends with Spooks. It has a backstory that traces the reflecting mirrors of literature and espionage to Elizabeth I and Christopher Marlowe. Thereafter, the scent goes cold, for about 300 years, until Britain’s long imperial sunset.

When British spying makes its first quasi-official showing in the works of Robert Baden-Powell, it’s as the “jolly larks” described in books such as Reconnaissance and Scouting (1884), where it’s described as a job for amateurs. “The best spies,” writes the first boy scout, “are unpaid men who are doing it for the love of the thing.”

Charles Henry Maxwell Knight slips into this narrative as an Edwardian whose boyhood became obsessed with animals: lizards, mice, hedgehogs and tortoises. Did he learn his cunning from his favourite childhood pet, a white rat named Agatha? Who knows? According to Henry Hemming, in this lively contribution to a maverick literature, Maxwell – “Max” – Knight was not just a charming oddball, he “may have been the greatest spymaster ever employed by MI5”.

Hemming’s “may have beens” haunt a biography that promises rather more than it delivers. Rich in sub-plot and cameo characters, its main theme is not as good as its overture. An engaging, damaged Englishman, Max may indeed be the model for “M” but he’s too tainted by fascist sympathies and confused sexuality to sustain the role of master spook with complete conviction.

After some unpromising beginnings as a naval reservist, London clubman, and jazz band leader, Knight’s first undercover job in 1923 was to penetrate the extreme right “British Fascisti” movement. The BF was a far cry from the jackboots of Hitler or Mussolini. Its founder was a lesbian former ambulance driver named Rotha Lintorn-Orman. Its membership included the captain of the England cricket team and the Irish fitness fanatic William Joyce, who would resurface later in Knight’s career as “Lord Haw-Haw”.

He flirted with pulp fiction, making friends with Dennis Wheatley, dabbling in the occult, and nurturing his menagerie

By the mid-1920s, Maxwell Knight was pursuing parallel lives, as a British fascist and as an agent-runner for MI5. Such a blurring of roles was typical of interwar counter-espionage, a triumph of the amateur principle that gave the young spy plenty of time for his pets (parrots, toads, grass snakes and a mongoose).

The first test for the fledgling security services came with the General Strike of 1926. The failure of a British revolution was possibly a vindication of the secret state, but it sponsored an identity crisis that sent Max (now married to a woman named Gwladys) into internal exile on Exmoor, as a publican.

By the time he re-emerged as “Captain King” or, behind as desk, as “M”, Maxwell Knight had become a fixture in the organisation that referred to itself as “the Office”, and was becoming renowned for recruiting glamorous young women from posh backgrounds. Knight, who was sexually ambiguous, worked well with female agents, and lucked out when he hired a Daily Mail journalist’s daughter named Olga Gray.

When Knight’s story becomes Gray’s story, Hemming seems to be in two minds about where his biographer’s loyalty lies. Or about the true character of “MI5’s greatest spymaster”. It’s not clear, for instance, how close Knight was to British fascism after the 1920s or, indeed, to William Joyce. He was certainly distracted. As well as running agents, he was also flirting with pulp fiction, making friends with Dennis Wheatley, dabbling in the occult, and nurturing his domestic menagerie.

The British spy: how our national obsession led to Bond and Smiley Read more

By the 1930s, Britain’s security services faced two challenges, from right and left: Mosley’s Blackshirts and the Soviet Union’s recruitment of the Cambridge spies. Maxwell Knight’s record in both departments is at best patchy. Besides, he was in the midst of his own personal crisis with the mysterious death of his wife, Gwladys. Natural causes or suicide? Hemming’s faltering narrative line leaves too many questions, both professional and personal, unanswered here.

Appropriately, for a tale of smoke and mirrors, Hemming does conjure the shadowy outline of an archetypal British spook from the MI5 archive. Maxwell Knight’s belated success in the Woolwich Arsenal spy trial, and his service in the second world war, morphs into a cold war career as “M” who recruited the young David Cornwell. Finally, in retirement, Knight found yet another career as a TV naturalist, and author of animal books (illustrated by Cornwell), working alongside David Attenborough. He died in 1968.

If the reality of “M” offers rather less than meets the eye, his appearance in John Le Carré’s A Perfect Spy as the “tweedy, unscalable” Jack Brotherhood stands in plain sight. It’s a nice irony that this shadowy, and puzzling, figure should enjoy an afterlife as a great cameo in the supreme British spy story of the late 20th century.

• M: Maxwell Knight, MI5’s Greatest Spy Master by Henry Hemming is published by Penguin (£20). To order a copy for £17, go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99