Maxwell Knight seemed unemployable, until he found his metier in the British Secret Service in the 1920s. He was a failure as a son, a husband – he was impotent, though his third wife didn’t mind – a naval rating, a civil servant, a paint seller, a prep school games teacher and a novelist. His abiding interest was in exotic animals, ranging from spiders, toads and snakes to parrots and a brown bear, which he kept in his rooms, living happily with the stench, filth and noise they brought with them. But that didn’t seem to offer him much of a career. He was also a jazz fiend, which was thought to be radical and even depraved in the 1920s; and a fascist, which probably helped him get into MI5.

MI5 has been much criticised for its upper-class, rightwing bias between the wars, which – it is claimed – caused it to lose sight of the German Nazi menace in its obsession with communism. Hemming’s book offers a different reading. Despite his eccentricities, or perhaps because of them, “Max” proved to be supremely good at his new job in countersubversion, earning Hemming’s accolade as “MI5’s greatest spymaster” – equally effective against both the major extremist threats of his day.

He was, admittedly, a little late on to the second of them. He had joined the British Fascisti because he believed in them, and in their value in countering the Red Menace – though Hemming suggests it may also have been to impress a girlfriend. His greatest chum there was William Joyce, later “Lord Haw-Haw”, whom he probably tipped off in 1939, just before Churchill’s roundup of domestic fascists, so enabling him to flee to Germany to undermine Britain’s war effort from there. (Joyce was hanged for treason in 1946.) That, Hemming thinks, was because Knight shared the famous (or notorious) EM Forster view, that it was better to betray one’s country than a friend. Meanwhile he achieved great success in placing his agents high up in communist and other leftwing circles, usually as secretaries.

Hemming credits him with two particular innovations as a spymaster: first, his use of women, who he thought were more to be trusted than men; and second, his emphasis on patience. Agents should not be too pushy or inquisitive, which could arouse suspicion, but must wait to be asked by their targets to take on greater responsibilities, and so become privy to deeper secrets. He was also good at spotting likely recruits for his subterfuges. (Hemming hints that this might have come from his empathy with animals.) The results of all this were a number of impressive coups, including the unmasking in 1938 of what came to be called the Woolwich Arsenal Spy Ring, through his leading female agent, Olga Gray.

Anthony Blunt … Knight suspected him of spying activity. Photograph: Jane Bown/The Guardian

It was only around then that Knight fully cottoned on to the danger from the other end of the political spectrum. Others on the right didn’t, arguing for neutrality or even an alliance with the continental fascists against the greater menace they feared from Russia. Why Knight didn’t take this line is not entirely clear. Hemming argues that it was because he never took on the nastier aspects of fascist ideology, such as racism, and was not even particularly political, being “throughout his life … mistrustful of intellectuals and elaborate political creeds; indeed there were times when he seemed to believe in little more than his country”. (Another contemporary rightist dismissed his ideology as merely “Conservatism with knobs on”.) This would explain his turning against the fascists and their fellow travellers. Ironically, his previous association with them allowed him to penetrate far more deeply and effectively than if he had not been one of them himself.

His main coups on this front were his reports on fellow-travelling organisations, such as the “Right Club”, that might have developed into a “Fifth Column”. These strengthened Churchill’s push for the mass arrest of all enemy aliens and fascists in 1940, against resistance from the Home Office on “civil liberties” grounds (as an ex-fascist, of course, Knight had little truck with such protests). It also led to the crucial arrests of Tyler Kent and Anna Wolkoff before they could bring to fruition their plot to keep the US out of the war (by alerting the isolationist American public to Roosevelt’s secret correspondence with Churchill). By the end of the war, Hemming claims, Knight had done more than anyone to bring about the death of organised fascism in Britain.

Meanwhile he didn’t take his eye off the Reds. Early in the war he wrote a famous internal memorandum, “The Comintern is not Dead”. It wasn’t taken seriously at the time – the USSR was now an ally, of course – but turned out to anticipate the Soviet penetration that became the major British spy scandal of the postwar years. He also suspected one of the Cambridge moles, Sir Anthony Blunt, before anyone else did. But that was just a “feeling” he had.

The book may be too celebratory, in fact. Could Hemming be one of 'them'?

Knight relied on his instincts a lot and on his ability to get his agents to warm to him: he became something of a legend for years afterwards among the broader intelligence community. (Hemming identifies penetration agents that have not been named before.) In the 1950s he exploited his charm in a second career – pursued contemporaneously with his MI5 one – as a popular writer and broadcaster on his first love, animals. A BBC national treasure, he was hard-line Tory and strongly in favour of corporal punishment – “My views will be called old-fashioned or sadistic by a few of our wishy-washy reformers” – but avuncular at the same time. I remember hearing him on the radio.

Hemming is too young to have heard him; but he has clearly fallen under his spell. (He likes eccentrics, having written a previous book about them.) He writes that the aim of this book is to “celebrate M and his agents”, and to defend them against the obloquy that is often visited – especially no doubt by those “wishy-washy reformers” – on people whose careers are founded on deception and double-dealing. On the contrary, Hemming claims, M’s shadowy spooks were “brave men and women who chose to let go of a part of themselves – who gave over their lives, really, anonymously and for very little reward – to a spymaster they trusted, and for a country they believed in”.

The book may be too celebratory, in fact. Could Hemming be one of them? Probably not; but he is very secretive about his background. (Google throws up almost nothing.) Apparently he read history at one of the universities I taught at, but obviously not in my time, as I can’t remember him from the History of the Secret State course I put on there, which was rather less celebratory. Whatever its provenance, however, this is a terrific book, well researched and superbly written. Apparently it’s going to be dramatised for TV soon. That should be fun. More seriously, its take on the question of MI5’s rightwing bias is original and important. Just don’t fall too uncritically for its charm.

• M is published by Preface. To order a copy for £17 (RRP £20) go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.