When Rudolf Hess bailed out of his Messerschmitt into a field in Scotland on the night of 10 May 1941, he was, on account of his method of entry, travelling light. If official reports are to be believed (and there is nothing about the Hess adventure that is not contested), he was carrying a flight map, some photographs of him with his son, and the business cards of two German friends. There were no other documents, no identification (Hess initially gave a false name to his captors, a befuddled group comprising a farmer with a pitchfork and the local Home Guard). However, as Daniel Pick recounts, his pockets were "stuffed full of pills and potions, including a curious elixir that had been given to him by the Swedish explorer Sven Hedin, who had in turn received it from a Tibetan lamasery."

Examined by the Medical Research Council a fortnight later, this was described as "a remarkable collection of drugs", apparently designed to ward off "all assault of the devil as far as his flesh was concerned". It included opium alkaloids, aspirin, atropine, Pervitin, barbiturates, saline mixture and a host of homeopathic products "so dilute that it is impossible to say what they are". The report concluded, with calculated understatement, that Hess had "a curious outlook on medical science."

The mystery of Hess and his "mission" went far deeper than his pharmacophilia, of course, and at first it was the task of intelligence officers to establish his motives. But, as Pick shows in this fascinating study, his role as emissary (for a negotiated peace between Germany and England, or so he claimed) and as intelligence source "was soon to be complemented and eventually eclipsed by his status as patient". Looking very much like "a caged great ape", as one of his psychiatrists put it, "Deputy Führer" Hess was to spend the rest of the war under close observation as a rare live specimen of that elusive quarry, "the Nazi mind".

Whether or not such a thing as the Nazi mind could be said to exist, let alone recovered or explained, lies at the heart of this book, which examines how psychoanalysis was harnessed to political thought about Nazism, and the legacy of that encounter. Just as the teams of Bletchley Park and the US Army Signals Intelligence Service sought to crack the enemy's secret codes, so psychoanalysts and psychiatrists were mobilised to decipher the unconscious encryptions and fantasies that were thought to drive Nazi ideology.

The analogy has its limitations: one approach is empirical and scientific, the other is amorphous and speculative. The Enigma Code could be broken, whereas the enigma of the unconscious cannot. Indeed, in four years of forensic probing, the psychoanalysts assigned to Hess were unable to reach any coherent opinions as to the subterranean contours of his mind. Was there a repressed homosexual identification with the Führer? Did he exert a Svengali-like influence on Hitler, or was it the other way around? Did he have a mother fixation? Did his rise in the Nazi party derive from a triumph of the will or its elimination? Was he insane? "No discrete diagnostic view of Hess lasted for long without some amendment," Pick writes. "He was conceptualised variously, or in combination, as obsessional, hysterical, paranoid and schizoid; a malingerer, manipulator and fantasist; highly neurotic; dissociated and confused; perverse and phobic."

Hess's mental condition deteriorated day by day. "It dawned soon enough on his doctors that his anxieties could never be assuaged by realistic assurance," Pick notes. This is rather glib. Hess had failed in his mission, he was a prisoner of war, estranged from his family, his country, his beloved Führer; he was subject to constant monitoring (even of his nocturnal emissions). What realistic assurance could be given in these circumstances? This demands a wider discussion, curiously absent here, of the psychological impact of captivity. As Pick acknowledges: "It was not always clear if he was being interrogated or psychoanalysed, debriefed or diagnosed."

In the attempt to explore Hess's "reality", however aberrant that might have been, and to lift his subconscious processes to the surface, did his physicians in some measure tamper with that reality? The use of "truth drugs" in 1944, albeit with Hess's consent, was an obvious move in that direction. But there was the more nuanced impact of psychoanalysis, which is based on the idea of a self that does the feeling with the same self also having to observe those feelings. It can be an exhausting process, and in a subject as fragile as Hess it may have created its own pathology, or even advanced an accentuated form of splitting off. Six months after his arrest, he retreated from this complication by developing (real or faked) amnesia. Thereafter, he was a most unproductive subject.

While Hess fell apart in Abergavenny (he was moved to a military hospital there in 1942), Hitler was being psychoanalysed in America. It is normally a requirement that the analyst meet the analysand, but as Hitler was busy with other matters this could not be arranged. With some reservations, but grateful for the income, the Freudian shrink Walter C Langer accepted a commission in 1943 to write a secret report on Hitler's mind. This "wild" analysis (in which the patient is absent) was conducted under the auspices of America's wartime intelligence service, the OSS (later reconstituted as the CIA), and heralded the start of a long partnership between US intelligence and psychoanalysis.

Langer, a German-speaker who had been analysed by Anna Freud in Vienna in the late 1930s, cobbled together his report from second-hand sources. The result was inevitably highly speculative, and Langer's caveat emptor – that Hitler was "elusive to the diagnostician" – did not protect him from future criticism. Equally inevitable for a Freudian of the period, Langer framed Hitler's (presumed) dreads and desires as products of the oral and anal world in which he was caught up; these were his "fixations", "the obsessive areas within which his normal developmental process had stalled". He was deemed incapable of normal genital satisfaction. Again, there was no evidence for this, but the possibility that Hitler was sexually frustrated, or perhaps monorchid, was eagerly taken up by Langer's colleagues at OSS. One of the schemes to come out of the dirty tricks department was a plot to smuggle sex hormones into Hitler's food, in the hopes that he would lose his hair and develop a soprano voice.

Arguably, Langer's report revealed more about psychoanalysis than it did about Hitler. It was circulated within the higher echelons of OSS but was soon left to gather dust (it was eventually published in 1972 as The Mind of Adolf Hitler). Its impact in wartime may have been minimal, but its legacy can be traced to the psychological profiling of foreign leaders that thereafter became commonplace in intelligence work.

Pick doesn't consider what might have been yielded had Hitler himself been on the couch, nor is the wider question of the self-psychologising of the Nazis examined. Hitler's comparison of his own self-assurance to the "precision and security of a sleepwalker" speaks to an awareness of unconscious processes, as do the photographs by Heinrich Hoffmann in which the Führer rehearses his extreme poses for the camera. Hitler may have been a hysteric – HG Wells's "screaming little defective in Berlin" – yet these images of controlled convulsion strongly imply a conscious identification with his own hysterical performance.

Not so Rudolf Hess, who by the time of the Nuremberg trials had become a bundle of tics and contortions. Psychiatrists for the prosecution judged him to be sane enough to stand trial, but many onlookers (and fellow defendants) were shocked by his psychic destruction. He was, Rebecca West wrote, "so plainly mad … He looked as if his mind had no surface, as if every part of it had been blasted away except the depth where the nightmares lived." All attempts to reanimate his memory during interrogation failed: "This is all a fog," Hess whimpered.

Pick's account of the psychiatric dimension of Nuremberg is gripping. It was first and foremost a judicial process, "but the clinical gaze was never absent", and was directed at the prisoners both as individuals and as a pathological group. Presented with this "unique laboratory" for examining the "chemistry" of the Nazi elite, eminent shrinks "clamoured for access to the prisoners". Some even proposed dissecting their brains (which would necessitate execution by a shot to the chest so as not to damage brain tissue), but the US chief prosecutor, Robert Jackson, fearful that shooting might imply death with honour, opted for hanging.

The vast literature thrown up by nazism – which includes a counter-literature that says we shouldn't go on paying it so much attention – speaks to a kind of pathology in itself. As Pick writes: "The question of how nazism caught the subject's desire was explored again and again. It was as though this very question might itself be a traumatic problem to resolve." Pick's considerable achievement is to reach far beyond the study of nazism to an extended meditation on different systems of thought and knowledge, and how these are applied to man's search for meaning.

• Frances Stonor Saunders's The Woman Who Shot Mussolini is published by Faber.