Towards the end of 1892, ‘Miss Lucy R’, a pale and delicate English governess living in Vienna, made her way to the surgery of a young neurologist on Berggasse 19 for the treatment of a ‘suppurative rhinitis’. Miss Lucy was tired, in low spirits and complained of ‘a muzzy head’. And though she had lost her sense of smell, she was endlessly tormented by the smell of burnt pudding.

Sigmund Freud was 36 years old when he began attending to Miss Lucy. Trained at the Salpêtrière Hospital in Paris by the great neurologist Jean-Martin Charcot, Freud had already published monographs on hypnosis, epilepsy, and cocaine, which he continued to self-administer for ‘vitality and capacity for work’. Now he was applying his able and imaginative mind to the mystery of hysteria – whose bewildering array of symptoms were still considered hereditary ‘stigmata’. Upon examining the 30-year-old governess, he found her physically healthy, save for her nose’s insensitivity to touch. What struck him most about this case was the recurrent smell of burnt pudding.

Freud rejected the possibility of an organic explanation, even though acrid or burning smells are commonly associated with migraines, epilepsy and sinus infections. Instead, he deduced that Miss Lucy’s hallucination was a ‘memory-symbol’, a psychic trace standing in for a forgotten or repressed trauma, possibly related to sexual seduction or abuse. ‘What I suspect,’ he told her bluntly, ‘is that you are in love with your master, the director, perhaps without being aware of it yourself, and that secretly you are nursing the hope that you really will take the place of the mother.’

Studies on Hysteria (1895), co-authored with his physician friend and mentor Josef Breuer, would prove to be Freud’s breakthrough work. The book, based on Miss Lucy and four other cases, led him to two important insights. First, the physical symptoms of hysteria were caused when intolerable ‘ideas’ were evicted from the conscious mind. Second, the most effective antidote to hysteria’s psychic befuddlement, the best way of returning the patient to ‘ordinary unhappiness’, was what Breuer’s patient Anna O dubbed ‘the talking cure’. Forget the hypnotic overtures that Freud had been dabbling with since his time with Charcot at La Salpêtrière – from now on, free association coupled with attentive listening would be the proprietary salve of psychoanalysis.

Freud’s early writings on hysteria garnered little fanfare from his clinical peers. On reading Studies, Richard von Krafft-Ebing, the chair of psychiatry at the University of Vienna, dismissed the so-called theory of hysteria, including Freud’s contention that symptoms often came from childhood molestation or abuse, as ‘a scientific fairy tale’. Similar misgivings were voiced by laboratory psychologists working to place their discipline on empirical foundations. To use a sobriquet coined by the combative psychologist Edward Scripture, founder of the Yale Experimental Psychology Laboratory, Freud was an ‘armchair psychologist’, and his serial ruminations on ‘the unconscious’ – on dreams, on infantile sexuality, on jokes and parapraxes – reflected an equally unscientific ambition: that psychoanalysis would evolve as ‘a profession of lay curers of souls who need not be doctors and should not be priests’.

Four decades after treating Miss Lucy, Freud had permeated Western thought. He’d built a therapeutic empire by identifying the id, ego and superego as the forces of a ‘power struggle’ between instinct and morality ‘going on deep within us’. Yet as Freud’s cultural stock rose, his writings remained testament to an elective blindness, showing imperial disregard for most of his philosophical precursors and peers. In all his major publications on the unconscious, from Studies through to Civilization and its Discontents (1930), Freud barely acknowledged the pioneering Pierre Janet, the French psychiatrist well-known for his theory that traumas caused personality to dissociate into conscious and unconscious parts. There was no mention of Friedrich Nietzsche, who held that the unconscious mind yielded the deepest truths, or of Arthur Schopenhauer, who identified will itself as unconscious. Freud all but ignored the experimental work on unconscious inference that Hermann von Helmholtz had undertaken since the 1840s. And he showed curmudgeonly disdain for the rival theories of his one-time acolytes and ultimate critics Alfred Adler (who put stock in feelings of inferiority) and Carl Jung (a proponent of archetypes inhabiting the unconscious).

In fact, despite Freud’s renown, several approaches to the unconscious had already been established before the advent of psychoanalysis. According to the Canadian psychiatrist and historian Henri Ellenberger, Freud & Co were merely the latest representatives of the ‘mythopoetic’, who sought reality in dreams and fantasies. Earlier theorists had regarded the unconscious as a secret recorder of impressions and sensations that lay beyond the narrow beam of consciousness, an incubator for creative, innovative and inspirational insights, and a gateway to the secondary or submerged personalities linked to somnambulism, hypnotism, hysteria and fugue states.

Other investigators grounded their work in mechanistic physiology, using the language of neurons and cortical excitation. By the mid-19th century, these researchers had much to say about the ‘latent’ and ‘automatic’ nature of acquired habits and actions, sparking a long-running debate on the whys and wherefores of ‘unconscious cerebration’. In The Physiology of Common Life (1860), the English philosopher George H Lewes observed:

In learning to speak a new language, to play on a musical instrument, or to perform unaccustomed movements, great difficulty is felt, because the channels through which each sensation has to pass have not become established; but no sooner has frequent repetition cut a pathway, than this difficulty vanishes; the actions become so automatic that they can be performed while the mind is otherwise engaged.

While Freud’s Gothic version of the unconscious was steeped in dream symbols, repressed wishes and hidden traumas, the unconscious as described by Lewes and other commentators chimed with the pragmatism found in works of Victorian psychology and self-help literature: the importance of turning useful actions into unthinking habits, of actively deciding upon one’s ‘second nature’. In his landmark textbook The Principles of Psychology (1890), William James insisted that almost all of our personal habits, including gait, voice and gesture, are fixed by the age of 20: ‘The more of the details of our daily life we can hand over to the effortless custody of automatism, the more our higher powers of mind will be set free for their own proper work.’

The first experimental demonstrations of this automatism came from an unlikely source. At the height of Britain’s table-tilting craze, in 1853, Michael Faraday set out to investigate why so many eminent and educated séance-goers attributed the uncanny movement of the table to electricity, magnetism or some other imponderable force. After commissioning a Regent Street instrument-maker to construct a table with levers that would secretly register downwards and oblique pressure from the hands of the sitters, Faraday’s trials at the Royal Institution furnished proof positive that table-turning was produced by ‘unconscious muscular action’.

Of course, Faraday’s ingenious ruse barely scratched the surface of the unconscious. More than a century after Gottfried Leibniz’s New Essays on Human Understanding had proposed that subliminal ‘petite perceptions’, not conscious will, accounted for most of our actions, mental philosophers and physiologists were still reaching for insights into the secret agency of the unconscious in everyday life. How exactly did a newly learned task become an unthinking habit? Why did a name or phrase that could not be recalled invariably spring to mind once one’s attention was directed elsewhere? Why were our emotions apparently, as the English physiologist William Carpenter wondered, so often ‘determined by circumstances of which the individual has no idea’?

With the rise of laboratory-based psychology in Germany, the questions that Leibniz had raised were effectively sidelined and the unconscious was, after a period of neglect, approached through an industrial doorway. Efficiency, not health or happiness, was the burning issue for a new generation of scientific psychologists who admonished the likes of Freud and James for ‘vague observation, endless speculation, and flimsy guesswork’.

As the new discipline travelled from Germany to the United States in the 1880s, former students of Wilhelm Wundt, who founded the first psychological laboratory at Leipzig, continued to attack the ‘mystical’ theorists of the ‘the unconscious’. Yet the subliminal processes involved in the acceleration of memory, perception and learning could not be ignored. ‘Rapid thought and quick action sometimes make all the difference between success and failure,’ Wundt’s erstwhile student, Edward Scripture, reminded his readers. ‘A man who can think and act in one half of the time that another man can, will accumulate mental or material capital twice as fast.’

Patients were invited to repose on a couch and talk about their problems while tethered to a galvanometer

Reaction-time research and learning-curve studies, which dominated scientific psychology in its early years, gave birth to a vast catalogue of tabulated statistics on the best methods of teaching telegraphy, shorthand, foreign languages, baseball and the piloting of aircraft. As Henry Ford inaugurated the era of mass production, opening the first moving assembly line at Highland Park in Michigan, American psychology was about to radically re-engineer the science of human behaviour.

Inspired by Ivan Pavlov’s research on the conditioning of animal reflexes at the Institute of Experimental Medicine in St Petersburg, John Broadus Watson, a psychologist at Johns Hopkins University in Baltimore, published his landmark article ‘Psychology as the Behaviourist Views it’ (1913), in which he argued that environmental triggers were of far greater importance in shaping human action than heredity or constitution. Cementing his behaviourist claims with his infamous experiments on ‘Little Albert’, a nine-month-old boy who was conditioned to fear various objects, Watson shoved the hereditarians and depth psychologists aside. There was ‘no such thing as an inheritance of capacity, talent, temperament, mental constitution and characteristics’. No hidden unconscious, no latent potentialities. The message was simple: anyone could, given the right training, be what they wanted in the new Fordist US.

The rise of behavioural psychology did not put an end to the unconscious. In the land of the typing pool, department store and nickelodeon, habituation and automaticity were central to industrial and educational psychology. The possibility of uncovering unconscious mental processes drew its greatest impetus from technologies that could flash, flicker and hum at below the liminal threshold of perception, and instruments that could identify the physiological markers of sleep and its borderlands. For example, the invention of the electroencephalography by Hans Berger in 1929 revealed something that Freud and his fellow dreamworkers had missed. There were two different types of nightly mental activity. In REM sleep, there was muscle and eye movement that suggested the tracking of inner imagery. In non-REM sleep, there was little in the way of visual imagery or sequential narrative.

Back in 1900, soon after Freud published The Interpretation of Dreams (1899), a little-known New York neurologist named James Corning was wondering whether an electrified consulting room might help him to treat hysteria and nervous irritability. Corning’s strategy involved playing ‘spiritual’ music and projecting swirling images onto a screen at the foot of the chaise-longue while his patients slept. Corning reported some success with his multi-sensory treatment – particularly when he played the role of dream narrator, whispering suggestions to his sleeping patients – until his experiments were abandoned due to time and cost.

But other researchers carried the baton. At the Burghölzli Clinic in Zurich, Carl Jung, the future ‘crown prince’ of psychoanalysis, had employed a galvanometer, a device that registered changes in the electrical resistance of the skin via electrodes, to identify a shadowy, complex cluster of unconscious feelings, beliefs and thoughts. At Johns Hopkins University, the psychologist Joseph Jastrow constructed a so-called ‘automatograph’ to study unconscious motor activity. And, in an experiment that was warmly received by Freud, the Viennese neurologist Otto Poetzl used a series of flashing images to help support the idea that dreams were composed from the debris of ‘unremembered day’.

Inevitably, some of these psychological instruments found their way into the therapist’s consulting room. In the mid-1920s, Harold Lasswell, a psychoanalytically trained political scientist who went on to become a major figure in the study of propaganda, turned his office at the University of Chicago into a ‘therapeutic laboratory’. His patients were invited to repose on a couch and talk about their problems and preoccupations, all while tethered to a galvanometer and a foreboding bank of instruments that tracked their eye movements, shifting posture, blood pressure and pulse rate. Laswell’s objective was to meld reams of physiological data with therapy’s viva voce insights into the unconscious. Predictably enough, his experiments were not well-received by the Freudian community. As soon as the New York and Chicago psychoanalytic institutes caught wind of Laswell’s trials, they issued a ban on such technological intrusions in the consulting room.

Beyond the couch and the lab, inventors and hobbyists were also contriving ways of tapping the unconscious. In 1927, as Laswell undertook his first trials at the University of Chicago, a Czech-born entrepreneur brought ‘the Psycho Phone’ to market. Essentially a wax-cylinder phonograph connected to an alarm clock, Alois Benjamin Saliger’s ‘automatic suggestion machine’ claimed to ‘direct the vast powers of the unconscious mind during sleep’. Retailing at a little over $200, it played a dozen or so specially produced ‘affirmation records’ with titles such as ‘Prosperity’, ‘Normality’ and ‘Mating’. According to testimonials, the phone could deliver profound life changes, including success in business, improved health, even charisma.

Ultimately, interest in sleep-learning waned. But mid-century pop psychology continued to seed the benefits and risks of subliminal influence and learning. L Ron Hubbard’s Dianetics: The Modern Science of Mental Health (1950) recommended purging oneself of the hidden traumas, or engrams, that were stored deep within the ‘reactive mind’ to achieve a ‘state of Clear’. Norman Vincent Peale’s The Power of Positive Thinking (1952) coached its readers to develop self-confidence through the same kind of affirmations and autosuggestions that Saliger had used. And Vance Packard’s The Hidden Persuaders (1957) exposed the subliminal skullduggery of US advertisers and marketeers.

Among the rollcall of the experts featured in Packard’s overheated exposé was James Vicary, described as ‘the most genial and ingratiating of all the major figures operating independent depth-probing firms’. Vicary worked for a roster of blue-chip clients, and his dubious insights into the consumer mindset included the notion that women bake cakes as a surrogate for childbirth, and that the surfeit of consumer choice was triggering a ‘hypnoidal state’ in the supermarket aisles.

Just as Packard’s book was having its day in the sun, Vicary held a New York press conference to proclaim that his firm had boosted beverage and snack sales in a New Jersey movie theatre by subliminally flashing messages (‘Drink Coca-Cola’ and ‘Eat Popcorn’) on the screen. More than 45,000 moviegoers had been exposed to a secret hailstorm of 1/3,000th-second adverts flashed every five seconds, Vicary claimed, and the point-blank messages had increased sales of Coca-Cola and popcorn by more than 18 and 57 per cent respectively.

Vicary’s advertising experiments whipped up a media frenzy. Within weeks, a rival company, Precon, headed by the psychologist Robert Corrigan, announced a patent-pending subliminal device that also boasted educational and psychotherapeutic applications. Before the year was out, a deluge of complaints led members of congress to demand assurances from the Federal Communications Commission regarding the use of ‘deceptive advertising’.

The unconscious is an enabler, rolling up its sleeves to expand the number of operations we can perform without thinking of them

By 1958, it was apparent that Vicary’s subliminal research was a hoax, but the marketplace for tales of mind control, especially those of a scientific or literary bent, was unaffected by the exposé. Aldous Huxley’s Brave New World Revisited (1958), published in the aftermath of the subliminal advertising controversy, rushed to endorse the mythology of subliminal manipulation that he’d laid out 30 years earlier in his dystopian novel. Giving rather too much credence to tired propaganda that insisted that brainwashing had been taken to new Pavlovian extremes by the Chinese Communist Party, Huxley, an avid anti-Freudian, claimed that experimental psychologists had various modes of subconscious persuasion at their disposal, including drugs, hypnopaedia and subliminally flashed images. If nothing else, Huxley’s odd little diatribe proved that ‘the unconscious’ had become a factitious repository, home to a phantasmagoria of imagined spells and scientific half-truths.

The most credible insights into the unconscious were rather more measured, stemming from a loose collective of cognitive scientists who were beginning to map the mind as an information-processing system. At the forefront of this new movement was the Cambridge-educated psychologist Donald Broadbent. A specialist in human-machine interaction, who had studied the work of pilots, air-traffic controllers and Post Office letter-sorters, Broadbent’s book Perception and Communication (1958) outlined the sundry processes involved in ‘filtering information from the senses’, ‘passing it through a limited capacity channel’, and then storing it. Broadbent’s dust-dry writing was no match for the soaring prose of Freud or James, but his flowcharts of the cognitive filters and buffers involved in handling complex information helped shine a light on what has become known as the cognitive unconscious – the ability to perform multiple tasks, to make inferences without deliberation, to demonstrate memory without recollection, to selectively attend to one phenomenon from a whole theatre of sensory possibilities.

The eminent Freudian scholar Peter Gay notes that the ‘fundamentals’ of Freudian theory garnered ‘impressive experimental support’ in the mid-20th century. While a generation of psychologists, many of whom had undergone analysis, used the language of psychoanalysis as a crutch for their studies of perception, learning, imitation and frustration, there was no substantive evidence of deep, dynamic processes. For example, Jerome Bruner and Leo Postman’s highly influential study of ‘perceptual denial’ – the Freudian theory that recognition of anxiety-provoking stimuli in the laboratory was delayed due to repression – missed a far simpler explanation. Taboo words were less common than their neutral counterparts, and it was expectation that accounted for their participants’ delayed recognition.

The subterranean channels explored by cognitive psychology have proved more revealing. Over the past three decades, experiments in implicit memory and priming have hinted at the existence of different memory systems. The Princeton psychologist Anne Treisman and others have revised Broadbent’s initial filter-theory of attention, proposing that unattended stimuli are not completely filtered but attenuated. And there is now a vast literature on ‘mere exposure’, which suggests that repeated exposure to a stimulus, above or below the threshold of conscious awareness, plays a large part in the formation of personal preferences.

Freud’s ghost might still haunt a small corner of the modern-day psychological laboratory, but the lexicon of censorship and repression has not retained its explanatory currency. Studies of waking and sleeping unconscious processes suggest that deception is not, and has never been, the second self’s true forte. As the mathematician and philosopher Alfred North Whitehead sagely observed in the early days of psychoanalysis, the unconscious is essentially an enabler, quietly rolling up its sleeves to expand ‘the number of important operations that we can perform without thinking of them’.