THESE are thrilling days for behavioural research. Every week seems to yield a new discovery about how bad people are at making decisions. Humans, it turns out, are impressionable, emotional and irrational. We buy things we don't need, often at arbitrary prices and for silly reasons. Studies show that when a store plays soothing music, shoppers will linger for longer and often spend more. If customers are in a good mood, they are more susceptible to persuasion. We believe price tends to indicate the value of things, not the other way around. And many people will squander valuable time to get something free. The sudden ubiquity of this research has rendered Homo economicus a straw man. Yet such observations are not new. Analysts have been studying modern man's dumb instincts for ages. Sigmund Freud argued that people are governed by irrational, unconscious urges over a century ago. And in America in the 1930s another Viennese psychologist named Ernest Dichter spun this insight into a million-dollar business. His genius was in seeing the opportunity that irrational buying offered for smart selling. “You would be amazed to find how often we mislead ourselves, regardless of how smart we think we are, when we attempt to explain why we are behaving the way we do,” Dichter observed in 1960, in his book “The Strategy of Desire”. He held that marketplace decisions are driven by emotions and subconscious whims and fears, and often have little to do with the product itself. Trained as a psychoanalyst, Dichter saw human motivation as an “iceberg”, with two-thirds hidden from view, even to the decision-maker. “What people actually spend their money on in most instances are psychological differences, illusory brand images,” he explained. At a time when national companies were aggressively jockeying for position among Americans—a suddenly cash-happy and acquisitive bunch—Dichter promised a way to influence consumers' brains. If shopping was an emotional minefield, then strategic marketing could be a gold mine for companies.

Between the late 1930s and 1960s Dichter became famous for transforming the fates of businesses such as Procter & Gamble, Exxon, Chrysler, General Mills and DuPont. His insight changed the way hundreds of products were sold, from cars to cake mix. He pioneered research techniques such as the focus group, understood the power of word-of-mouth persuasion and earned startling fees for his theories. By the late 1950s his global business reached an annual turnover of $1m ($8m today), and he enjoyed a reputation as the Freud of the supermarket age.

Dichter's radical approach to goading shoppers, called “motivational research”, was considered so successful that he was even accused of threatening America's national well-being. Americans have become “the most manipulated people outside the Iron Curtain,” complained Vance Packard, a sociologist and virulent critic, in his 1957 book “The Hidden Persuaders”. Even so, Dichter's fame waned long before he died in 1991. He spent his later years as a discarded guru in Peekskill, New York, scribbling the occasional book about management or motivation. Media research moved on; his name has largely been forgotten. Yet many of his ideas about the role of the unconscious in sales are now back in fashion.

Well before Dichter and Freudian analysis, American businesses had spent decades trying to decipher market patterns. Such efforts date from the late 19th century, when new polling techniques were used to suss out the needs of consumers. This trend picked up after the first world war, as national brands began dominating the marketplace and companies invested more time and money in edging out competitors. In the late 1920s and 1930s advertising executives were encouraged to mingle with the masses to discern preferences and interview shoppers.

But these surveys were ultimately slapdash and speculative. Businesses were recognising the limits of quantitative studies (dismissively described as “nose counting”), which offered little genuine insight into how customers behaved. Asking shoppers why they bought particular products was like “asking people why they thought they were neurotic,” quipped Dichter.

In fact, he believed, most people have no idea why they buy things. They might answer questions in an effort to be helpful (particularly in the early 20th century, when consumers were chuffed to be asked to share their thoughts). But these were attempts to make sense of decisions retrospectively. To understand what truly motivated people, Dichter said, it was necessary to get them to talk at length about their everyday habits. Instead of subjecting many people to quick questionnaires, he preferred a deep, psychoanalytical approach with fewer participants: “If you let somebody talk long enough, you can read between the lines to find out what he really means.”

Dichter had arrived in America in 1938, the perfect time for these ideas to take off. The intellectual environment was just opening up to new and unorthodox concepts; Freudian psychology was becoming sexy. The idea that there was more to human behaviour than meets the eye was becoming widespread. Freud's notions of subconscious urges and socialised inhibitions seemed to make intuitive sense. Even Albert Einstein, America's patron saint of the hard sciences, sought out Freud's counsel in the run-up to the second world war, praising him for his “critical judgment, earnestness and responsibility”.

Psychoanalysis became particularly popular in post-war America, a time of abundance, conformity and anxiety about being “abnormal”, observes Lawrence Samuel, author of “Freud on Madison Avenue”. Nothing makes people more neurotic than the expectation that they should be enjoying themselves.

For advertisers, argues Mr Samuel, Freud was a “godsend”. When goods were scarce and people bought what they could get, it was hardly necessary to understand consumer psychology. But in an age of prosperity, when supply outstripped demand and countless indistinguishable goods were competing for buyers, companies had to rely more heavily on branding and advertising. There was a clear need to improve upon existing campaigns, which often simply announced the benefits of a product with grand promises and sparkling smiles (for example a print ad from 1925 for Johnson's Liquid Wax: “This New Easy Way to have Beautiful Waxed Floors”).

American businesses suddenly needed to understand and channel the desires of consumers. And Dichter—well, he “made their heads explode”, says Mr Samuel.

A 31-year-old Jewish psychologist fleeing Hitler's rise in Europe, Dichter came to New York with his wife Hedy and $100 in his pocket. The city was suddenly full of fellow Austrian Jewish PhDs; with the help of Paul Lazarsfeld, his former statistics teacher, Dichter quickly got a job at a market-research firm in Manhattan. Dichter is credited with popularising motivational research, yet it was Lazarsfeld who first discovered the value of qualitative research based on lengthy interviews. He understood the psychological roots of consumer decisions, but he remained wedded to finding statistical proof for his theories, and went on to found Columbia University's Bureau of Applied Social Research.

Dichter saw this as “naive empiricism” and decided to venture off on his own. A gifted self-promoter, he saw the city as a welcoming place for a young man with big ideas about subliminal desires, sex and sales.

In 1939 he wrote to six big American companies, introducing himself as “a young psychologist from Vienna” with “some interesting new ideas which can help you be more successful, effective, sell more and communicate better.” Mindful of Freud's nascent appeal on Madison Avenue, he emphasised that in Vienna they had lived on the same street. (Dichter had studied psychoanalysis in Austria, but with Alfred Adler and Wilhelm Stekel, not Freud himself.) With help from Lazarsfeld, he heard back from four companies. Having secured these contracts, Dichter came up with the concepts that would make him famous.

The consumer on the couch

What makes soap interesting? Why choose one brand over another? Dichter's first contract was with the Compton Advertising Agency, to help them sell Ivory soap. Market research typically involved asking shoppers questions like “Why do you use this brand of soap?” Or, more provocatively, “Why don't you use this brand of soap?” Regarding such lines of inquiry as useless, Dichter instead conducted a hundred so-called “depth interviews”, or open-ended conversations, about his subjects' most recent scrubbing experiences.

The approach was not unlike therapy, with Dichter mining the responses for encoded, unconscious motives and desires. In the case of soap, he found that bathing was a ritual that afforded rare moments of personal indulgence, particularly before a romantic date (“You never can tell,” explained one woman). He discerned an erotic element to bathing, observing that “one of the few occasions when the puritanical American [is] allowed to caress himself or herself [is] while applying soap.” As for why customers picked a particular brand, Dichter concluded that it wasn't exactly the smell or price or look or feel of the soap, but all that and something else besides—that is, the gestalt or “personality” of the soap.

This was a big idea. Dichter understood that every product has an image, even a “soul”, and is bought not merely for the purpose it serves but for the values it seems to embody. Our possessions are extensions of our own personalities, which serve as a “kind of mirror which reflects our own image”. Dichter's message to advertisers was: figure out the personality of a product, and you will understand how to market it. Soaps could be old or young, flirty or conservative. Ivory, Dichter inferred, had a “sombre, utilitarian, thoroughly cleansing character”. It was the mother-daughter of soaps, whereas a brand like Camay was a seductress. Such insights led to the slogans “Be smart and get a fresh start with Ivory Soap” and “Wash your troubles away”.

But the innovation that put Dichter on the marketing map involved a problem Chrysler was having with its relatively new line of Plymouth cars—which consumers were shunning. Dichter headed out to Detroit, conducted extensive interviews with a couple of hundred people, and deduced that the company's first problem was its existing advertising, which boasted that its cars were “different from any other one you have ever tried”. This evidently triggered an unconscious fear of the unknown among buyers, for whom familiarity in a car meant safety.

He also learned from interviews that whereas convertibles made up only 2% of sales in 1939, most men, particularly middle-aged ones, dreamed of owning one. When convertibles were placed in the windows of dealerships as “bait”, more men came in. But when they returned actually to make a purchase, they typically came with their wives and chose a sensible sedan (the Plymouth line offered both).

Dichter gathered that the convertible symbolised youth, freedom and the secret wish for a mistress: an idle bit of temptation. He suggested that Chrysler beef up its convertible advertising—and, in recognition of spouses' role in the final decisions, begin marketing cars in women's magazines. Meanwhile, a new and more reassuring campaign emphasised that it would take “only a few minutes” to feel at home with the new Plymouth.

Reporting on the campaign's success, Time described Dichter as “the first to apply to advertising the really scientific psychology” that tapped “hidden desires and urges”. He chalked up his own success to the importance of examining “not how people should behave but how they do behave”.

Don't go overboard

Business was good for Dichter. In 1946 he set up his Institute for Motivational Research in Croton-on-Hudson, outside New York City; it was soon followed by a dozen satellite offices in America and abroad. His unique approach to marketing—a mix of observation and imagination—was particularly popular in the 1950s, when industry leaders would queue up “like desperate patients for a miracle medical doctor,” writes Ronald Fullerton, a marketing specialist and academic. A charismatic salesman, Dichter flipped Freud's pessimistic ideas about childhood traumas and buried urges into opportunities for seeking pleasure and fulfilment in everyday purchases. At a time when on-screen married couples still slept in separate beds, his colourful take on the sexual relevance of ordinary objects got attention.

Dichter's case history is a fascinating mix of outlandish ideas and now-conventional solutions. To elevate typewriter sales, he suggested the machines be modelled on the female body, “making the keyboard more receptive, more concave”. People smoke, he explained, because it is both a sign of virility and a legitimate excuse to interrupt the day for a moment of pleasure, “comparable to sucking at the nipples of a gigantic world breast”. A phallic shape to lipstick increased sales by the way it offered a subconscious invitation to fellatio (“but one has to be careful not to go overboard and make the parallels too obvious,” Dichter cautioned). Prunes had an image problem as they were seen as a “symbol of old age”, like “dried-out spinsters”. To inspire demand, Dichter suggested they be branded the “California Wonderfruit” and feature fresh, supple plums on the packaging. Prunes have been marketed using images of their younger selves ever since.

Baking is an expression of femininity, he explained, so when a woman pulls a cake or loaf out of the oven, “in a sense it is like giving birth”. Thus mixes that require the cook to add only water are threatening to women, since their role is marginalised; these days, thanks to Dichter, nearly all such mixes involve adding eggs, a symbol of fertility. In her book “Food is Love”, Katherine Parkin notes the way he encouraged advertisers to impress on women the importance of cooking as a way of showing love.

Dichter recognised that in the new world of America, brands had become “a substitute for nobility and a family tree”. People seek out products that correspond with the group they want to associate with. Every object has a special meaning—one that often relates to sex, insecurity or a desire for prestige. He also understood that consumers felt guilt after buying self-indulgent products, so marketers had to sell such things as tobacco and candy as “rewards” for the deserving. Much of this is taken for granted now, but when Dichter was whispering these sweet nothings in the ears of CEOs, they were revelatory.

For Dichter, the ability to express oneself through shopping was a matter of great importance. After a childhood that had been defined by serious poverty, near-starvation and prejudice, he saw America as a beacon of democratic capitalism, with limitless opportunities to grow and express individuality. (Soon after immigrating he took lessons to drop his Viennese accent so he would sound “all-American”.) But he also viewed Americans as a puritanical bunch, incapable of embracing change without anxiety, or spending money without remorse. He worried that this inability to enjoy progress would cripple the country's economy, which served as a crucial bulwark against the Soviet Union. If consumer culture was America's best defence against communism, then motivating materialistic desires was an investment in the country's future.

Dichter's colourful take on the sexual relevance of ordinary objects got attention

To ensure national security, Americans needed to learn how to please themselves in realistic but ephemeral ways. “To some extent the needs and wants of people have to be continuously stirred up,” he argued, so that everyone will work hard to buy what they desire. In the early 1950s he discerned that, when Americans borrowed money, they preferred to do so from loan sharks at high interest rather than from a bank, because they saw bankers as judgmental father figures, whereas loan sharks lacked the authority to moralise. He advised one bank to advertise checking accounts with overdraft facilities, recognising that people wanted more money than they had but didn't want to take out loans. As for credit cards, Dichter presciently called them “magic” for the way they provided “the American consumer with a symbol of inexhaustible potency.” One can only imagine what he would make of America's latter-day spendthrift habits.

Yet the abundant 1950s also gave rise to criticism about the emptiness of affluence and the perversity of advertising. In “The Hidden Persuaders”, Packard targeted Dichter for exploiting the emotions of consumers to inspire a national glut of self-indulgence. He claimed that motivational researchers such as Dichter, with their scientific cunning and Freudian voodoo, had unleashed the “chilling world of George Orwell and his Big Brother”. Ironically, Packard's dramatic assessment of Dichter's dark powers ended up bringing him plenty of new business.

A year later John Kenneth Galbraith released “The Affluent Society”, in which he described a country increasingly riven by economic divisions but too selfish to care. Then in “The Feminine Mystique”, in 1963, Betty Friedan took Dichter to task for being “paid approximately a million dollars a year for his professional services in manipulating the emotions of American women to serve the needs of business.” Though women were taking important steps beyond the home, Friedan accused advertisers of exploiting their homemaking insecurities in order to keep them in the kitchen with such gal pals as Betty Crocker and Aunt Jemima.

Back to Vienna By the early 1960s Freudian analysis was falling from favour. The talking cure suddenly seemed too soft, too unscientific and sometimes too weird. This poisoned the market for Dichter's insights, which were often profound but occasionally bizarre and impossible either to validate or disprove. In his book “Marketing Myths That Are Killing Business”, Kevin Clancy describes hiring Dichter to weigh in on a Pepsi campaign. “Kevin, stop!” Dichter blurted. “You are showing Pepsi in all these commercials with ice…You must not do this…You are associating your client with death!” His verdict labelled “too nutty”, Dichter was sent packing. The rise of the computer in the 1960s introduced new quantitative methods of researching consumer attitudes, which promised greater scientific purity. Instead of presuming that secret motivations for buying detergent lurked in the traumatised subconscious, it seemed safer and wiser to study behaviour and create mathematical models based on income and geography. Meanwhile the development of the cognitive sciences from the late 1950s through to the 1970s offered new empirical methods for considering such things as memory and problem-solving. Psychologists, anthropologists and behavioural scientists studied human behaviour to help understand mental development. By the 1980s scientists exploring the brain were examining how people handle information and make decisions.

But more recent developments in neuroscience have inspired fresh questions about instincts and desires, unconscious prophesies and gut decisions. New information about human cognition has led the hard sciences back to the same sort of concerns that preoccupied psychoanalysts in Vienna a century ago.

Thus what was once the domain of Freud and Dichter has been appropriated by researchers in lab coats. Yet many of the theories sound remarkably similar—albeit with rather less emphasis on Oedipal urges and castration anxieties. “Recent published findings in neuroscience indicate it is emotion, and not reason, that drives our purchasing decisions,” reported Mobile Marketer magazine earlier this year. The quantitative trends that tossed Dichter aside have ultimately led back to his ideas.

Companies selling insight into the consumer unconscious are now quick to couch their approach in empirical terms. “Over 85% of consumer buying behaviour is driven by the non-conscious,” explains Buyology Inc, which launched in 2009 in New York as “the world's leading strategic neuro-insight company”. Using “statistically rigorous, large-sample web-based tools”, the company promises to unearth the real drivers of buying decisions. Then there is the Zaltman Metaphor Elicitation Technique (ZMET), a patented interview process used by companies around the world that guarantees to go “beyond what people say to reveal what they actually mean.” Rooted in neuroscience, ZMET's one-on-one exchanges use images and metaphors to get at the deeper thoughts and feelings of a consumer. “A lot goes on in our minds that we're not aware of,” explained Gerald Zaltman, a business professor at Harvard, after he came up with the idea.