Even admen have souls, and some of them are enduring dark nights. Jeff Goodby is co-chairman of Goodby, Silverstein & Partners, a San Francisco advertising agency responsible for some of the most famous campaigns of the 1990s, including ones for Nike and Budweiser. On his return from this year’s annual ad industry awards festival at Cannes in June, Goodby wrote a rueful piece for The Wall Street Journal. In the past, he said, the only true measure of success was whether the public knew and cared about your work. “You could get into a cab and find out, in a mile or two, whether you mattered in life, just by asking the driver.” Now, “No one knows what we do any more.”

Cannes, said Goodby, used to be a showcase for some of the most famous ideas in the world. This year, it felt more like a convention of industrial roofing specialists, discussing the latest in fibreglass insulation technology. Sure, there was ingenuity on display: content delivery systems, co-ordinated interactive installations. But when he got home and tried to tell friends from outside the industry what he had seen, they looked bored, even pitying.

An industry that used to compete with Hollywood is starting to wonder if it has become a colonial outpost of Silicon Valley. The prime spots on the Cannes beachfront this year belonged to Facebook and Google. A recent report from Accenture said that “marketing is so inextricably linked to technology that, by 2017, chief marketing officers are projected to spend more on information technology and analytics than chief information officers.”

Ten years ago, ad agencies thought they had hit on a new formula for success: move on from the blunt instruments of conventional advertising and embrace the laser-sharp selling tools of digital media. The industry’s about-turn was made partly out of an instinct for self-preservation, but also from a yearning to be aligned with the zeitgeist, in a world where glamour has migrated from Madison Avenue to Palo Alto. To some, however, it feels as if they have given up the ghost.

Goodby’s cri de coeur echoed the anxieties of others. In April this year, Giles Hedger, chief of strategy for Leo Burnett Worldwide, wrote a polemic for the industry’s UK trade journal, Campaign, bemoaning the domination of advertising by technology: “The belief that [the] marketing contract can be stripped of all its joyful subjectivity until all that remains between consumer and brand is transaction . . . is the fallacy of our time.” Hedger described his views as “heresy” but said he believed “the voices of sanity are growing around us”.

There is a schism in the church of brands. Ad agencies have fervently embraced the digital gospel. But by doing so, say the doubters, they lost sight of what made the industry valuable in the first place.

Coca-Cola’s 1971 “I’d like to teach the world to sing” ad marked a switch towards advertising that aimed to “lift the heart”

The series finale of Mad Men ended, fittingly, with an ad. A multiracial group of hippyish young people stand on a hilltop and sing a hymn-like song, while holding bottles of Coca-Cola. “I’d like to teach the world to sing”, which first aired in 1971, was the perfect ending to Mad Men for several reasons, but one of them is that it ushered in a new era of advertising.

Until the 1970s, advertising was regarded as a branch of sales. It was thought that ads existed to persuade the consumers you couldn’t meet in person of your brand’s unique attributes: Persil washes whiter. Then, Coke and others started to find success with ads that didn’t try to persuade anyone of anything. “I’d like to teach the world to sing” said nothing about how the drink tasted or whether it was more refreshing than its rivals. It simply lifted the heart a little and got everyone singing its song.

A mid-1980s Levi’s ad: the blend of sex, music and Americana made each new one a national event

From the 1970s onwards, the ad industry entered a creative golden age, making ads which millions enjoyed, quoted and sang. When I started in advertising in the 1990s, the industry was still confident. It had passed the point of peak hubris (which we can date to 1987, when the Saatchi brothers, bored one afternoon, tried to buy a bank) but it knew what it was for. We made famous stuff, and we made stuff famous. Every ad in the Levi’s campaign, for instance, with its combustible blend of sex, music and Americana, was a national event.

Early in the 21st century, the digital era arrived and the fame factory was repurposed. Big TV campaigns suddenly came to be seen as inherently inefficient and primitive. Finance directors, who had always regarded the money spent on advertising as suspicious — it seemed to work, but they couldn’t quite see how — seized the opportunity to hold ad agencies more accountable. Online, it was possible to measure exactly how many “impressions” an ad received, and (in some cases) which ones led to a sale.

Best of all, the internet appeared to solve what was known as the wastage problem. Put an ad for your coffee brand in a Coronation Street break and you’ll immediately reach a large section of your target audience. But not everyone who watches Coronation Street is thinking of buying coffee, or even drinks it — yet you’re paying for them anyway, since the bigger the audience, the higher the cost of the airtime. Online, you could just target the people who had bought your brand before, or were in the market for it; if you sold tents, Google’s data allowed you to target 18-year-old festival-goers. Marketers could even tailor their messages for different consumers, so that each selling claim was never less than relevant. No more wastage.

3 in 10,000 The clicks through that ads get from ‘engaged’ Twitter fans

As if that weren’t enough, social media offered new ways to talk to consumers. The buzzword was “engagement”, a concept that Google and Facebook began assiduously promoting. Advertising is conventionally a one-way communication, unless you count shouting at the TV. Social media allowed marketers to co-opt consumers by inviting them to share ads, enter competitions, and tweet or blog about the brand. Every self-respecting brand outfitted itself with a Facebook page, YouTube channel and Twitter hashtag. Accountability, efficient targeting and free media: it was almost too good to be true. Marketing directors, cheered on by CFOs, management consultants and analysts, boasted about the increasing percentage of their budget devoted to “digital”. Ad agencies scrambled to reinvent themselves, hiring data scientists and social media specialists.

Everyone agreed that TV ads were not just wasteful, but crude and intrusive, and that viewers would use new technologies to avoid them (an undercurrent of self-loathing is never far from the industry’s surface). The entire edifice of mass marketing seemed to be crumbling. Why would you want to send the same message out to millions of people at a time, via a technology that first became popular when Dwight Eisenhower was in the White House? Why would you want to teach the world to sing, when you could offer your most cherished customers a personal encounter with your brand?

5% Pepsi Cola and Diet Pepsi’s drop in market share the year Pepsi moved to digital ads

In 2010, Pepsi embarked on an audacious new marketing strategy. Foregoing its slot in the Super Bowl, America’s annual showcase for lavish TV ads, it diverted its TV budget into a social media campaign: the “Pepsi Refresh Project”. Consumers were invited to propose ideas that would have a positive impact on society, and Pepsi promised to fund the ones that gained the most votes, via Twitter, Facebook and YouTube. “We wanted to explore how a brand could be integrated into the digital space,” Frank Cooper, then Pepsi’s “chief consumer engagement officer” explained.

The Refresh Project accomplished everything a social media campaign is supposed to accomplish: millions of Facebook likes and thousands of new Twitter followers. But it didn’t sell Pepsi. Pepsi Cola and Diet Pepsi both lost about 5 per cent of their market shares over the course of the year — a calamitous decline. The brand returned to TV. Bob Hoffman, a veteran American adman who blogs as the Ad Contrarian, has gained a large following for his savage critiques of digital hype. After Pepsi Refresh, he concluded, “only zealots and fools will continue to bow down to the gods of social media.”

The Pepsi debacle emboldened the ad industry’s other contrarians to point out that, despite the dazzling promises of digital, nearly all brands in mature categories still rely on conventional media. At the risk of being labelled Luddite, they suggested that although the internet has changed how the game is played, it has not changed its fundamental rules: mass marketing works; fame works; emotion works — and “legacy media”, especially TV, still do all of this better than the new.

Like many insurgencies, this one has been fired by a book. How Brands Grow (OUP, 2010), is by Professor Byron Sharp, of the Ehrenberg-Bass Institute at the University of South Australia. Most marketing books are long on airy assertion and short on rigour. How Brands Grow is the opposite. It is empirical, closely argued and, in its sober way, incendiary. Sharp marshals a vast array of evidence from many different categories — soft drinks, motorbikes, concrete mixers — and identifies universal laws of brand purchasing. To date, nobody has seriously challenged his findings, though plenty have ignored them.

Sharp’s first law is that brands can’t get bigger on the back of loyal customers. Applying a statistical analysis to sales data, he demonstrates that the majority of any successful brand’s sales comes from “light buyers”: people who buy it relatively infrequently. Coca-Cola’s business is not built on a hardcore of Coke lovers who drink it daily, but on the millions of people who buy it once or twice a year. You, for instance, may not think of yourself as a Coke buyer, but if you’ve bought it once in the last 12 months, you’re actually a typical Coke consumer. This pattern recurs across brands, categories, countries and time. Whether it’s toothpaste or computers, French cars or Australian banks, brands depend on large numbers of people — that’s to say, the masses — who buy them only occasionally, leave long gaps between purchases and buy competing brands in between.

The industry yearned to be aligned with the zeitgeist, in a world where glamour has migrated to Palo Alto

If you work for a brand owner, the implications are profound. First, you will never increase your brand’s market share by targeting existing users — the task that digital media performs so efficiently. The effort and expense marketers put into targeting their own customers with emails and web banners is largely wasted; loyalty programmes, says Sharp, “do practically nothing to drive growth”. What seems like a prudent use of funds — focusing on people who have already proved they like the brand — is actually just spinning wheels.

Second, and paradoxically, a successful brand needs to find a way of reaching people who are not in its target market, in the sense of people who are predisposed to buy it. The brand’s advertising must somehow gain the attention of people who are not interested in it, have never bought it, or who bought it so long ago they can’t remember — so that when they are ready to buy, it automatically springs to mind. In the wastage is the value.

87% The amount of television UK viewers still watch ‘live’

Advertising, says Sharp, works best when it doesn’t try and persuade, but merely makes us remember the brand at the point of purchase. He call this “mental availability”. Human beings have powerful spam filters: we screen out nearly all of the 30,000 or so brands in a supermarket. Advertising opens people’s eyes; you are simply more likely to notice a product in a store, or on a webpage, if you already know it. Gordon Brown, founder of market research company Millward Brown, suggested that advertising’s role is to make a brand “interesting” as it sits on the shelf.

The most effective ads don’t sell, but they do make people buy. By keeping the brand alive in your mind, Coke ads change the probability of you buying it in the next year by a minuscule proportion, a nudge so small that you almost certainly won’t notice it, which is why people often say that advertising doesn’t affect them. But that tiny effect adds up to millions of cans.

The John Lewis Christmas ads, which put emotion centre stage, have proved a particularly effective use of TV

Brands are not the rich sources of differentiation marketers like to think of them as, but short cuts through the complexity of decision-making. Most consumers aren’t aware of, or interested in, the difference between Nescafé and Kenco and don’t want to spend longer than they need to thinking about which they prefer. They just want to get coffee and get home. Marketers are usually surprised to hear this and find it hard to accept — they like to imagine that people who buy their brand are deeply attached to it. But the data show that even people who regularly favour one brand over others will pick a competitor if it happens to be more easily available or cheaper that day. In the words of Sharp’s mentor, Professor Andrew Ehrenberg: “Your customers are customers of other brands who occasionally buy you.”

Brand advertising, at best, does something very different to a search result, an email or a Facebook update

All of this makes “engagement” largely pointless. Light buyers aren’t fans of your brand. They don’t think of it as special or even unique. They aren’t much interested in whether your vodka is from Russia or Sweden, or how many times it has been distilled. No surprise, then, that they almost certainly don’t follow your brand on Twitter or visit its Facebook page, or that they can think of a thousand things they’d rather do than share a “digital experience”, let alone sign up to a “project”.

Even the people who do join brand pages on Facebook hardly ever click on them. The US company Forrester Research has found that the rate of engagement among a brand’s Facebook fans is seven in 10,000; for Twitter it is three in 10,000. People might watch ads on Facebook or YouTube, but that’s about all the interaction they want (Facebook itself recently conceded this point). A senior marketer at the drinks company Diageo, where Sharp’s book has been influential, put it to me bluntly. “After 10 or 15 years of f***ing around with digital we’ve realised that people don’t want to ‘engage’ with brands, because they don’t care about them.’

What if you were to invent a way of getting light buyers to recall your brand just as they are about to choose? Ideally, it would reach millions of people who aren’t particularly thinking about your product. You’d want them to see the same thing at around the same time, so that they can talk to each other about what they’ve seen, reinforcing each other’s memories of it. You would need to sneak up on them, since they have near-zero interest in hearing from you, indeed don’t want to. You’d need a form of content requiring negligible mental effort to process: one which comes in bite-sized chunks, but which is still capable of moving and delighting. It turns out there is an app for that: the TV ad.

TV is in healthier condition than anyone predicted 10 years ago. The average viewer watches nearly as much TV, on TV sets, as he or she always did, and now they watch programmes on mobiles and tablets too. We aren’t skipping ads any more than we used to: 87 per cent of viewing in the UK is “live”. A recent US study found that ad-skipping is declining; people are too distracted by their phones to bother. The passive nature of TV turns out to be its hidden weapon: it facilitates a détente between viewer and advertiser. The best ads make us pay attention and look up from our phones.

In the past few years, the success of the retailer John Lewis has gone some way to restoring the status of TV advertising. Few marketers have failed to notice how effective its Christmas ads have been. In one, a boy’s impatience for the big day turns out to be caused by his desire to give his parents their present. John Lewis largely forgoes persuasion for emotion, and for good reason. In the past 30 years, scientists have shown that we are much more likely to retain something mentally when we have a strong emotional response to it. Ads imprint themselves on the cortex when they touch the heart.

Les Binet, from John Lewis’s agency Adam & Eve DDB, is one of the industry’s most respected experts on advertising effectiveness. In 2013, on behalf of the IPA, Binet, along with Peter Field, conducted an analysis of the most successful UK campaigns of the past 30 years. They found that “the most effective advertisements of all are those with little or no rational content”, and that TV is the emotional medium par excellence. An online banner ad, however smartly targeted, is unlikely to make anyone grin, gasp or weep.

John Lewis ads are instantly recognisable, as are those from, for instance, the long-running campaigns for Walkers crisps or Jack Daniel’s. Since the amount of cognitive energy that people expend on brands is so tiny, they generally stick with what they know and reach for familiar things: a name, a pack, a logo. The challenge for any brand, then, is to be both interesting and consistent. Lucy Jameson, the CEO of Grey London, told me that her agency creates “long ideas”: tightly defined themes that are endlessly varied and refreshed.

Marketers consistently undervalue consistency. Diageo recently carried out an audit of all the endlines that it had attached to one of its biggest brands, Guinness, and were embarrassed to discover it had used more than 20 different slogans in 15 years. What’s more, when it asked people to recall an endline, the only one they remembered was “Good things come to those who wait”, which hadn’t run since 1999. Vast sums of money had been spent on campaigns which probably had short-term effects but barely left a trace in consumer memories.

The fame of a brand takes years of investment to build but, properly maintained, it becomes a gusher of cash, cascading down the generations — Warren Buffett, one of Coca-Cola’s biggest shareholders, has earned billions from this principle. But in the short term, the effects of advertising on consumer memories are very hard to measure. No wonder marketing directors get addicted to what Martin Weigel, of Nike’s agency Wieden + Kennedy, calls the “crack” of instant data: likes, shares, impressions.

Laurence Green, of London agency 101, distinguishes between two classes of advertising. There are ads for when you know what you’re looking for — if I want to find a plumber, I used to look in the local paper. Google is unbeatable at this, which is why it has wreaked havoc on the newspaper business. Then there is advertising which burnishes the aura of a brand in the minds of people who don’t yet want to buy it, by imbuing it with meaning. A soft drink can come to represent optimism, a chain of stores the warmth of middle-class homes. Google and Facebook, says Green, still haven’t shown that they can do this.

Don Draper’s successors instinctively understood the commercial value of fame, emotion and consistency. But they never quite defined it. We now know that brand advertising, at its best, does something very different to a salesperson, a search result, an email or a Facebook update. It injects a brand into the cultural bloodstream and, by doing so, books a spot in the most important media of all: people’s brains.

The question is whether this revelation has come too late to save the ad industry. The admen have allowed their lunch to be eaten by a host of specialists in digital, social media and data management. According to a report on the industry from the accounting firm Kingston Smith W1, ad agency profits averaged 10.7 per cent in 2014, their lowest level for seven years.

The contrarians are gaining ground, however. The vaunted efficiency of digital marketing has been undermined by the prevalence of fraudulent data. It is estimated that 50 per cent of online ad impressions are from “non-human traffic” — that’s to say, digital robots. The introduction of adblocking software to Apple’s latest operating system is talked of as a potentially fatal threat to online ads. Meanwhile, TV adspend is at a record high, boosted by the campaigns of digital businesses like Amazon, Facebook and Google.

If TV is in good shape, that is partly because it has given itself an upgrade: HD, bigger screens and better content — such as Mad Men. In fact the only part of the TV package that hasn’t improved is the ads. According to Kantar Media, in 1986, 28 per cent of people in Britain said they enjoyed the ads more than the programmes. Today, only 12 per cent say so.

Perhaps there is a moral here for any industry threatened by technological disruption. In the rush to reinvention, don’t forget your USP.

Ian Leslie is a writer who also works as a freelance strategist in the advertising industry. Twitter: @mrianleslie

Illustrations by Francesco Ciccolella

Letter in response to this report

Making ads stay long in our consciousness / From Trish Evans