For David Chase, HBO was the last port of call. He had spent the previous year shopping his idea for a new drama series to the big broadcasters. One by one, they passed. Fox wanted simpler storylines. CBS asked why the main character had to be in therapy. ABC said there was too much swearing. All of them found it hard to imagine audiences warming to the show’s protagonist, a surly gangster called Tony Soprano.

HBO, a cable network, had less money, fewer viewers and lower prestige than the broadcasters but was willing to take a risk on The Sopranos. Its executives were unsure about giving the show’s creator the control he sought over the project, however — until HBO’s key decision-maker, Chris Albrecht, agreed to hear Chase out in person. “He talked about his vision,” Albrecht told the writer Brett Martin. “By the time he was finished, I thought, ‘Wow.’”

HBO’s bet proved one of the most lucrative in television history. The Sopranos, which ran for six series and won 21 Emmys, drove up subscriptions and earned the network millions in DVD sales. Within three years of the show taking to air in 1999, the upstart cable channel was more profitable than any of the broadcasters.

The Sopranos didn’t just transform its parent channel, it transformed TV. Today, HBO competes with FX, Showtime, Netflix, Amazon and others in a creative race to the top. “Our job is not to get people to watch the 8.30pm lead into the 9pm news,” Joe Lewis, head of original programming at Amazon Studios, told me. “Our job is to make the greatest TV shows on the planet.”

HBO’s The Sopranos (1999-2007)

Chase originally conceived of The Sopranos as a movie. For more than 20 years, he had worked as a writer and producer of TV drama but had come to hate its suffocating formal constraints. In middle age, he yearned to satisfy his artistic soul. At the time, that meant making feature films, in the tradition of his idols: auteurs such as Francis Ford Coppola, Stanley Kubrick and Roman Polanski.

Today, a writer who cherishes a personal vision would have no qualms about taking it to the small screen. TV is the place to go if you want to make art and money at the same time. Paul Weitz, a screenwriter and director whose credits include American Pie and About a Boy, is currently at work on the fourth season of Mozart in the Jungle for Amazon Studios. “Your agent used to say, ‘Don’t do TV, it will hurt your film career,’” he recalls. “Now it’s the other way around.”

Every creative industry is engaged in a civil war. The creatives — whether writers, painters or musicians — want to play with new forms of expression; the capitalists prefer to go with what worked last time. But sometimes the two sides come together, on equal terms, in gloriously fertile equilibrium. We call these periods golden ages.

***

Fabien Accominotti, a sociologist at the London School of Economics, has studied the Paris art market between 1870 and 1930, when a succession of radically innovative movements emerged — Impressionism, Cubism, Fauvism — along with a new class of superstar artists, such as Monet and Picasso. Reading art history books, it is easy to assume that this flourishing was a happy accident coinciding with the arrival of some exceptionally gifted artists. Accominotti argues that to understand the art, you need to follow the money.

The Parisian golden age emerged out of the collapse of a system that penalised artistic innovation. For most of the 19th century, the Académie des Beaux-Arts, a state-sanctioned institution, dominated the production and consumption of French art. A jury of academicians decided which paintings were exhibited at the Salon, the main forum for collectors to view new work. The academy set strict rules on artistic expression, and preferred idealised scenes from classical mythology to anything resembling contemporary life. For the most part, the art that resulted was staid and predictable, painted by skilled but anonymous technicians.

Since the academy decided which paintings made it into the Salon and how prominently they were displayed, any artist trying to make a living from his work had little choice but to follow its rule book. But the rapidly growing population of young artists in Paris chafed against such strictures. After a loose-knit group of renegades, later to become known as the Impressionists, began staging their own exhibitions, the academy’s monopoly was supplanted by a decentralised network of galleries, dealers, critics and artists. Paris’s golden age began.

Commercial galleries sprang up around town, increasing more than twentyfold over the period studied by Accominotti. The galleries, each with their own distinctive character, served as shop windows for artists to display their work to a new audience of middle-class collectors.

Independent critics became highly influential, since they helped customers discern who was doing the most exciting work. A new breed of dealers, such as Paul Durand-Ruel and Ambroise Vollard, temperamentally sympathetic to artists, acted as middlemen.

The artists themselves, liberated from the constraints of the system, were free to pursue radical new ideas. Individual artists were celebrated — and remunerated — as they never were under the reign of the academy. Paris became a global magnet for artistic talent. Creative originality became a route to commercial success instead of a barrier to it.

***

The current golden age of TV also emerged from a crumbling system. Since its beginning, the US TV industry had been dominated by NBC, ABC and CBS (the “big three” were joined by a fourth, Fox, in 1986). The broadcasters had a template for storytelling that they expected writers to adhere to.

The action had to be shaped around an invisible grid of ad breaks and schedules. A drama series needed to be a string of self-contained narratives that could be watched in any order (hence the prevalence of shows set in police stations and hospitals — a crime can be solved, a patient saved, within the hour). That made it easier to sell the series on, since whoever bought it could air it out of sequence, and viewers could dip in and out as they liked.

© Marcus Marritt

Since many of a show’s viewers arrived by accident, and there was every chance they would leave at any moment, it was crucial they should never be made to work hard. The more familiar the storylines and character types, the better — hence the prevalence of mavericks who get results. Some creatives were able to smuggle in depth and novelty but, for the most part, they were required to deliver the least objectionable work rather than the best. No wonder a writer with creative ambition dreamt of making movies.

By the 1990s, the broadcasters’ hegemony was threatened by a convergence of new technologies. Cable and satellite channels were multiplying, and DVD viewing was increasingly popular. The first TiVo boxes — digital video recorders — shipped in 1999. A sea change in TV viewing was under way. As Laura Kennedy, an executive vice-president at Lionsgate, put it: “Streaming started with DVDs. That’s when people started to get used to viewing content in multiple bites, and on their own terms, rather than on the schedule of a broadcaster.”

AMC’s Mad Men (2007-15)

Viewers who chose what and when they wanted to watch were more likely to demand something extraordinary, and to pay it close attention. This mode of viewing was terrible news for the broadcasters, who relied on large numbers of people watching whatever was on that night — including ads. “Television’s economics were eroding from underneath it,” says John Landgraf, then at NBC, now chief executive of the Fox-owned cable network FX. “People wouldn’t watch repeats, which was how the broadcasters made money.”

If a large and inattentive audience was more valuable to broadcasters than a small and loyal one, then for HBO it was the other way around. Funded by subscription, it didn’t have to worry about ratings but it did have to make viewers feel it had something worth paying for. The rise of what the cultural critic Anna Leszkiewicz calls “the avid viewer” therefore represented an opportunity. HBO already had a niche in the market; now there was a market in the niche.

455: Scripted original programmes aired on US TV in 2016. That number is expected to rise this year. 24: Hours per week the average British adult spends watching TV 93 million: Netflix subscribers across the world 23 million: Average multiplatform viewers per episode of series six of Game of Thrones

HBO sold itself directly to viewers. Basic cable channels, such as FX or AMC, did so indirectly, charging carriage fees to cable service providers such as Comcast, which meant they were dependent on viewers demanding their brand as part of a package. For the cable channels, just being there wasn’t enough; they needed to offer something special. Until The Sopranos, however, they relied largely on a mix of old movies, repeats and wrestling. “Americans did not sit down at the end of the day and turn on cable and expect to see something new,” says Amanda Lotz, professor of communication studies at the University of Michigan.

It took David Chase’s creative ambition to explode the latent commercial possibilities of pay-TV. “The Sopranos was a form-breaker,” says Landgraf. “Every now and again in TV a new form is invented: the first game show, the first late-night talk show. The Sopranos opened up a whole new way of storytelling.” Chase melded TV with the kinds of movies he loved, like The Godfather, where the fundamental narrative involves the development of a central character. Sure, viewers might want to watch a standalone episode of The Sopranos but true satisfaction came from following the journeys of Tony Soprano and those around him.

Chase’s novelistic, multi-layered narrative perfectly answered the needs of TV’s new audience. Avid viewers were more receptive to storylines that unspooled across several episodes rather than playing out within one. They enjoyed grappling with complex, difficult characters, and genre-busting shifts in tone. Crucially, they were also willing to pay for this kind of storytelling. The Sopranos was not just a new form, it was a new business model. “It was like finding an oilfield nobody had drilled in before,” says Landgraf.

AMC’s Breaking Bad (2008-13)

In The Sopranos’ wake, FX took a gamble on The Shield, a series whose central character was a corrupt and violent cop. AMC made Mad Men, created by Matthew Weiner, an alumnus of The Sopranos’ writing room, and followed it with Breaking Bad. These shows did not reach big audiences on first airing but the acclaim and attention they received transformed the reputations of their parent brands, who were then able to raise their fees to cable providers.

Every now and again in TV a new form is invented: the first game show, the first late-night talk show. ‘The Sopranos’ opened up a whole new way of storytelling

The cable channels could not offer creative talent the prestige, or budget, of a broadcaster or movie studio but they could offer them freedom. Executives advised on, but did not interfere in, storylines or casting decisions; plotlines were not picked apart by focus groups. Chris Albrecht, who greenlighted The Sopranos at HBO (he now heads the cable network Starz), told me: “Our strategy was simple. Find creative people and give them a platform to realise an idea they’re passionate about.” The term “showrunner” became used to describe someone who not only creates the show but oversees its production.

Netflix’s House of Cards (2013-present)

Just as Parisian artists were able to take their work to dozens of galleries instead of one institution, writers and production houses now had many more buyers for even their craziest ideas. Since nobody really knew what worked or didn’t work, it made sense to experiment. Shows that did not win big audiences would get renewed if critics feted them or a small but vociferous audience of loyalists made itself heard online. A frontier spirit prevailed.

In 2013, just as things threatened to get predictable, Netflix, the online streaming service, decided to make its own shows. For all the hype around its data-driven methodology, it essentially adopted the HBO model: invest in creative talent, give it free rein, and seek out passionate niches. House of Cards, a political drama starring Kevin Spacey, transformed Netflix’s reputation from a DVD rental company into a TV brand. Netflix was soon joined by Amazon Prime, which competed for critical kudos by creating its own award-winning shows such as Transparent — a groundbreaking, tragicomic series about a transgender parent.

Amazon’s Transparent (2014-present)

“Streaming is the culmination of a change that has been coming for decades,” Joe Lewis, of Amazon Studios, told me. “If there are just three TV networks, and one TV in the house, then your goal is to get viewers to not turn you off. When there’s a huge number of great shows worthy of your time, and lots of TVs, or TV-enabled devices, in the house, the challenge becomes getting someone to turn your show on — which means turning that person on.”

Lewis is a businessman who speaks with unabashed fervour about creativity, like a latter-day Vollard. “I tell my executives to think like artists who fight for truth. We want to make things that last. Any true work of art accrues value and meaning over time. Transparent will still be valuable in 10 or 20 years, like The Godfather is now.”

***

When Chase’s movie heroes made their best work, Hollywood was in the midst of its last golden age. The Godfather, Taxi Driver and Chinatown emerged from the “New Hollywood” era of the 1970s, which saw a profusion of innovative, challenging films, many of which are now regarded as classics.

I tell my executives to think like artists who fight for truth. We want to make things that last

Today, more than ever, the movie industry is dominated by one, highly formalised genre: the multi-sequel blockbuster. In 1999, when The Sopranos was first broadcast, the list of the year’s most successful movies was topped by Star Wars: The Phantom Menace and Toy Story 2 but also included The Sixth Sense, Shakespeare in Love and Analyze This. Blockbusters mingled with adult dramas and comedies. In 2016, by contrast, every one of the top 10 grossing films was a sequel, spin-off, superhero movie or cartoon. Only two original movies have been number one in the past decade: Frozen and Avatar. As Amanda Lotz puts it: “Hollywood has a story problem.”

Blockbusters require skill to make, and can be fun to consume, but they operate within a set of tightly drawn narrative rules. If you have artistic ambitions, that can get very dull. In a 2010 interview with The New York Times, Mad Men creator Matthew Weiner bemoaned the film industry’s creative impoverishment: “It’s a bummer to see movie after movie where . . . so much money is spent, and they’re just bland, lifeless, familiar, fake.” Films, he said, did not speak to him any more. “I’m not a superhero, it’s not one of my interests.”

***

Golden ages contain the seeds of their own demise. Towards the end of the period studied by Accominotti, the rate of artistic innovation in Paris slowed. Accominotti argues that this relative decline was the function of a more settled market. A hierarchy of galleries had become established, with the big ones exerting disproportionate market power. Artists were snapped up by dealers when they were young and encouraged to develop distinctive, consistent styles. This made them easier to market to top galleries, and better off, but less likely to engage in the kind of collaborations with fellow artists that spawned new movements. In short, once the dealers and the artists figured out reliable ways to make money, the art became more predictable and less interesting.

We’re not seeing any slowdown in the appetite for new content

New Hollywood was born from the collapse of the old studio system, which required creative talent to follow tried and trusted templates. By the late 1960s, box office revenues were cratering. Colour television had given people a reason to stay home in the evening, and the Woodstock generation wasn’t interested in another Doris Day and Rock Hudson movie. In desperation, Hollywood’s executives handed money and control to a new breed of talented and wilful directors.

Amid the creative free-for-all that resulted, a new formula for success materialised. In 1977, Star Wars, created by George Lucas, became the biggest grossing movie ever. It featured dazzling special effects and characters who made for great toys. Star Wars showed the business that an ascent to the mythic realm of comic books, and away from fine-grained depictions of American society, was a safer route to success.

In the following decades, the movie industry slowly but inexorably consolidated around the formula Lucas invented, a process that has accelerated in recent years, as non-western audiences, particularly in China, have become more important to Hollywood’s business, putting further emphasis on splashy effects and archetypal characters that travel easily. These global blockbusters are prohibitively expensive to make and market. Meanwhile, DVD sales, an important secondary source of income for the studios, have been devastated by on-demand services. In response, the studios are making fewer and more costly films. “Each theatrical release is now a very big bet,” said Laura Kennedy of Lionsgate. “There’s a lot more pressure on it to work — and that puts pressure on the creative content.”

William Goldman’s famous maxim about moviemaking (“nobody knows anything”) is incomplete: nobody knows anything for sure. Studios know that if the last movie in a series was successful it’s a fair bet the next one will be too. It is possible Rogue One: A Star Wars Story could have flopped but, as an investment, Rogue One was a lot less risky than handing $200m to today’s equivalent of the young George Lucas and telling him to make something up. Creative innovation is now a risk the movie business cannot afford.

So is talent. Fewer movies has meant less work for big-name actors and directors. There has been a steady migration of high-profile creatives from one side of Hollywood’s storytelling business to the other. This year, Joel and Ethan Coen, perhaps Hollywood’s last auteurs, announced they were making a TV mini-series, while Peter Morgan scripted Netflix’s £100m-budget series The Crown.

When Martin Scorsese was making the pilot of HBO’s Boardwalk Empire, he told a friend: “I feel like I’m back in the ’70s, doing my own thing.” Young talent, such as the Duffer Brothers, the twins who created Netflix’s retro sci-fi hit Stranger Things, is bypassing the movie industry altogether.

Netflix’s Stranger Things (2016-present)

In the long run, an industry hostile to new ideas and talent will either die or be reinvented. If it is the latter, a moment of crisis may be necessary. Laura Kennedy told me there was a sense in which the movie business had not been disrupted enough. “The economic model has barely changed, compared to TV. People still go to movie theatres, even if they go less often. The industry hasn’t been forced to undergo a technology transition.” No matter how smart it gets about distribution or marketing, the industry requires a visionary to solve its story problem. Golden ages are kickstarted by someone brilliant and cussed enough to invent the future, like a late-career writer who needs creative fulfilment more than a pay cheque.

***

We used to complain that there was never anything good on TV. Now we complain about the opposite. Spoilt for choice as we may be, we should not forget to be astonished that an industry once regarded as an artistic wasteland has turned itself into the world’s most prolific source of new stories.

Whether we are in the middle or nearing the end of television’s golden age is hard to tell. Our sense of being overwhelmed by new shows is not illusory. According to FX Networks research, in 2016 there were 455 original series in TV, up from 182 in 2002. Yet we haven’t necessarily reached what John Landgraf has called “peak TV”. “We’re not seeing any slowdown in the appetite for new content, from all kinds of buyers,” says Laura Kennedy.

But the US cable business, which led the revolution, is facing a shakeout. Growing numbers of households with broadband are cancelling their cable subscriptions or not getting one in the first place. “The number of cable networks will shrink dramatically,” Erik Brannon, an analyst at IHS Technology, told me. Netflix and Amazon may find themselves coming under more financial pressure too unless their giant bets on the future pay off soon.

The stakes are rising and, amid fierce competition for the attention of critics and viewers, the market for more subtle ideas may be shrinking. An executive at a London production house, who asked not to be named, told me that while there were plenty of buyers for new shows, the specs were getting tighter.

“Everyone wants something high concept: an idea that can be summed up in a headline. They always ask, ‘Will it cut through?’” The search is on for a storytelling format that minimises risk. HBO’s Game of Thrones has shown the industry that expensively produced shows set in mythical realms, with global appeal, can be enormously profitable.

HBO’s Game of Thrones (2011-present)

“People like me make creative decisions, and we’re also responsible to shareholders,” says Landgraf. “Right now, those two goals are in harmony. But when they conflict, capital will have its way. It has to.”

Only when nobody knows anything does a creative industry bet on its creators. Golden ages come to an end when everybody knows something.

Ian Leslie is the author of ‘Curious: The Desire to Know and Why Your Future Depends on It’ (Quercus).

Twitter: @mrianleslie

Illustrations by Marcus Marritt