In 1970, if you had said that London would one day become the centre of the international art world, the successor to Paris before the first world war and New York after the second, most people would have thought you mad. The gleaming commercial galleries, the art fairs, the record-breaking sales at Christie’s and Sotheby’s, the arrival of the super-rich from every corner of the globe – all of this was decades away. Large parts of the city were still pitted and scarred from the bombs of the blitz. The port and docks on the Thames in east London were so completely derelict that people assumed they would be like that for ever. Most people didn’t even notice the power station that crouched opposite St Paul’s Cathedral – for there was no Southwark tube station, no elegantly engineered footbridge across the river, no glassy apartments, no Shakespeare’s Globe, no scenic path along the water’s edge to Tower Bridge. No one imagined that this behemoth, then still a decade away from being decommissioned, would one day become the world’s most popular museum of modern and contemporary art.

Tate, now an empire of four museums, and a global brand, was then a single entity: the Tate Gallery, which occupied the building now known as Tate Britain, in Pimlico. It played second fiddle to the grander National Gallery, from which it had recently become independent, and had a rambling and uneven collection divided into “British art” and “modern foreign paintings”, as if contemporary art were a vice conducted mainly overseas. It had some great pictures, and hosted some memorable exhibitions: among them was 1964’s Painting and Sculpture of a Decade, a survey of the previous 10 years of contemporary art that, for an 18-year-old Hampstead schoolboy named Nicholas Serota, had fanned the flames of an interest in art; five decades later, he recalled its “bright colours and American art and a sense that things were changing”. But for most British artists, particularly those of the rising generation, the Tate Gallery was marginal. “The best you could hope for there was a one-man show the year before you kicked over,” recalls sculptor Richard Deacon, who was a student in 1970.

The British art world itself was tiny: a “cult”, Deacon told me. Exhibition openings were small and ardent, like early-Christian prayer meetings. There were maybe three commercial galleries in London selling contemporary art. There was nothing remotely like the Pompidou Centre, which would open in Paris in 1977.

But over the next four decades, remarkable changes occurred to the culture in Britain, which were themselves tangled up with the story of its economy, its society, and the growth of that complex organism, London. Artists began to move to the city’s East End, to the condemned slums and the old factories, where they found space to make and show art. In the mid-1980s, a fresh generation emerged from the art schools: they could cushion their poverty with the dole, and they were also shrewd and canny and impatient for success. A modernising government, intensely relaxed about people getting filthy rich, came to power. Some of those filthy riches began to be spent on contemporary art. The middle classes stripped their floorboards and bought sleek modern furniture. The sharp edges and concrete of mid-century architecture stopped being ugly and started being fashionable. The country was shaking off its old fustiness. Taste was transformed. The idea of Tate Modern, a cathedral where the people would gather to marvel at the new and the strange, became something that it was dimly possible to imagine. And when it finally opened in 2000, it became both an embodiment of the changes that had occurred in British society, and a catalyst for further transformation.

Tate Modern, however, was not inevitable. It was Serota’s dream, his creation; and there is no one in the British cultural world more single-minded, more monkishly devoted to the arts as a civic and public necessity, more able to bend events to his will. There is no individual who has done more to change the way this country sees art.

Nicholas Serota, outgoing director of the Tate galleries and incoming chair of Arts Council England. Photograph: Alicia Canter/The Guardian

When he arrived at the Tate in September 1988, it was an affectionately regarded and faintly parochial museum; he left it earlier this month one of the most powerful forces in the international art world. Tate Modern’s popularity – five million visitors a year – immediately outstripped the expectations of even its most ardent cheerleaders. And then came the crowning achievement: last year, Serota opened an enormous extension, swelling the museum’s already gargantuan size by 60%.

The days of the cult were over. Contemporary art had become an established religion. Among the British bourgeoisie, an afternoon looking at, say, Rachel Whiteread’s casts of mattresses or the insides of boxes has become not a swaggering act of rebellion so much as a respectable family outing. Serota is the new religion’s pontiff, its primate and its prince.

Now, at 71, he has taken up a yet more challenging role: chair of Arts Council England, the public body that funds English arts organisations. There he will widen his gaze beyond visual art, turning his attention to music and theatre, to dance and literature. He will be judged by his ability to stave off the predations feared in an era of Brexit and economic uncertainty. More fundamentally, it will be a test of whether the national transformation he has stewarded – from a country largely indifferent to modern art to one that, in certain quarters at least, appears to take it seriously – is an enduring change, or merely a temporary infatuation.

Recently, in a glass-walled meeting room in the cheerless Arts Council offices in Bloomsbury, I asked him why he had never taken a job in the US, where he has been courted over and over again. His opposite number at the Museum of Modern Art in New York (MoMA), was paid $2.1m in 2013; Serota was on £178,000 when he left Tate earlier this month. He said, after one of his characteristic long pauses (he speaks only after silently considering, and bombproofing, every utterance): “I am not an incredible patriot, but somehow working here means more than doing it somewhere like Chicago. Making a great museum in Chicago is an academic exercise. Doing it in London is about changing society.”

To build the great popularising art machine that is the Tate, Serota has put himself through a regime of unremitting labour that most people would quail at. More than anything else, he is a man whose energies are always fixed on a distant future point: a steely-eyed strategist driven by a religiously held set of beliefs about the importance of the arts to both civic life and to the individual. He is a shy, quietly spoken man who warms his sometimes oblique sentences to just above room temperature with a wry smile; his face is composed of adamantly straight lines, like a Vorticist drawing. Tall, slender and punctilious of dress, he moves swiftly and nimbly, like a limber aquatic creature darting past weeds. To most he offers the front of the skilled and canny administrator. James Purnell, a former culture secretary, refers to his “slightly cardinal-like demeanour”. To only a few – usually artists – he reveals a more relaxed side; even, sometimes, a sliver of vulnerability. It is this relationship with artists that provides the key to his vision of Tate: a museum, in his view, should be refracted through the eyes of those who create.

In 1970, when Serota joined it as a young graduate, the Arts Council’s head office was a grandiose building at 105 Piccadilly, Mayfair. The visual arts department, however, was the least magnificent part of the building. Tucked up in the eaves, above the more prominent sections for music and theatre, it was presided over by a chain-smoking, gruff war veteran who had lost his leg in north Africa and won the DSO, a medal for distinguished service in combat. Serota started out by hanging travelling exhibitions in museums and libraries across a beat quaintly called the “north-east”, which actually stretched from Essex to the Scottish border. “There was an extraordinary level of public ignorance and often actual hostility to contemporary visual art and architecture. And we always we felt we were the Cinderella art form – the joke,” remembers Francis Pugh, who later married Serota’s younger sister, Judith. Pugh also worked at the Arts Council at the time. He recalls walking across Green Park one lunchtime with his future brother-in-law, of whom he was slightly in awe. It was around 1972; Serota would have been 26. “We were bemoaning the state of things. And then Nick said: ‘What we really need in London is a museum of modern art.’”

The year he turned 27 was the last time in Serota’s career that he was not in charge of an institution. In 1973, he became director of the Museum of Modern Art in Oxford, a gallery without its own collection, despite the grand title. He set about turning the small, damp former brewery into a place with international ambitions. In 1974, he curated a show of drawings by the German artist Joseph Beuys. It was the kind of exhibition that people travelled to see; Richard Deacon still remembers it vividly, 43 years on. Beuys’s work had been seen in Edinburgh, but for most Britons his politically committed, avant-garde art was an exotic rumour. (His most famous work is, perhaps, his 1974 performance I Like America and America Likes Me, for which he shut himself into a New York gallery for three days with a live coyote.) For Anthony d’Offay, who later became one of London’s most influential dealers, the exhibition was the first time he had clapped eyes on Beuys’s work, and it was a revelation. “You couldn’t think of an artist any less in the Slade tradition” – meaning any less British. Serota was determinedly broadening horizons, showing art from beyond the island. There was no dark secret to this, he recalls: he would simply write to artists he liked, preferably bypassing their gallery. To this day, a fixed habit is composing letters in a spiky hand with his Rotring fountain pen; every December, he writes 1,100 Christmas cards.

Serota with Damien Hirst in 2003. Photograph: Linda Nylind/The Guardian

His then deputy Sandy Nairne (who, 20 years later, would reprise that role at the Tate) remembers that, while Serota was happy to delegate to colleagues on some matters, on questions of aesthetics, whether the layout of a show or the design of a catalogue, he would brook no opposition. “I knew that there was no point my having ideas about design, about how something should look,” he said. Once, when a new gallery floor was laid in Oxford, Serota found it, in his words, “ghastly and shiny”. That meant a weekend for him and Nairne on their knees, wet-and-dry sandpapering until he was satisfied. Serota’s aesthetic exactitude (and willingness to get his hands dirty) has remained a constant. Frances Morris, director of Tate Modern, told me she often used to see him gliding through the galleries there, stooping to pick up visitors’ dropped litter. And when cultural historian Robert Hewison returned to Tate Britain one morning in 2000, after a day hanging a show, he found that Serota had redone it overnight – and made it better. To this day, Serota hangs all the pictures in his daughters’ and stepdaughters’ homes.

After only three years in Oxford, Serota left to run the Whitechapel, an adventurous gallery in east London. He still had a shock of sandy curls, but, signalling his gradual movement towards the establishment, he became Nicholas rather than Nick, lost the black, beatnikish polo neck of the Oxford years, and took to wearing corduroy suits with check shirts and knitted ties. He promoted young British artists who are now household names: Howard Hodgkin, Antony Gormley. From overseas he brought work by artists then barely known in Britain: Eva Hesse, Cy Twombly, Philip Guston, Bruce Nauman.

In a sense, he began to run the gallery as a kind of silent rebuke to the Tate. In 1976, for example, the Sunday Times ran an article headlined “Tate drops a costly brick” mocking the museum for purchasing Carl Andre’s minimalist sculpture Equivalent VIII, which consisted of 120 bricks placed in a rectangular pattern on the floor. The museum was portrayed as having frittered away taxpayers’ money on rubbish; the work was nothing but a con, the paper suggested. The affair of the “Tate bricks” was picked up everywhere, and the museum engulfed in a scandal that seems unthinkable now. Norman Reid, the Tate director at the time, went into retreat, failing to mount a robust defence of the art work, even forbidding staff members from speaking about it to newspapers. So Serota did what he thought Reid should have done: instead of falling back, he advanced. He staged a whole exhibition of Andre’s sculptures to “allow people to see, make a judgment, make their mind up”. For a man whose apparent froideur masks a natural diffidence, Serota has never been afraid to stir up controversy.

Under its next director, Alan Bowness – who laid plans for Tate galleries in Liverpool and St Ives – the museum was to enter a period of recovery. But there was no doubt that Serota had the museum in his sights. “Round about 1983,” he told me, “I began to think that, when Alan goes, maybe I should try to have a go at the Tate.” He was an unlikely pretender to the job in some ways – it had usually been held by distinguished art historians, not buccaneering youngsters – which, despite his art history degree from Cambridge and MA from the Courtauld Institute, he certainly was. But he had a dawning sense that the Tate was a sleeping giant; that it could be more than London’s second-most important art museum.

The scandal of the bricks was a particularly unhappy chapter in a long history of British ambivalence towards contemporary art – especially contemporary art from overseas. History judges museums by the quality of their collections, and an institution whose job it is to amass new art works will always be faced with the fact that a something later judged to be a masterpiece may be widely seen at the time of its making as bewildering – or plain rubbish. The director’s job is to stay ahead of public taste, but that is particularly difficult in Britain because the public (that is, the taxpayer) funds the museum.

It is partly owing to this, and partly owing to successive directors’ and trustees’ timidity, that the history of the Tate can read like a chronicle of squandered opportunities. In the early 1920s, works by Cézanne were repeatedly turned down. The entire estate of Henri Gaudier-Brzeska, a pioneering French sculptor who died in the first world war, was refused in 1926. Kandinskys and Gauguins were waved away. The trustees refused to buy six Van Goghs in 1927. No purchases were made at the notorious auction, held in Switzerland in 1939, of “degenerate art” confiscated by Germany’s Nazi government, where Munchs and Picassos were sold at bargain prices. At the same time, although Tate was founded in 1897 through the generosity of the sugar magnate Henry Tate, the UK lacks a serious philanthropic culture, certainly compared to that of the US. There Alfred Barr, the first director of MoMA, was able, in the mid-20th century, to persuade collectors to donate modernist art that swiftly eclipsed the quality of Tate’s.

In addition to such structural difficulties, Tate’s past has been peppered with crisis. Directors have had to deal with serious physical threats to the institution: the building that is now Tate Britain in Pimlico is still scarred from the blitz (though the art had been stored safely out of London). More seriously, in 1928, the director almost drowned and the collection was imperilled when the Thames embankment collapsed. Directors show their mettle at such moments: one of Serota’s longtime deputies, Alex Beard, remembers that when Tate Modern caught fire in 2003, owing to an ambitious but ill-fated pyrotechnic display by Chinese artist Cai Guo-Qiang, Serota calmly gathered a couple of technicians and unscrewed Jackson Pollock’s unglazed Summertime from the wall to protect it from the sprinklers. In addition, nearly every director, including Bowness, has engaged in open and damaging warfare with their trustees. Such problems have led to successive directors collapsing into “nervous exhaustion” – what we would now call severe mental illness. Most spectacularly, as Frances Spalding’s history of Tate records, JB Manson departed in 1938 after drunkenly “cock-a-doodle-dooing” a speech given in Paris by the French culture minister, before attempting to sexually assault the British ambassador’s wife.

Tate Britain on Millbank in London. Photograph: Tate

Undeterred by the example of ignominy offered by some of the previous holders of the office, in 1988 Serota applied to become director of the Tate. To support his case, he produced an astringent document, typed over exactly two sides of A4, titled Grasping the Nettle. The introduction began with the statement that “The Tate approaches its centenary in a state of impending crisis”, and laid out why: funding gaps, rising prices in the art market, and creaking, old-fashioned management. The rest was divided into eight sections (“Collections”, “Exhibitions”, “Buildings”, and so on). It continued in a similar vein, each subsection beginning with a verb in the imperative (“Create”, “Establish”, “Abandon”, and so forth). Section 2.1 exhorted: “Divide the collections into British and Modern Foreign components”. Not quite a blueprint for Tate Modern – but nearly, especially as it would mean overturning plans for a sculpture museum and a museum for “new art” on the Pimlico site, then the favoured solution to the museum’s space problem. The document concluded by asserting that for the Tate to be a success it would need to be “cherished” by artists. “The Tate is loved, but not sufficiently respected,” he wrote.

Despite the later challenge of building Tate Modern, Serota told me his early years were his hardest at Tate. “Beginning to get the momentum going, to turn the battleship – that was the most difficult thing,” he said. Though some young curators, such as Frances Morris (now director of Tate Modern), were excited by the fresh air blowing through the gallery, and their “stylish, urbane, slightly intimidating” new boss, others were hostile, sensing that the supremacy of the historic British collection would be challenged.

Still, things were changing in the art world, and Serota was in the vanguard. The artist Richard Wentworth has a theory that “the year the past ended” was 1986. That was when Joseph Beuys, Henry Moore and Andy Warhol all died and hastened their journey into the canon; that was when the “Big Bang” deregulated the City of London. A year later, east London’s empty docks, soon to be swept up in a property rush, became the setting for an exhibition curated by Damien Hirst, Wentworth’s former student at Goldsmith’s, in an old Port of London Authority building. The show was called Freeze, and it launched a generation of artists. Charles Saatchi, whose eccentric, New York-style hobby was collecting contemporary art, began to buy their work and show it at his gallery in an old paint factory in north London.

One of Serota’s early acts, aside from a clear-out, tidy-up and rehang of the congested Tate Gallery spaces, was to reboot the Turner prize. It had been founded in 1984, and started slowly, as an award that could honour writers and curators as well as artists. Serota changed the rules so that only artists under 50 and based in Britain were eligible, gave the shortlisted work a proper exhibition, and made a deal with Channel 4 to get it on television. It coincided with – and, for many years, was defined by – the rise of the YBAs, as the new crop of young British artists became known. “It gave the general public the chance to see some really contemporary art, which they had not been able to do at the Tate before,” Serota told me. “You had to look really hard to see this kind of work. Most people weren’t going to warehouses in the East End, or to commercial galleries. White Cube [the gallery most strongly associated with the YBAs, founded in 1993] was upstairs in a small office building in St James’s.”

The prize became the culture wars’ most bloody battleground. The watershed was 1993, when a 30-year-old named Rachel Whiteread won: that was the moment when her work House, a concrete cast of the last remaining house of a condemned East End terrace, transformed the national debate about what an art work could be, about what it meant to make a monument. The more thoughtful critics wondered why there was such a stream of animosity in national attitudes to art: was it because the repressed British preferred to keep at arm’s length anything to do with emotions, or, worse, anything that could be thought of as “intellectual”?

The prize-bashing intensified through the 1990s. “For 1,000 years, art has been one of our great civilising forces. Today pickled sheep and a soiled bed threaten to make fools of us all,” wrote the Daily Mail in 1999 – a typical headline, expressing that peculiarly British anxiety that contemporary art was, somehow, a swindle or a conjuror’s trick. “We were always accused of deliberately sensationalising and deliberately bringing out artists who were provocative,” said Serota. “And yet, if you look at the list of winners, it’s a pretty good list. There are very few whose shows you wouldn’t go to now.”

In all this, Serota was like a battle-hungry medieval knight-bishop; his shock-troops were artists. If it was sometimes painful for them to be thrust so nakedly into combat, “they have been, fortunately, robust enough to survive it”. He invited figures such as Brian Eno, Paul Smith and Madonna to present the cheque, giving the event an aura of fashion and glamour. Brian Sewell, the traditionalist art critic of the London Evening Standard, attacked Serota repeatedly, calling for his resignation, dubbing him (on 13 separate occasions) a panjandrum and identifying what he called the “Serota tendency”. That was a label that stuck – it referred to Serota’s supposed appetite for conceptual art, at the expense of all else, though he has never openly advertised his personal taste. Another label with staying power came from art dealer Leslie Waddington, who called him “a Jansenist in Comme des Garçons”. (In fact, the suits were Issey Miyake; these days they are Margaret Howell, with Armani for best.)

As these battles raged, Serota had his eye set on a distant goal. One night in March 1993, he took a trip to the south bank of the Thames, to look at a “looming hulk” of a building. His deputy at the time, Francis Carnwath, had mentioned the building to him, the result of a suggestion thrown out by the architectural historian Gavin Stamp. The place had been abandoned in 1981 and it was a scene of complete dereliction. “When I got there it was dark. I couldn’t work out how big it was, so I paced it out from the chimney to the end of the building, and worked out the footprint.” There was, he said, “that extraordinary sense of being right on the river, and opposite St Paul’s.” He thought to himself: “Maybe it’s manageable.”

Serota acquired his love of art, but he was born into the politics and public-service principles that have dictated the direction of his life. His parents, Beatrice and Stanley Serota, were the children of Jewish immigrants from eastern Europe who had grown up in neighbouring streets in east London. Eschewing the family cabinet-making business, which, as Serota Bespoke Furniture, still makes library chairs and bookcases, Stanley became an engineer. “We used to do experiments in the bath, trying to create a floating drilling rig that would not capsize in the gulf of Nigeria or wherever,” said Serota. But it was his mother who provided the stronger influence. Beatrice, who studied at the London School of Economics, was politicised in the late 1930s in the shadow of the Spanish civil war. She rose, via Hampstead council and the Greater London Council, to become a Labour peer and a health minister under Harold Wilson; and prefigured one of her son’s present roles by becoming a governor of the BBC. “In the moment after 1945, people like my mother saw the chance to change the world,” he said. Serota himself was born in 1946, the year the Arts Council was founded – part of the postwar blast of state-led idealism that also led to the founding of the NHS.

The schoolboy Serota was captain of swimming and rugby. He was head boy, twice. He remains ferociously competitive, even on trivial matters – Alex Beard remembers Serota “tight-lipped” when beaten in a “fun” quiz night at Tate. Though it’s now difficult to imagine the spectral figure of Serota in a scrum, he likes to roll up his white shirtsleeves to signal his concentration, carefully placing his cufflinks in his jacket pocket, revealing wiry, busily veined forearms. (His father, a hobbyist silversmith, once made him a pair of cufflinks in the form of Carl Andre’s bricks.)

At Cambridge, he started out reading economics, with no thought of working in the art world. But then he discovered a place called Kettle’s Yard. Now a public gallery, it was, then, the home of Jim Ede, who had worked at the Tate as a young man, and helped rescue works from the 1928 flood. He had left, frustrated by his bosses’ timidity in relation to contemporary art – and had bought the Gaudier-Brzeskas the museum had rejected. “He would open the house every day between 2 and 4pm,” said Serota. “In the attic he had these cupboards in which he kept Gaudier-Brzeska drawings that you could borrow and hang in your room. The compelling thing was here was a person who had lived with artists. It was that very direct connection.”

Ede caused Serota to sense the possibilities of art, of working with artists, of understanding the worldview of another person through the art they had made. For a man who radiates containment and control, Serota’s relationship with art is visceral rather than cerebral and scholarly. His experience of art is, he told me, “very often simply physical. It hits you in the stomach if it’s really great.” For all his appearance of asceticism, he is a sensualist. “You enter a Richard Serra sculpture,” he said, referring to the giant, labyrinthine steel structures made by the American artist, “and your body changes. Its feeling and its shape and its form. It makes you aware of yourself, but it also makes you aware of the person who has made it. And then you understand more about the way in which other people’s minds work and their emotions and their experiences. [Art] is about encountering other people’s experience, other people’s language, other people’s view of the world.”

It was the chance to make this kind of experience available to a wider public that sharpened Serota’s ideas about the future Tate Modern as they began to take shape in the early 1990s. At the heart of his most cherished beliefs is the conviction that a museum supported by the state and owned by the citizenry is a more powerfully beneficial force than a museum governed, like MoMA, by a small group of wealthy trustees. There were also deeply practical considerations. The premises in Pimlico could show only about a quarter of the collection, which was, necessarily, always growing. In 1992, when Serota had a colleague, Richard Francis, draw up a plan for the extra space needed if the Tate were to expand on its original site, it spreadeagled right out into the Thames – an impossibility. The building was at capacity in terms of its visitors, too: it sometimes had to shut its doors before closing time owing to overcrowding. At the same time, Serota, always highly competitive, was all too aware that every other great world city had a museum of modern and contemporary art, and so why not London? For all the hostility vented in some quarters, he could sense that there was also an appetite, almost entirely unmet, for such an institution. If he built it, he figured, they would come.

Tate Modern’s turbine hall before its refurbishment. Photograph: Tate

Things moved fast after Serota’s nocturnal trip to Bankside. In July 1993, he took the trustees to look at the site. In July 1994, an architectural competition was announced, which was eventually won by the little-known Swiss practice of Herzog & de Meuron, chosen over contenders including Rem Koolhaas and Renzo Piano. The purchase of the power station for £8.5m was agreed. In 1995, machinery began to be stripped out of the building. In 1997, construction began. By all accounts, Serota revelled in the work – despite the fact that at the start the project had neither political support nor an obvious means of being funded. David Mellor, a former secretary of state for national heritage, even referred to the “clapped-out old power station” they had chosen. (More conventionally wise heads had looked to the plot where the London Eye stands now – in many ways this was the obvious solution, as it was near other cultural attractions and well-connected by public transport). But Serota ploughed on, working, as he often does, from gut instinct: the rawness and scale of the building reminded him of the industrial spaces in which, in the early 1990s, artists were improvising exhibitions.

The project cost £134m, around half of which had to be raised from private sources. Courting wealthy private donors has not been Serota’s favourite aspect of the work. “He is not worldly and does not particularly value money,” said Frances Morris. “And so I think he finds it difficult to understand why others find it so difficult to give. He genuinely believes that one should share one’s good fortune.” The museum finally opened on 12 May 2000, which also happened to be Stanley Serota’s birthday – and that of Joseph Beuys.

It quickly became clear that Tate Modern was offering something more than a solitary encounter with art, something different from the museum’s traditional role of solemn edification. Tate Modern was fun, and social, and cool. It changed the way people behaved in front of art, and towards each other. It changed what people thought art actually was, and what a museum was: art could be something that swallowed you up and placed you at the heart of your own immersive drama; the museum could be the magical space that contained and enabled such metamorphic experiences. The clearest sign of this was in 2003, when artist Olafur Eliasson’s Weather Project transformed the basilica-like turbine hall into a mystical place where a vast orange sun shone through wreaths of dry ice, and a mirrored ceiling caused the crowds to gaze upwards, enchanted by their own reflections. It provoked what the Guardian called at the time “an almost psychotropic transformation of human social behaviour”. Strangers talked to each other (a psychotropic transformation in itself, in London). They performed for themselves and each other, observing their movements under the giant mirrors, becoming part of the art, abandoning all the fear and deference usually associated with museum exhibits. They picnicked under that artificial sodium sky, as if they were at a music festival. Tate Modern had become part of popular culture.

For Serota, the most important signs of Britain’s growing acceptance of modern and contemporary art are not to be found in the cavernous spaces of Tate Modern. “In Britain we’ve shown a greater appetite for the visual arts, for the art of the 20th century, than I have ever thought was possible,” he told me. “The most exciting development of the past 15 years has been that you can get 400,000 people a year going to a gallery in Margate, 200,000 to a gallery in Nottingham, 600,000 to a gallery in Liverpool. Compared with what people thought was possible 25 years ago, it’s astonishing.”

One evening, soon before he stepped down from Tate, I joined Serota as he visited one of the fresh crop of regional institutions. His job of the evening was to “lend endorsement” to a fundraising auction at Nottingham Contemporary, one of the most successful of the post-Tate Modern wave of public contemporary galleries, which opened in 2009. As we walked there together from another Nottingham gallery, the New Art Exchange, where he had been discussing fundraising over tea and cake, he mistook a trail of steam coming from its roof for a byproduct of its climate-control system; it was in fact an artwork by Lara Favaretto titled Thinking Head. It was somehow reassuring that even Serota should occasionally make such an error, as mere mortals quite often do.

At the gallery, there was a group of dealers and wealthy collectors up from London, who would bid on most of the lots – everything from cashmere dressing gowns (yours for a grand) to limited-edition artworks. Serota may have shifted thinking about art, but he has done it in a political and economic era that his mother would not have recognised, or probably liked very much. Tate was birthed into a neoliberal age when, in the words of cultural historian Robert Hewison, “visual art became a part of a social experience and a spending experience”. It may be perfectly obvious and perfectly natural, now, that a visit to Tate Modern will include a browse in the gift shop and a trip to the viewing platform to admire the panorama of London – that it is, in some intangible way, about absorbing the aura of its brand, of “being there”.

Elizabeth Price, who won the Turner prize in 2012, told me: “I don’t like the idea that art has to be fun all the time. I don’t subscribe to the idea that, to make something engaging, you have to engage people as if they are infants. Sometimes at Tate there is a sense that in order to get people through the doors you have to turn up the volume.” But she is otherwise an admirer of Serota – as artists, or at least those artists whose careers have prospered on his watch, almost always are. Martin Creed spoke of his conspiratorial warmth: a wink once sent across the room at an opening that seemed to express, “‘I know this is all a pile of shite, the bureaucracy. But there is this other thing, the work, and I’m with you on that.’”

Work by Andy Warhol at Tate Liverpool in 2015. Photograph: Christopher Thomond/The Guardian

Some think Serota should have left Tate soon after 2000, in the glow of Tate Modern’s triumph. Some question the necessity of building the museum’s extension, recently rechristened the Blavatnik wing after its most generous private funder. (It cost £260m, of which £58m came from the public purse; when Maria Balshaw replaced Serota as director of Tate on 1 June, it was still just under £20m short of its fundraising target.) Some people think that no institution is well-served by its leader staying for 29 years. Tellingly, there have been five directors of Tate Modern working under Serota in 17 years; it can be frustrating reporting to him on his own favourite patch. According to his account: “There is undoubtedly a slight issue that it’s a very big ship, but you’re not the sole captain, and for many people being the sole captain is something they feel the need of.” For all those who speak of his wry warmth, there are those too who mention a certain sang-froid, a ruthlessness, an uncanny ability to get his own way. There has been, perhaps, collateral damage in Serota’s war. But the war has been won.

At least for now: contemporary art’s relationship with wider society has sometimes felt a little shaky, especially after the financial crisis of 2008. There are plenty of people who find contemporary art inexplicable or pointless; plenty who do not automatically agree that it should receive public funding. Local-authority cuts have hit institutions hard across the country; private funding is difficult to find. Reports came last November, for example, of a threat of closure to the New Art Gallery Walsall in the West Midlands, which opened in 2000 and which holds an important collection of modern and contemporary art. Even in London, one might wonder how sturdy and deep-rooted the art world really is. The delicate set of relationships knitting together artists, institutions, public funders, dealers, collectors and philanthropists are still functioning strongly – London is still a place that artists want to live, despite ruinous property prices, and despite the attractions of, say, Glasgow. But it is London’s status as the centre of the European financial world that accounts for the number of commercial galleries in the capital, and it is unclear what will happen after Brexit. Art collecting has never penetrated very deeply into Britain’s well-off professional classes. Artists will always make art, everywhere. But the art world may move on.

It has now become Serota’s job to see off all these potential threats – and their equivalents in other art forms – in his role at Arts Council England. The job will not, by most people’s standards, be exactly enjoyable; he will have to make a tough argument that culture can help find Britain a new economic role and status in a post-Brexit world. It remains to be seen how much time the post-Brexit settlement will have for the kind of progressive, liberal, internationalist values that many arts organisations espouse –and for the outre-seeming work that many of them champion. It also is possible that Serota’s reign at Tate – 1988 to 2017, the era stretching from the fall of the Berlin wall to Britain’s EU referendum – may in retrospect seem like a particularly propitious time for the running of a populist mega-museum in an overconsuming city fuelled by international finance. His job at Arts Council England – where his career comes full circle – will test whether he can convince the English to value the arts as fundamental to ordinary existence, not just as branches of industry or drivers of tourism, or aspects of social policy, or flourishes of fashion.

“I think the current climate is going to make the position of the arts more difficult,” said Serota, “because greater credibility is being given again to doubt”. Doubt, that is, of the value of the new. In uncertain times, cultures can prefer to look backwards, to decline into nostalgia. Dennis Stevenson, the chair of Tate from 1988 until 1998, tried to talk his friend out of the role. “It’s brilliant for the Arts Council, brilliant for everyone else. But I think he’s idiotic to do it – you go around being kicked all the time,” he said. (Serota’s reply to this warning was an inscrutable smile.) His aim at the Arts Council, he said, was to make the arts “integral to life”; and for everyone, not just the few. He is completely secure in his devoutly held belief that the arts – books, music, visual art, the theatre, dance – have the capacity to make lives better. It seems an unfeasibly idealistic goal that he has set himself, one doomed to fail. Or perhaps not. The battle-scarred zealot has fight in him yet.

Main photograph by Dan Chung for the Guardian

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.

This article was amended on 26 June to correct a factual error. Serota’s parents grew up in east London, not Hampstead.

