What has changed? Are things better or worse? These are the questions I am asked constantly as I prepare to give up my job as a nightly aisle-squatter after some 10,000 nights at the theatre. They are difficult to answer. British theatre is incredibly resilient, yet radically different from when I took up my post at the Guardian in 1971. Even the job of being a critic has altered in all sorts of ways.

When I joined the paper, a review was normally transmitted in one of two ways. Either you rushed back to the office straight after a show and banged out your copy on a typewriter by 11pm, or you frantically sought a working phone box and spelled out your 350 words to an easily bored copy-taker. (This helps to explain why my predecessor woke up one morning to find his review of The Merchant of Venice featured a character called Skylark.) Now, you tap out the review on a laptop and it whizzes through the ether in a fraction of a second. The deadline has also shifted – unless the show is extremely hot news, as when a Cumberbatch or Branagh plays Hamlet – to 9.30am the morning after the first night. That means I seem to write much of the review in my sleep, something that may not come as a surprise to regular readers.

But if the process – and the people who get to write the reviews – has changed, the role of the critic remains much the same: you still have to describe, interpret and evaluate a theatrical event for readers who were not there and may have no chance of going. You are not writing for scholars, posterity or the cast of the show, although it would be nice if they all read your words. You are addressing that morning’s reader with as much clarity and elegance as you can muster, and seeking to put a transient event in its theatrical and social context. It is a constant challenge that has kept me going for nearly half a century and that has left me feeling that all you can do, in Samuel Beckett’s words, is “fail better”.

If the critic’s role has several constants but has changed in multiple ways, the same goes for the theatre. Some things have got better, some worse. If I had to pick out one thing I regret, it would be the virtual disappearance of the regional rep company. Time was when many theatres boasted a permanent ensemble: the most famous was the Liverpool Everyman of the 70s, when Julie Walters, Bill Nighy, Jonathan Pryce, Antony Sher and Pete Postlethwaite were part of a dazzling troupe. Financial constraints and cultural fragmentation make it hard to assemble and retain that kind of team today.

‘The role of the critic remains much the same: you still have to describe, interpret and evaluate a theatrical event for readers who were not there’ ... an extract from Billington’s first review for the Guardian, from 2 October 1971. Photograph: Richard Nelsson/The Guardian

I would rather, however, accentuate the positive. One thing that has emerged in my lifetime is the move towards gender equality. In the first year of my job, I salivated at the prospect of reviewing new plays by Tom Stoppard, Edward Bond, Alan Bennett, David Storey and Alan Ayckbourn. One fact was obvious: they were all men. One of the first features I wrote asked why there were so few female playwrights. I blush now to reread the piece, which offered all kinds of phoney explanations, but at least I felt I was posing the right question.

It is one that seems redundant today. I would attribute that partly to Caryl Churchill, who, from early work such as Owners and Objections to Sex and Violence, has acted as a guiding light to aspiring dramatists. Where she led, others followed, including Pam Gems, Charlotte Keatley, Shelagh Stephenson, Bryony Lavery, Catherine Johnson and Winsome Pinnock. Surveying the scene today, what strikes me is that many of the most ambitious, rule-bending plays are by women. Lucy Kirkwood’s The Children, Laura Wade’s The Watsons, Lucy Prebble’s A Very Expensive Poison, Ella Hickson’s Oil, Natasha Gordon’s Nine Night and Lolita Chakrabarti’s Red Velvet are works that tackle a huge range of social, political and historical issues. They also experiment with form in a way that echoes the example of Churchill.

Gender equality now affects casting, too. There have long been striking examples of gender-flipping. I saw Frances de la Tour play Hamlet in 1980; other performers of the moody Dane include Angela Winkler, Maxine Peake and Ruth Negga, with Cush Jumbo still to come. I have also seen, and admired, Kathryn Hunter and Glenda Jackson as King Lear, as well as a number of single-sex productions, such as the recent Richard II at the Sam Wanamaker Playhouse in London, cast entirely with women of colour.

It may be a reflection of my age, but I find an inflexible quota system, with a 50/50 gender split, more problematic, especially in Shakespeare. A play such as Troilus and Cressida, which explores the latent homoeroticism of male combat, loses more than it gains from gender parity, as a recent RSC production proved. I am, however, always open to persuasion.

What I have no hesitation in saying is that greater sexual equality has given criticism a long-overdue shake-up. When I started out, the theatre critics were like a travelling male club, trundling from stall to stall in baggy, egg-stained suits. Today, on any first night, you will see as many female critics as men. I once had the temerity to ask a female colleague if she thought gender made a difference: she suggested that reviews by women were more impressionistic, those by men more conventionally linear. Be that as it may, the glass ceiling in criticism has cracked irrevocably. What is sorely lacking is critics of colour, which is why the appointment of Arifa Akbar as my successor takes on extra significance.

In the theatre at large, diversity has made huge advances. There have always been outstanding plays by writers of colour: take Moon on a Rainbow Shawl by Errol John, Skyvers by Barry Reckord and Play Mas by Mustapha Matura. What is striking today is that plays by BAME writers are taking place on main stages, are wide-ranging in theme and derive from a large talent pool: I could make a long list, but Bola Agbaje, Gurpreet Kaur Bhatti, Tanika Gupta, debbie tucker green, Kwame Kwei-Armah and Roy Williams represent a rising new generation.

‘Who today can say, with total and unshakeable confidence, what a play is?’ ... Billington outside the Globe theatre (now the Gielgud) on Shaftesbury Avenue, London, in 1977. Photograph: Sally Greenhill/The Guardian

There would be shock and outrage today if you saw a classic revival with an all-white cast. The days when black actors were confined, in Shakespeare, to playing Othello or Aaron the Moor in Titus Andronicus are long gone. I have seen actors such as Sophie Okonedo, Sharon D Clarke, Cecilia Noble, Hugh Quarshie, Adrian Lester, David Oyelowo and Chiwetel Ejiofor prove they can play anything and everything. All this represents real progress.

The mantra of all self-respecting companies right now is, of course, to create a theatre that represents the infinite variety of the British public. Who could argue with that? But I get nervous when I hear an Arts Council desk-wallah say that, when it comes to funding, “relevance takes precedence over excellence”. That strikes me as a ludicrous false dichotomy. No amount of outreach, community or educational activity will be of any use if the work itself is not first-rate.

I am often asked how playwriting has changed in my time. One thing I have learned over the years is that politics and aesthetics are theatrically indivisible and that the plays that make the most impact are invariably the best written. A few examples will have to suffice. David Edgar’s Destiny (1976) memorably caught what Dennis Potter called “the nostalgia, the disappointment, the dumbly aching resentments” of the extreme right in Britain. Churchill’s Top Girls (1982) pinned down the contradictions of a Thatcherite philosophy where public advance was achieved at private cost. Pravda (1985), by Howard Brenton and David Hare, not only exposed the newspaper industry’s craven submission to Rupert Murdoch, but the perilous isolation of power. The Colour of Justice (1999), edited by Richard Norton-Taylor from the transcripts of the Macpherson report into the killing of Stephen Lawrence, revealed the layers of institutional racism within the Metropolitan police as inexorably as a Greek tragedy. James Graham’s This House (2012) grippingly portrayed the daily machinations of Westminster politics while implicitly putting the case for electoral reform.

What fascinates me is the way plays uncannily reflect the temper of the times; you can chart Britain since the second world war through the history of its theatre. That is still true: the social divisions created by Brexit, the problems of identity politics and the exponential rise in mental illness are constant dramatic themes.

I believe passionately in criticism's unending mission to record, interpret and evaluate what happens on stages

The most radical change has been in dramatic form. When I started out, there was a rough consensus as to what constituted a play. Admittedly, some writers were pushing the envelope: Ayckbourn showed in How the Other Half Loves that you could put overlapping action in two households on to a single stage; John McGrath in The Cheviot, the Stag and the Black, Black Oil used the form of a ceilidh to promote a message about the exploitation of Scotland’s resources.

Who today can say, with total and unshakeable confidence, what a play is? Harold Pinter pointed the way, showing in One for the Road (1984) that in slightly more than 30 minutes you could offer as powerful a portrait of tyranny as anything in Arthur Koestler. Now, the pungent one-act piece that provides a metaphor for the wider world is everywhere: Churchill’s Far Away, Mike Bartlett’s Cock, Sarah Kane’s 4.48 Psychosis and Jez Butterworth’s The River, to name just a few. What is true of Britain is also true of the US. F Scott Fitzgerald said there are no second acts in American lives and now there are few second acts in American plays.

Yet you can immediately contradict that by pointing to our corresponding appetite for the inordinate. Matthew Lopez’s The Inheritance is in two parts, lasting six and a half hours. Butterworth can also write long, as he proved in Jerusalem and The Ferryman. Right now, April De Angelis’s two-part version of Elena Ferrante’s My Brilliant Friend is playing at the National Theatre. The old rule that a play was an event lasting 150 minutes, with an interval, has been thrown out of the window. Today, form follows function; a play can be as long or as short as its subject dictates.

I have dwelled so long on playwriting that I have said too little about other aspects of theatre. Has acting significantly altered during my tenure? I would say it has, in accordance with changes in taste, technology and dramaturgy. One of the first pieces I wrote for the Guardian was a celebration of three great performances: Laurence Olivier in Eugene O’Neill’s Long Day’s Journey Into Night, Paul Scofield in Carl Zuckmayer’s The Captain of Köpenick and Ralph Richardson in John Osborne’s West of Suez. Great acting still exists, but in a different form. Classics are often re-gendered. Living dramatists tend to write, with obvious exceptions, democratically structured plays rather than ones built around star parts. The influence of the TV camera, as well as the ubiquity of studio theatres, means actors are trained to communicate through small gestures. To put it simply, the ironic has replaced the heroic.

‘One thing I have learned over the years is that politics and aesthetics are theatrically indivisible’ ... Billington in conversation with David Hare in 2012. Hare co-wrote Pravda, which exposed the newspaper industry’s submission to Rupert Murdoch. Photograph: Alicia Canter/The Guardian

One of the pleasures of a critic’s job – as well as one of its greatest challenges – is trying to describe performances. I have been lucky to have seen some unforgettable ones. It is impossible to list them all, but I would like to mention a few that remain part of my imperishable memory: John Wood, and later Tom Hollander, relishing the intellectual acrobatics of Henry Carr in Stoppard’s Travesties; Ian McKellen and Judi Dench as the Macbeths in Trevor Nunn’s grippingly intimate production; Eileen Atkins as the fiercely independent Maddy Rooney in Beckett’s All That Fall; Maggie Smith capturing the piercing recollections, as well as the moral evasiveness, of the heroine in A German Life; Simon Russell Beale conveying the troubled testiness of Shakespeare’s Prospero and Mark Rylance the whimsical passion of Olivia; Paapa Essiedu and David Tennant both highlighting the quick-wittedness and humour of Hamlet in different RSC productions. I could go on and on. One of British theatre’s prime assets is its ability to produce, decade after decade, new generations of exciting actors.

If there is an aspect of theatre of which I have been critical in recent years, it lies in the elevation of the director to the role of supreme creative artist. I should add quickly that I am not remotely anti-director. I was brought up on the work of Peter Brook, Tyrone Guthrie and Joan Littlewood. I could also list numerous instances where a director has given me new insight into a familiar work. At the start of my career, John Barton directed a Richard II that, by having Ian Richardson and Richard Pasco alternate the roles of Richard and Bolingbroke, saw them as tragic parallels rather than the usual polar opposites. Last year, Marianne Elliott gave us a whole new perspective on Stephen Sondheim’s Company by turning it into a story of a modern woman’s (rather than a man’s) quest for a permanent relationship. My quarrel lies with directors who see themselves as creators rather than interpreters – a byproduct of the fact that many of them, in order to make a living, frequently work in Germany, where the theatre is far better funded and the director is regarded as god.

But I have no wish to end on a sour note. I count myself lucky to have been witness to an amazing period in British theatre. I shall probably go on haunting playhouses until I fall off my perch. We hear a lot about the death of criticism, but I believe passionately in its unending mission to record, interpret and evaluate what happens on stages. The means of communication may change in the light of evolving technology, but the function remains the same. I have been privileged to do the job for the past 48 years, but, as I move from the world of daily deadlines to that of periodic commentary, I can only wish my successor the same good fortune I have enjoyed. In the words of Hector in Bennett’s The History Boys, I am happy to pass the parcel.

Michael Billington will be talking to Arifa Akbar at a Guardian Live event at Kings Place, London, on 3 February