Back in April, a six-part drama called Undercover premiered on BBC1. Perhaps you were one of the five million people who watched it: the story was audacious and continent-hopping, enfolding a narrative about a man on death row in the United States with an all-too-believable tale of a Metropolitan Police officer who marries a woman he is meant to be keeping under surveillance.

The reason the programme attracted so much attention, however, was not what it was about, but whom. Starring Sophie Okonedo and Adrian Lester, Undercover was widely reported as the first mainstream British television drama with black actors in the lead roles. This wasn’t true: as James Cooray Smith wrote on the New Statesman website, that milestone was passed in June 1956 by Mrs Patterson, a BBC adaptation of a Broadway play starring Eartha Kitt.

Yet Undercover was still a breakthrough. Smith, casting his mind back over more than six decades of British television, could not think of more than a handful of other examples. Writing in the Observer, Chitra Ramaswamy expressed her feelings with quiet devastation: “In 2016, it is an outrage that it’s a big deal to see a successful, affluent, complicated black family sit at a ­dinner table eating pasta.” Think about that. In 2016 in Britain, a country where more than nine million people describe themselves as non-white, it is news that a black, middle-class family should not only feature in a prime-time BBC drama but be at its heart. Undercover exposed how white most British television is.

Actors of colour have appeared on British film and TV screens for decades, and they have been visible on British stages for centuries – yet they have been shunted into the margins with depressing regularity. In January the actor Idris Elba urged British MPs to take the matter seriously. “Although there’s a lot of reality TV,” he argued, “TV hasn’t caught up with reality.”

In February, there was renewed uproar over the lack of racial diversity in Hollywood at the 88th Academy Awards, and the infuriated hashtag #OscarsSoWhite blossomed again on social media. A month later, Lenny Henry argued that black and minority ethnic (BAME) talent was being “ghettoised”. The term could hardly be more charged. Speaking at the London premiere of Mira Nair’s film Queen of Katwe, the actor David Oyelowo said: “What we need now is for a change to come. I think the talk is done.”

There has been some change. In March, the Royal Shakespeare Company opened a production of Hamlet starring Paapa Essiedu, an actor of Ghanaian heritage raised in London. It was the first time that a black performer had taken the role for the company. A new set of BBC diversity targets both on- and off-screen was unveiled in April. Noma Dumezweni is playing Hermione in Harry Potter and the Cursed Child in the West End, and in October the BFI launched Black Star, a nationwide season celebrating black talent in film and TV. But what does the picture really look like, in late 2016? And what, if anything, needs to change?

The first challenge is that many in the film and TV industry find it difficult to talk about the subject. Researching this article, I lost count of the number of people who demurred to go on the record, or of actors who seemed eager to speak but were then dissuaded. Fatigue might be partly to blame – it’s exhausting to be asked repeatedly about diversity because you didn’t go to Harrow and your skin isn’t white – but I got the sense that there’s more going on.

One man who passionately believes this is the screenwriter Trix Worrell, the creator of the pioneering Channel 4 sitcom Desmond’s, which brought an African-Caribbean barbershop in south-east ­London to Middle England’s living rooms in the late 1980s and early 1990s.

“TV is very difficult to break into. There’s a protectionism there,” he says with a shrug, when we meet for coffee on the seafront in Hastings, where he now lives. “People are nervous about rocking the boat.”

Though cheerful about most of the things we discuss, Worrell admits to feeling a roiling anger when it comes to this particular matter. Does he think that diversity has improved since he was pitching Desmond’s, three decades ago? “No. I say that with absolute certainty and surety.”

It is hard to underestimate the influence that Desmond’s had. The series ran for 71 episodes and at its peak it had five million viewers, remarkable for a sitcom. Starring the veteran actor Norman Beaton alongside a largely British-Guyanese cast, it made that community visible in a way that has not been rivalled in Britain in the 22 years since it came off air. It did so with the deftest of touches, addressing problems of interracial relationships and tensions within the black community through warm comedy.

“Up to that point, black people were ­never seen on TV,” Worrell recalls. “The only time we appeared in any media was in the red tops – muggings, vice. The idea was to show a black family who were just like any other.” Yet it seems that, apart from the spin-off comedy series Porkpie, occasioned by Beaton’s sudden death in 1994, Channel 4 has regarded the idea of portraying a normal black family in a sitcom as too great a gamble in the years since, despite an increase in the number of non-white roles in its other drama output.

Worrell smiles, but it is clear that the ­matter isn’t a joke. “The thing that’s said among black people is that there’ll only be one black sitcom every ten years.”

***

When I phone Paapa Essiedu while he’s on a lunch break from Hamlet, I am prepared to get a more positive perspective. Just 26, Essiedu has had a spectacular and seemingly unimpeded rise. A graduate of the prestigious Guildhall School of Music and Drama in London, he joined the RSC in 2012 and then hopped to the National Theatre in Sam Mendes’s King Lear, before returning to Stratford. The Telegraph greeted his debut as Hamlet with the notice that every actor dreams of: “A new star is born”.

But Essiedu seems ready to implode with frustration. “It’s ridiculous,” he says. “This stuff has been here for decades and decades: we’re lying to ourselves if we think there’s been a lack of awareness until now. Lots of people are talking and talking, but we need action.” Has he experienced racism directly? “Put it this way: quite often, I’ve been in a room where everyone else is white.”

A major issue, he says, is the apparently unshakeable addiction of British TV and film to corsets-and-cleavage period drama, which has left many BAME actors locked out of the audition room. The BBC is in the middle of a run of literary spin-offs, from War and Peace to The Moonstone. Over on ITV, we have had Victoria and the invincible Downton Abbey.

It still feels as though much of British drama is stuck in an airbrushed version of the country’s past. Though partly set in contemporary Egypt, BBC1’s adaptation of The Night Manager by John le Carré had only a handful of non-white actors in significant roles. Allowing for exceptions such as the BBC’s version of Andrea Levy’s Windrush-era novel Small Island, broadcast in 2009, you could be forgiven for thinking, had you never visited Britain, that people of only one skin colour live in this country. That the largely white drama series are successful on the export market only helps to extend the cycle.

“Producers say, ‘Oh, we commission stuff that people want to watch,’” Essiedu tells me. “But it’s such a narrow version of history – middle-to-upper-class Caucasian men, generally. Period drama can be from anywhere in the world: Africa, Asia. Where are those stories?”

Drama is just a sliver of broadcasting output, but other genres aren’t much better. Journalists from ethnic-minority backgrounds have made steady progress in television newsrooms – but not fast enough, Channel 4’s Krishnan Guru-Murthy has ­argued; there is a glaring absence, however, when it comes to lifestyle and entertainment TV. The recent success of the intrepid youth TV star Reggie Yates notwithstanding, it is difficult to ignore or account for the dearth of BAME presenters in documentaries and “serious” factual programming; and no major current British chat show has a permanent anchor who isn’t white.

Adil Ray’s BBC1 comedy Citizen Khan, which focuses on the escapades of the overbearing Muslim patriarch Mr Khan and his family in the Sparkhill area of Birmingham, is a rare exception. It has just returned for a fifth season. A worthy successor to Desmond’s in its tongue-in-cheek approach to potentially inflammatory issues (the 2014 Christmas special featured the birth of Mr Khan’s grandson, Mohammad, on Christmas Day) the programme also resembles its forebear in a more depressing way: it appears to be one of a kind.

When I ask Ray why he thinks this is, he selects his words carefully. “It’s not prejudice exactly,” he says, “but in the TV business, there are a lot of formulas. If you’re doing curry, get an Asian person. If it’s hip-hop, someone who’s black. If you’re doing a walk in the countryside, or drinking tea in the Cotswolds . . .” He leaves the sentence hanging.

What appears on screen is only the visible part of the problem. Actors get cast in roles only if writers write them; projects get made only if commissioners commission them. TV and film are notoriously incestuous and competitive industries. Careers are unstable. Knowing someone who knows someone is often – too often – the only way of getting work.

According to figures produced this year by Creative Skillset, many media companies fail dismally when it comes to representation. Just 24 per cent of those in senior roles in cable or satellite firms are female; 4 per cent of employees in positions in senior terrestrial broadcast are BAME; and, if the numbers are to be believed, there are no BAME people at all working on the senior production side of independent film companies. The figures aren’t entirely robust – they rely on organisations filling in forms and returning them – but if they’re anywhere near the truth they make for grim reading.

The BBC’s statistics are more encouraging (according to the latest figures, BAME people make up 13.4 per cent of staff overall and hold 9.2 per cent of leadership roles) but don’t include freelancers, an area in which it is reasonable to suppose that, without quotas to fill, representation will be worse. In September, the media regulator Ofcom put broadcasters on notice that they could face “harder-edged” regulation if they did not improve diversity.

Chi Onwurah, the MP for Newcastle upon Tyne Central, who has been vocal about these matters in parliament, says that the BBC has a special duty to up its game. “It’s not doing enough,” she tells me. “If it was, there wouldn’t be a problem. It was very interesting watching the [European Union] referendum; all the efforts broadcasters have gone to to make sure there was balance. If they went to half that effort for BAME, gender and disability, it would be a different world.”

The BBC is keen to show that it is paying attention. Last year, it appointed Tunde Ogungbesan as its new head of “diversity, inclusion and succession”, and in April his team announced eye-catching targets: gender parity across every part of the corporation; 8 per cent of staff disabled; 8 per cent of staff lesbian, gay or trans; 15 per cent of staff from BAME backgrounds. Those numbers will be replicated on screen, lead roles included, and are roughly equivalent to averages for the overall population of Britain.

Yet the idea that established BBC presenters will go quietly seems optimistic. Take the ruckus that the comedian Jon Holmes recently raised when his contract with The Now Show (Radio 4) wasn’t renewed. Holmes asked in the Mail on Sunday: “Should I, as a white man . . . be fired from my job because I am a white man?”

Ogungbesan – a former head of diversity for Shell – has a businesslike attitude to the challenges he faces, which are, he concedes, considerable. “We’ve got four years to do this, and we know there’s a hell of a lot of work to do.” That is why his team has given itself a deadline. “Hopefully, when we hit those targets in 2020, we’ll be the most diverse broadcaster in the UK.”

How does he respond to Onwurah’s suggestion that the BBC is skilled at announcing targets but less good at making change happen? “We’re publishing our results,” he says. “You’ll be able to hold us to it.”

And what if the targets aren’t met? Ogun­gbesan laughs, for perhaps a touch too long. He will not consider the possibility. “I’m like a boxer. I refuse to look at it.”

***

If British TV and film don’t get their act together soon, there may be no one left to cast. Increasingly, black and Asian stars are decamping to America to make their career there. Among those who have joined the brain drain are Archie Panjabi and Cush Jumbo (The Good Wife), David Oyelowo (Selma) and Chiwetel Ejiofor (12 Years a Slave). Idris Elba, who brooded brilliantly in BBC1’s crime procedural Luther, would likely never have been cast in a big British series if he hadn’t already made a name in the United States with The Wire. Before she appeared in Undercover, Sophie Okonedo said in an interview that the scripts she was offered from the US far outnumbered those from the UK.

Visiting Los Angeles recently, I tracked down Parminder Nagra, who made her name in Bend It Like Beckham before being spotted by a producer for the long-running medical drama ER. In 2003 she was offered the role of the Anglo-American doctor Neela Rasgotra, which she played until the series ended in 2009. A big part in the NBC crime drama The Blacklist followed, along with other film and TV work.

She never intended to move, she says, laughing ruefully, when we meet at a café in a well-to-do suburb of LA populated by movie folk. She has worked occasionally elsewhere but, 13 years on, she is still on the west coast. “The jobs I’ve got, like most actors, haven’t come about in a conventional way. It’s generally because someone is open-minded enough to look at you.”

Although she is careful to make it clear that the US is far from a utopia in terms of how it portrays race, sexuality or gender on screen – she tells a gruesome tale of a white writer who sent her his attempt at an “Asian” character – Nagra senses that things are more open in the US. “It’s a bigger pond here, because of the sheer size of the country,” she says. “There are writers of colour in the UK, but what happens is that you’ve only got one or two people at the top who are making decisions about the taste of the country . . . Those people are white.”

The landscape is certainly more open in the US. Leaving aside the allegations about Bill Cosby, NBC’s Cosby Show (1984-92) was a force for good, with its focus on a middle-class African-American family and with the numerous ethnically diverse shows it made possible: A Different World, The Fresh Prince of Bel-Air, In Living Color, Scandal (the last was commissioned by the influential black writer-producer Shonda Rhimes). Back in the early 1980s, the gentle NBC sitcom Gimme a Break! – starring Nell Carter – explored issues of racism, too.

US cable and online subscription ­services are even more courageous. Netflix’s Orange Is the New Black has an ethnically kaleidoscopic cast and plotlines that vault across almost every conceivable question of gender, sexuality, body image and politics. Where it has apparently taken the BBC until 2016 to realise that families can be both black and upper middle class, ABC in the US was years ahead: in 2014 it commissioned Black-ish, which offers a subtle portrait of an advertising executive who frets that he is losing touch with both his Obama-era kids and his inner-city origins.

Nagra nods. “There still are a lot of issues here, but if you’re an actor of colour, there is more work. All those British period dramas are really well done, but there’s a yearning there: ‘Can I please just see somebody like me on TV?’”

The reason all this matters is that TV, theatre and film have a duty to show us not merely who we are, but who we can become. In Undercover, Okonedo becomes Britain’s first black, female director of public prosecutions: this may seem unlikely, given the state of the UK’s judiciary, yet seeing it on TV helps to shift perceptions. No one would argue that Okonedo’s co-star Dennis Haysbert got Barack Obama into the White House by playing a black president of the United States in 24, but perhaps it made such a world marginally more imaginable.

The time is overdue for British TV to abandon its fetish for bodices and show us what our nation actually looks like, in all its variety – and to be more imaginative about the kind of history it presents. Colour-blind casting is mainstream in theatre. Actors of various heritages appear in Pinter or Chekhov and no one raises an eyebrow.

Anthropologists argue that race and gender are forms of performance, sets of shared codes, rather than something intrinsic to who we are. Is it so difficult to imagine a Jane Austen production with performers of black or Asian heritage? Is that any harder to believe than the thousand impossibilities we witness every day in TV drama?

I ask Essiedu if he is optimistic. Yes, he says forcefully. “I have to be. Optimism is the only way we initiate change.”

When I put the same question to Nagra, she pauses to think. “I remember being asked about this when I started ER, and I was a bit tired of the issue even then. Yet here we still are.” Her expression is wry. “So ask me in ten years’ time.”