Rupert Graves’s eldest son got a nasty shock recently when he went on a school trip to see the 1987 Merchant Ivory adaptation of EM Forster’s gay love story Maurice. “He didn’t know I was in it,” the actor says over a morning cappuccino in a north London brasserie. “He went, ‘Argh! No way!’ and ran out.” He actually fled the cinema? “Yeah. ‘I can’t watch my dad naked on screen in front of all my mates. Can’t do it.’” He might also wish to steer clear of A Room with a View, which his father made with the same team two years earlier — his film debut, in fact — and in which he is seen throwing off his clothes to cavort in the lake and scamper through the undergrowth. “He’s not prudish. But seeing your dad naked is a lot for a lad to take.”

Some people would be positively thrilled at the prospect of a naked Rupert Graves: his devoted fans, the GravesDiggers, for example, or Mark Gatiss, screenwriter and star of Sherlock, in which Graves plays Inspector Lestrade. “I’ve had a crush on Rupert Graves since I was 14,” Gatiss admitted in 2014, then added: “Who hasn’t?” Graves, 56, who is today wearing round-framed glasses and a bedhead of salt-and-pepper hair, keeps his kit on in his new film, Horrible Histories the Movie: Rotten Romans, a big-screen version of the popular CBBC franchise which comes on like Monty Python Jr. He plays a centurion whose unexpected pauses keep wrong-footing his underlings. Thinking he is telling them to attack, they gather up their weapons in preparation, when what he has really said, if they’d only waited, is: “‘We attack!’ … would be the words of a rash man.” There’s nothing here to embarrass any of Graves’s five children, give or take the rap he delivers on horseback (sample line: “I’ve got 99 problems but a bridge ain’t one”).

Facebook Twitter Pinterest Tough rap … Graves, second left, with fellow Rotten Romans in the Horrible Histories movie. Photograph: Nick Wall

He shrugs when I ask what the secret is to playing comedy. “I haven’t really done any. It’s stylised reality, isn’t it? Picking out bits and exaggerating them. But it’s got to be true.” Of course, he’s been funny in films before. He was superb as the lodger romantically involved with his landlady and her daughter in Intimate Relations. There was that bit, I remind him, where he wanted to accuse Julie Walters’s character of being a hypocrite but he couldn’t think of the word so he ended up shouting: “You’re a hippo!” He smiles back at me blankly. He doesn’t remember it. And last year he was in Swimming With Men, a kind of underwater Full Monty. “Yeah but I wasn’t playing an insane centurion doing a rap,” he laughs. “OK, I was an insane guy with a nose-clip on doing synchronised swimming, but it’s not quite the same.”

If there has been any pigeonholing, it came at the start of his career, when his performances in those Merchant Ivory films, as well as in two by Charles Sturridge, Where Angels Fear to Tread (Forster again) and A Handful of Dust (Waugh), encouraged misconceptions about his background. His name only sealed the deal. “Do you know what? Genuinely, it helps in this business being called Rupert. I’ve sniffed that attitude in acting: the Oxbridge thing. Making movies isn’t a cheap exercise. You need money and the knock-on from that is the industry is populated by a lot of posh people. It’s very hard to break into if you’re not middle class and privately educated.” Or if you don’t have a name that suggests you are. But if being called Rupert boosted his chances, it was a blight on his early years. “I hated it as a kid. I really wished I was called Pete.” Were there any significant Petes in his life? “Only Peter Purves on Blue Peter. Rupert is a ridiculous name in 1970s Weston-super-Mare. Ridiculous! It’s like being called Basil.”

Facebook Twitter Pinterest ‘No way, dad!’ … with Simon Callow and Julian Sands in A Room With a View. Photograph: Allstar/Goldcrest Films

Weston-super-Mare was a decaying, druggy sort of place with no hope. I was always being chased by skinheads

Graves speaks in short bursts and clipped sentences as though he’s slightly out of breath, but he is at his most fluid when conjuring up the landscape of his youth. “It was one of these strange and dying seaside towns where the package holiday had killed off most of the trade, so the hotels had to lower their prices. It was the end of a line. A decaying, Thatcherite, druggy sort of place with no investment and no hope. I was always being chased by skinheads. It had some peculiar qualities, being by the sea and near Cheddar, some beautiful land. And it was blasted by the weather, which comes up the Bristol Channel and thumps Weston first. As a kid, it was great to hide behind the buildings as waves would smack them. The weather was physical and exciting. It gave the town character.”

At school, he was dreadfully shy and suffered from a stammer, but was also an incorrigible show-off. “I was hyper. My shyness meant I overcompensated, just to get anything out of my mouth at all. I had to summon up so much courage to even speak and then I got carried away because I’d mustered all that energy.” He went to a speech therapist to overcome the stammer and it was she who spotted his acting potential. Turns with the local operatic society followed; he also performed at the end of the pier, miming to songs and delivering Victorian comic monologues. What did he get out of it? “The warmth of the approval.” He found himself an agent from the back of the Stage (“I don’t even think we met”) and took on bits and bobs of TV work while still at school, including his turn as a prefect — his first toff — in the spy series Return of the Saint.

Facebook Twitter Pinterest Banshee-like … as a cross-dressing criminal in Open Fire. Photograph: United Archives GmbH/Alamy

He adored working with James Ivory. “He would say things like, ‘Put your pipe in upside down.’ ‘Why?’ ‘Because it’s funny.’ He was right. It was funny. In a Magritte sort of way.” But people started cooling on the costume-drama craze at the end of the 1980s. “There was a backlash. I thought, ‘This is really unfair!’ I didn’t go to any fucking posh schools and I was getting lumped in with that.” The director Alan Parker famously derided the Merchant Ivory output as “the Laura Ashley school of film-making.” Graves scoffs. “That was horrible. Really stupid. Forster was actually trying to say something.”

But he was looking for a new direction around that time anyway. “The whole industry thought I was Posh Rupert and I wasn’t. I didn’t know people who lived in Fulham and Chelsea, I didn’t know what they thought or what they ate for breakfast.” He reacted by taking on parts which were worlds away from parasols and linen trousers. The most notable was Open Fire, a 1994 TV film written and directed by Paul Greengrass, in which Graves was ferocious and banshee-like, all nails and hair and heels, as the real-life cross-dressing criminal David Martin. He also played a laddish motorcycle courier reunited with a transgender school pal in Different for Girls. It was theatre that really excited him, though, and still does. “You don’t tend to get typecast in theatre.” He was in Torch Song Trilogy with Anthony Sher, the Broadway run of Patrick Marber’s Closer, Pinter’s The Caretaker with Michael Gambon. When I ask whether there is any screen work that he would like people to catch up with, he says: “I can think of plays.” But they’re gone. “Argh! I know. That’s the beauty of it. And the pain.”

Facebook Twitter Pinterest Beauty and pain … Graves with Michael Gambon in Pinter’s The Caretaker in 2000. Photograph: Tristram Kenton/The Guardian

I'm not good at being polite to the right people. But I find [networking] a bit grubby

He thinks he could have been a bigger star if he’d worked at that side of things. “I’m just not ambitious in a career-building way. I’m not good at being polite to the right people. I’m not strategically engaged.” He can’t think when he last did something that would count as networking. “People do it. But I never did. I find it a bit grubby. I can see the benefits. I’ve been lucky that I can keep afloat and not be either too hassled on the street or have to do any of the business side of things, which I’m shit at.” Afloat is an understatement: he has excelled in some of Sally Wainwright’s best work (as a dodgy barrister in the first series of Scott & Bailey and as Derek Jacobi’s long-lost son in the third Last Tango in Halifax), and appeared on stage in the acclaimed season of Pinter’s short plays at the start of this year. He will soon be seen as Mr Weston in a starry new film of Jane Austen’s Emma, alongside Anya Taylor-Joy, Johnny Flynn, Bill Nighy and Miranda Hart.

Throughout our conversation, he has been charming company while also giving the impression of a runner on the starting blocks, poised to scarper the second he hears the pistol. Once we’ve wrapped up, he rises to his feet quickly and knocks a knife off the table, catching it before it hits the floor. “That’s the thing about getting old,” he says, returning the cutlery to its place. “I’ve become a dropper. I’m always dropping things.” Is he a forgetter yet? “Oh yeah. But I’ve always been a forgetter. Never remember names. Not even in scripts.” And there I was expecting you to recall a line of dialogue from nearly 25 years ago, I say. He laughs and makes a noise I’ve never heard before — “Nyyyyyyr! Nyyyyyyr!” — and then — on your marks, get set — he’s gone.