“Shagging Eddie Izzard isn’t something that I envisaged would ever happen in my career,” says Olivia Williams with a throaty chuckle. We’ve been discussing the movie that she’s just completed, Victoria And Abdul, penned by Billy Elliott writer Lee Hall, directed by Stephen Frears (The Queen, Philomena), with Judi Dench in the lead role.

“I’m afraid it’s another Queen Victoria story,” says Williams, “but a really interesting angle because she had another servant” – other than John Brown, that is, the subject of the 1997 film Mrs Brown. He was the Indian servant Abdul Kareem – a young Muslim who taught the Empress of India Urdu, and exchanged letters with her that were destroyed when she died.

“I play Lady Churchill... an amalgamation of two Lady Churchills,” says Williams. “There was one who was a lady-in-waiting, which is the one I was largely playing, and Lady Randolph, who ended up being Winston Churchill’s mother, but who was actually shagging Bertie the future king [Edward VII, played by Izzard]. They just wanted to get a bit of shagging in basically.”

Williams sounds self-deprecating about her role. This suggests she has few illusions about the film industry – she has seen the business from just about every side. One early break was co-starring with Kevin Costner in the 1997 post-apocalyptic flop The Postman. She was the object of Bill Murray's affections in Wes Anderson’s 1998 film Rushmore. A year later she played Bruce Willis’s wife in box office hit The Sixth Sense. More recently she appeared as Carey Mulligan’s teacher in 2009’s An Education, and then as the Cherie Blair-like figure in Roman Polanski’s The Ghost Writer in 2010.

A scene from film 'Sixth Sense', with Olivia Williams and Bruce Willis

“There's a new enemy on the block, which is ‘business affairs’,” says Williams. They are “the people who negotiate your deal”. “The producers can want you, and the writer and the director can want you, and the other lead actor wants you, so it all ‘yes, yes, yes’. And then the agent starts dealing with ‘business affairs’ and they say ‘your last three movies haven’t made any money... you’re this age... your last income was this, so we’re not paying you that because that movie didn’t make any money. And we’re not paying for your flight. It’s the opposite of that Andie MacDowell [L'Oréal] advert... ‘because you’re not worth it’.”

Perhaps it’s unsurprising then that Williams, like so many thesps who have established themselves primarily as movie actors, has been lured in by the ongoing golden age of television drama, completing two seasons – 23 episodes – of Manhattan, the US TV drama set in 1943, among the scientists at the Manhattan Project, who developed the first atomic bomb.

“Long-form television... this astounding thing where they give roles to 48-year-old women that would make a teenager beg,” she says. “That did not happen when I was young. Women weren’t protagonists, they weren’t doctors and lawyers, and they are now. I got the last of the rich seam in movies and the first of the rich seam in television. I’m so very lucky and I don't underestimate that.”

Williams is remaining in the 1940s for her latest role, in ITV’s The Halcyon, playing Lady Priscilla Hamilton, the long-suffering wife of the owner of a grand London hotel during the Blitz. Her husband, played by Alex Jennings, is a serial philanderer, while the well-heeled guests mainly despise Winston Churchill. This is the Downton Abbey upper-class compromised by appeasement – a spot more Kazuo Ishiguro than Julian Fellowes – and as such, perhaps not as wholesome as ITV’s world-beating ratings juggernaut.

“That’s what I liked about the show”, says Williams. “It wasn't just trying to play the Dunkirk spirit, there are also Mosleyites and Love in a Cold Climate [the 1949 Nancy Mitford novel about an eccentric upper-class family], and there also the people who weren’t doing good works during the war but dancing their way out of misery.

“According to my 101-year-old step-grandmother-in-law, you kept dancing. She would come down to the Hammersmith Palais to dance with black Canadians till her feet bled, and when bombs fell some people would shout ‘take cover’ and crawl under the tables while others kept dancing.”

Whether or not The Halcyon fulfils ITV’s hopes of being a new Downton, Williams herself has never seen a single episode of the Crawley family saga. “I simply read a script that seemed very dynamic, it was talking about swooping into a room with great music playing, through lots of bodies dancing. From what I understand about Downton, that’s not how many episodes of its began.

But like Downton Abbey – and indeed ITV’s recent Victoria – the cast of characters include both upstairs and downstairs – the bellboys and doormen. Somewhere in the middle is a general manager played, with his usual sly demeanour, by the great Steven Mackintosh. “People left the country because there weren't enough people to staff the houses and moved into hotels in London”, says Williams. “Lady Priscilla’s not very likeable, which I like about her. ‘The customer is always right’ is not a phrase that would come easily to her lips.”

Williams herself was born in London in 1968, her parents both barristers. After graduating in English from Cambridge, she studied drama for two years at the Bristol Old Vic Theatre before working for a further three at the Royal Shakespeare Company. “The astounding thing about the RSC was how much of your time you spent just acting,” she says. “You were acting solidly from ten in the morning until ten-thirty at night, three plays on the go at one time... I acted more in those three years than I ever will again. You go to a movie and you’re lucky if you clock up four minutes of acting a day.

“A lot of actors I’ve met – particularly financially successful actors – don’t like acting. It’s very odd. And they seem to spend a lot of their time quite unhappy, and I realise that a lot of people of set out on this route to change their lives, to become someone else, and [I] think that’s very dangerous.”

Once asked in a newspaper questionnaire ‘What advice would you give a young actor?’, Williams replied: ‘Do something else’. Was she being merely facetious, or is there some truth in her jest? “The bitchy answer is ‘the business is overcrowded – don’t join it and take away my work’,” she says. “But then there’s the serious thing that I’m not sure it’s a path to happiness. It’s made me very happy and I can wish that for someone else, but it’s a crap-chute.

“If you’re sensitive you can go under. I have a wonderful family and group of friends and resources that I was given by my privileged upbringing and education. But I also made some resolutions, like I’m not ever paying for a flight to LA – and I don’t want to be in a place where all I have is a hotel room where I’m waiting for my agent to call. How many people do you hear about who go under in a hotel room?”

Her next flight to Los Angeles has been paid for by the increasingly ambitious Starz channel, as she relocates to California to film a new American TV drama Counterpart, which is described as “an espionage thriller with a metaphysical twist”. JK Simmons, the Oscar-winning actor from Whiplash, stars as a man who discovers that the UN agency he works for is guarding a secret: a crossing to a parallel dimension.

“It’s science-fiction-y, alternative universe with an interesting Scandi-noir feel about it,” says Williams. “You get to play yourself on both sides of the divide, so you get to play two people, and the portal to the parallel universe is inconveniently – for an American TV show – in the former Soviet Berlin.”

Married to the American-born British actor Rashan Stone, with whom she has two children, aged nine and 12, Williams says that when she watches herself on the screen, she tends not to get involved with the story, but to treat them as home movies, with memories of people and places. “For me it’s like a really expensive movie of a great holiday... doing Victoria And Abdul we’d been to some astounding places – we shot up in the Cairngorms in Scotland.

“Sometimes the worst movies produce the best memories. Paul Bettany’s a dear friend and he and I made what I thought was an excellent film, Dead Babies from a Martin Amis novel, and which died an ignominious death, but I wouldn’t have missed it for the world. And we filmed Last Days On Mars – a zombie Mars movie – in Jordan, and you can’t go to Jordan now.”

It doesn't sound like a bad life, really. “Yes,” says Williams. “ Although I do want to eliminate ‘business affairs’...”