The royal family is no stranger to scandal, but the suicide of Prince Andrew’s friend, accused pedophile and convicted sex offender Jeffrey Epstein has ruffled feathers at the highest level. The royal family is braced for more damaging headlines as the FBI prepares to investigate Ghislaine Maxwell, Epstein’s former girlfriend and a close friend of Prince Andrew’s.

While Buckingham Palace has attempted to distance Prince Andrew from the disgraced businessman, who was accused of trafficking girls as young as 14 for sex with wealthy businessmen, Andrew’s connections with Epstein cannot be erased. The Duke of York was dragged into the scandal in 2015, when Virginia Roberts Giuffre claimed that she was forced to have underage sex with the duke in 2001. The duke, who can be seen with his arm around Giuffre, then 17, in a picture taken at the time, has previously said that the allegation is “categorically untrue.” Over the weekend Buckingham Palace released another statement, reading in part, “It is emphatically denied that The Duke of York had any form of sexual contact or relationship with Virginia Roberts. Any claim to the contrary is false and without foundation.”

Andrew was at Balmoral, the Queen’s Scottish residence, when news broke of Epstein’s suicide in a New York jail cell on Saturday. The duke was with his ex-wife, Sarah Ferguson (who previously accepted £15,000 from Epstein to clear some of her debts, which she says she regrets), and his daughters, Princesses Beatrice and Eugenie, when he learned of Epstein’s suicide, and sources say they have been “his rock.” Meanwhile the Queen was photographed traveling with Andrew to church on Sunday, while the Duke of Edinburgh also travelled to Aberdeenshire earlier than expected, in a show of solidarity. Courtiers are understood to have discreetly removed newspapers from the breakfast room at Balmoral to spare Andrew any embarrassment.

But despite the denials and show of unity, the duke’s friendship with Epstein has ruffled feathers at the highest level. According to one source, Prince Charles is less than happy that the royal family has been dragged into such a tawdry story.

“Andrew welcomed Epstein at Windsor Castle, Sandringham, and Balmoral at different points during their friendship, which led to tensions between him and Charles,” according to the source. “Charles has disapproved of quite a lot of Andrew’s friends and acquaintances over the years, and Epstein is no exception. I was told he was very concerned about the friendship for a long time and saw it as a disaster waiting to happen. This story isn’t going away, and no one knows what will come out next.”

The Queen and Prince Philip, meanwhile, continue to accept Andrew’s denials that he participated in anything illegal. “Andrew has had not easy conversations with both his mother and father, but he has always insisted he has done nothing wrong and that the charges are fallacious. His parents believe him,” adds the source.

That would explain the Queen’s warm smile and show of solidarity on Sunday, as news about Epstein’s death dominated headlines around the world. It was interpreted by some as an act of defiance, which upset some royalists who felt it was poor judgement on the Queen’s part.

In the U.K. a handful of British M.P.s have called for a full investigation into the duke’s friendship with Epstein. Given his senior position in the royal family, some feel that the duke must explain his friendship with the late sex offender.

Concerned courtiers, meanwhile, are hoping that the story will move on without the prince being a part of the narrative.

More Great Stories from Vanity Fair

— Our September cover story: how Kristen Stewart keeps cool

— Marianne Williamson explains her brand of magical thinking

— The surprisingly normal way Prince George celebrated his sixth birthday

— Lil Nas X breaks a major record—and drops some golden tweets too

— Why Samantha Morton doesn’t regret working with Woody Allen

Looking for more? Sign up for our daily newsletter and never miss a story.