Questions have swirled for years about Prince Andrew, the Duke of York’s relationship with convicted sex offender Jeffrey Epstein—including an allegation that the royal had sex with a 17-year-old girl who was trafficked by the disgraced financier. The Queen’s second son, with the backing of Buckingham Palace, has always forcefully denied wrongdoing. But just days after Prince Andrew gave a disastrous BBC interview about his relationship with Epstein, which included him saying he did not regret the “very useful” friendship, for the palace to react. On Wednesday, Andrew released a statement saying he would “step back” from public duties for the “foreseeable future.”

“We all think she’s this lovely and charming old lady, but she is [the] monarch and she can get very, very angry. It takes quite a lot to get her angry, and I think that perhaps she feels her son is not telling the truth and perhaps she feels he has to pay for it,” says Ingrid Seward, editor-in-chief of Majesty magazine, adding that the Queen likely consulted her eldest son Prince Charles, who is poised to succeed her on the throne, on what action to take. “I expect [Charles] is absolutely furious, because it is damaging—whatever way you look at it, it’s damaging.”

The hour-long interview on Saturday was meant to put to rest allegations that have surfaced in court documents since at least 2015. Following Epstein’s death by suicide in August, pressure has been mounting on Andrew to answer questions about his relationship with Epstein. Instead, it resulted in the famously-discreet royal family dominating headlines around the world, as well as to U.S. lawyers seeking justice for Epstein’s alleged victims calling for the Duke to answer questions under oath.

Following Wednesday’s announcement, U.S.-based lawyer Gloria Allred, who is representing some of Epstein’s victims, suggested that the Duke could be served with a subpoena if he did not willingly speak to U.S. authorities investigating Epstein.

“It was unsustainable after the interview he gave that he could continue in any sort of public position,” says Stephen Bates, former royal correspondent for the U.K.’s Guardian newspaper. “The fact that he didn’t think it necessary on screen, as his first priority, to sympathize with the victims of Epstein—I would have thought had he taken any advice at all, that would have been the first thing he would have been told to do.”

It’s not the first time that the Duke or other members of the royal family have faced controversy. In a 1994 television documentary, Prince Charles admitted his infidelity during his marriage to Princess Diana. The following year, Diana gave an explosive television interview to the BBC, revealing more details about their marriage. Both instances gained significant, and largely unwelcome, attention from the British press and the public.

But this time round, the scandal is of quite a different nature, with allegations involving sex trafficking by Epstein and calls for law enforcement from the other side of the Atlantic to get involved. Dickie Arbiter, a royal commentator and former press secretary to the Queen, tells TIME that the family hasn’t faced such backlash “on this scale” since Edward VIII abdicated the throne in 1936 to marry American divorcée Wallis Simpson; a scandal that caused a national crisis.

It’s also new territory for the Duke; the royal who was nicknamed “Air Miles Andy” by the British press in the 2000s due to his expenditure on foreign trips is now facing pressure to speak to U.S. prosecutors who are still investigating Epstein’s alleged crimes. “Andrew has always been pretty obtuse. He has a sense of entitlement and generally speaking an imperviousness to criticism,” says Bates. “I suspect what he was expecting was an easier ride than he got, but he probably felt he dealt with it okay, which is just delusional.”

Royal observers say Wednesday’s swift announcement appears to be “damage limitation,” amid several businesses withdrawing their support for Andrew’s charitable endeavors as well as students at the University of Huddersfied lobbying to remove him as chancellor. “Normally the palace doesn’t move that quickly, but because of the appalling publicity surrounding the interview, the Queen and her advisers decided that the only solution to the problem was a quick solution,” says Seward. “To actually have derision heaped upon him like Prince Andrew has [had], I don’t think it’s ever happened before.”

In addition, Arbiter and other experts suggest that Andrew likely didn’t clear the interview in detail with other members of the royal family, or seek the palace’s advice on how to handle the questions.

“As a mother, she will be devastated, and as the Queen, she’ll be annoyed that all this has happened, annoyed with the acres of coverage and annoyed she hadn’t been told more about the interview before it happened,” Arbiter says.

The damage caused by the Duke’s apparent lack of remorse over his friendship with Epstein in Saturday’s interview is extensive. But his withdrawal from public duties is perhaps more of a symbolic move, given his limited existing role in the public eye. “He hasn’t been carrying out many really high profile public engagements for quite a long time,” says Bates, the former Guardian correspondent. “It’s not as if he was seen every day out and about around the country.” In a 2018 survey published in The Times of London, the Duke was reported to have carried out the least amount of public duties over the year, when compared to his three siblings.

Backlash to the interview has made the front pages of British newspapers all week and dominated conversation on social media, likely contributing to the quick timing of Wednesday’s announcement. The question of whether the monarchy is fit for purpose was even raised in this week’s general election debate between current U.K. Prime Minister Boris Johnson and leader of the opposition Jeremy Corbyn.

However, commentators think that while this incident continues to be highly damaging for Andrew in particular, the long-term reputation of the monarchy as a whole is likely to remain intact; last year, a poll pointed to 7 in 10 Brits supporting the monarchy indicating widespread favorable attitudes towards the institution. But the Duke’s role in the Epstein scandal, and his conduct during Saturday’s interview, has undeniably raised questions about the role of the wider royal family at the very least. “The monarch herself is secure and very popular,” says Arbiter. “The question is, how do we cope with the extended family?”

Get The Brief. Sign up to receive the top stories you need to know right now. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Contact us at letters@time.com.