Future historians may conclude that Prince Andrew’s defining achievement was to gift the nation a new verb.

Following a tumultuous week when his car-crash interview shook the House of Windsor so vigorously it seemed its palaces were in danger of losing their crenellations, the Duke of York now finds himself banished from duties. His fate is the 21st-century equivalent of that which befell the difficult minor royals of previous eras who were locked up in asylums, away from the public gaze.

“Prince Andrew has been de-royaled, if there is such a word,” said the historian and biographer Robert Lacey, an adviser to the acclaimed Netflix series The Crown. “At the risk of sounding melodramatic, I really would compare it to 1936 and the abdication of Edward VIII. What we are talking about is effectively the removal of a member of the royal family as a result of public opinion.”

The duke can take some comfort from the knowledge that if he had been around a few hundred years ago, things could have been worse. “One can even compare it to 1649, when Charles I was executed,” Lacey said. “This is a reminder that what was an institution of absolute power now depends ultimately on the consent and approval of the communities it seeks to represent, and Prince Andrew failed in this respect.”

They should have just sent him off to Australia. That would have been a bloody good idea. Out of sight, out of mind Palace source

Arrogant, aloof and slow-witted, according to some who have encountered him, the duke’s interview last weekend with Newsnight’s Emily Maitlis is now seen as a textbook example of how not to conduct a damage-limitation exercise.

Perhaps he should have heeded the lessons of history. Princess Diana’s bombshell Panorama confession did huge damage to the royal family. Prince Charles’s decision to admit to adultery via a television interview greatly reduced his standing in the eyes of the public.

“Andrew is a bit of a plonker, everybody knows that,” said one source close to the palace. “There’s no way he should have been allowed to do that interview. They should have just sent him off to Australia. That would have been a bloody good idea. Out of sight, out of mind.”

The duke’s former adviser Jason Stein, who resigned shortly before the broadcast after only a month in the job, was one rare voice who was opposed to the interview. But Andrew’s private secretary, Amanda Thirsk, who has now been moved from her post, disagreed with Stein, seeing the interview as a chance for the duke to put some distance between him and his late friend, the disgraced sex offender Jeffrey Epstein.

Ultimately, the decision came down to the duke and, confident in his abilities to swing public opinion, he agreed to be interviewed. It was to prove a catastrophic example of hubris.

“What he has managed to do is achieve the worst of both worlds,” one lawyer said. “He has done the interview, but he’s left lots of things unanswered. And he has not given a coherent explanation of what happened.”

It appears Buckingham Palace was incapable of spotting the bear traps that the interview presented – a sign, according to insiders, of institutional decay. “There’s a lack of discipline there at the moment,” the source said. “Sir Christopher Geidt [the Queen’s private secretary from 2007 to 2017] was a real steadying hand. But he’s not there now.”

Prince Andrew during his controversial Newsnight interview. Photograph: BBC

Geidt, a battle-hardened former soldier, was seen as a good adviser to the monarch. “He had the measure of her,” the source said.

Others who might have been prepared to speak truth to power are also long gone.

Two decades ago, as the royal family sought to modernise post-Diana, it brought in several sharp-witted outsiders, including former No 10 press adviser Colleen Harris and Mark Bolland, a PR executive, to act as troubleshooters. “But the family hated them,” the source said. “Because they imposed quite a lot of discipline. I do think some of the staffing has gone a bit awry. They don’t have people who can stand up to them now. They’ve gone back to the Hooray Henrys who go ‘yah, yah, yah’ and don’t confront them properly.”

This would not be such a problem if there was leadership at the top. But the Duke of Edinburgh is frail and has retired from the fray. The Queen is 93.

“The royals are like most families,” the source said. “You need a head of the family to step in – somebody tough and stern – who sorts it all out, but they don’t have anyone at the moment. It’s all sliding again. The Queen is just tired. It needs a stronger hand at the helm.”

From the Duke of Edinburgh’s infamous car crash to the apparently growing divide between princes William and Harry, the royal family could be forgiven for wishing to see the back of 2019. That the Queen felt she had no choice but to sacrifice Andrew, whom many consider her favourite child, will have made it a particularly bleak year.

“The circumstances suggest he may well have been her favourite,” Lacey said. “It is well established that the Queen regretted that her early accession to the throne did not enable her to parent Charles and Anne as she would have liked, nor to enjoy the experience of having children. It’s no secret that she and Prince Philip took the decision that they would like a second, younger family and Andrew was the first of those.”

That the Prince of Wales was heavily involved in the decision to excommunicate his brother is evidence of a monarchy in flux.

“This will be seen as the moment that marks the transition from one reign to another, when Prince Charles clearly stepped in,” Lacey said. “Although Prince Andrew was allowed to say he jumped, he was in fact pushed. The royal family are absolutely ruthless when it comes to protecting the institution.”

But a monarchy in transition can be a fragile, fractious thing.

“The Prince of Wales has got his team,” a royal source said. “He has always done his own thing, but now they are looking across the park and want more of a say, and this causes friction.”

Lacey believes it is significant that the Prince of Wales was away on royal duties when the decision to use Buckingham Palace for the interview was taken.

“With Prince Philip out of the scene, Prince Andrew had effectively become the man in charge of the palace,” Lacey said. “From the inquiries one can make, it is not really certain that the Queen gave her personal approval for the use of the palace.”

Prince Andrew and his former wife, Sarah Ferguson, at Royal Ascot this year. Photograph: Tim Rooke/Shutterstock

The choice of location was both telling and a substantial error of judgment, according to Lacey.

“In a way, it [the ensuing post-interview furore] is nothing to do at all with whatever Prince Andrew’s relations were or were not with Jeffrey Epstein. It all starts with last Saturday night and a projection of his attitudes.

“Most people do not feel they want to be represented by someone like that. It was made worse by his decision to use Buckingham Palace as a backdrop. It invited questions and confrontation. It went along with the lack of contrition in what he said, and in his tone.”

As it became apparent just how far Andrew’s interview had backfired, the palace drafted a statement that it was hoped would paint the duke in a more empathetic light. Announcing his intention to stand down from public duties, the duke confirmed that he was willing “to help any appropriate law enforcement agency with their investigations, if required”.

But, as with his television interview, the response raised more questions than it answered.

“He’s not committing to giving evidence to anyone who asks him. It’s carefully caveated – deliberately so,” said one legal expert who asked to remain anonymous. “The duke says ‘if required’ – so he’s not prepared to do it voluntarily. It may well be a get-out. Epstein is dead, so there is not going to be a criminal prosecution of him. It may well be that law enforcement agencies are looking at prosecuting his associates involved in the trafficking of young women, but there may end up being no criminal prosecution.”

Epstein’s victims are pursuing civil damages claims against the financier’s estate, but lawyers believe it is significant that the duke has not signalled any desire to help them by agreeing to provide a witness testimony. “If he’s genuinely concerned about helping victims, he needs to commit to assisting those who are representing them, as they have called upon him to do,” said Richard Scorer, a specialist in sexual abuse cases at the law firm Slater and Gordon. “At the moment, his statement doesn’t include that.”

John Cooper QC, a barrister who knows about rehabilitating reputations, having sought to have the conviction of Dr Crippen overturned on the strength of new DNA evidence, suggested there was no reason why the duke would not accede to the lawyers’ demands.

The spectre of Epstein will loom large over the duke for years to come

“Hopefully, as a man of public duty, he would be quite content to help them,” Cooper said.

The now notorious photograph of the duke with his arm around the waist of one of Epstein’s teenage victims, Virginia Roberts, apparently taken in 2001 at the London mews home of Epstein’s girlfriend Ghislaine Maxwell, has been cited as evidence that he could provide valuable insight.

But there are claims the photo is a fake. The duke used his Newsnight interview to say he had no recollection of it being taken, that he did not recall meeting Roberts, and to categorically deny having sex with her.

In 2015 the Metropolitan Police conducted a review of “available evidence” after receiving a complaint over claims lodged in court papers that a girl – now known to be Roberts – was “forced to have sex with Prince Andrew”. But the Met decided that the matter “would not progress to a full investigation”, according to Channel 4 News.

That decision perplexes some in legal circles. Several experts spoken to by the Observer point out that, in 2015, Scotland Yard was throwing substantial resources into investigating claims of a Westminster paedophile ring. “What was different about this claim?” asked one.

A spokeswoman for Scotland Yard told the Observer: “We acknowledge the considerable interest and concern around this case and have revisited that decision-making and believe it remains entirely appropriate. Therefore no further action is being taken. The Met will always take seriously any allegation concerning sexual exploitation.” But until the duke provides testimony under oath, it is likely that the controversy surrounding the photo will continue to haunt him.

“Can the prince be compelled to give testimony under oath in a civil case?” asked Dominique Penson, a New York-based attorney who has handled many sex abuse claims. “The answer is yes. If he’s compelled to give a deposition, they are taken under oath.”

Penson has served subpoenas on witnesses in European countries compelling them to give depositions for cases heard in the US, using a supranational legal mechanism called the Hague Treaty.

“He could be served with a subpoena,” Penson said. “He would not be compelled to come back here to give testimony but he could provide sworn testimony in the UK.”

Lawyers may also want to obtain documents that confirm the duke’s whereabouts on key dates. He was provided with a security detail who were supposed to keep logs of his movements and carry out risk assessments of people he met. Do they exist and if so where are they?

With his television interview and his subsequent statement leaving so many unanswered questions, it seems that the spectre of Epstein will loom large over the duke for years to come – years in which an enfeebled monarchy will attempt to redefine itself under a new king.

But it may be that the duke is now beyond rehabilitation.

“I suppose he could try to do a Mother Teresa – or the classic example is John Profumo, who is generally said to have redeemed himself,” Lacey said. “But it would have to be a pretty massive and visible act of redemption for people to say ‘ah yes, well, we’d love to have Prince Andrew as our patron’. And I simply can’t see that.”

The centuries of scandal

This is not the first time the British monarchy has been rocked by scandal. Even before social media and the tabloid press, the royals were regularly the object of public outrage and ridicule. Here, Kate Williams, professor of history at Reading University, looks at how the palace has dealt with disgrace in the past.

Frederick, Duke of York, 1809

Frederick, Duke of York.

George III had seven sons and six daughters. By 1800, they had between them 56 illegitimate children and one legitimate child, Charlotte, daughter of the Prince of Wales. The brothers and sisters generated scandal apace – including the Prince of Wales’s shocking attempts to divorce his wife for adultery. But society was rocked by the acts of his brother, Frederick, duke of York.

His mistress, Mary Anne Clarke, needed money to maintain her position and began selling army positions. The married duke was the commander-in-chief of the armed forces – thus the nursery rhyme, The Grand Old Duke of York – so this was corruption on a huge scale. MPs investigated and Clarke appeared looking marvellously chic and read out the duke’s love letters.

He was widely mocked and derided and was forced to give up his position and live quietly away from the public glare – no more mistresses.

Edward VII, 1869

Queen Victoria’s eldest son, known popularly as “Bertie” was spoiled and debauched. He visited brothels, openly consorted with women and had dozens of mistresses, including Alice Keppel, the great-grandmother of Camilla Parker Bowles – who was even invited to his deathbed – and the actress Lillie Langtry.

In 1869, the young Scottish aristocrat Harriet Mordaunt gave birth to a girl who seemed as if she might be blind and Lady Mordaunt panicked about sexually transmitted diseases. She confessed to her husband her various affairs, including the prince, and Lord Mordaunt said he would divorce herfor adultery. A prince called as the third party in a divorce case, would be a real threat to the monarchy.

Harriet’s family also feared the scandal and so she was declared insane and locked in an asylum for the rest of her life. Bertie’s public reputation was saved – and he acceded to the throne after Victoria’s death in 1901.

Edward VIII, 1936

Edward VIII, as the Duke of Windsor, after his wedding to Wallis Simpson.

In the 1920s, Edward, Prince of Wales, handsome, youthful and idealistic, was seen as the future of the monarchy. He was fond of older married women and by his mid-thirties had fallen in love with a witty American, Mrs Wallis Simpson. When he ascended the throne in 1936, he began to talk seriously of marriage. Prime minister Stanley Baldwin told him it was politically impossible for him to marry the soon to be twice-divorced Simpson – the Church of England would not accept it and public opinion would be vehemently opposed.

The king was on a collision course with his ministers and Baldwin feared that the matter would bring down his government. Finally, Edward decided to abdicate and, on 10 December 1936, made the announcement to the nation.

He was essentially exiled for the rest of his life and died in his home near Paris in 1972.

Divorce, 1978

Princess Margaret’s marriage to society photographer Lord Snowdon was crumbling when she met Roddy Llewellyn a handsome young aristocratic landscape gardener, who, at 25, was 17 years her junior.

They began a passionate love affair, much of it conducted at her beautiful villa in Mustique. In February 1976, the News of the World published front-page pictures of Margaret and Roddy enjoying themselves on the beach – and a huge scandal erupted.

Snowdon insisted on divorce (despite his own many affairs) and the press and many politicians attacked Margaret for being a “floozy” and a “royal parasite” and there was talk of taking her off the civil list.It was the first royal divorce since Princess Victoria Melita in 1901 (who had a very unhappy marriage).

Margaret was devastated and heartbroken and, although Roddy eventually married another, her public reputation suffered greatly.

Annus horribilis, 1992

A series of tapes secretly recorded by Princess Diana detailing the breakdown of her marriage to Prince Charles were the main source for Andrew Morton’s 1992 bestseller, Diana: Her True Story.

As she herself put it, 1992 was the Queen’s annus horribilis. In spring, the Duke of York and Sarah Ferguson announced their separation – and the notorious “toe-sucking” photographs of Sarah sunbathing with her financial adviser caused huge scandal in the summer. Then the marriage of Charles and Diana, the Prince and Princess of Wales, imploded.

The bombshell book, Diana, Her True Story, by Andrew Morton was published in May 1992, laying bare Diana’s suicide attempts, eating disorders and suspicions about Camilla Parker Bowles. Then the leaks began – a phone exchange between Diana and James Gilbey, who called her “Squidgy”, detailed in the Sun and a transcript of an intimate conversation between Camilla and Prince Charles. In December of that year, prime minister John Major announced the “amicable separation” of the pair.

The war of the Windsors continued and the Queen pushed them to divorce in 1995. The divorce was supposed to stop the talk; it did not. Diana continued to attract a huge amount of press wherever she went. Her death in 1997 turned what had been a royal scandal into a tragedy.