The unprecedented banishment of Prince Andrew to the outer reaches of the royal firm, without a useful future role or official engagements, free – presumably – to spend more time with his golf clubs, comes at a critical time for the family.

After the calamities of the 1990s and the patient rebuilding of reputation that followed, all seemed set fair only 18 months ago. The Queen was still soldiering stoically on into her 90s, working on her red boxes of official papers every day as she had done for the previous seven decades and attending 293 engagements during the year. Prince Harry had just married Meghan Markle to popular acclaim, seemingly inaugurating a new era for the royal family. And the succession was secure to three generations, if they played their cards right.

How easily destabilising uncertainties can intrude. From the Duke of Edinburgh’s car crash in January, through to Harry and Meghan’s evident unhappiness with their royal role and their fissure with Prince William and Duchess Kate, the great hopes of the royal family, it has been a difficult year, if not yet another annus horribilis like 1992. On top of that came Andrew’s self-inflicted crisis, which has angered other members of the family, especially Prince Charles, reopening old sores and throwing the smooth running of the institution into chaos.

Andrew’s fall from grace has had a slow motion inevitability. His relationship with the convicted child sex offender Jeffrey Epstein had been known since the pair were pictured strolling together in Central Park in 2011, but was ignored, especially once a Florida judge ruled that the allegations involving the prince were unnecessary to know. But following the latest complaints about Epstein and his suicide in prison in August, Andrew’s association with him was bound to float into view once more, even though no criminal allegations have been laid against him and he denies any personal wrongdoing.

His ill-judged television interview last weekend precipitated his fall, highlighting as it did well documented flaws: his sense of entitlement, freeloading, obtuseness and insensitivity. The British public can bear a lot from its royal family but resent extravagance and arrogance, especially over the expenditure of money and particularly from someone whose dedication to the common good has been sometimes hard to detect.

Prince Andrew at a British trade fair in Kuwait in 2000. Photograph: Yasser Al-Zayyat/AFP via Getty Images

The prince’s timing was also at fault: distracting attention from the general election (which royals are not supposed to do) and, from Charles’s point of view, deflecting more positive publicity from the Waleses’ visit to New Zealand – not that that was generating much excitement anyway.

Making matters worse: Andrew did not have the excuse that the interview was sprung on him. It was his decision to take part, evidently without proper briefing, rehearsal or forethought. Perhaps he believed he could wing it, despite his ill-concealed disdain for the media and the warnings of his newly appointed communications adviser Jason Stein, who lasted four weeks before leaving for the City PR firm Finsbury. The future of his private secretary, Amanda Thirsk, who lobbied in favour of the interview as a clear-the-air exercise must now be in doubt.

The royal family operates to a complex grid system, not only in terms of official visits (“whose turn is it to go to the west country? ... when was the last visit to Leicester?”) but also in their association with particular interests: William and Kate with the arts, Camilla, duchess of Cornwall with health and literacy issues.

Andrew’s turf was business. For a time he relished the title of British trade ambassador, giving him scope to travel in some comfort and no little expense by chartered jet to often agreeable parts of the world, a role he continued to carry out until this month, though he lost the title some time ago because of his association with Epstein. If the trade results were nebulous – did foreigners really sign deals because Prince Andrew had turned up at a British embassy reception? – the contacts did him no harm at all. One such is Timur Kulibayev, the son-in-law of the former dictator of Kazakhstan, who bought the Yorks’ derelict former marital home near Ascot for £15m – £3m above the asking price – in 2008.

Years of the media ridiculing the cost of his trips – last year’s best effort was £22,208 for a three-day visit to Hungary – had no effect whatsoever. “The trade role was perfect for him because he felt he was having an impact,” a former palace official told me. “The problem with Yorkie is he can be very arrogant and petulant. I think that’s down to insecurity.”

Timeline Jeffrey Epstein, Ghislaine Maxwell and Prince Andrew Show In the early 90s, Ghislaine Maxwell, the daughter of British media tycoon Robert Maxwell, met investment banker and financier Jeffrey Epstein. Their relationship was initially romantic, but it evolved into something more akin to that of Maxwell being a close friend, confidante and personal assistant. The Duke of York, Prince Andrew, was reportedly introduced to Epstein through their mutual friend Maxwell in 1999, and Epstein reportedly visited the Queen’s private retreat in Aberdeenshire. Some have suggested the introduction was made earlier. A 2011 letter to the Times of London from the prince’s then private secretary, Alastair Watson, suggests Andrew and Epstein knew each other from the early 90s. Andrew, Maxwell and Epstein are seen together at Donald Trump’s Mar-a-Lago club in Florida. Later that year, Epstein and Maxwell attend a joint birthday party at Windsor Castle hosted by the Queen. Andrew and Epstein holiday together and are pictured on a yacht in Phuket, Thailand, surrounded by topless women. The Times of London reported the prince’s holiday was paid for by Epstein. In the same year, Virginia Giuffre, then 17, claims to have had sex with Andrew in Maxwell’s home in Belgravia, London. Giuffre, whose surname was Roberts at the time of the alleged incidents, says she slept with Andrew twice more, at Epstein’s New York home and at an “orgy” on his private island in the Caribbean. Epstein is jailed for 18 months by a Florida state court after pleading guilty to prostituting minors. Soon after his release, Epstein is visited by Andrew in New York. The pair are photographed together in Central Park. Footage emerges years later, reportedly shot on 6 December, that appears to show Andrew inside Epstein’s Manhattan mansion waving goodbye to a woman from behind a door. Andrew quits his role as UK trade envoy following a furore over the Central Park photos. Allegations that Andrew had sex with Giuffre emerge in court documents in Florida related to Epstein. The papers say she was forced to have sex with Andrew when she was 17, which is under the age of consent under Florida law. Buckingham Palace denies the allegations. The claims against Andrew are later struck from US civil court records following a federal judge’s ruling. Andrew is accused of sexual impropriety by a second alleged Epstein victim, Johanna Sjoberg. She claims he touched her breast at the billionaire’s Manhattan apartment in 2001. Buckingham Palace says the allegations are 'categorically untrue'. Epstein is found dead in his jail cell after being re-arrested and charged with sex trafficking. A medical examiner says the death was a suicide. A pilot on Epstein’s private jet later that month claims Andrew was a passenger on past flights with the financier and Giuffre. Andrew takes part in a disastrous BBC TV interview during which he claims he could not have had sex with Giuffre because he was at home after a visit to Pizza Express in Woking, and that her description of his dancing with her beforehand could not be true because he was unable to sweat, and that he had "no recollection of ever meeting this lady". After several days of negative reaction, Andrew announces he is to step back from public duties 'for the foreseeable future'. US prosecutor Geoffrey Berman gives a public statement suggesting there has been 'zero cooperation' with the investigation from Andrew. After Berman again claims the prince has 'completely shut the door' on cooperating with the US investigation in March, lawyers for Andrew insist he has repeatedly offered to cooperate and accuse US prosecutors of misleading the public and breaching confidentiality. Maxwell, who has seldom been seen in public in recent years, is arrested by the FBI on charges related to Epstein. Unsealed testimony from a 2015 civil case reveal a series of claims about her role in Epstein sex-trafficking ring, including allegations that she trained underage girls as sex slaves.

The writing was really on the wall for Andrew’s future when companies started questioning their sponsorship of his Pitch@Palace initiative, a sort of Dragon’s Den with chandeliers, where young entrepreneurs once a year attend Buckingham Palace and present their ideas to business leaders.

They would scarcely want to be associated with someone tarred by friendship with a notorious sex offender. Andrew also has links to more than 200 charities, 33 of which are involved with education and young people. He won’t immediately shed his honorary military titles, as an admiral (except perhaps as admiral of the Sea Cadet Corps), honorary colonel of several regiments and an air commodore in the RAF, but they won’t like the taint either and he may have to stand down for the good of the services. Talk that Andrew’s purdah will only last a few months is wishful thinking: Epstein’s ghost will always hover behind him now.

Andrew’s absence will leave a hole in the royal grid: who will fulfil the 394 engagements he carried out last year? Who will step in to his role with business, particularly with Harry and Meghan also taking a back seat? The ambition of Prince Charles and some advisers to slim down the working membership of the family to a smaller active core is being achieved by natural wastage, not design.

Age is also having its effect: the 98 year-old Duke of Edinburgh has retired, the Queen herself is slowing down at 93, Charles is 71 and even William is nearly 40: “He’s just another bald bloke in a suit,” one royal photographer lamented. The latest poll this week, YouGov for the Times, was ominous: 54% supporting the sovereign, 20% not and 25% indifferent. The strategy will need a rethink if the royals are, in the Queen’s words, needing to be seen to be believed.

• Stephen Bates is the author of Royalty Inc: Britain’s Best-Known Brand (Aurum)