Randy Andy, Air Miles Andy, and now the Duke of Hazard: headline writers have always found evocative ways to shorthand the Queen’s second son at different stages in his royal career.

As the Duke of York, 59, is spotted teeing off at the Real Club Valderrama, the exclusive golf club in Sotogrande, southern Spain, coverage of his controversial former friendship with the disgraced billionaire financier Jeffrey Epstein continues to make uncomfortable reading for Buckingham Palace.

Once the pilot prince, photographed with a rose clamped between his teeth at a hero’s homecoming following the Falklands war, his public image is now a disaster. The latest optics are grainy video of him at Epstein’s New York mansion, amid reports of royal foot massages dispensed by young women.

The duke, first named in US court documents in 2015, has issued outright denials over allegations he had sexual encounters in 2001 with Virginia Roberts, now Giuffre, when she was a 17 -year-old masseuse for Epstein. The claims were struck from the US court record by a judge who described them as “immaterial and impertinent”. Also vehemently denied are allegations, in recently released court papers of the Guiffre case, made by Johanna Sjoberg that Andrew touched her breast while sitting on a couch at Epstein’s home in 2001.

Facebook Twitter Pinterest Prince Andrew, Virginia Roberts, aged 17, and Ghislaine Maxwell at Ghislaine Maxwell’s townhouse in London in 2001. Photograph: REX/Shutterstock

When the video footage of Andrew waving off a young woman from the convicted sex offender’s residence emerged this month, an equally strong statement, through Buckingham Palace, said the duke was “appalled” by the recent sexual abuse claims levelled at Epstein, 66, who killed himself in jail on 10 August while facing sex trafficking charges.

“His Royal Highness deplores the exploitation of any human being and the suggestion he would condone, participate in or encourage any such behaviour is abhorrent,” it said.

There can be no denial, however, about Andrew’s crass lack of judgment. That video, and a photograph of him walking with Epstein in New York’s Central Park, were taken in 2010 – two years after the billionaire was convicted of soliciting a minor for prostitution and was a registered sex offender.

“The situation changed when Epstein was convicted. It was [Andrew’s] conscious decision to stay loyal, to not criticise his friend the convicted paedophile. And, while that may be fine for him, it’s a step too far. What it’s saying is the royal family tolerates this sort of behaviour,” said one veteran royal commentator, who has met Andrew on several occasions, and believes he should now step down from his public roles.

Time and again, Andrew, who a courtier once reportedly described as having “a pompous level of self-importance”, has demonstrated an eye-watering lack of judgment. Palace staff have rated him the rudest of royals, according to reports. A secret cable, published on WikiLeaks in 2010, revealed a US ambassador describing Andrew speaking “cockily” during one official lunch, leading to a discussion that “verged on the rude”.

He earned the soubriquet Air Miles Andy after the National Audit Office censured his helicopter habit in 2005, which included him spending £32,000 in one year flying to Scotland to the Royal and Ancient club at St Andrews in his capacity as captain.

In 2011, following pressure over the Epstein connection, he stood down from his role as UK special representative for trade and investment. It did not help when his ex-wife, Sarah Ferguson, admitted having accepted £15,000 from Epstein to help pay off her debts.

His links have often raised eyebrows; meetings with the Libyan dictator Muammar Gaddafi’s son Saif; entertaining the son-in-law of Tunisia’s ousted president Zine al-Abidine Ben Ali at Buckingham Palace; his relations with Timur Kulibayev, son-in-law of the then president of Kazakhstan, who purchased the duke’s Sunninghill Park home for £3m more than its £12m asking price in 2007.

“Rightly, or wrongly, there is a perceived air of arrogance about Andrew, and he is described by various people as boorish and very self-centred,” said Joe Little, managing editor of Majesty Magazine. “Perhaps we shouldn’t be judging the book by its cover. But, certainly, that’s the impression he has given for a very long time.”

Facebook Twitter Pinterest Andrew (r) has remained loyal to his former wife, Sarah Ferguson. Photograph: Adam Davy/PA

Andrew’s loyalty to his seemingly perpetually broke former wife, famously photographed having her toes sucked by her “financial adviser” before their 1996 divorce, remains clear. The two live under the same roof, Royal Lodge in Windsor Great Park, and are currently holidaying together in Sotogrande. Even after Ferguson was caught trying to sell access to her ex-husband to an undercover reporter for £500,000 – of which the duke denied any knowledge – they remain close.

Andrew’s friendship with Ghislaine Maxwell, daughter of the notorious press baron Robert Maxwell, and a close friend of Epstein’s, first surfaced in 2000 when they appeared together at private parties and celebrity functions in the UK and US.

Today he is patron of some 200 charities and organisations, and the Prince Andrew Charitable Trust supports young people, healthcare issues and the participation of the disabled and wounded service personnel in sporting activities. His Pitch@Palace initiative, which links tech entrepreneurs to people who can help their business, is his “passion project” which he has taken global.

Long-rumoured to be the Queen’s favourite son, she has made her support clear. The day after Epstein was found dead in a Manhattan jail, Andrew was photographed beaming alongside his mother on the way to church in Crathie, near Balmoral. She made him a Knight Grand Cross of the Royal Victorian Order in 2011, one month after it emerged he had seen Epstein during a private trip to New York.

His fate is that of the typical second son, struggling to find a role. “I think despite knowing he was always going to go down the line of succession, the fact that in his youth he was second in line to the throne, and now he is wherever he is, that must be quite a blow to your confidence and feeling of importance,” said Little.

• This article was amended on 26 August 2019 to correct the spelling of Timur Kulibayev’s name and clarify that his father-in-law is no longer the president of Kazakhstan.