In 2017, Prince Philip announced his retirement from his official royal duties a month before his 96th birthday. Since then he has largely stayed out of the public eye and the headlines, with the major exception of a January 2019 car accident, but there have been signs that he is still paying attention to the public image of his family behind the scenes. Last week Prince Charles reportedly went to Sandringham to seek the counsel of his father regarding the Prince Andrew fallout. On Thursday the Telegraph reported that Philip also spoke to Andrew about the scandal over his friendship with Jeffrey Epstein, and said he had to “take his punishment,” per the outlet’s source.

According to the newspaper, Andrew drove the 140 miles to Sandringham on Monday to have lunch with Philip and Charles, just hours before the BBC aired an interview with his accuser, Virginia Roberts Giuffre. “It was a tense meeting, and Philip told his son he had to take his punishment,” one Sandringham insider told the Telegraph. “There was no screaming or shouting, but Philip told him in no uncertain terms that he had to step down for the sake of the monarchy.”

Charles also had strong words for his brother. “It was all very civilized, but Charles calmly read him the riot act and told him there was no way back for him in the near future,” the insider said. “Andrew thought he was being treated harshly as nothing has yet been proved against him, but he accepted the decision.”

According to the source, Philip was sympathetic to Andrew’s concern, but was more focused on the interests of the family. “Philip doesn’t like trial by the media, but he is realistic enough to realize that Andrew’s actions were a danger to the very fabric of the royal family,” the insider added.

When Andrew and Charles were children, Philip had a reputation for being a brusque disciplinarian. In 1986, Vanity Fair reported that Philip and Andrew had a close relationship, and that Philip admired his son’s “macho action-man image.” But that closeness has not prevented Philip from punishing Andrew for past transgressions. As of 2018, he reportedly still harbored ill will toward Sarah Ferguson after the 1992 toe-sucking scandal landed the royals the worst headlines they had gotten in decades.

Charles returned from a trip to the Pacific on November 26 and soon headed to Sandringham, reportedly to spend time with his father. He ultimately spent about a week at the Norfolk estate before heading back to London for events connected to the NATO summit, which began Tuesday. A Buckingham Palace reception for the world leaders was the first major royal event to take place after Andrew stepped down from his public role.

More Great Stories From Vanity Fair

— Kate Middleton is having a moment, and here’s why the queen is a fan

— Caught between boomers and millennials, is Generation X really our last, best hope?

— Were Prince Harry, Meghan Markle, Kate Middleton, and Prince William ever actually the fab four?

— As the college-admissions scandal rolls onward, Felicity Huffman’s and Lori Loughlin’s paths diverge once again

— Prince Andrew’s fallout continues as he seems unable to stay away from giving interviews

— From the Archive: The American nightmare and ballad of Richard Jewel

Looking for more? Sign up for our daily newsletter and never miss a story.