President Donald Trump is returning to Buckingham Palace — and again, American princess Meghan Markle won’t be there.

The president is heading to London for a two-day NATO summit, including a drinks reception at Buckingham Palace on Tuesday evening hosted by Queen Elizabeth. While other royals will be in attendance — including Kate Middleton, Prince Charles and Camilla, Duchess of Cornwall — Meghan and Prince Harry will miss the event as it falls during their “family time” off from royal duties.

The Duke and Duchess of Sussex revealed they would take about six weeks off towards the end of the year amid their concerns that the intense scrutiny they’ve faced has been tough on the couple.

Image zoom Chris Jackson/Getty

“The Duke and Duchess have a full schedule of engagements and commitments until mid-November, after which they will be taking some much-needed family time,” a royal source confirmed to PEOPLE.

Meghan, 38, and Harry, 35, are also planning to skip the annual holiday festivities at Sandringham — including their public walk to church on Christmas morning — with Queen Elizabeth and the rest of the royal family.

Image zoom Meghan Markle and Prince Harry on Christmas 2018 Tim Rooke/REX/Shutterstock

Although Kate is expected to attend the palace party with President Trump, her husband Prince William is currently in the Middle East on a four-day solo tour of Kuwait and Oman.

RELATED: Why President Trump Doesn’t Have to Bow to the Queen — and Melania Doesn’t Have to Curtsy

Image zoom Kate Middleton attends state banquet for President Trump Victoria Jones- WPA Pool/Getty Images

Image zoom First Lady Melania Trump, Queen Elizabeth, President Donald Trump and Prince Charles Chris Jackson/Getty

President Trump, 73, and First Lady Melania Trump, 49, went to the U.K. in June for a days-long state visit full of pomp and circumstance and protests. He and his family, including his four adult children, also attended a state banquet then with the Queen, Prince Charles, Camilla, Kate and William at Buckingham Palace.

During the visit, Prince Harry joined other members of the royal family for a palace exhibit at Buckingham Palace with the President and First Lady. At the time, Meghan was on maternity leave after welcoming son Archie the previous month.

Image zoom Ivanka Trump and Prince Harry MANDEL NGAN/AFP/Getty

After attending a private luncheon hosted by the Queen, Harry joined President Trump, the First Lady, Ivanka Trump and her husband Jared Kushner to view a special exhibition in the Picture Gallery, which showcased items of historical significance to the United States from the Royal Collection. Upon entering the room, Harry soon broke away to chat to Earl Peel, the Lord Chamberlain.

The outing came just hours after Trump insisted he had not said anything “bad” about Meghan. The Sun previously published an Oval Office interview with the president in which he reacted with surprise at learning Meghan had criticized him during the 2016 campaign, labeling him “misogynistic.”

“I didn’t know that she was nasty,” Trump said of Meghan.

Can’t get enough of PEOPLE‘s Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

Image zoom President Donald Trump; Meghan Markle Julie Douxe/REX; Chris Jackson/Getty Images

Trump soon disputed that he had made the remark at all, despite the interview being recorded. He also praised Meghan and wished her well following the birth of her son.

But, speaking with reporters at the White House as he prepared to head to London, he declined to apologize or clarify what he said — again insisting there was no quote to discuss.

“I made no bad comment,” he said.