Stephen Colbert enlisted the help of esteemed royals obsessive John Oliver to discuss the drama over Prince Harry, Duke of Sussex and Meghan Markle, Duchess of Sussex’s decision to scale back their royal family roles.

Taking a moment away from his own talk show desk, comedian Oliver visited The Late Show with Stephen Colbert this week. And Colbert asked the Brit if he was “transfixed” by this shakeup.

“It was a piece of cultural annexation,” Oliver joked. “You sent an American over there and you took a prince.”

Then the comedian went on to take a more serious tone to comment on Meghan and Harry’s choice to step back from royalty, as they announced in January.

“I think they’ve absolutely done the right thing. I think it was a horrible situation for them to be in,” Oliver said.

Oliver recalled a recent visit to Colbert’s show, before the royal wedding in May of 2018, when he predicted something like this could happen to the new family. “I was saying, ‘This is going to be rough.’ The British tabloid press is rough on royalty.”

Meghan and Harry have made clear that the tabloid coverage of their family played a role in their choice to step back, in the hopes of giving their son, Archie Harrison Mountbatten-Windsor, a more normal life. The family will split their time between Canada and the U.K. “This geographic balance will enable us to raise our son with an appreciation for the royal tradition into which he was born, while also providing our family with the space to focus on the next chapter,” they said in an Instagram post.

Watch Stephen Colbert’s full video interview with John Oliver here.

Get The Brief. Sign up to receive the top stories you need to know right now. Please enter a valid email address. Sign Up Now Check the box if you do not wish to receive promotional offers via email from TIME. You can unsubscribe at any time. By signing up you are agreeing to our Terms of Use and Privacy Policy . This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply. Thank you! For your security, we've sent a confirmation email to the address you entered. Click the link to confirm your subscription and begin receiving our newsletters. If you don't get the confirmation within 10 minutes, please check your spam folder.

Write to Rachel E. Greenspan at rachel.greenspan@time.com.