There is a "growing frustration" among members of the U.K. royal family over international attention on Prince Andrew's ties to the Jeffrey Epstein sex trafficking case, The Sunday Times' Roya Nikkhah told "CBS This Morning." Authorities publicly called out the prince for refusing to cooperate with authorities in a Monday press conference.

Five Epstein accusers said they were outraged over Prince Andrew's alleged refusal to help federal investigators. Andrew had said that he was willing to speak to authorities with his attorneys' backing in a November interview with the BBC, but reportedly claims that nobody had reached out to him.

"I have to say I'm not really that surprised," Nikkhah said. "I've seen the reports overnight of Andrew saying actually, 'you know, this isn't true, I will provide testimony if I'm asked to'."

Reports of nobody reaching out to him are "absolutely not true," Nikkhah added.

She revealed that Gloria Allred, who is representing several accusers, gave The Sunday Times a letter she sent to Andrew in early December asking him to assist U.S. prosecutors with the case against Jeffrey Epstein. The Duke of York has not yet replied.

Still, Nikkah said, Queen Elizabeth still holds her second son in high regard. "She sticks by him. We saw her out riding with him, we saw her at church with him last week," but the extended family is frustrated that he seems to be providing "fuel" to the ongoing controversy.

As for Andrew's older brother, future monarch Prince Charles, Nikkhah said that his son Harry and his new family are very much on his mind. Charles hoped for a "streamlined" royal family under his reign.

"The modern monarchy has to seem to be working for a purpose" to stay relevant, Nikkah said. "But for him, that streamlined monarchy always was going to include Harry and whoever he chose to be his wife, so he'll be readjusting that."

When asked about rumors of the queen abdicating her throne early among the other controversies surrounding the family, Nikkhah said simply that "it's not going to happen."