(CNN) A federal judge expressed deep skepticism Friday of special counsel Robert Mueller's investigation in the bank fraud case against former Trump campaign chairman Paul Manafort, at one point saying he believes that Mueller's motivation is to oust President Donald Trump from office.

Although Mueller's authority has been tested in court before, Friday's hearing was notable for District Judge T.S. Ellis' decision to wade into the divisive political debate around the investigation.

"You don't really care about Mr. Manafort's bank fraud," Ellis said to prosecutor Michael Dreeben, at times losing his temper. Ellis said prosecutors were interested in Manafort because of his potential to provide material that would lead to Trump's "prosecution or impeachment."

Prosecutors have not disclosed that they're interested in having Manafort flip, but they have indicated that they continue to investigate him after his former deputy Rick Gates agreed to cooperate.

"That's what you're really interested in," Ellis said, citing how prosecutors regularly turn cooperators in drug cases. He was appointed to the bench in Virginia by President Ronald Reagan in 1987.

Read More