Paul Manafort’s lawyers asked a federal judge to toss the charges against him, saying Thursday that Special Counsel Robert Mueller exceeded his authority when he sought an indictment against the former chairman of President Trump’s campaign.

The lawyers said Mr. Mueller was appointed to investigate Russian collusion with the Trump campaign, but the charges against Mr. Manafort stem from his own business dealings before the campaign. It’s the latest bid by Mr. Manafort to try to scuttle the case against him.

“Mr Manafort’s activities in the Ukraine have nothing to do with the campaign or collusion with the Russian government,” Kevin Downing, Mr. Manafort’s lawyer, told U.S. District Judge Amy Berman Jackson.

But prosector Michael Dreeben said there was a connection between Mr. Manafort’s activities in the Ukraine and alleged Russian collusion with the Trump campaign.

“Mr. Manafort worked with Russian-backed Ukrainian politicians,” he said. “Did they provide back channels to Russia?”

Mr. Manafort’s legal defense dovetails with complaints some conservatives have lodged against the special counsel, saying there hasn’t been any proof of collusion, but the probe is continuing anyway, including spurring an investigation into Mr. Trump’s personal lawyer for payments during the campaign to an adult film actress who claims to have had sex with Mr. Trump a decade before he was running for office.

The Trump administration has denied Russian collusion claims, and the president has called Mr. Mueller’s investigation a “hoax” and a “witch hunt.”

Judge Jackson appeared skeptical of Mr. Manafort’s line of argument, at one point telling Mr. Downing that Justice Department rules for appointing a special counsel give Mr. Mueller “a broad authority” to investigate other possible crimes arising out of the Russia investigation.

“That’s in the regulations as an option,” she said.

But she ended the nearly three-hour hearing without issuing a decision.

Mr. Manafort is accused of money laundering, making false statements to federal investigators, tax fraud in Washington, D.C. He has similar, but separate charges filed against him in Virginia.

The allegations stem from Mr. Mueller’s investigation into collusion between the Russian government and the Trump campaign during the 2016 election. Prosecutors allege that Mr. Manafort failed to report income from his lobbying work for the Ukrainian government in a scheme to avoid paying taxes. He was among the first to be indicted by Mr. Mueller.

Mr. Manafort is slated to face trial in Washington, D.C., in September. A trial in Virginia is scheduled for June.

Thursday’s courtroom debate centered around Deputy Attorney General Rod Rosenstein’s May 17 order naming Mr. Mueller as special counsel and the authority it gave him. Mr. Downing said if special counsel had that authority, it would have been specifically stated per Justice Department rules.

Those rules require special counsel must probe a “specific factual matter,” Mr. Downing said.

Earlier this month, the Justice Department filed a copy of the memo Mr. Rosenstein wrote last August detailing the breadth of the special counsel’s authority. The memo revealed Mr. Mueller had been authorized to probe Mr. Manafort’s work with the Ukrainian government.

Mr. Dreeben said the memo did not broaden Mr. Mueller’s authority, but, instead, confirmed his powers from his original appointment. He added that investigators rarely start out with a specific crime, but rather go where the facts lead. Therefore, any crimes stemming from the Russia investigation would be part of the special counsel’s jurisdiction.

The issues debated Thursday were just a portion of Mr. Manafort’s challenge to the special counsel’s authority. A separate hearing on the legality of a search warrant used to seize evidence from Mr. Manafort’s storage locker in Alexandria is scheduled for next month.

Prosecutors have been looking at Mr. Manafort for some time, according to court records. Last week, court documents showed that the special counsel is interested in some of his activities while he was chairman of the Trump campaign.

The documents revealed investigators were looking into possible campaign finance law violations during a June 2016 meeting between Trump campaign officials and Russia operatives.

The meeting at Trump Tower was attended by Mr. Manafort, the president’s eldest son, Donald Trump Jr. and son-in-law Jared Kushner. The three meet with Aras Emin Agalarov, a Russian billionaire and his son.