Maryland asked a federal judge to bar Matthew Whitaker from serving as acting attorney general, the first of what experts say will likely be a series of court challenges to his appointment.

A flurry of legal action, they said, could force the Trump administration to decide whether defending Mr. Whitaker’s installation is worth a potential Supreme Court test of the president’s appointment powers.

In court papers filed Tuesday, Maryland’s attorney general, Democrat Brian Frosh, argued that President Trump bypassed the Constitution and federal law when naming Mr. Whitaker to succeed Jeff Sessions, who was ousted as attorney general last week.

Related Defunct Patent Firm That Whitaker Advised Posted Video Featuring Trump

Maryland’s lawyers want Mr. Whitaker replaced by Deputy Attorney General Rod Rosenstein, whom the filing calls Mr. Sessions’ “proper successor.” If successful, the legal challenge could invalidate actions Mr. Whitaker takes as attorney general.

The challenge escalates the legal debate surrounding the installation of Mr. Whitaker, who also faces questions about his oversight of special counsel Robert Mueller’s investigation into alleged Russian interference in the 2016 elections. Mr. Whitaker criticized the investigation publicly before coming to the Justice Department as Mr. Sessions’ chief of staff.

Inside the White House, Mr. Whitaker is viewed as more than a temporary placeholder, officials said. The president likes Mr. Whitaker and has told advisers that he would consider nominating him to the position if he performs well in coming weeks, the officials said.

That could set up a tough balancing act for Mr. Whitaker. Mr. Trump has repeatedly disparaged the Mueller investigation and signaled he would like it to be reined in, but if Mr. Whitaker curtails it too aggressively, he could risk a backlash from the senators who would have to confirm his nomination.

Senate Minority Leader Chuck Schumer (D., N.Y.) said on the Senate floor that Mr. Whitaker is expected to meet with ethics officials this week. “We expect that Congress will be notified about the results of those discussions,” the senator said.

Trump: I Don’t Know Matt Whitaker President Donald Trump told reporters at the White House that he doesn't know Matthew Whitaker, his new acting attorney general, but that he is "a very highly respected man." Photo: AP

Mr. Schumer also said that if Mr. Whitaker doesn’t recuse himself from the Russia investigation, Democrats will seek to add a measure to a coming spending bill that would block him from interfering with Mr. Mueller’s probe.

In general, an acting official can serve up to 210 days without Senate confirmation, or longer if a nomination of is pending. Even if it is short-lived, Mr. Whitaker’s appointment could have far-reaching consequences, as he oversees criminal and national-security issues as well as civil-rights and immigration policy.

A Justice Department spokeswoman declined to comment on the court challenge. Department officials have previously said Mr. Whitaker’s appointment is lawful under the Vacancies Reform Act, which outlines how the president can make interim appointments to high-level posts. Critics say that law wasn’t intended to allow acting appointments to such a high-level post for an extended period.

Maryland’s request for an injunction—made before District Judge Ellen L. Hollander, an Obama appointee—comes as part of a lawsuit against the Trump administration over the Affordable Care Act, which names Mr. Sessions as a defendant. With Mr. Sessions gone, Maryland says it needs the judge’s ruling to prevent Mr. Whitaker from illegitimately influencing the department’s legal positions.

Tuesday’s court filing suggests the dispute could affect other federal litigation, including immigration, an area of prime interest to the president.

Newsletter Sign-up Capital Journal Scoops, analysis and insights driving Washington from the WSJ's D.C. bureau. PREVIEW

Immigrants appealing deportation orders to the U.S. circuit courts must name the attorney general as defendant. Those immigrants could raise the same objection Maryland did—that Mr. Whitaker isn’t a proper party to the lawsuit since he isn’t the lawful attorney general.

“Questions about whether Whitaker is a legally appointed attorney general could be raised in every federal court immigration appeal now pending or filed in the future,” said Lucas Guttentag, a professor of immigration law at Stanford University.

Some 5,000 such appeals were filed in the federal circuit courts in the year that ended in March, according to the Administrative Office of the U.S. Courts.

The legal fight over Mr. Whitaker could also stymie the Trump administration’s effort to speed up deportations by increasing the number of immigration judges, Mr. Guttentag said. These judges are appointed by the attorney general, and immigrants could challenge their status if they were named by Mr. Whitaker.

Maryland’s lawyers argue that Mr. Whitaker’s appointment violates another, more specific statute that says the deputy attorney general is second in the line of succession at the Justice Department. The Constitution also says the duties of the attorney general can be carried out only by someone with Senate approval, the state argues.

The Justice Department’s Office of Legal Counsel, which offers advice on legal issues, is expected to issue an opinion in coming days in support of Mr. Whitaker’s installation, a person familiar with the matter said. The opinion will likely echo guidance issued in 2003 that addressed the problem of Senate confirmation by defining an acting director as an “inferior officer,” who under Supreme Court precedent doesn’t require Senate approval.

—Michael C. Bender contributed to this article.

Write to Sadie Gurman at sadie.gurman@wsj.com and Jess Bravin at jess.bravin@wsj.com