WASHINGTON – President Donald Trump's controversial selection of Matthew Whitaker to replace ousted Attorney General Jeff Sessions faced its first major challenge Tuesday in federal court from the state of Maryland, which claimed that the appointment is "illegal and unconstitutional."

Maryland Attorney General Brian Frosh, a Democrat, argued in court documents that Deputy Attorney General Rod Rosenstein should be installed in Whitaker's place.

"The Constitution and Congress have established vitally important processes for filling high-level vacancies in the federal government," Frosh said Tuesday. "Few positions are more critical than that of the U.S. attorney general, an office that wields enormous enforcement power and authority over the lives of all Americans."

If the court were to rule in Maryland's favor, the Trump administration would be almost certain to appeal the ruling to the Supreme Court.

More:Acting AG Whitaker to consult with DOJ ethics lawyers on recusal as Dems press his disqualification on Russia probe

More:Two Republican former AGs raise questions on Trump's naming Matthew Whitaker to lead Justice Department

Maryland's challenge tracks a flurry of arguments leveled by Democrats and some Republicans who assert that the White House skirted established rules of succession by installing the former Iowa federal prosecutor, who is not confirmed by the Senate.

Lawmakers also have raised concerns that the appointment of Whitaker, who served as Sessions' chief of staff prior to his appointment, signaled Trump's intent to derail special counsel Robert Mueller's investigation into Russian meddling in the 2016 election and potential coordination with the Trump campaign. Before joining the Justice Department, Whitaker was sharply critical of Mueller's probe, publicly expressing doubt about Russia's role and suggesting the special counsel's office could be defunded.

Maryland's legal action is being taken as part of a lawsuit filed in September, which asked a federal judge to uphold aspects of the Affordable Care Act – including protections for pre-existing conditions. In court documents, Sessions had argued that provision should be struck down in a lawsuit brought against Obamacare by a group of Republican states.

The White House has said Whitaker's appointment was justified under the 1998 Federal Vacancies Act, but Maryland argues that law applies to routine positions and not one as important and powerful as the nation's top law enforcement officer.

"The attorney general's succession statute and the Constitution protect the country against exactly what President Trump has attempted to do here –pluck an unqualified and unconfirmed partisan to be the nation's chief law enforcement officer in order to protect himself rather than the rule of law," Frosh said.

The Justice Department declined comment Tuesday.

More:Trump does know Matthew Whitaker, he's just not a close friend, Kellyanne Conway clarifies

More:Kellyanne Conway's husband calls Trump's promotion of Whitaker 'unconstitutional'