Texas led a coalition of states Tuesday in a lawsuit challenging the legality of the Obama-era DACA deportation amnesty, opening up a new legal front in the increasingly messy battle over illegal immigration.

The lawsuit, filed in federal court in southern Texas, asks judges to rule against the 2012 program that’s granted tentative legal status to more than 800,000 Dreamers.

Rather than ending the Deferred Action for Childhood Arrivals program, though, the seven-state coalition says it would be open to a two-year phaseout similar to what President Trump tried to do last year. No new DACA permits or renewals would be allowed, but those already approved could serve out the rest of their two-year permits.

“Texas has argued for years that the federal executive branch lacks the power to unilaterally grant unlawfully present aliens lawful presence and work authorization,” Texas Attorney General Ken Paxton said in filing the lawsuit. “Left intact, DACA sets a dangerous precedent by giving the executive branch sweeping authority to ignore the laws enacted by Congress and change our nation’s immigration laws to suit a president’s own policy preferences.”

Mr. Paxton had threatened to file this type of lawsuit last year unless the Trump administration voluntarily ended DACA. Mr. Trump complied, announcing a six-month phaseout.

But as the phaseout was proceeding multiple federal courts stepped in and ruled the phaseout illegal, saying Mr. Trump cut too many corners for his move to be legal.

Texas and its fellow states — Alabama, Arkansas, Louisiana, Nebraska, South Carolina and West Virginia — say it was the original 2012 DACA program that cut corners and is illegal.

The lawsuit puts Mr. Trump in a curious position, forcing him to defend a program that his own top lawyers have said is illegal and unconstitutional.

Josh Blackman, a professor at the South Texas College of Law, suggested the administration may decide not to defend DACA at all.

“After all, [Attorney General Jeff Sessions] has already argued it is unconstitutional,” he tweeted.

He said in that case one of the activist groups that backs DACA would likely step in and mount the defense.

The Justice Department, which represents the government, declined to comment.

This isn’t Texas’s first go-around in challenging federal immigration policies.

The state sued the Obama administration in 2014, after the president proposed expanding the DACA program to include still-more Dreamers, and to also rope in millions of parents who weren’t Dreamers, but who had children who were legal U.S. residents or citizens. The 2014 expansion was dubbed “DAPA.”

First a federal district court and later the 5th U.S. Circuit Court of Appeals sided with Texas, saying Deferred Action for Parents of Americans was illegal. The Supreme Court let the appeals court ruling stand, effectively thwarting Mr. Obama’s immigration plans.

Judge Andrew S. Hanen, the first to rule on the case, said the Obama administration broke procedural laws by making a major policy change without going through the normal regulatory process. The 5th Circuit went further in its 2-1 decision, saying the 2014 policy violated immigration laws. In both cases, the judges said the 2014 expansion was essentially the same as the 2012 DACA policy.

But neither of those courts actually struck down DACA itself, leaving the policy in place when Mr. Trump took office.

One other federal court in Mississippi has said DACA is illegal — but that judge eventually concluded he didn’t have jurisdiction to issue an injunction.

The crux of the legal question is whether a president can carve an entire class of people out of danger of being deported under existing laws, and then grant them work permits entitling them to driver’s licenses and some taxpayer benefits.

Mr. Obama himself had repeatedly said a president didn’t have that power — then reversed himself in June 2012, just as he was preparing to face voters for re-election.

During the 2016 campaign Mr. Trump gave competing signals, saying he wanted to end DACA but also vowing to treat the Dreamers protected by the program with leniency.

As he announced his phaseout in September 2017, Mr. Trump said what he really wanted was for Congress to step in and provide a permanent solution for DACA recipients.

Mr. Trump earlier this year even suggested a plan that could have legalized up to 1.8 million immigrants who are in the U.S. illegally — far broader than the current DACA population — and tied that legalization to funding for his border wall, limits to the chain of family migration and an end to the Diversity Visa Lottery.

That plan didn’t gain traction among Democrats, who countered with their own more generous amnesties — without the major border security and legal immigration changes.

No plan was able to garner the 60 votes needed to clear the Senate. The GOP-led House, meanwhile, has ducked the debate altogether on its chamber floor.