The president's plan to build a wall along the southern border could come crumbling down before the structure is even built, thanks to a lawsuit filed by the State of California being argued Friday in San Diego.

The wall's first substantial legal challenge will be heard by a federal judge, Gonzalo Curiel, whom President Trump criticized during the 2016 campaign over the judge's Mexican heritage.

"Now, he is Hispanic, I believe. He is a very hostile judge to me," Trump said on "Fox News Sunday" back on Feb. 28, 2016, referring to Curiel's handling of a class-action case against the now-defunct Trump University.

Curiel could halt construction of the wall or any fencing along the U.S.-Mexican border, should he decide that California's complaint has merit and can proceed to trial.

"If he does that, it could definitely put a wrench in the works as far as the administration is concerned," said former Justice Department lawyer Thomas Dupree.

California filed a lawsuit against the Trump administration, claiming its proposed wall would cause "irreparable harm" to the state.

The administration must follow up to 30 different state and federal environmental regulations before constructing any barrier along the border, including the National Environmental Policy Act, which would require a lengthy Environmental Impact Statement before contractors can initiate any fence building.

"Will it end the whole debate on the fence or the wall? No. But it could be a significant victory for one side or the other," said Andrew Gordon, a former Department of Homeland Security lawyer during the Obama administration.

The Trump administration argues its authority over border construction is absolute, based on a 1996 law that allows the federal government to "take such actions as may be necessary to install additional barriers ... to deter illegal crossings in areas of high illegal entry into the United States."

The president also argues that Congress, through the Real ID Act of 2005 and the Secure Fence Act of 2006, gave the Department of Homeland Security a waiver from any environmental rules to build that barrier. Government lawyers claim the waiver remains in effect.

"When Congress gave them the waiver authority, the waiver authority was supposed to encompass not just the projects that were on the books at that moment but any other projects that could have been done under the act," Gordon said.

The act called for 800 miles of double reinforced fencing, although today less than 100 miles of double fencing exists. Regardless, California argues that the waiver authority expired in 2008 and does not last forever.

"What the challengers are arguing is that when Congress passed this law, it didn't intend to give the DHS a blank check to make whatever border security improvements it wanted," Dupree said. "If Congress wants this administration to streamline the process to build the wall, it can pass legislation to that effect. But don't try to cite this statute that was passed years ago in furtherance of that goal."

Dupree and Gordon said both sides have a strong case.

"Fundamentally, and ultimately, the question is: How broad is the waiver authority Congress gave Homeland Security?" Gordon asked.

The California challenge is already slowing down Trump's border security effort. State lawmakers are considering a bill to blacklist any company that bids on the border wall or any other border project, including a plan to replace 14 miles of old, rusting, eight-foot-high primary fence outside San Diego.

The project, approved under the Obama administration, replaces a Vietnam-era steel landing mat starting near the Pacific Ocean and heading 14 miles east.

According to those familiar with the project, construction was to have begun in March but several contractors backed out for fear of losing future business with the state of California.

"Some of members have opted not to bid on these projects," said Peter Tateishi, CEO of the Associated General Contractors of California. "Penalizing the contracting community is not the right direction to go. Having fewer bids means fewer options and when we have less competition in these spaces, we tend to have higher prices."

The Border Patrol said the replacement fence is needed to protect the safety of its agents.

"Right now, the existing primary fence is almost like no fence at all," said Chris Harris, a spokesman for the agents' union. "It's eight feet high and, in some places, even less. We get rocked by the Mexican side. I've been hit. Other agents have suffered serious injuries. Why the state of California would want to endanger agents by objecting to a fence where one already exists is beyond me."

California Attorney General Xavier Becerra did not respond to inquiries this week from Fox News.