A collection of civil rights advocates filed suit against Ben Carson and the Department of Housing and Urban Development (HUD) on Tuesday over its delay of an Obama-era regulation designed to help cities and counties address segregation.

The Affirmatively Furthering Fair Housing (AFFH) rule was designed in 2015 to enforce a rule from the Fair Housing Act of 1968 that went largely ignored for 40 years. The AFFH rule requires municipalities to develop concrete plans to address segregation in their communities. Failure to do so would result in denial of money from the Community Development Block Grant program, which disperses $3 billion annually.

In January, Carson suddenly announced he was delaying the AFFH rule until 2020. But because the rule requires anti-segregation plans that are tied to the Assessment of Fair Housing report, which municipalities file every five years, the timing of the delay effectively pushes the AFFH rule’s implementation back to 2024 and 2025, if not indefinitely.

The National Fair Housing Alliance, the Texas Low Income Housing Information Service, and Texas Appleseed—the plaintiffs in the suit—argue that HUD and Carson violated the Administrative Procedure Act (APA) when it delayed the AFFH rule without following the same notice-and-comment procedure that created the rule under the Obama administration.

The suit marks the second time Carson and HUD have been sued for delaying an Obama rule. In October, civil rights groups sued after Carson tried to delay the Small Area Fair Market Rent (SAFMR) rule, which sought to give recipients of Section 8 housing vouchers more choice in where to live. Carson lost the suit in December on the grounds that it violated the APA. Losing that suit apparently didn’t convince the department to adhere to the the APA in its attempt to delay the AFFH rule.

In its original announcement delaying the AFFH rule, Carson cited that 17 of the 49 Assessment of Fair Housing submissions from municipalities had been rejected by HUD as insufficient. Carson said this was evidence that municipalities were ill-equipped to implement the rule and needed more time.

Advocates for the rules say that 49 submissions weren’t enough to accurately assess the rule given that more than 1,200 communities are eligible for federal block grant money. They also believe that a back-and-forth between HUD and the municipality was precisely how the AFFH process was suppose to work, and that the submissions that had been rejected were done on the grounds that the municipality simply hadn’t followed the instructions.

“It’s a little bit like you’re in school and you’re writing a paper that suppose to be 10 pages and you write a three page paper, or they’re suppose to cover World War II but you’re writing about World War I,” said Michael Allen, a partner at the civil rihts law firm Relman, Dane, and Colfax. “They didn’t follow the pretty clear instructions.”

The AFFH rule was born out of a 2006 case in which Westchester County, an affluent area north of New York City, took more than $50 million in federal money under the false promise of addressing its segregation problem. The Obama administration settled the case in 2009, requiring Westchester to return $62.5 million to the federal government.

The new lawsuit comes from two Texas groups who hope that when the Houston area receives disaster relief money for Hurricane Harvey from the federal government, it can force those funds to be dispersed in a way that improves segregation issues, particularly as it relates to infrastructure such as flood drainage in areas that are predominantly minority.