Housing and civil rights groups filed suit Tuesday in a federal court to force the Department of Housing and Urban Development to reinstate a fair housing rule that Secretary Ben Carson moved to delay in January.

The complaint, filed in the U.S. District Court for the District of Columbia, alleged that agency is neglecting its duty to pursue fair housing goals.

“Each day HUD holds up requiring jurisdictions to fully comply with the law is another day that millions of people are being denied fair housing opportunities,” said Lisa Rice, president of the National Fair Housing Alliance. “HUD’s action is a clear example of ‘justice delayed, justice denied.’”

The rule in question, known as Affirmatively Furthering Fair Housing, implements a section of the 1968 Civil Rights Act by giving localities an assessment tool and requiring them to use it to draw up plans to address disparities in housing and economic opportunity facing minorities and other protected classes.

The rule was finalized in 2015 under the Obama administration, and welcomed by advocates who saw it as a step toward curbing segregation and exclusionary housing trends.

Carson, however, had criticized the rule as social engineering. Last year, he said he aimed to “reinterpret” the rule.

In delaying the rule’s applicability to towns and counties in January, HUD said it was too burdensome and that localities needed more time to comply.

The American Civil Liberties Union and the NAACP Legal Defense and Educational Fund, among other entities, are representing the housing groups in the suit.