A collection of civil rights groups is suing the U.S. Department of Housing and Urban Development, along with its secretary, Ben Carson, over the delay of a housing voucher rule designed to combat segregation among those who use “Section 8” vouchers to subsidize their rent.

Open Communities Alliance, a Connecticut nonprofit that advocates for “interracial access to opportunity and housing choice,” filed the lawsuit on behalf of plaintiffs Crystal Carter and Tiara Moore over Carson’s decision to push back mandatory implementation of the Small Area Fair Market Rent (SAFMR) rule by two years.

Originally set to take effect on January 1, 2018, SAFMR aims to give housing voucher users greater choice in where they live. Currently, the vouchers subsidize rent payments above about 30 percent of voucher recipients’ income. The subsidy, however, is capped by a “fair market rent” rate calculated by HUD by averaging the rent across the entire metropolitan area in which the voucher user lives.

But because rents vary dramatically over any given metro area and are typically lower along the perimeter, the average is driven down and the subsidy ends up being too low to cover rent except in areas that are impoverished. The SAFMR rule recalculates the fair market rent on the basis of a nearby zip code, which raises the fair market rent and, thus, the subsidy. This gives voucher users the opportunity to move to a better neighborhood with better schools.

Because the vast majority of Section 8 voucher recipients are people of color, civil rights activists charge the fair market rent status quo with reinforcing segregation, and say the SAFMR would encourage integration.

In 2012, HUD launched a pilot study for SAFMR in five states that, ultimately, led to HUD deciding to adopt the policy on November 16, 2016, with an effective date of January 1, 2018. But Carson, who in May came under fire for saying poverty was largely “a state of mind,” has delayed mandatory implementation of SAFMR by two years, prompting Open Communities Alliance’s lawsuit.

HUD spokesperson Brian Sullivan told The Intercept, which broke the news of the lawsuit, that the delay does not indicate a change of policy, and he pointed to a HUD blog post from August that announced the delay. But the post also indicated that any of the public housing agencies (PHAs) that are ready to implement the rule may do so at their discretion.

“As the original deadline approached, it became clear to us that PHAs needed more time to integrate this big change into their voucher programs,” the blog post reads, citing “research” as a reason for the decision. “We contemplated this possibility during the rule-making process, and the final rule allows the Secretary to give local agencies additional time if it was determined that they needed it—and they needed it.”

The lawsuit comes on the heels of what’s been a busy week for Carson. At the Mortgage Bankers Association Annual Meeting in Denver this week, Carson called on Congress to reform Fannie Mae and Freddie Mac. And on Tuesday, a video of Carson getting dressed down by Rep. Al Green at a Congressional hearing went viral.