Attorneys for two New Orleans residents who sued Brad Pitt’s Make It Right Foundation for allegedly selling them poorly constructed homes are asking a federal judge to deny Pitt’s request to be removed from the suit.

The lawsuit, filed in September and now being heard in the Eastern Louisiana U.S. District Court, alleges that Make It Right, a charity formed by Pitt in 2007 to help Lower Ninth Ward residents return after Hurricane Katrina, built “defective” homes that leaked, were filled with mold and fell apart. In their suit, the two residents, Lloyd Francis and Jennifer Decuir, accuse Pitt and Make It Right of unfair trade practices, breach of contract and fraud.

Pitt’s attorneys have denied the allegations, though they admit that there are problems with the homes, which they blame on the project’s architect in a separate lawsuit and which they said will take $20 million to fix.

In November, Pitt’s attorneys filed a motion requesting the residents’ claims against Pitt be dismissed and his name removed from the lawsuit. The attorneys argued that even if the claims were credible, Pitt wasn’t to blame for the construction and should not “simply be lumped together with other defendants and held liable for alleged conduct in which he is not even alleged to have participated.”

Neither Pitt’s attorneys nor Make It Right returned requests for comment. Attorneys for the Lower Ninth Ward homeowners declined to comment.

In Tuesday’s motion, lawyers for the Lower Ninth Ward homeowners argued that as the founder of the board, and a member, Pitt bears responsibility for Make It Right’s directors and employees on the ground who oversaw construction, made inadequate repairs and kept homeowners in the dark regarding the deficiencies in their homes. The motion claims Pitt knew of homeowners’ complaints as early as 2009.