WASHINGTON, D.C. -- Advocacy groups for the LGBTQ community are calling Housing and Urban Development Department Secretary Ben Carson’s remarks about transgender people discriminatory and inaccurate.

“There are some women who said they were not comfortable with the idea of being in a shelter, being in a shower, and somebody who had a very different anatomy,” Carson told a House Appropriations subcommittee during testimony on Tuesday.

Carson’s comments came in response to an inquiry posed by Rep. Mike Quigley, D-Illinois, about the removal of materials designed to prevent discrimination against LGBTQ individuals from HUD’s website. Progressive advocacy group People for the American Way says it has filed a lawsuit against HUD for removing those materials from its site.

“Remember, it is complex,” Carson said, adding, “we obviously believe in equal rights for everybody including the LGBT community but we also believe in equal rights for the women in the shelters and shelters where there are men and their equal rights.”

3:34

LGBTQ advocacy groups fired back, calling Carson’s remarks discriminatory toward transgender people.

"Ben Carson’s comments about people being made uncomfortable by transgender people is more evidence of the discriminatory stance against the LGBTQ community and marginalized communities from Trump administration HUD officials,” People For the American Way’s executive vice president Marge Baker said in a statement. “This is a disturbing move away from the department’s original purpose to administer fair housing and help the homeless.”

GLAAD, an organization focused on LGBTQ representation in media, pushed back on Carson’s remarks, calling them “blatant and factually inaccurate anti-transgender rhetoric.”

“It is because of derogatory myths like this, which have been debunked time and time again, that the transgender community faces disproportionate levels of discrimination and homelessness,” said Sarah Kate Ellis, president and CEO of GLAAD.

A HUD spokesman did not respond to a request for comment on Wednesday.

Carson faced scrutiny earlier this month for proposing alterations to anti-discrimination language in the HUD mission statement.