Washington (CNN) Housing and Urban Development Secretary Ben Carson made dismissive comments in a recent meeting that multiple people present interpreted as an attack on transgender people, The Washington Post reported Thursday.

Carson expressed concern about "big, hairy men" attempting to enter women's homeless shelters, according to three people present, the Post reported. Two agency staffers also told the paper that during the meeting in HUD's San Francisco office, Carson complained about the blurring of gender roles, a refrain that upset many staffers, the Post said.

"That was the first time any of us heard him use such derogatory language," one San Francisco staffer told the Post. "He's more tactful when he's talking before Congress, whereas this sounded like a slur to me."

HUD did not immediately respond to CNN's request for comment.

A senior HUD official told The Washington Post that "the Secretary does not use derogatory language to refer to transgendered individuals. Any reporting to the contrary is false." The official added that Carson was referring to men who pretend to be women in order to get into shelters for women who are victims of domestic violence, not transgender people.

Carson has a long history of spawning controversy with his comments about the LGBTQ community. In 2016, he compared being transgender to changing ethnicities, and during his tenure as HUD secretary the department has proposed a rule that would allow federally funded homeless shelters to consider sex and gender identity when deciding whether to accommodate someone.

One staffer at the HUD meeting told the Post that Carson said "transgender people should get the same rights as everyone else, but they don't get to change things for everybody else."

News of Carson's comments comes the same day that 61 House members denounced what they call "anti-LGBTQ" changes made to the 2019 HUD Notice of Funding Availability.

"The Trump administration is engaged in a relentless campaign to attack the health and safety of LGBTQ Americans," said Rep. Jennifer Wexton,a Virginia Democrat, in a news release. "HUD's actions to weaken housing and shelter protections pose a real -- even deadly -- threat to the LGBTQ community, particularly transgender individuals who are already subject to alarming rates of violence and abuse."

Rep. Ayanna Pressley, a Massachusetts Democrat, echoed that sentiment, adding, "The occupant of the White House and Secretary Carson are using HUD to advance their agenda of cruelty and discrimination."

Before launching his ultimately unsuccessful 2016 presidential campaign, Carson told CNN he considers LGBTQ issues "personal."

"It seems to be a topic -- a person's sexual orientation -- that is of fair amount of concern to you. I don't find it to be anywhere near as interesting," Carson said at the time. "I think it's a personal issue and we ought to leave it as a personal issue."