Washington (CNN) Housing and Urban Development Secretary Ben Carson said Tuesday that political correctness is "going to destroy our nation," when pressed by a Democratic lawmaker to apologize for comments he reportedly made at a meeting in September.

At a House Financial Services Committee hearing on housing policy, Democratic Rep. Jennifer Wexton of Virginia said she wanted to give Carson a chance to apologize for comments made during a meeting last month with department staff in which he expressed concern about "big, hairy men" attempting to enter women's homeless shelters. The comments, reported by The Washington Post , were interpreted by three people present as an attack on transgender people, the paper said.

Two agency staffers also told the Post that during the meeting in HUD's San Francisco office, Carson complained about the blurring of gender roles, a refrain that upset many staffers.

"First of all, I didn't describe transgender women that way," Carson said Tuesday.

"I was relating a story that a women's group told me about big, hairy men -- who are not transgender women, by the way -- coming into your facility and having to be accepted because of the rules that were in place," he continued.

Read More