Secretary of Housing and Urban Development Ben Carson on Friday night defended the disparaging comments he had made during a HUD meeting about transgender women.

According to the Post, Carson sent out an email to HUD staffers on Friday following the backlash over his comments deriding homeless transgender women as “big hairy men” who were trying to infiltrate women’s shelters.

“My point was that we have to permit policies that take into consideration the rights of everybody, including [cisgender] women — many of which have suffered at the hands of male domestic abusers — who believe there are men who might hurt them,” Carson wrote in the memo.

“Our society is in danger when we pick one issue (such as gender identity) and say it does not matter how it impacts others because this one issue should override every other common-sense consideration,” he also wrote. “I think we have to look out for everyone, and we need to use our intellectual capabilities to find common good rather than attempting to always stir up controversy through identity politics.”

Carson continued to defend his comments on Fox News host Tucker Carlson’s program on Friday night.

“I simply pointed out the fact that, you know, we have to have policies that take into consideration everybody’s rights,” Carson told Carlson. “I say everybody has equal rights. Nobody gets extra rights.”

Watch Carson on Fox below: