Jessica Estepa

USA TODAY

Housing and Urban Development Secretary Ben Carson referred to slaves as "immigrants" while speaking Monday to department employees.

"That's what America is about, a land of dreams and opportunity," Carson said. "There were other immigrants who came here in the bottom of slave ships, worked even longer, even harder for less. But they too had a dream that one day their sons, daughters, grandsons, granddaughters, great-grandsons, great-granddaughters, might pursue prosperity and happiness in this land."

Twitter users — including the NAACP, the Anne Frank Center for Mutual Respect and Chelsea Clinton — weren't having it:

A HUD spokesman told Good Morning America that no one present for Carson's speech "was confusing voluntary immigration with involuntary servitude. Please."

Carson sent a tweet later Monday saying, "Slaves didn't just give up and die, our ancestors made something of themselves." The tweet included a link to a SoundCloud file from a radio interview with Carson on the Armstrong Williams Show in which he explains to caller what he meant by his remarks about slaves.

"There were numerous people who were brought over here on slaves ships and it is a horrible thing," Carson said. "I'm not saying that it wasn't a horrible thing, but what I am saying is that those people were strong. They were strong-willed. They didn't just give up and die like many of the other people who they tried to enslave. And one of the reasons that they didn't just give up and die is because they used the brain that God gave us and they figured that a time would come when there would be freedom. A time would come when their children could achieve."

HUD did not immediately respond to a request for comment from USA TODAY.

Carson, a retired neurosurgeon who vied for the 2016 GOP presidential nomination, was confirmed and sworn in as HUD secretary last week. He's never held public office before, nor is he versed in housing policy.

"Ben shares my optimism about the future of our country and is part of ensuring that this is a Presidency representing all Americans," Trump said in December when he announced Carson as his pick to head HUD.

Watch a video of Carson's full remarks here.