Washington (CNN) Ben Carson said Wednesday night that he was "irritated" by the comparison between the fight for same-sex marriage rights and the Civil Rights Movement because there's no overt segregation against gays.

Speaking on Fox News' "Special Report", Carson elaborated on his remarks on CNN in March that he believes being gay is a choice because people "go into prison straight -- and when they come out, they're gay."

The GOP presidential contender told host Brett Baier that he "shouldn't have allowed my emotions" into the conversation, but was reacting to CNN host Chris Cuomo's line of questioning on the issue.

"I was a little bit irritated that he was equating the whole [gay marriage] issue with the Civil Rights movement. Because, quite frankly, I didn't remember any times when there were signs up that says, you know, 'everybody else here and gay people have to drink at this fountain,'" he said.

JUST WATCHED Why Ben Carson's campaign is different Replay More Videos ... MUST WATCH Why Ben Carson's campaign is different 02:09

Read More