(CNN) Researchers say Dr. Ben Carson may be a brain surgeon, but science shows he's dead wrong about how sexual orientation works.

But scientists say decades of research overwhelmingly shows sexual preference is inborn, not a choice.

"It doesn't seem to be the social environment, it doesn't seem to be the parents or peers that make you gay," said Gerulf Rieger, a lecturer in the department of psychology at the University of Essex in England. "It seems to be something that comes from within."

"Sexual orientation is biologically determined," added Cynthia Struckman-Johnson, a professor of psychology at the University of South Dakota. "I don't think any educated person aware of the research would argue differently."

JUST WATCHED Ben Carson apologizes for comments on gay people Replay More Videos ... MUST WATCH Ben Carson apologizes for comments on gay people 03:13

In a statement Wednesday, Carson apologized for what he said on CNN, and softened his words, saying the verdict is still out on whether people are born with their sexual preference.

"I do not pretend to know how every individual came to their sexual orientation," he added.

While science is seldom definitive about anything, especially something as complicated as why we love who we love, researchers say the preponderance of the evidence shows sexual orientation is not chosen.

They point to studies showing children are aware of their sexual orientation at a young age. Plus, genetic studies suggest homosexuality runs in families.

Finally, researchers have found that the very structure of the brain -- specifically, regions of the hypothalamus -- differs in gay and heterosexual men.

"Ask a bunch of straight guys [if they could switch to being gay] and they would tell you, 'Are you kidding me?" Rieger said. "So the other way around doesn't work either."

So what was Carson's scientific support for his argument that being gay is a choice? He pointed to prisons.

"A lot of people who go into prison go into prison straight - and when they come out, they're gay. So, did something happen while they were in there? Ask yourself that question," Carson told CNN's Chris Cuomo.

Experts on sexual activity in prisons challenged the very premise of Carson's statement, calling it "preposterous" and "appalling."

"To, in any way, suggest that a person would go in as a heterosexual and then somehow become a homosexual as a result, that's simply preposterous. There's no evidence to support that," said Robert Dumond, a mental health counselor who's testified to the Department of Justice about rape in prison.

Prisoners do engage in homosexual practices, but that doesn't mean they've gone from straight to gay, the experts said.

Prisoners are often raped, they say, or coerced to engage in sex in exchange for protection from violent prisoners.

"They're physically forced or forced through blackmail to engage in sex," Struckman-Johnson said. "To have someone say this is chosen is just showing such great disregard for what we know about the world of sexual assault in prison."

The prisoners who are perpetrating the rape are often either gay to begin with, or are heterosexual and rape to subjugate other prisoners.

"It's more about power than sex," said Chris Hensley, a criminologist at the University of Tennessee who's done extensive research on sexuality in prison. "These men don't see what they're doing as homosexual, and they will leave their incarceration heterosexual."

Other prisoners do willingly engage in homosexual activity, but the experts say it's more of a way to release sexual tension and have some degree of intimacy.

"It doesn't mean they would choose someone of the same gender in any other situation," Struckman-Johnson said.

Critics of these studies point out other research they say indicates that orientation is a choice, namely studies that show sexual orientation can change over the course of a lifetime, and that "socialization" can alter orientation.

Overall, the scientists with whom we spoke said they were shocked at Carson's arguments.

"I'm deeply saddened on multiple levels that someone in the 21st century would take this position," Dumond said.