(CNN) With his average build, impish smile and mop of brown hair, adult film star James Deen has carved out a niche in the industry as the guy who doesn't look like he belongs there .

"I'm like a guy a chick might actually meet in a bar," he told GQ in 2012, describing his appeal.

He not only looked the part but also appeared poised for crossover success into mainstream media as an actor and a spokesman for sex positivity

Now, his boy-next-door image is under threat amid allegations he raped an ex-girlfriend, beat up one actress and tried to force himself on another. He has not been charged with a crime, but the backlash has been intense.

Deen's former girlfriend and co-star Stoya was the first to speak up on Saturday. In a tweet, she accused Deen of holding her down and raping her.

"That thing where you log in to the internet for a second and see people idolizing the guy who raped you as a feminist. That thing sucks," she said

Stoya's allegation led others in the adult film industry to speak out against Deen using the hashtag #SolidaritywithStoya.

"He's dead on the inside and dead to me. He's literally the worst person I've ever met. That's all I'll say for now," industry veteran Joanna Angel said in a tweet

In an essay published Monday on The Daily Beast, former adult actress Tori Lux said Deen pinned her down and hit her in the face on set at a porn studio in June 2011. In another Daily Beast story published Monday, adult actress Ashley Fires said she refuses to work with Deen because he tried to rape her in the communal bathroom of a studio.

Deen did not return CNN's request for comment, but he addressed the allegations on Twitter on Sunday, calling them "egregious."

"I want to assure my friends, fans and colleagues that these allegations are both false and defamatory," he said

"I respect women and I know and respect limits both professionally and privately," he said

Poised for crossover success

Because of Deen's nice-guy image, the allegations are drawing comparisons to the sex abuse scandal involving Bill Cosby -- another controversy involving allegations seemingly incongruous with Cosby's public persona as a moralizing father figure.

The fallout for Deen has been swift.

He has stepped down from the board of the Adult Performer Advocacy Committee, the group said in a tweet . San Francisco-based Kink.com, a network of BDSM and fetish sites that featured Deen as a performer, is ending its relationship with him.

"For the Kink.com community, as well as the larger BDSM community, consent and respect are sacrosanct. Effective immediately, Kink.com will cease all ties with James Deen, both as a performer and a producer," the company said in a statement.

Deen's appearance, which defies the cliche of the male porn star with bulging biceps, spray tan and frosted tips, has made him the subject of numerous profiles dubbing him an "Internet sensation" and "the porn star next door." His appeal -- often likened to a high school crush -- has earned the label of cipher for female desire, an average-looking guy who makes porn palatable for women.

His allure extends to teens and college students who share pictures and videos of him on Tumblr and elsewhere.

The labels set him on the cusp of crossover success, beginning with his turn in Paul Schrader's 2013 film "The Canyons" opposite Lindsay Lohan , followed by "What Would James Deen Do?" his sex column for women on The Frisky.

The women's blog announced it would be ending his column in light of the allegations.

Allegations that he violated consent made it "impossible" to continue working with him, Frisky editor-in-chief Amelia McDonell-Parry said.

"I very much liked James Deen. I enjoyed working with him on WWJDD. I asked him to do an advice column because I liked his directness and his confidence, but most of all, I liked his emphasis on communication, honesty and, most of all, CONSENT," McDonell-Parry said in a post explaining the decision

"No amount of good rapport between us or traffic to his columns would EVER supersede the fact that I BELIEVE WOMEN."

'I was afraid'

Lux said fear of not being believed kept her from reporting the allegations to law enforcement or speaking up sooner. CNN has reached out to the other women, but it is not immediately clear whether they went to authorities.

People tend to believe that sex workers place themselves in harm's way, and therefore can't be assaulted, Lux wrote in The Daily Beast.

"Despite porn being a legal form of sex work, and it occurring in a controlled environment such as a porn set, this blame-the-victim mentality is still inherent in much of society. In turn, sex workers are silenced and our negative experiences are swept under the rug as we try to protect ourselves from the judgment of others -- or worse, a variety of problems ranging from further physical attacks to professional issues such as slander and/or blacklisting.

"Simply put: I was afraid."

As the editor of a site that frequently features stories of sexual assault and victim blaming, The Frisky's McDonell-Parry said she stands with women such as Lux and Stoya.

"The court of public opinion is not a court of law, and I don't need Stoya or any woman to 'prove' that she has been raped for me to believe her. Women who come out as rape victims are far, far, far too often not believed."