Thomas Ravenel's Southern Charm costars have all but cut ties with him since filming on season 5 wrapped, a source tells PEOPLE

Thomas Ravenel‘s Southern Charm costars have all but cut ties with him since production on season 5 wrapped, a source tells PEOPLE.

Get push notifications with news, features and more.

“He doesn’t have friendships with the [main] cast anymore — everyone is done,” the insider explains. “They’re disgusted with him and the way he and Ashley treated Kathryn this season. His eye-opening behavior and the drama surrounding those allegations caused a huge wedge in his friendships.”

“There’s too much dishonesty there, too much manipulation,” the source continues. “They’ve all just been keeping their distance.”

WATCH: Southern Charm’s Thomas Ravenel Accused of Sexual Assault, Allegedly Paid Accuser $200,000

Representatives for Bravo had no comment, but the source adds that only Whitney Sudler-Smith and J.D. Madison — who recur on the Bravo reality show —have remained in contact with Ravenel, as has Dennis. The two share 50/50 custody of their daughter Kensie, 4, and son Saint, 2½.

Dennis was surrounded by some of her Southern Charm cast mates over the weekend, when Rose, pal Danni Baird, and former-foe-turned-friend Patricia Altschul — Sudler-Smith’s mother — helped throw her a surprise 27th birthday party.

“Best birthday that I have ever had… and also the only surprise party anyone has ever held for me!” Dennis tweeted. “Thank y’all for making my day so special!”

Ravenel was not invited to the party, the source says — though it appears unlikely he would have even attended.

“Thomas been like a hermit, staying close to home at his Brookland Plantation,” reveals the source. “He’s completely distraught over the allegations and what’s happened with the show.”

“He’s bounced back from scandal before, but this is much harder now that he has kids,” the source continues.

Image zoom Thomas Ravenel Rodolfo Martinez/Bravo/NBCU Photo Bank via Getty

Ravenel has had a history of trouble with the law, dating back to 2007 when — as South Carolina’s treasurer — he was indicted on federal cocaine distribution charges. He was sentenced and served 10 months in jail, and went to rehab as well.

Back in May, news broke that Ravenel was being investigated by South Carolina police for the alleged sexual assault of his nanny. Real estate agent Ashley Perkins has also accused him of assaulting her mom Debbie Holloway Perkins after they met on Tinder in October 2015. Ravenel has denied the accusations through his attorney.

Since then, Bravo and Haymaker (the production company behind Southern Charm) have launched an investigation into the claims, saying they “take allegations like these very seriously.”

“Once all the information is gathered and carefully reviewed, appropriate action will be taken,” they said.

No update into the investigation has been provided yet, but Ravenel did skip the season 5 Southern Charm reunion, which taped in June.

RELATED VIDEO: Southern Charm Kathryn Dennis on ‘Natural Chemistry’ with Thomas Ravenel: ‘There’s Common Ground from the Past’

When the reunion aired in July, Jacobs said that the allegations have changed Ravenel.

“I have seen a new Thomas,” she explained. “I mean, it’s humbling, of course, and now he’s [like] ‘Let’s stay in, let’s watch movies, let’s cook dinner, let’s just stay under the radar right now.’ And I’ve just seen a much softer, more sensitive, kinder, more supportive boyfriend. You know, everyone wants to be with you when you’re on top. Who’s with you when you’re down there? And I’ve told him, I’ve just said, ‘I’m not leaving.’ ”

But the rest of the cast wasn’t convinced by her insistence that she and Ravenel are in a healthy relationship and alleged Jacobs was being manipulated by Ravenel in the same way he once controlled her enemy, Dennis.

“You’re exactly where Kathryn was,” said Conover, with Eubanks adding, “You’re lying to yourself, Ashley … What does it say about a man who talks poorly about the mother of his children? … He’s been a not nice person.”