Thomas Ravenel tweeted that he will not appear in Southern Charm season 6

Southern Charm Cast Was 'Disgusted' with Thomas Ravenel, Cut Off Contact Before He Quit: Source

A week before Thomas Ravenel announced he’d quit Southern Charm, a source told PEOPLE that many of the reality show’s stars were icing him out.

Ravenel, 55, declared on Tuesday that he will not appear in Southern Charm season 6 as police continue their investigation of sexual assault claims against the former politician.

Get push notifications with news, features and more.

“I’m not doing the show anymore,” he tweeted. “In the contract they have the right to fictionalize your story. They took advantage of me. I decided I’ve got too much to lose and informed them I’m not coming back.”

Image zoom Thomas Ravenel Charles Sykes/Bravo/NBCU Photo Bank via Getty

He then deleted the tweet, and Bravo has not issued any official response.

PEOPLE confirmed in May that a victim reported an alleged incident with Ravenel that occurred in January 2015 to Charleston police. The incidents listed on the investigation report are “sex offense” and “forcible rape.”

RELATED VIDEO: Southern Charm’s Ashley Jacobs Posts About ‘Difficult Roads’ amid Thomas Ravenel Breakup Rumors

And “Nanny Dawn” — the ex-caretaker to his daughter Kensie, 4, and son Saint, 2½ — claimed to PEOPLE that she is the one who came forward. She said she took action after hearing the story of real estate agent Ashley Perkins, who publicly accused Ravenel of assaulting her mom Debbie Holloway Perkins after they met on Tinder in October 2015. (Ravenel denied all the allegations through his attorney.)

Last week, PEOPLE’s source said Ravenel’s costars had been ostracized from his circle of friends over the past few weeks due to the sexual assault allegations and the way he treated girlfriend Ashley Jacobs and mother of his children Kathryn Dennis.

Image zoom Ashley Jacobs (left) and Kathryn Dennis Paul Cheney/Bravo/NBCU Photo Bank via Getty

Representatives for Bravo had no comment, but the source added that only Whitney Sudler-Smith and J.D. Madison — who recur on the Bravo reality show — remained in contact with Ravenel, plus Dennis as they co-parent their children.

“He doesn’t have friendships with the [main] cast anymore — everyone is done,” the source explained. “They’re disgusted with him and the way he and Ashley treated Kathryn this season. His eye-opening behavior and the drama surrounding those allegations caused a huge wedge in his friendships.”