Southern Charm is back, but minus a face many fans associate with the show.

For the first time in the series’ history, Thomas Ravenel is not a cast member on the Bravo hit. The one-time politician announced his exit from the series last August, after sitting out the season five reunion. The network opted not to have Ravenel, 56, attend the taping after multiple allegations of sexual misconduct against him surfaced, including an alleged rape in 2015. Ravenel has denied any wrongdoing. A month after leaving the show, Ravenel was arrested and charged with assault and battery in the second degree, stemming from the alleged rape. Ravenel has yet to enter a plea and awaits trial.

"The big change, I thought that it would be, the absence, people actually watching him being on camera, seeing what he's doing," star Craig Conover tells ET of filming season six. "What's interesting is, right when I walked in the first dinner party, truth is we didn't realize how much -- how many things he gave us to talk about. Even if he wasn't there, a lot of the gossip and stuff we do talk about was based around stuff that he did, his antics."

ET spoke with Conover and his co-stars, Shep Rose and Austen Kroll, at the King Charles Inn in Charleston, South Carolina.

"Say what you will about him, the guy was larger than life," Rose offers. "He brings a lot because, I mean, you just never know what he's going to say. Or he shows up in a certain kind of a mood and all of a sudden there's chaos. Yes, he's gone. Yes, there's a void, but we were cautiously optimistic that we could figure it out without him. So we just went in with our head down and tried our best."

Despite his absence from the group, Ravenel is still a large part of Southern Charm’s sixth season, at least in the premiere. The episode kicks off with the remaining cast discussing the charges against him.

That cast includes Ravenel’s ex, Kathryn Dennis, with whom he shares two children, 5-year-old Kensie and 2-year-old Saint. The pair is currently in the middle of a custody battle, which has been sealed by the courts. Parts of the battle have been made public, though, due in part to a lawsuit Ravenel filed against Bravo and the production company behind Southern Charm, aiming to block the use of unaired footage of his and Dennis’ children on future episodes of the show.

"Well, as you know, I have no problem giving my opinion on anything whether anybody asks me or not," Patricia Altschul notes. "But I cannot say anything about Thomas, because he served me with papers. I'm subpoenaed, so you won't get anything from me."

Seeing as Altschul has been subpoenaed as part of the suit, the other cast members admit to being wary of speaking out about it. In his filing, Ravenel claimed producers encouraged Dennis to fight for full custody of the kids in order to stir up drama for the show. Dennis has denied these claims, and Bravo has asked the court to throw out the lawsuit. Dennis says people can talk all they want about the drama surrounding Ravenel, admitting she’s "numb to it" at this point.

"Nothing surprises me," she shares. "It's unfortunate that negative things happen and they're talked about, but it's a part of my life at this point. So I keep living in my bubble and move on. Moving forward!"

"It was a little difficult to kind of figure out," co-star Naomie Olindo tells ET from her family’s restaurant, NICO. "Sometimes we would be talking about these things because it would be very relevant and you know, he was in the paper, and all the stuff with J.D. [Madison]. It was kind of both of them going down together."

Ravenel’s friend, former recurring Southern Charm cast member John David "J.D." Madison, was accused of sexual assault in August 2018. He has yet to address the allegations publicly or respond to requests for comment.

"We wanted to talk about it, because we thought it was important to talk about it," Olindo continues. "For it to be out there and stuff, but we didn't want to upset Kathryn because [Ravenel’s] still the father of her children."

"Look, I empathize with Kathryn, with her situation with custody, with this and that and other," Rose says. "But we don't sit around talking about him, nor do we, I wish him the best let's just put it that way."

"That is the sentiment that we all share, and we kind of leave it at that," Kroll adds.

"Somebody wrote [on social media], 'Well, Thomas isn't going to be on a show. It's going to be boring,' which I took umbrage at," Altschul says. "I think this is going to be the best season because -- and I don't think I am letting the cat out of the bag -- but the guys who have gotten away with murder, are really going to get taken down a peg. There's gonna be a reckoning with all of them."

Even without Ravenel mixing things up, the cast describes the season as lit, real weird, bewildering, riveting and peligroso, the Spanish word for dangerous. That one came courtesy of Rose.

"I've been pretty enthusiastic about the show for the last five seasons and now I'm, like, nervous all of a sudden," Rose confesses. "I’ve been in the ocean for five seasons treading water and I haven’t gotten eaten yet, you know? I’m waiting for the rescue boat so I can maybe get out of season six without getting eaten."

Well, that remains to be seen, as does the possibility of a Ravenel return, which the cast isn’t ruling out.

"I don't see that happening, but honestly there's a lot of things that have happened, that I didn't see happening," Olindo says.

"Thomas is still very much in the mix," Cameran Eubanks admits to ET, sitting inside Gwynn’s of Mount Pleasant department store. "I’m going to say never say never, with Thomas. He has said he would never come back, but I think if he and Kathryn got to a good place, I think he would absolutely consider it."

"I hope, one day maybe," Dennis says of getting to a good place Ravenel. "But I have been hoping for a long time."

Ravenel’s presence is also felt, in a way, thanks to his off-again, on-again girlfriend Ashley Jacobs, who does appear on screen in season six. Stick with ET for more on her return in the coming weeks.

Southern Charm airs Wednesdays at 8 p.m. ET on Bravo, starting May 15.

RELATED CONTENT: