"The network and producers are going to have to decide if he's such a liability that they have to take stronger action," a source tells PEOPLE

Today Show Coworkers Want Billy Bush Out: 'It Really Seems Like It's Time for Him to Go'

The Today show has been rocked by the recent Billy Bush/Donald Trump controversy.

A source tells PEOPLE the atmosphere around set Monday morning feels “surreal” after Bush’s suspension following the leak of a tape in which the star made lewd comments about women while filming a segment with Trump for Access Hollywood in 2005.

Get push notifications with news, features and more.

The source says there are lots of “gossipy conversations” about Bush flying around set of the NBC talk show, and that the prevailing attitude is that they don’t want him to come back, especially the women.

According to the insider, Bush, 44, “hasn’t really endeared himself to the rest of the cast and crew anyway, so they’re fine with him leaving, if that’s what the decision is.”

“Ever since [Ryan] Lochte, a lot of people have thought that he was a liability anyway,” says the insider. “Tamron [Hall] doesn’t like him and they have always been uncomfortable around each other. Al [Roker] doesn’t speak to him, like at all. There is no love lost there. Matt [Lauer] is okay with him, not friends, but okay. He doesn’t have much of a relationship with Savannah [Guthrie]. So it won’t be a loss to any of them personally if he doesn’t come back.”

“The network and producers are going to have to decide if he’s such a liability that they have to take stronger action,” adds the source. “Does anyone tune in to Today to watch Billy? Probably not really. So if he leaves, it’s not a big deal.”

WATCH: Billy Bush Suspended from ‘Today’ Following Release of Lewd Comments with Donald Trump

And while the source says they would hate to see Bush fired for something he said 11 years ago, this is “more than that.”

“The feeling is, ‘Is he so compromised that he can’t be a neutral journalist? Can people see him as being a professional?’ ” says the source. “It really seems like it’s time for him to go. We’re all waiting to see what happens.”

On Monday, Savannah Guthrie and other Today show anchors throughout the broadcast repeated the news that Bush had been suspended from the show pending a review.

On Friday afternoon, Bush issued an apology for the comments he made in the tape.

“Obviously I’m embarrassed and ashamed,” he said. “It’s no excuse, but this happened eleven years ago — I was younger, less mature, and acted foolishly in playing along. I’m very sorry.”

In an email to the Today show staff this weekend, show executive Noah Oppenheim confirmed that the the co-anchor had been suspended indefinitely from the show pending a review.