Matt Lauer probably won’t attempt a comeback once the dust has settled from his termination from the Today show, a former and longtime colleague tells PEOPLE.

“I think he’ll retire. He should anyway. He had made his money and said all he wanted to do is be with the kids and golf,” the source says. “He wanted to run away and live with the money he had made and be with his family. This is definitely not the way he wanted to go out, but I don’t think he feels the need to be on air like some people, doing specials. I think he’ll have a quiet life and go golfing a ton.”

Get push notifications with news, features and more.

NBC announced early Wednesday that Lauer, 59, had been fired after the network received a “detailed complaint from a colleague about inappropriate sexual behavior” on Monday, with reason to believe “this may not have been an isolated incident.” A source told PEOPLE that Lauer was let go due to sexual misconduct throughout 2014, including at the Winter Olympics in Sochi, Russia. (A second source with knowledge of the situation says Lauer had viewed this relationship as “consensual” and was “dumbfounded” by the Monday accusation.) Women have also anonymously accused him of sexual harassment and assault.

Image zoom Matt Lauer Charles Sykes/Invision/AP/REX/Shutterstock

In a statement released Thursday morning, Lauer said, “Some of what is being said about me is untrue or mischaracterized, but there is enough truth in these stories to make me feel embarrassed and ashamed. I regret that my shame is now shared by the people I cherish dearly.”

“He was a flirt. A philanderer. He did his thing. So that marriage was never good. They lived totally separate lives,” his colleague says. Still, “He loves his kids. When you see him as a dad to those kids, he is a solid father with no holes in that record.”

The source adds: “He is a player. Not a predator but a player. He didn’t have to leer or hunt, he was just charming and people wanted to listen because he could hold a room with his sense of humor.”

Still, this source says he was the picture of professionalism when they worked together at NBC, where Lauer covered news for 25 years. The source considered him a mentor and “amazing leader” known for quiet acts of kindness, like offering up his agent to handle someone else’s contract or paying the fee for behind-the-camera staffers to receive their Emmys.

“This is the head of the show, and he was doing good deeds behind the doors,” the source explains. “People would say, ‘Go to Matt.’ He would pay off people’s bills. His driver passed, and he paid for all his funeral bills so the family wouldn’t have to.”

And when a crew member’s wife learned she had a tumor, “they were trying to get a laptop for her so she could do something during chemo,” the source says. “Matt bought the laptop and told them to keep all the money they raised for medical bills.”

The source also lauded his work ethic: “He never stopped prepping and following up and working hard as hard as he climbed. He treated the security guard the same as he would treat a CEO.”

The source described Lauer’s relationship with the woman who filed a sexual harassment report as consensual for months” but acknowledges in this post-Harvey Weinstein climate, “How can you really understand consent when he is the main anchor? Unless it was going to be a super serious relationship and be disclosed, then he was more interested in realizing nothing was forever. He loved his kids and had an arrangement with his wife. … I don’t doubt that Matt had inappropriate relationships with people at work. That part I’m not denying. He would admit it.”

Still, the source says, “I never witnessed him being anything other than, ‘I got your back.'”

Image zoom From left: Jack Matthew Lauer, Romy Lauer, Thijs Lauer and Matt Lauer James Devaney/GC Images

Overall, the source says those who worked at Lauer’s side are saddened by the allegations.

“He had so much good. He was careless, reckless, distasteful, but it’s not what we’re reading,” the source says.

The source has emailed with Lauer since the news broke, and he sent a brief response.