David and Yolanda Foster‘s marriage broke down in part over her pursuit of fame on The Real Housewives of Beverly Hills, according to a source close to the family.

In the days since the couple announced the end of their four-year-marriage, there has been much speculation over exactly what lead to their split.

Get push notifications with news, features and more.

And a source tells PEOPLE how David, 66, struggled with his wife’s bid to star on the Bravo show – and indeed, only appeared on the show to make her happy.

“Yolanda’s obsession with fame for her and her children is the cause of the split,” a source tells PEOPLE. “David didn’t want to be part of the show, but he knew how important it was to her. He wanted to make her happy. ”

A source close to Yolanda counters: “David enjoyed being on the show. He gained a whole new set of fans through Housewives.”

David and Yolanda announced their split last week after nine years together and four years of marriage.

“What’s so hurtful now is that there are these stories perpetuating that he’s a horrible guy, who’s left her out to dry,” says a source.

“David would have probably worked on the marriage,” adds the source. ” He’s 66 and he doesn’t want to be alone and on his fourth divorce.”

Yolanda has three children from a previous marriage to Mohamed Hadid: Bella, 19, and son Anwar, 16, and supermodel daughter Gigi, 20.

RELATED VIDEO: Yolanda Foster’s Heartbreaking Confessions