"On their recent trip, they already knew they were splitting up," a source tells PEOPLE of Kristin Cavallari and Jay Cutler

Kristin Cavallari and Jay Cutler surprised fans with their split announcement but a source tells PEOPLE that the pair were struggling months before.

"Kristin and Jay had been having problems for a long time. They hadn't spent much time together in recent months," the source says.

Get push notifications with news, features and more.

On Sunday, Cavallari, 33, announced she and the former NFL quarterback, 36, are getting divorced after seven years of marriage and a decade-long romance. They share sons Camden, 7, and Jaxon, 5, and 4-year-old daughter Saylor.

"With great sadness, after 10 years together we have come to a loving conclusion to get a divorce," Cavallari wrote. "We have nothing but love and respect for one another and are deeply grateful for the years shared, memories made, and the children we are so proud of. This is just the situation of two people growing apart. We ask everyone to respect our privacy as we navigate this difficult time within our family."

Image zoom Cindy Barrymore/Rex USA

Prior to announcing their split, Cavallari and Cutler returned home after a three-week trip to the Bahamas, which was supposed to be a spring break vacation with their children as well as her hairstylist Justin Anderson and his fiancé Austin "Scoot" Rhodes.

"On their recent trip, they already knew they were splitting up," the source tells PEOPLE. "The reason they announced it now is because it’s a quiet time for them to work through all of this and try to move forward."

Image zoom Kristin Cavallari/Instagram

During the latest season of her E! reality series, Very Cavallari, fans saw the demise of Cavallari's friendship with her longtime best friend Kelly Henderson. Also spotlighted was trouble in Cavallari and Cutler's marriage, specifically rumors that he was having an affair with Henderson. (While Cavallari insisted she never believed the rumor, she and Henderson had a falling out.)

The source tells PEOPLE Cavallari and Cutler's split had "absolutely nothing to do" with Henderson or rumors of cheating. "Kristin and Jay are a lot different. She has so much on her plate, with her brands and with the show," the source says.

"Jay is not as interested in all of that. She's so passionate about her work and they couldn’t relate to each other as much. So it pulled them apart," the source adds.

Another aspect of their relationship highlighted in Very Cavallari was how Cavallari and Cutler's roles reversed at home since his football career came to a halt in 2017 when he retired.