A source tells PEOPLE that the family's estrangement is in part due to their differing religious beliefs

Aaron Rodgers‘ questions about his Christian faith may have pushed his family further away.

The Green Bay Packers quarterback, 36, recently sat down with his girlfriend Danica Patrick for a conversation on her Pretty Intense podcast, during which he opened up about not feeling connected to his religious community as a child.

Get push notifications with news, features and more.

After hearing Aaron’s’ sentiments about his Christian upbringing, a source close to the family tells PEOPLE they felt like he had “turned his back on them.”

“There’s clearly a lot more to it, but that’s how he is perceived by his family,” the insider adds. “His comments are very hurtful to the family.”

Image zoom Aaron Rodgers Harry How/Getty

The NFL star and his family have been estranged for several years — including his younger brother and Bachelorette star Jordan Rodgers. While Aaron has not spoken publicly about the reasons behind the estrangement, the insider tells PEOPLE that religion is a part of it.

“They were dismayed. The family is very dedicated to their Christian faith,” the source tells PEOPLE of Aaron’s recent comments. “To them, his comments are basically a slap in the face to the fundamentals of who they are. It’s basically him turning his back on everything they have taught him.”

The insider adds that the family “still loves Aaron very much,” but disagrees with him about fundamental things. “They have these times where things start to thaw out, but then something like this happens, and then it’s back to square one. It’s sad.”

In a 2017 interview with the New York Times, the family patriarch, Ed Rodgers, claimed that Aaron’s rise to fame also caused the rift between the family, telling the outlet that the quarterback had allegedly stopped talking to his family members at the end of 2014, shortly after he began dating actress Olivia Munn.

Image zoom Gregg DeGuire/WireImage

During her own interview in 2018, on Andy Cohen’s SiriusXM radio show Andy Cohen Live, Munn said that Aaron had already been estranged from his family months before they became an item.

“I was friendly with Jordan, and I met the parents only a couple of times,” Munn told Cohen at the time. “Before he and I started dating, he hadn’t spoken to the parents and one brother for like eight months before we started dating.”

Later that year, Aaron sparked some hopes of a reconciliation between the family when he revealed he had celebrated his 35th birthday with his family in December 2018.

However, things seem to have remained tense since then.