Tampa Bay QB Tom Brady got super candid about his relationship to Gisele Bundchen and their past problems in a new interview with Howard Stern.

Speaking on The Howard Stern Show on Sirius XM on Wednesday, the six-time Super Bowl champion and former New England Patriots quarterback recalled a time his marriage to the Brazilian supermodel was in jeopardy.

"A couple of years ago...she didn't feel like I was doing my part for the family and she felt like I would play football all season and she would take care of the house," Brady said. "And all of a sudden, when that season ended, I'd be like, 'Great, let me get into all my other business activities, let me get in to my football training,' and she's sitting there going, 'Well, when are you gonna do things for the house? When are you gonna take the kids to school and do that?' And that was a big part of our marriage, that I had to like, check myself because she's like, 'I have goals and dreams too...so you'd better start taking care of things at the house."

Tom Brady &His Kids' Cutest Moments

"So two years ago, as it related to your football for me, I had to make a big transition in my life to say, I can't do all the things that I wanted to do for football like I used to," he said. "I gotta take care of things in my family because my family- the situation wasn't great. She wasn't satisfied with our marriage. So I needed to make a change in that. Her point was, 'Well, yeah, of course this works for you. It all works for you. But it doesn't work for me.' Because you can get caught up in your life where you think a relationship's great because it only works for you, and the point of a relationship [is] it has to work for both. You'd better work on both 'cause if you don't, then ultimately, it's not sustainable."

Brady said at one point, his wife wrote him a "very thought-out letter" about their marriage, that he still keeps in a drawer.

"It's a good reminder for me that things are gonna change and evolve over time," he said. "There's family decisions that are made and I think most men want to focus on their career, and it's very easy for men to focus on their career."

Brady and Bundchen have been married since February 2009 and share two children, son Benjamin, 10, and daughter Vivian, 7. He also shares a 12-year-old son, Jack, with ex-girlfriend and "Blue Bloods" star Bridget Moynahan.

"I didn't want to have kids or get married until my late 30s," Brady told Stern. "I remember thinking, I don't want anything to get in the way...once my football career kinda got rolling, I thought in my mind, why don't I just wait until I'm basically done playing, which I thought would be my late 30s, and then I'll figure out the rest from there. But that wasn't in the plan."

Brady said he and his wife met in December 2006 through a mutual friend and had their first date at a wine bar in New York City.

"I was just getting out of a relationship, so for me, it wasn't like I was so ready to look for a different person, but she just came into my life at the right time," he said. "And it was an interesting time because we found out after that that my ex-girlfriend was pregnant with my oldest son. We were forced into this very important thing that was happening in our life at a new part of our relationship. So in a lot of ways I found out a lot about Gisele through this experience. She found out a lot about me."

"Again, I didn't think I was gonna have kids for another 10 years," he continued. "And next thing you know, I found out that Bridget was pregnant with our son. So that was a very unique time and it...challenged me in a lot of ways. It was very hard for my wife to think that she fell in love with this guy and then...his ex-girlfriend's pregnant. And it was very challenging for my son's mom, 'cause she didn't envision that either. But I think we all made the best of the situation and fortunately, he's the most amazing son at 12 years old. My son Jack is like my, I mean there's like, my heart explodes when I think of him. He's the greatest kid you could ever ask for."