The pair talk to Teen People about their 8-year age difference and plans for the future

Rocker Joel Madden, 26, and his pop star girlfriend Hilary Duff, 18, say their much-discussed eight-year age difference is no big deal now – but it wasn’t always so easy, Teen People reports in a new interview with the couple.

“I was like, ‘I don’t know what I’m doing,'” Madden says of his early courtship with Duff. “I didn’t care what the press would say. I cared what my mom would say. But I couldn’t get (Hilary) off my mind. We were just friends for four or five months.”

Get push notifications with news, features and more.

In fact, Madden tells the magazine that if he had a daughter who wanted to date a guy with such a big age difference, “I wouldn’t let it happen.”

Even after they became a couple, the pair took things slowly. “Joel is the most respectful boyfriend I’ve ever had,” Duff says. “We had five or six dates before we even kissed.”

As for their relationship now, Madden says he’s totally committed to Duff. “I think it’s because I came from a broken home, but my whole life, all I wanted was some stability,” says Madden, who was 16 when his father left his mom and the rest of the family. “For the first time in my life, someone has given that to me, and that’s priceless. So I don’t even think about not being faithful. To me, cheating is lame. And I think flirting is cheating’s ugly cousin.”

When asked about whether they plan a walk down the aisle, the couple says don’t expect wedding bells anytime soon. “We think about the future, but not marriage,” Madden says. “I just hope we’re always happy.”

Adds Duff: “I don’t ever think about being apart from Joel. We just feel so comfortable with each other, and we know that we’re not going anywhere.”