"I would've loved to have had that conversation a little bit sooner, not leading into the week of Fantasy Suites," Peter Weber told PEOPLE

The Bachelor‘s Peter Weber is opening up about the serious conversation he had with Madison Prewett.

Exclusively speaking with PEOPLE at The Bachelor: Women Tell All taping last week, the 28-year-old pilot addressed Prewett’s perspective about potentially not moving forward in their relationship if she found out he had slept with the other contestants during Fantasy Suites.

Get push notifications with news, features and more.

“I would’ve loved to have had that conversation a little bit sooner, not leading into the week of fantasy suites,” Weber told PEOPLE. “But I also don’t fault her at all for what she told me, because I was very clear with all the women from the very first night to please share what’s ever on your heart.”

“Do not hold anything back, and if it’s on your heart, I need to know that,” he added. “And that’s all that she did. People try to say it’s an ultimatum. It wasn’t. She simply felt I needed to know what she was feeling and I could see how uncomfortable it was for her to have that conversation with me. But at the end of the day, I’m a hypocrite if I tried to say that that wasn’t right for her to do that.”

Image zoom Madison Prewett / Peter Weber ABC

During Monday night’s episode, Prewett — who had revealed she is saving herself for marriage — pulled Weber aside to address her “expectations” heading into the Fantasy Suite dates.

“I don’t think Peter knows how much I’m struggling with moving to the next level of the Fantasy Suites and an engagement,” she admitted. “That is a really huge, major step. My faith is literally everything to me and I have saved myself for marriage. I can’t keep moving forward and continuing this relationship if I don’t get the clarity and peace and confidence that I need.”

“In no way do I want to give you an ultimatum,” Prewett said. “But for me, actions speak louder than words.”

After two dreamy dates with Hannah Ann Sluss and Victoria Fuller, Weber revealed to Prewett he had been intimate with other women.

“I don’t want this to feel like it’s all about sex, but this is really a big thing right now for us,” he told Prewett during a sit down dinner. “I don’t know how else to do this. I’ll just be very honest and I won’t give any details, but I have been intimate, and I can’t lie to you about that.”

Will you accept this rose? Sign up for PEOPLE’s free weekly Bachelor Nation newsletter to get the latest news on The Bachelor, The Bachelorette and everything in between!

Prewett, then, excused herself.

“I feel really, really hurt and really let down,” she said in a confessional. “He knew coming into this week, because I had made it clear, that there was a chance that he could lose me if he made certain decisions and he made them anyways. I’m so frustrated with him, I’m so mad at him.”

After going after her, Weber said, “If you can honestly see you and me together forever, do not walk away. Please don’t. I know that I hurt you and I’m sorry for that, but please don’t walk away from this.”