In 2005, Jessica Simpson wore the famed Daisy Duke shorts for her scene-stealing role in Dukes of Hazzard. She was praised at the time for her body — and her legs in particular — but the singer says it set her up for years of “scrutiny” any time her weight changed.

Writing in her new memoir Open Book, out today, Simpson, 39, says that the Daisy Duke moment “created a gold standard Jessica, the ‘before’ for every ‘is she fat or is she thin’ story for the rest of my career.”

And those body expectations did follow her for the next 15 years, through three pregnancies and the infamous “mom jeans” look in 2009 that had people calling her “fat.” Simpson tells PEOPLE that she never expected that level of judgement when she signed her record deal at age 17.

RELATED: Jessica Simpson Reveals in Brave New Memoir: ‘I Was Killing Myself with All the Drinking and Pills’

Warner Bros/Kobal/Shutterstock More

“I thought it was about my voice. I didn’t know that it was going to end up being about how I looked in a dress,” she says. “It’s heartbreaking and I mean, I punished myself for it. I took diet pills. I heard it and I couldn’t not hear it in the back of my mind every time I was on stage, every time I walked out the door.”

Simpson says her own insecurities — and ones she had never thought of — were constantly thrown at her.

“It is not easy, it hurts,” she says. “And still, I feel like we all look in the mirror and are not 100% all the time. I mean, we all see our flaws. Some, the others don’t see. And mine were just out there for the world to rip apart, when they weren’t even flaws. When they were made into flaws that I didn’t know I had.”

Jessica Simpson | Mike Rosenthal More

Now, as a mom of three and after another tough pregnancy that put her at over 200 lbs., Simpson says that she “just can’t read” the comments about her body anymore, even the ones that are veiled compliments about her weight loss.

“Even now, people [are] commenting on my Instagram, ‘Oh, snap back?’ [about her weight loss] No, it wasn’t a snap back and I don’t even know what that word means,” she says. “It’s like, I work hard and when I work out, a lot of it is to release anxiety. That’s one of my tools for sobriety. Just walking, just going and talking, walking and talking with my husband. Even some of my biggest fans… They’re saying it as a compliment, but it’s like, that’s not what I was trying to get with this picture but okay.”