On the cover of The Sound of Gravel is a photograph: four tow-haired children squinting into the sunlight, smiling, hopeful – the kind of Polaroid that could have been pulled from just about any family album. But there’s a devastating story behind this picture and this family: one that Ruth Wariner recounts in her powerful and poignant memoir about growing up in a polygamist community.

Wariner is, as she writes, her mother’s fourth child and her father’s thirty-ninth. Her father, the founding prophet of the colony, was murdered when she was three, and her mother was remarried to another member of the community. The details of their life are harsh: The family lives in a dilapidated farm in rural Mexico – one without electricity and modern plumbing. The men pick up odd jobs; the sister-wives bear their children, and travel back and forth between their home in Mexico and the U.S. to collect welfare checks.

Get push notifications with news, features and more.

A vicious fight between Ruth’s mother and stepfather sends the family fleeing to her maternal grandparents’ home in California. It is there that Wariner discovers that there is a better world beyond the colony – one with hot showers, Scooby-Doo cartoons, and playgrounds filled with swing sets and friendly kids.

But her happiness is short-lived as the family returns to the compound. Fed up with a life of poverty, her lack of education (she’s pulled out of school to help care for her siblings), and constant abuse at the hands of her stepfather, Wariner starts to question the rules of the church and her way of life.

“I knew my life wouldn’t be happy if all it amounted to was having several children by a shared household,” she writes. “I wanted love, but Mom couldn’t teach me how to get that because she didn’t know herself. She couldn’t show me how to be happy; only how to survive.”

It is a devastating tragedy that convinces a 15-year-old Wariner to break away from the only world she has ever known, in search of a better life for herself and her three younger sisters, then ages four, two and seven months. They return to the States, where Wariner raises her siblings while getting her GED, and puts herself through college and graduate school.

Now happily married and a Spanish teacher in Portland, Oregon, Wariner talks to PEOPLE about her heartbreaking, haunting, yet ultimately uplifting story.

Why did you decide to write the book – beside the fact that you had an amazing story to tell?

When I was in my early 20s, my three little sisters – Elena, 12, Leah, 10, and Holly, 8 – who I was raising at the time in Oregon, started asking questions about our mother and what happened to her. I realized, “My gosh – my sisters don’t remember our mother!” The idea to write the book happened then. Years later, when I finished college and started teaching, I went back to school and started taking writing classes. By that time, my sisters had grown up and moved out, but I wanted them to know our history: where we came from, what happened to our mom.

There are so many intense and emotional scenes in the book. It must have been hard to recall those memories.

Writing about my mother was hard. I remember looking at photographs, and writing down the details of what she looked like, what she smelled like, what our conversations were like, and I would just bawl. I would literally have to step away from the computer for days. At the same time, it was cathartic; there was a release of a lot of pain. Writing about my mother, as an adult, helped me to understand her situation better and have more compassion for her. As a kid, you don’t realize that your mother is craving love, and that she has this belief system that is holding her back. She has all of these kids – three of them special-needs children – and her religion is telling her to have more, and doesn’t give her much choice about her own life. I was able to see her in a different light.

The living conditions in the colony were dismal. There was constant tension and jealousy among the sister-wives. Why do you think your mom stayed in the community?

I think she felt like she belonged there. Once she married my father, a prophet, she felt anointed. They brought her in and made her feel special, and she had a place in the world. I don’t know if she was satisfied with that way of life, but that was her comfort zone – the only life she had ever known. Also, when you go to church every Sunday and they tell you this way of life is the only way you’re going to see God again, it influences you. That said, I think she felt lonely, even with all of us kids. Writing about her made me realize that she didn’t have a lot of self-love. She didn’t feel like she deserved better, so she kept falling back into this situation. In a way, her lifestyle and belief system validated the way she felt about herself. Of course, this is just me, trying to make sense of it all.

One of the most disturbing parts of your story: You telling your mom, repeatedly, that your stepfather has been sexually abusing you. Her response is stunning – basically telling you to forgive him and get over it.

It completely crushed me. I don’t completely understand why she stayed, but again, I think it falls back on her belief system. The women in that church felt like they had to be married in order to get into the kingdom of heaven. They also rationalized the abuse: It was like, “Hey, it’s no big deal. It’s not like he’s raping these girls.” In their minds, it just wasn’t that bad. As a child and teenager, I had so many conflicting feelings about my mom. She was a loving person, who always tried to make things fun for us – buying us little gifts on Christmas and Valentine’s Day, even though we didn’t have a lot of money. But when she decided to stay with my stepfather when he was physically abusing her, and then when she learned he was abusing me, that was hard to take. I adored her, so it was hard to see her make those choices. I was angry, yet I felt guilty about that anger because she was a good person.

After the horrific beating at the hands of your stepfather, your mother flees with you and your siblings to your grandparent’s house in California. That must have been a huge culture shock.

It was! But in a lot of ways it was a good shock. I had started school in Mexico and couldn’t understand what the teacher was even saying, since I grew up in an English-speaking household. In Strathmore, I could understand the people around me. My teacher knew that I came from a poor family, and looked after me. And I finally had a chance to be a kid – having fun on the playground, eating Lucky Charms cereal, watching cartoons, listening to Simon and Garfunkel and Elton John. All things we hadn’t been exposed to before.

The family moved around a lot – going from the Mormon community in Mexico to a cramped trailer in El Paso.

By the time I was in 7th grade, I had gone to four different schools in two different countries! I was always behind. Actually, the reason I decided to go back to school when I finally left home wasn’t just to make a better life for me and my family – I also loved learning. School has always been a safe, fun place for me. At Southern Oregon University, I’d discuss poetry and listen to other people’s views on philosophy and world religions. I realized, “I don’t have to live one specific lifestyle!” It gave me a sense of freedom.

At one point, when the family is living in a cramped trailer in El Paso, people from social services stop by. They seem to have a sense of the dire situation, but don’t do much about it.

At the time, I was relieved they didn’t take us! I didn’t want them to intrude in our lives. But as an adult, writing about that experience, I was like, “Wow, I can’t believe they left us there!” But kids in this country are left in abusive households all the time, and the law can only do so much. I recently met a cousin who I’d never met before, and she talked about being left alone with her siblings all the time, while her mother went off with her father, never knowing where she was. It was what women in our community did.

How were you and your three younger sisters able to survive when you moved to the States?

We stayed with my grandmother for four years, and she and I took care of my sisters while I started taking classes. Then we stayed with an aunt and uncle in southern Oregon for a while, and spent summers with my older brother Matt and his first wife, Maria. We moved out on our own when I was 19, and my sisters were 4, 6, and 8. I rented a trailer and supported the four of us by working as a secretary at a wrecking yard. The Farmer’s Home Association was giving out loans, so low-income families could buy homes. My boss helped me fill out the application and a week later, they called. My grandmother paid the $700 down payment. I was 19 when my sisters and I finally had a home of our own.

How where you able to study, while raising your sisters?

I took classes at the community college and worked part-time while they were at school. I wanted to keep us together. In my heart, it was the right thing to do. Sometimes I didn’t sleep at night because I needed to get my homework done. We got welfare and I took out student loans. It was hard, but I was in survival mode.

Are you still religious?

I consider myself a spiritual person. I believe in God, but for me, it’s much more personal. I like to read books about improving yourself and connecting with yourself – books by Marianne Williamson, for example – rather than going to a church and having somebody tell me how to live my life. Organized religion is tough for me, to be honest, probably because I have had it thrown down my throat so much.

How has your past helped you become the woman you are today?

Being an older sister in a large family and having my brothers and sisters to look after gave me a sense of purpose and a strong work ethic that has served me well – getting through college and becoming a teacher. In spite of how hard that life was, I was able to make a better life for myself and my sisters. It also gave me perspective. I’m grateful for my life right now, particularly after what I came from. You realize what’s really important, and for me it is family. My siblings and I remain close.

At times, it must feel like you’ve been through a war together!

Sometimes I wonder if I have PTSD! Actually, I’m kidding, though my body still tenses up at times when I talk about my past. We’re all really close now. My younger brother and sister, Aaron and Elena, live in Seattle, and work at Boeing. My second brother, my special-needs brother, Luke, lives in Oregon. He has his own apartment and a job, and is involved in the Special Olympics. My aunt helps him with things like budgeting and grocery shopping. My older brother Matt is a devout fundamentalist, who still lives in Mexico and El Paso. And yes, at various times in his life, has had multiple wives. He and his second wife are still together, though. That’s the way it is: You are in and out of these polygamist relationships. If things don’t work out, the wives just leave.

What has been your siblings’ reaction the book?

They loved it – it made them cry – and they feel like they got to know their mother better, which is why I wrote the book. Even Matt, though he’s still in the church, is excited about the book. It was healing for him to understand my perspective of our lives, and why I have decided not to return to LeBarone, which is something that has been hard for him.

Did your experience in a polygamist community affect your own personal relationships?

Absolutely. When I was in my teens and early 20s, I’d end up with boyfriends who were apathetic and emotionally available. I began therapy and started recognizing a lot of the same patterns that my mom had made in her relationship – being with a man who was never around, who kind of love her but kind of didn’t. Eventually, I started questioning the choices I was making, and started dating nicer men. I met Alan after my sisters had moved out. I had a career and was feeling comfortable with myself. He was everything I wanted, and everything my mom wanted but never had in a relationship.