A never-before-seen photo surfaces of the forgotten Kennedy, who, after a disastrous lobotomy, was rarely heard from again

There was one Kennedy that the world never got to know: Rosemary, the third of Joseph P. Kennedy and Rose’s nine children.

Born in 1918, she was mentally impaired and never truly fit in with her family.

Get push notifications with news, features and more.

“The shame of her disability was our family secret,” says her nephew Timothy Shriver, 55, who includes her story in his new book, Fully Alive.

“She grew up in a time where there was enormous shame surrounding children with special needs,” Shriver, the son of Rosemary’s sister, Eunice Shriver, tells PEOPLE. “People didn t want to admit it.”

This included the highly ambitious Kennedys. At age 18, Rose was still struggling at the fourth grade level. Prone to rages and running away in the middle of the night, she became hard to control as she grew older.

At age 23, her father scheduled a lobotomy, without telling his wife, Rose. “It went terribly wrong” and left her with limited motor and speaking skills, Shriver says.

Rosemary spent the next six decades hidden from the public in a Wisconsin Catholic institution, where she was cared for by nuns. She died there in 2005 at age 86. Her father never visited her again, and her siblings rarely spoke of her.

“I don’t know how other than just from sheer devastation he [Joe Sr.] could allow her to disappear so much from his life,” Shriver says. “He had to be destroyed by it. That’s the only way I can explain it. It’s an incredibly heartbreaking story of a dad trying to help his daughter and hurting her. What could be worse?”

Shriver, now chairman of Special Olympics, which his mom founded in 1968 for people with intellectual disabilities, long grappled with the silence that surrounded Rosemary.

“It was a relentless question lurking behind the curtain,” he says. Shriver decided to include Rosemary’s story, along with those of others with intellectual disabilities who’ve inspired him, in his book, excerpted in this week’s PEOPLE. He credits her as the inspiration for his family’s focus on service.

“Why are we hiding from this story?” he asks. “There’s nothing to fear here. It’s a human story. It’s a heroic story. Her role is a powerful part of my life.”

For more on Rosemary Kennedy, pick up this week’s PEOPLE, on newsstands Friday

Want more stories like this?

Sign up for our newsletter and other special offers:

Thank you for signing up!