One day about seven years ago, Joel Nowak sent a letter to friends and coworkers announcing his plans. Ten years before, he had begun identifying as a woman: he’d changed his name, taken hormones, had surgery. Now he would be transitioning again—retransitioning, as he puts it. He would be a man once more.

“I can remember when I first started buying clothes from the men’s rack at Old Navy again,” he says now. “It felt kind of scary, but was also kind of a rush.”

Nowak had concluded that he was female at his core and decided to transition when he was in his late twenties. “This is something I had felt since I was a kid,” he says. “I had questioned my gender identity since I can remember.”

“It was really an exciting time,” he said. “I felt self-actualized. It was something I had wanted to do all my life.”

But after transitioning, Nowak felt haunted by the gender he’d been assigned at birth. “I started to have doubts,” he says. “I had come to believe that I was basically a female in a male body—but as time went on, I realized, well, no, in a lot of ways I do feel that I am male, and I am actually kind of okay being male. … I was in this closeted mode of trying to deny being male and trying to cover up maleness and being uncomfortable with my history and my past and the physical remnants of being male.”

So, Nowak began to detransition. “It felt liberating for me to give all that up and say to heck with that,” he said, referring to the constant worry of appearing male instead of female.

In the wake of Caitlyn Jenner’s Vanity Fair premiere, a “sex change regret” narrative has entered the mainstream media. Earlier this month, CNN interviewed Walt Heyer, an activist who transitioned to female in his 40s and then back to male less than a decade later. Heyer, who writes for the conservative website The Federalist, regrets his initial transition and argues that transgender people who seek surgery are categorically suffering from “delusional psychological problems.” His message has gained traction among right-wing commentators, including Rush Limbaugh, and outlets like Breitbart News. His story is used by anti-LGBT activists to invalidate the very idea of transgender identities. But those who detransition are not inherently conservative — in fact, some have very radical ideas about gender fluidity. And, as in Nowak’s case, not all examples of detransitioning are so ideologically driven. (There is such a wide range of experiences that this article focuses specifically on those who have transitioned from male to female to male.)

De-transitioning after surgical interventions, as in Nowak’s case, is exceedingly rare. Research has often put the percentage of regret between 1 and 2 percent. A Swedish study of trans men and women who went through what is increasingly referred to as gender affirming surgery (as opposed to “sexual reassignment surgery” or “sex change”) found zero cases of regret out of 59 people; 95 percent of patients felt their lives had improved post-transition.

Sex researcher Christine Milrod says the infrequency with which detransitioning happens is in part because of how determined one has to be to jump through the many hoops required in order to undergo gender affirming surgery, which includes therapy, hormone treatment and living full-time as the target gender. That’s not to mention insurance requirements, financial costs and the difficulty of the surgical process itself. “You’re in the hospital for five days straight, then you get out of the hospital and you’re at home recovering for a month, you’re sitting on a donut. It’s a tough recovery,” Milrod says. “In order to get there, you really need to have thought this through.”

Nowak, who started the blog REtransition to share his story, believes that part of why he initially transitioned was that he confused being turned on by the idea of himself as a woman with actually being a woman at heart. This interpretation jibes with the controversial concept of autogynephilia, which holds that some male-to-female transitions are the result of an erotic attraction to the idea of oneself as a woman. But that experience in no way defines those who detransition post-surgery.

A 41-year-old man who runs the website Third Way Trans and prefers to go by the pseudonym “TWT” says his transition to female at the age of 19 arose partly out of the bullying he had to endure as a “soft, nerdy” kid.

“It was the deep rooted idea that I could not simultaneously be myself and be a man at the same time, and so I had to become a woman in order to be fully myself,” he says. “This was a legitimate need to escape the false self I had created in childhood, but instead resulted in trying to live a different false self.”

At the age of 39, he began to transition back to male. In explaining the name of his website, TWT writes, “The third way is a middle way. It … neither suggests that trans experience is invalid, nor says that transition is the only solution to gender issues.”

Nowak is of a similar mind. “There’s always been repressive people who say ‘Repent!’ or ‘Take that dress off!’— not that kind of B.S.,” he says. “I’m hoping that as we talk about gender and move forward, maybe there will be more options for people.”

Detransitioning is actually far more common in the stages before surgery, when people are still exploring their options. “There are people who take hormones and then decide to go off hormones,” says Randi Ettner, a therapist who has served on the board of the World Professional Association for Transgender Health. “That is not uncommon.” This can happen for a number of reasons, but a common one is the difficulty of transitioning. “Some people begin a transition and decide that the social or economic costs are too high,” she says. Those social costs can include strained relations with unsupportive significant others, friends and family members.

Ami Kaplan, a therapist specializing in gender issues, says that most people who begin a transition are “truly transgender,” but that some get discouraged along the way. “It might be too difficult,” she says. “Not everybody can do a complete transition, really, because of the stigma and discrimination and the losses they would encounter if they were to go ahead.”

Ettner says people also sometimes start and stop hormones as part of an exploration of a more fluid gender identity. “There’s a new-ish phenomenon which is genderqueer—or people who are non-binary,” she said. “In other words, they’re not really suffering or experiencing clinically significant stress, but they do perhaps want to masculinize a bit or appear more feminine. So they may experiment with hormones.”

In Milrod’s experience, regret after taking hormones is exceedingly rare. Over the past two decades, she has counseled hundreds of clients through transitions and those who are merely “engaging in gender play,” as she puts it, generally demure when she offers the option of taking hormones. “In my twenty years, and I’m really thinking here,” she says, pausing, “I have had one person regret and stop the hormones.”

“We used to have this idea that the only people who could have hormones or transition were people who fell into this very small category of being true transsexuals,” Ettner says. “Now we realize that this is a spectrum and that gender is sort of like a fingerprint. It’s really a very individual concept.”

This presents some challenges when dealing with doctors who often view gender in strictly binary terms, though.

“That’s what we struggle with every day, trying to find a surgeon who will do breast surgery on a female to male person who doesn’t want hormones, but who does want the obvious secondary sex characteristics of female bodies,” she says. “It’s these mosaic presentations that we have to sell to the surgeon—and sometimes they buy in and sometimes they don’t.”

In that sense, there are some detransitioning narratives that can actually sound kind of progressive. And for every Walt Heyer, there is at least one Joel Nowak, who says he doesn’t regret transitioning in the first place. Sure, his life would have been easier, but he says, “I think I made the decision that I needed to make at that time. It is what it is.”