Last week, doe-eyed alt-rocker Beck denied he was a Scientologist, telling the Sydney Morning Herald his previously avowed beliefs were all just a “misconception.” The declaration came on the heels of his 14th studio album, and arrives months after he initiated divorce proceedings against his ex-wife, who also practiced Scientology.

Now survivors of the dangerous celebrity-studded sect like actress Leah Remini and Mike Rinder, a former senior executive of the Church of Scientology International, are bringing Beck to task over how he’s decided to announce his sea change.

“He’s acting like he was never a Scientologist, which is his way of not publicly speaking out and therefore not getting labeled like us and attacked like us,” Remini told The Underground Bunker, a blog run by journalist Tony Ortega, a longtime critic of the church.

“Pussy move,” Remini added. “You can quote me on that.”

Beck specifically told the Herald: “I think there’s a misconception that I am a Scientologist. I’m not a Scientologist. I don’t have any connection or affiliation with it. My father has been a Scientologist for a long time, but I’ve pretty much just focused on my music and my work for most of my life, and tended to do my own thing … I think it’s just something people ran with.” (The Church of Scientology has yet to provide comment.)

It’s not the first time the 49-year-old Beck (who took his mother’s last name, Hansen) has disavowed Scientology—or parried questions about his creed.

As early as May 2001, Beck denied being an adherent to science-fiction scribe L. Ron Hubbard’s sketchy religion. That month, the New York Post reported that “Beck’s closeness to the church has cost him a few band members,” including Joey Waronker and Smokey Hormel, who quit because “they were alienated” after the musician converted. “Beck was drawn into the religion by his father, David Campbell—a Scientologist for the past 10 years—and his bass player, Justin Meldal-Johnson [sic],” the Post noted then. (Meldal-Johnsen is listed on The Underground Bunker as a Scientologist, and Nine Inch Nails songwriter-producer Trent Reznor even addressed Meldal-Johnsen’s faith in an interview, saying the bassist asked him if his belief in Scientology would affect keeping him in the band. Reznor assured him it wouldn’t.)

But in 2004, Beck married Marissa Ribisi, who along with her twin brother, actor Giovanni Ribisi, is a second-generation Scientologist. (In a 2014 interview on Marc Maron’s WTF podcast, Giovanni dismissed the notion of aliens in the church, saying, “I have never ever heard of aliens in Scientology, and I’ve been a Scientologist all my life.”)

Remini told Ortega’s website, “You can’t be married to a Scientology family such as the Ribisi family and not be fully on board. Beck was there every day doing courses and auditing. He was very much a Scientologist. To say that he was more focused on his thing and the press just ran with it is a pussy move.” (For the record, Beck filed for divorce from Marissa Ribisi in February, after 14 years of marriage. It’s unclear if Scientology played a part in the split.)

“ Beck was there every day doing courses and auditing. He was very much a Scientologist. To say that he was more focused on his thing and the press just ran with it is a pussy move. ”

“This is his way of not publicly stating that he was very much in and now he’s not. It’s very clever of him to use these words,” Remini added. “It allows him to evade publicly denouncing Scientology. It’s his way to get around being attacked as we all have.”

Rinder also condemned Beck’s style of defecting from Scientology. “Beck took the chicken’s way out,” he told The Underground Bunker. “To say, ‘I’ve never really been a Scientologist’ is an ‘acceptable truth.’ There is no doubt he was a Scientologist. But it’s a way of avoiding questions like ‘Why did you leave?’”

“For him to actually distance himself is indicative he cannot say nothing any longer and just how toxic it is to be identified as a Scientologist,” Rinder continued.

Beck called Scientology “useful” in a 2005 profile in The New York Times Magazine. “My dad’s been doing it since before I was born,” he said, in his first public acknowledgement of the faith. Asked how Scientology helps him, Beck answered obliquely: “It’s a personal thing. I’m a musician. I’m not, like, a personality. I’ve never really pretended to perform that kind of function.”

Yet, according to one 2008 Vanity Fair article, a friend had suggested Beck wanted to leave the church. Back then, the magazine was looking into the suicides of art-world couple Jeremy Blake and Theresa Duncan, who believed Scientologists were conspiring against them. Duncan emailed a confidant claiming Beck “told me he wanted to leave the cult desperately.” In response to Duncan’s allegation, Beck told Vanity Fair: “That’s ridiculous. Totally false. Had we been closer and discussed anything as personal as religion, I would have only had positive things to say about Scientology.”

In 2012, Beck told New York that Scientology is “just something that I’ve been around.” He added, “Some people do yoga, some people get into meditation, a whole half of my family are all Presbyterian and they’re very churchgoing, and my grandfather on my mother’s side was smoking hash and drinking beer every night.”

The seven-time Grammy winner suggested the religion was kinda, sorta part of his life. “Yeah, people in my family do it. I’ve read books, and I’ve learned about it. I mean, what I’m doing—I have a job, raising kids, I have friends, I have my interests, so I think my life is pretty full. I’m not off doing some weirdo stuff.”

Ortega reported that actor Jim Carrey’s former girlfriend Cathriona White was taking Scientology classes before she killed herself in 2015. Ortega also claimed that White, an Irish makeup artist, was introduced to the religion through Beck’s friends, who’d encouraged her to move to the United States to take church courses.

“Her friends tell us that Cathriona originally got involved in Scientology about four to five years ago through members of musician Beck’s circle of friends,” Ortega stated in one September 2015 post.

Still, Beck is doubling down on his exit from the mysterious organization.

As a freshly published New Yorker profile notes, “the question of Beck’s religious affiliation has preoccupied the music press for years; it is cited in nearly every article written about him.” Beck addressed the topic as he often has—declaring songwriting, not Scientology, to be his priority.

“I’ve so devoted myself to music that it’s kind of my main thing, and religion hasn’t been a central part of my life,” Beck said. “There’s a misconception that I’m a Scientologist. There was a period of time, maybe in the early 2000s, where my family recommended I get some counseling. But, beyond that, it hasn’t been something I’ve actively pursued.”