The new leaders of the far-right Proud Boys “fraternity” are scrambling to improve the group’s image after founder Gavin McInnes claimed he was quitting last week, introducing new rules that downplay the group’s penchant for violence and prepare the group for the potential of future law enforcement action.

But not everyone in the Proud Boys is on board with the rules. One prominent member from the group’s militant wing who wants to take the group in a more racist direction claims he’s really the one in charge—and that he won’t let “a few losers who think they own the Proud Boys” take over.

Vice magazine co-founder McInnes launched the Proud Boys in 2016, claiming that the group would be a contemporary version of men’s clubs like the Elks, but devoted to “the West.” Like other men’s clubs, the Proud Boys came with a complicated, bizarre hierarchy that required members to submit to a number of rules, including a “beat-in” hazing ritual where members have to call out the names of cereal brands while they’re punched.

But the Proud Boys quickly attracted more controversy than an ordinary drinking group, especially after members began fighting with left-wing anti-fascist activists at pro-Trump rallies. McInnes insisted his group wasn’t racist, but Proud Boys kept appearing with racist figures or using the group as a gateway to white supremacist activism.

In October, the fringe group came under public scrutiny after a brawl between Proud Boys and anti-fascist activists in New York City resulted in riot and assault charges against several Proud Boys. In November, it got worse after police documents described the Proud Boys as an “extremist group with ties to white nationalism.”

McInnes attempted to disassociate himself from the group in a lengthy video published last Wednesday, just days after those police documents were revealed. McInnes claimed he was leaving the Proud Boys in an effort to help the legal cases of members facing criminal assault and riot charges in New York City.

Right-wing provocateur Milo Yiannopoulos, who has attended Proud Boys events, followed McInnes’ lead and quickly distanced himself from the group as well.

While McInnes insisted in his video that he was no longer the Proud Boys leader, his purported exit last week created a leadership vacuum which played out this week.

On the official Proud Boys site, an anonymous group calling themselves “The Elders” claimed that they were now the leaders and would be represented by Jason Lee Van Dyke, the Proud Boys’ attorney.

On Sunday, the Proud Boys voted on a new set of rules that Van Dyke claims will get them away from their reputation for violence.

“This organization was not founded for the purpose of going to rallies and beating people up,” Van Dyke said.

Van Dyke is an unlikely face for the Proud Boys’ new attempt at mainstream respectability. In September, Van Dyke was arrested and charged with filing a false police report over some stolen guns. In 2014, he posted a threatening tweet with a racial slur and a noose. And, while he was a student at Michigan State University, police found copies of anti-Semitic and racist tracts in Van Dyke’s dorm room.

But now Van Dyke and his allies in the Proud Boys are pushing rules that will separate the group from white supremacists—or at least will do so on paper. The new bylaws include bans on white supremacist or alt-right members, and prohibitions on racial discrimination in membership.

The bylaws approved by the Proud Boys also appear to nod at the new law enforcement attention on the group. The national Proud Boys organization won’t keep membership lists, and local chapters are required to tell the national group if their own membership lists are subpoenaed. Proud Boys are now allowed to voluntarily freeze their memberships, according to the rules, if it will help them in a criminal case.

The new rules also try to officially downplay past Proud Boys rules that encouraged violence against left-wing activists.

McInnes had reserved the Proud Boys’ highest level, the “fourth degree,” for Proud Boys who were in a fight for the group. But Van Dyke told The Daily Beast the “highly controversial” degree will be changing to a merely “honorary” title.

Even the Proud Boys’ initiation ritual, in which members are beaten until they can name five cereals, has been codified in the post-McInnes era—no punches to the head allowed.

Van Dyke also tried to distance the Proud Boys from Patriot Prayer, the far-right Pacific Northwest group that regularly holds rallies in Portland in an attempt to bait antifa activists into fights. Proud Boys have featured prominently at fights around Patriot Prayer rallies in the past, with viral videos of Proud Boys punching out their political opponents going viral on the right.

“We’re not the personal army for Patriot Prayer, or anyone else, for that matter,” Van Dyke said.

The Proud Boys rules also include new restrictions on members’ behavior. Proud Boys are banned from using opioids or crystal meth—although notably not cocaine, a drug McInnes has often extolled. Also now banned: wearing flip flops, fedoras, or cargo shorts to a Proud Boys meet-up.

In the past, “third degree” Proud Boys were prohibited from masturbating more than once a month. But the new rules, under a section in the bylaws dubbed “No Wanks,” extends that prohibition to all Proud Boys.

“No heterosexual brother of the Fraternity shall masturbate more than one time in any calendar month,” the new rule reads, adding later, “All members shall abstain from pornography.”

But not everyone is on board with the new Proud Boy “elders.”

Augustus Invictus, an alt-right figure and self-described pagan who spoke at the 2017 white supremacist rally in Charlottesville, claimed in a video that he was the new top Proud Boy. Invictus was a member of the Proud Boys through its now mostly defunct militant wing, the Fraternal Order of Alt Knights.

Invictus, who said in the video that his toddler son was now the Proud Boys’ chief of staff and slammed “a few losers who think they own the Proud Boys,” said he would lift the Proud Boys’ rules against associating with explicitly racist figures like white nationalist leader Richard Spencer.

“The days of Proud Boy cuckery are over,” Invictus said.

Invictus didn’t respond to a request for comment. Van Dyke insists that Invictus is no longer a Proud Boy, if he ever was.

With McInnes gone, the Proud Boys have lost their most prominent figure. But Van Dyke claims the club can outlast its founder’s exit.

“I don’t think a whole lot of members—with respect to Gavin—were there just for Gavin,” Van Dyke said.