After investigations into serious misconduct, findings of systemic bias cast a pall on law enforcement and question the city’s faith as a progressive beacon

A court filing by a US attorney pulled back the curtain of the San Francisco police department in March last year, revealing a shockingly ugly culture in which sworn officers of the law exchanged text messages expressing such sentiments as “All niggers must fucking hang” and “Cross burning lowers blood pressure”.

The ensuing scandal – which implicated 14 officers and compromised thousands of criminal cases – cast a pall on the police department and shook San Francisco’s faith in itself as a progressive beacon.

Elected leaders and the police brass quickly and forcefully denounced the officers involved and promised reform. But even as the city attempted to clean up the mess, another group of at least four San Francisco police officers was exchanging text messages that mocked the community response to the scandal, used racist slurs and denigrated LGBT people.

The revelation last week of that second batch of bigoted text messages has prompted another round of recrimination between city leaders and again raised the question: how can this be happening in liberal San Francisco?

“We seem to have a continuation of the problem,” said George Gascón, the city’s district attorney. “Last year the police department indicated that this was an isolated incident. I differed then because you don’t just have 14 people being racist without there being a wider problem.”

It is perhaps easier to identify that problem from a text message reading “White Power” than it is from a department’s overall work product, but statistics paint a damning picture of law enforcement in the city by the bay.

You don’t just have 14 people being racist without there being a wider problem George Gascón

San Francisco’s population is just 5.8% black, but black adults make up 40% of all arrests in the city and 56% of inmates in the San Francisco county jail. A 2015 study found that black adults were seven times as likely to be arrested as whites, and black women were 13.4 times as likely to be arrested as white women.

What is often overlooked in discussions of the racist text messages is the fact that both were discovered in the course of investigations into serious misconduct by SFPD officers.

The first batch was revealed in the course of a federal investigation into a group of officers who were caught on video illegally searching and stealing property from residents of single-room occupancy hotels. Six officers were ultimately indicted in the corruption case, which resulted in multiple convictions.

The second batch came out because of a criminal investigation into allegations of rape against officer Jason Lai. The investigation found insufficient evidence to support a sexual assault charge, but Lai was charged with six misdemeanor counts of misusing police records. Another officer, Lt Curtis Liu, allegedly interfered with the investigation into Lai.

Gascón pointed to another troubling case as evidence of systemic bias in the SFPD: Operation Safe Schools. The joint SFPD/DEA effort was supposed to target drug dealing near schools and resulted in the arrest of 37 people – all black. Leaked surveillance video of the operation revealed undercover officers eschewing arrests of non-black people engaging in the same criminal activity as the black people they did arrest.

Facebook Twitter Pinterest A protester stands on the steps of City Hall in San Francisco. Mario Woods was killed in a barrage of police gunfire that was caught on video. Photograph: Ben Margot/AP

A former San Francisco police chief himself, Gascón has increasingly positioned himself as the standard bearer of police accountability in San Francisco, squaring off against the police chief and police union, the San Francisco Police Officers’ Association.

Last year, Gascón empaneled a blue ribbon taskforce to investigate misconduct and the “culture of bias” in the police department. (In a sign of how politics work in this one-party town, the SFPOA has fired back at Gascón by, in essence, accusing him of being the real racist.)

“If you’re an African American member of the community, or if you’re LGBT, you’re going to have to start questioning what kind of treatment you’re going to get from police if this kind of behavior goes on unabated,” Gascón told the Guardian on Thursday. It’s a striking statement coming from a man who regularly sends people to prison based on the testimony of police officers.

But while everyone in San Francisco appears to agree that there’s a problem with the SFPD, few can agree on the path forward.

“There’s a certain thing in the culture where these people think that they’re defending and protecting, and they can be a law unto themselves, and it has to be reformed,” said supervisor Aaron Peskin, the mayor’s primary foil on the city’s board of supervisors.

“Everyone thinks that that can’t be in San Francisco, but it is. The best guy to reform it is [police chief] Greg Suhr, because’s he’s one of them,” Peskin added. “At least I used to think that.”

Suhr has faced noisy calls for his resignation since the police killing of Mario Woods on 2 December 2015. Woods was killed in a barrage of police gunfire that was caught on video, and his death has prompted ongoing protests against Suhr and San Francisco’s mayor, Ed Lee, with many pressuring Lee to fire Suhr.

Lee declined to answer queries from the Guardian as to whether he intends to fire or stand by Suhr. His staff released a statement reading: “Mayor Lee has zero tolerance for bigotry and racism in our San Francisco Police Department. Chief Suhr did the right thing in taking immediate disciplinary action and seeking immediate termination.”

Supervisor David Campos, a frequent opponent of the mayor who served on the police commission for three years, thinks that calls for the chief’s firing or resignation are “too simplistic”.

“The buck stops with the mayor,” he said. “The mayor is doing on police reform what he’s doing on homelessness or the housing crisis: taking passive steps only when he feels he has to, with the hope that the issue will go away. But the thing about these issues is that they don’t go away.”

Campos points to the mayor’s embrace of a Department of Justice inquiry into the police department as an example of a half-measure. The review will be carried out by the department’s office of community-oriented policing services, not the civil rights division, meaning that it won’t result in a binding settlement.

“A lot of people feel like it’s not a real review,” Campos said. “He [the mayor] does the right thing and then he steps back from it.”

Jeff Adachi, the city’s elected public defender and a longtime advocate of criminal justice reform, also points to a lack of leadership from City Hall.

“The mayor has not been vocal,” he said. “His style is to say let the police commission deal with it, or whatever.”

Adachi wants to see the department go beyond “lip service” and “window dressing” and truly invest in things like crisis intervention training and implicit bias training.

“The bigger question is not so much whether [Suhr] should be fired,” Adachi said, “but is he the person who can not only move the department beyond the scandals, but change the culture so that new line officers can say: this is the person that I want to model myself after.”

Sgt Yulanda Williams, a 30-year veteran of SFPD who leads Officers for Justice, a group representing minority and female officers, has been an outspoken supporter of reform efforts. For her, the latest revelations are a depressing setback.

“We’re the most diverse city in the United States, and it baffles me to think that we’re dealing with these issues,” she said.