In the 80s and 90s, police were found to have regularly ignored rape cases. The resulting crisis brought shame to the city – but now the department's transformation is a model for others

When Shannon Schieber moved into an apartment on a quiet, pretty little street in Philadelphia to pursue a doctorate at the prestigious University of Pennsylvania Wharton Business School nearby, she didn't know that a serial rapist was stalking single women in the area.

Neither did the police. Eight months later the man broke into her home in the middle of the night, raped the 23-year-old, then choked her to death as she put up a tremendous fight.

Schieber was his fifth victim. There were to be seven others viciously assaulted after her. But Schieber was the only one he killed, and the police blunders surrounding that case helped expose a scandal.

It emerged that the Philadelphia police department harboured a culture where rape victims were routinely belittled and their cases ignored by patrol officers and detectives, while predators got away with sexual assault and, literally, murder.

The crisis brought shame on the city but it also sparked comprehensive reform of the police.

That was in 1998, and today, 15 years after Schieber's death, the police department has been transformed from one that a previous chief admitted was "god awful" in its handling of sex crimes into one held up as an example to others.

Now Philadelphia is pressing other cities across America not to wait for the kind of crisis that blighted its reputation before dealing with the sort of failures that lead to it.

One of the city's big weaknesses was tolerating a system where for years detectives had got away with filing rape cases under a non-criminal classification code that was the equivalent of sweeping them under the carpet.

"A police department that has problems and is making mistakes has got to recognise the need to change, and it has to start at the top," said Charles Ramsey, Philadelphia's police commissioner since 2008 and the president of the Police Executive Research Forum, which aims to spread best practices nationally.

"If they don't take action, eventually it's going to come to light anyway, resulting in scandals where the police have intentionally misclassified crimes – and you're going to get a victim saying 'there's been no investigation, no follow-up'."

There is no shortage of current and recent examples of other police departments engulfed by similar controversy.

• Washington DC's Metropolitan Police Department is currently being re-assessed by the city council after an investigation by campaign group Human Rights Watch and a review by an independent law firm found that police work on sexual assault cases was "inadequate" and recommended independent oversight, re-training and providing advocates for victims by right, to prevent cases being shelved and victims maltreated.

• Cleveland police are repeatedly under scrutiny: currently over the case of three women held by a local man for years as sex slaves; in 2011 for a backlog of 1,000-plus untested rape kits and in 2009 for failing to catch serial rapist and killer Anthony Sowell before he murdered 11 women, despite other victims escaping and calling the police.

• The New Orleans police department was censured by the Department of Justice in 2011 for poor policing of crime, including an unprecedented finding of systemic gender bias in its failures to investigate sexual assaults.

• The police in Missoula, Montana, agreed last month to reform after a federal investigation into inappropriate responses to sex crimes.

The Guardian has previously reported on some of these and other similar problems across the nation.

But some cities police sex crimes markedly better than others. Human Rights Watch praised Philadelphia, Austin, Grand Rapids and Kansas City, Missouri, for making the biggest strides in the last 10 to 12 years, with "a victim-centred approach as opposed to one that emphasises quickly closing cases", HRW stated in a report earlier this year.

New York and Baltimore are examples of cities where policing rape has made some recent progress but still faces significant obstacles, according to various experts. Many other cities hide bad practices behind a lack of oversight at local or federal level.

'Our department is light years ahead of the majority now,' says Charles Ramsey. Photograph: Matt Rourke/AP

One of the unique innovations in Philadelphia that came out of the scandal is an external oversight committee that independently reviews police handling of rape cases.

The committee is made up of some of law enforcement's sharpest critics: staff from the city's rape crisis centre, many of whose clients were let down by the police for years; experts in women's legal rights; and children's advocates.

Carol Tracy, executive director of the Philadelphia-based Women's Law Project is a leading member of the committee now and was asked by the Philadelphia police in 1999 to help spearhead its creation.

"It brought transparency and independent scrutiny at a time when the public had lost confidence in the police. It is a model other departments should learn from," she said.

The oversight committee examines rape reports classified as 'unfounded', meaning the police believe the incident does not meet the criteria for rape or sexual assault, or that it was fabricated.

In the late 90s Philadelphia police would declare almost a fifth of cases to be "unfounded", in addition to the hundreds it simply buried by misclassifying them under bogus codes.

Now, as well as long ago eliminating the intentional misclassifications, its "unfounded" rate for rape cases has dropped to around 9% or 10%, closer to the national average range of 6% to 9%, which most experts deem fair.

One former detective from the 90s became notorious for nicknaming the Philadelphia sex crimes unit "the lying bitches unit" and publicly opining that women lie about rape "about half the time".

The current head of the unit, now called the special victims unit, Captain John Darby, swiftly scotched that.

"How often does a sexual assault victim lie about it? Everyone wants to quantify that – the best thing to say is the rate is very low," he said.

When pressed he estimated a mere 6%.

His detectives are trained to understand that even if sometimes a victim is not truthful about how many drinks he or she had or some other peripheral circumstance because they fear being harshly judged and blamed, it does not mean they are lying about being assaulted.

And Darby said that his team are not out to prosecute victims for underage drinking or drug use, for example, because overlooking that helps elicit an honest and more full account of what happened, which in turn improves the chances of arresting a rape suspect, and increases as victim's value as a potential witness.

As for arresting the victim themselves for the crime of false reporting, in the few cases where detectives deduce they were lying outright, Darby almost never resorts to that because it stops other victims coming forward in an area of crime where the Department of Justice estimates 60% of sexual assaults go unreported.

"I can count on one hand how often we've made such an arrest, and victims are not threatened with arrest," said Darby, who has led the SVU for 11 years. That does not mean that falsely accused suspects are being rounded up in droves either, he pointed out. When a woman, or more commonly a teenager, is lying about being raped, this will invariably emerge during a skilled interview without intimidation, said Darby. Meanwhile his main focus is on improving services for the other 94%.

He is currently preparing to move the SVU into a new state-of-the-art complex, where forensic medical exam rooms, sex crime investigators, child abuse investigators, a representative from the district attorney's office and other related services are all under one roof.

In the 80s and 90s, the sex crimes unit was first based in an unheated former horse stable, which smelt of manure in summer, and after that a disused military armory with barbed wire around dilapidated buildings. Victims waiting to be interviewed, who had already waited in the general hospital ER to have their intimate forensic exam, surrounded by shooting or car crash victims, would often see their alleged attackers sitting in the police station corridor in handcuffs.

The current forensic medical centre dedicated to victims of sexual assault is run by retired senior Philadelphia sex crimes officer, former Lieutenant Michael Boyle, who despite being one of the better detectives in the '90s did not see the scandal coming.

"People had become complacent. Officers were being careless and in many cases lazy and weren't looking at cases as individual women but just as a filing code or an overwhelming workload," Boyle said.

Many detectives were burnt out, there was poor training, poor supervision, almost no oversight, insufficient funding, and little respect from the rest of the department, which fostered an entrenched culture of bad behavior.

"I don't believe in many cases it was willful. We were working in a battlefield, in this neighbourhood particularly, with a constant wave of people's personal misery coming at you," said Boyle, indicating the deprived area outside his window, where drug dealing and prostitution are rife.

Some inherently sexist but sometimes mainly exhausted police developed a "lady, let's get this over with" attitude, Boyle said, which he now describes as "inexcusable and indefensible" but was routine before the scandal broke – and still is in many departments up and down the country.

Boyle is now a respected authority on dealing with rape victims and gathering evidence and addresses conferences and workshops, briefing judges and prosecutors.

Captain Darby said that a culture existed that allowed officers, whether male or female, who had a predilection towards bias against rape victims to manifest that in their investigations.

"It starts to become a cancer in the squad," he said.

Dismissing a rape victim because she is a sex worker or an addict does not help pinpoint serial rapists or help a woman emerge from a situation where "she's down on her luck temporarily", said Darby.

Despite the reforms, Darby said, there still had to be constant vigilance, from the chief down, to prevent lapsing into old ways or lack of further progress.

'They set Shannon up for murder'

It emerged during the investigation into Shannon Schieber's murder that two of the rapes prior to her attack had, initially at least, been filed by detectives under the classification 2701.

A 2701 was a "non-criminal" category that meant something so trivial it required no further police action.

Schieber was heard screaming for help by a neighbour, who called the police.

They arrived in six minutes, knocked on Schieber's door but, hearing nothing and seeing no signs of forced entry, they left.

Schieber's father, Sylvester Schieber believes, especially having chewed the case over with federal criminologists, that the attacker intended solely to rape his daughter, but when the patrol officers knocked on the door he strangled her to silence her, then fled out of the balcony door.

If the officers had known there was a serial rapist prowling the area they would probably have forced their way in and perhaps been able to save the promising young finance student – although she would not have chosen that area to live if the public had been warned, he said.

"They set Shannon up for murder," Schieber said.

The police were initially convinced her murder was a crime of passion by an ex-suitor.

It took eight months for investigators to link the five neighbourhood cases to one predator and two years and a series of Pulitzer prize-nominated investigations by the Philadelphia Inquirer newspaper for the police to admit there was anything more than "isolated sloppiness" at play, and that comprehensive reform was needed.

All the 2701 shelved cases going back five years (the statute of limitations) were re-examined. Six-hundred eighty-one were reclassified as rape, 863 as other sexual offences.

The Schiebers failed in a lawsuit against the city. But Sylvester Schieber takes some grim satisfaction from the reforms that took place.

"If Shannon screamed for help tonight they would break the door down. The probability that she would be alive today would be a lot higher. It took her murder and another one [by a different perpetrator] for policing policy to change," he said.

Schieber's killer wasn't caught until 2002. Troy Graves, then 29, had raped one more woman in Philadelphia then married and moved to Colorado, where he raped six women near Fort Collins. By that time Philadelphia police had been forced to recognise they were looking for a serial predator.

"A very skilled forensic medical exam on one of his victims in Colorado in 2001 helped crack it," said Boyle. "He had licked the woman's breast and the examining nurse was diligent enough to swab the breasts meticulously – and picked up his DNA.

"After Shannon, he had been careful not to leave sperm at the scene. But after that rape kit he popped up on the federal DNA computer system," said Boyle.

The police then eventually identified and tracked down Graves, who is now serving multiple life sentences.

Philadelphia has since improved its detection of linked cases and its outreach to the public.

Just in recent weeks, officers from the SVU were handing out flyers in the lovely Fairmount Park near the river and warning joggers not to listen to music through earphones and to stay alert because the mysterious, serial "Fairmount Park Rapist" is still on the loose more than a decade after his first attack.

The federal government has little power to force the 18,000-plus police departments around the US to adopt better practices.

"I think our department is light years ahead of the majority now. We're trying to keep the ball rolling forward and share information with other cities," said commissioner Ramsey (pictured).

He won't discuss other cities specifically, including Washington DC, where he was chief from 1998 to 2006.

But he acknowledged that, like Philadelphia, cities where problems are exposed can go through a period of resistance before admitting mistakes and devising often painful police reforms.

"It's like the stages of grieving – denial, followed by acceptance, followed by the will to move forward. It's a process," he said.