Police in Washington DC frequently fail to investigate reports of rape, and treat victims so dismissively at times, that they experience fresh trauma while the chances of the perpetrator being caught are undermined, according to a comprehensive report due out next week.

Campaign group Human Rights Watch is expected to uncover "disturbing evidence of police failure" in a 200-plus page report after a two-year investigation into law enforcement practices in the US capital.

But although shocking, the situation in Washington is far from isolated. There are widespread examples across the US of the police routinely neglecting crimes of sexual violence and refusing to believe victims.

"This is a national crisis requiring federal action. We need a paradigm shift in police culture, because rapes and sexual assaults are being swept under the rug, and too many victims are being bullied," said Carol Tracy of the Women's Law Project, a legal advocacy group that specialises in sexual violence cases.

Human Rights Watch began looking into the situation in Washington after discovering evidence that the city's Metropolitan police department (MPD) were refusing even to document a significant number of reports of sexual assaults coming in from the central hospital where victims are treated.

Full details and statistics will be disclosed by HRW in its final report, due to be published on 24 January.

But in a preliminary letter to the police department, HRW initially estimated that more than 37% of reports of serious sexual assault and rape were not being followed up on by investigators. Sara Darehshori, the senior counsel at HRW and author of the report, told the Guardian: "There are some good detectives. But some victims' stories are appalling. And a significant number of cases are falling through the cracks entirely."

The MPD has disputed the report's initial findings, but declared it is committed to improvements. Human Rights Watch will pledge to work with them. Washington's problems are serious but "it isn't unique," said Darehshori.

The Human Rights Watch report focuses solely on Washington, but in many cities across the US, the police record an alarming proportion of reported rapes as "unfounded" cases, meaning they decide the crime did not happen and the report was false or baseless.

The national average is 6%. But according to the latest internal FBI statistics, Pittsburgh shelves 34% of cases in this way, Atlanta 24% , Dallas 13%, Jersey City 18%, Lincoln, Nebraska, 19%, San Bernadino 34%, Durham, North Carolina, 31%. Chicago does not declare annual "unfounded" statistics but its average from 2000 to 2009 was 17%.

In New York City, the number of recorded rapes declined by 35.7% between 2005 and 2009. But over that period the number of sex crimes labelled as mere misdemeanors rose by 6%. Advocacy groups also expressed concern about high rates of rape cases being dropped as unfounded and reports of victims being treated dismissively. All this prompted police commissioner Ray Kelly in 2010 to form a task force to improve the investigation and victim interview procedures in sexual assault cases. Reforms are ongoing.

The New Orleans police department is under federal review for shelving 50% of sex attack cases as "non-criminal complaints".

"Any figure over 10% is alarming and should be looked into," said Joanne Archambault, a retired San Diego police sergeant who runs the international pressure group End Violence Against Women and trains police across the nation in handling sexual assault.

At the other end of the scale, some figures appear too low, in a national picture of confusion and inconsistency. Houston police department only declares 2% of its rape cases to be "unfounded".

"Women don't lie any more often in Pittsburgh than they do in Houston," said Dr Dean Kilpatrick, director of the national crime victims research and treatment centre and a professor at the Medical University of South Carolina. He said many departments' records were "outrageous".

Experts agree that an average of 5% of rapes are falsified. "This issue of investigators not believing large numbers of victims, then threatening them or arresting them is a very serious national problem," said Archambault.

Sara Reedy, 27, last year won a $1.5m settlement from the Cranberry police near Pittsburgh after they did not believe she had been raped at gunpoint and arrested her. She was about to stand trial when her perpetrator was caught elsewhere.

Sara Reedy, wrongfully arrested and charged with theft, false reporting, has won a $1.5m police settlement. Photograph: Cory Morton

Meanwhile many cities report backlogs of thousands of untested forensic evidence, including Houston, Detroit, Los Angeles County, San Antonio, Phoenix and Albuquerque.

Former Miss Arizona Hilary Peele is a rape victim-turned advocate after her rape kit – the evidence collected by medical professionals after her ordeal – was neglected following an attack in her apartment in 2004. "It's hard enough to report these crimes in the first place," she said. Police didn't test her kit for DNA until she had called every two weeks for eight months.

In Cleveland, Ohio, serial rapist and murderer Anthony Sowell was caught in 2011 after killing 11 women, six of them after a rape kit was disregarded. Milwaukee serial rapist Gregory Below was finally sentenced to 350 years in 2011, but three of his victims said the police dismissed their cases initially.

Across the US, five rapes are committed for every murder. But in concerns about under-recording, in New Orleans and Baton Rouge, Louisiana, and Newark, New Jersey, there are more murders than rapes, according to annual crime figures, while the numbers are almost the same in Detroit, Baltimore, and Washington DC.

In signs of a turning point in attitudes, however, some cities are taking strong action.

A decade of comprehensive reform in Philadelphia, where the sex crimes department was once nicknamed "the lying bitches unit", and Austin, Texas, have transformed their police units into models of best practice. Baltimore is undertaking serious reforms and Kansas City is regarded as a good example of policing.

Philadelphia and Austin police make building trusting relationships with victims and working closely with rape crisis centres and the community a top priority. Philadelphia allows annual scrutiny of its records by external experts. Austin pairs detectives and prosecutors to address grand juries, especially on the most common but most difficult-to-investigate cases involving 'non-strangers' or alcohol and drugs.

Carol Tracy of the Women's Law Project concluded: "We are looking at a chronic and systemic failure to investigate sex crimes properly. But we are also seeing a way forward, a growing acknowledgement of the problems, and no excuse not to undertake serious reform."