Police forces in England and Wales are failing to take antisocial behaviour seriously and urgent action must be taken to improve understanding of the toll it takes on communities, the head of the police inspectorate has said.

The chief inspector of constabulary, Denis O'Connor, said police forces' recording of information about harassment, criminal damage and verbal abuse was "inadequate" and must be improved immediately.

More than half the 43 forces in England and Wales could not automatically identify people who were repeat victims of antisocial behaviour, leaving police officers ignorant of vulnerable people in need of help.

Other damning results of a survey by HM Inspectorate of Constabulary (HMIC) included the finding that officers failed to turn up to almost one in four (23%) of complaints about antisocial behaviour.

The survey comes in the aftermath of the case of Fiona Pilkington, who killed her severely disabled 18-year-old daughter and herself after they were ongoing victims of antisocial behaviour by gangs near their home in Leicestershire.

"I certainly think, on the basis of what we have got here, we should take antisocial behaviour seriously," O'Connor said. "Because one of the tests we do in the background is how does this impact in terms of crime? It is constant and in their face, personalised, and people like a sanctuary. If they cannot go home in peace, imagine how unnerving it is. I think this undermines confidence if it is not dealt with seriously – confidence not just in the police, but in general."

Sir Hugh Orde, president of the Association of Chief Police Officers, said forces faced "hard choices" in prioritising their time and needed to work better on securing responses to reports of antisocial behaviour from other agencies such as local councils and housing associations.

"There are certainly cases where we could, without question, do better, and every chief constable in the country will be talking about this today to their communities," he told BBC Radio 4's Today programme. "But you can't actually say we are only talking about antisocial behaviour because the police can't just focus on one thing, we have to deal with the whole complexity of policing from antisocial behaviour ... right through to organised crime and international terrorism because that's what keeps communities safe in the totalities. We have to make hard choices."

O'Connor said antisocial behaviour was a deeply troubling issue for the public, but that there was no agreed framework among forces on how best to record and deal with it. "An awful lot of police forces have real problems," he said. "There is a lot of it [antisocial behaviour], a lot of it is under-reported and there is a problem with nailing the intelligence around it. It is like going back to the doctor's surgery but you see a difference doctor every time. The more times they suffer, the less confidence people have. There are some heartrending stories."

The home secretary, Alan Johnson, said the government expected all local agencies to prioritise antisocial behaviour: "Antisocial behaviour is to be tackled not tolerated - police and other agencies must protect victims and punish perpetrators."

But the shadow home secretary, Chris Grayling, blamed the prevalence of antisocial behaviour on "Labour's culture of bureaucracy" which he said forced police officers to fill in "endless forms" and limited the time they spent on the streets.

Officials at HMIC have begun further research to draw up a framework to assess the performance of police forces in tackling antisocial behaviour.

The West Mercia assistant chief constable, Simon Edens, the national leader on tackling antisocial behaviour, said officers must work closely with councils, housing providers and parents. "Chief officers recognise that what people want most for the community in which they live is to feel safe in the homes they live in, and on the streets they walk," he added.

"We know that antisocial behaviour matters to the public and that the police service is the only 24-hour, 365 day per year resource that the public can turn to. Of the 3,600 neighbourhood policing teams across the country, most – if not all – of them will have antisocial behaviour as a top or high priority."

The vulnerability of many of the victims of antisocial behaviour was underlined in the HMIC research, which found that one in five repeat victims classed themselves as disabled in some way.