Willesden, north-west London, 2.30am. A passer-by has seen a man try to break into some vehicles, chased him away and called 999. Minutes later, PC Paul Jones reaches the area and begins searching through bins, down alleyways and behind fences until he spots someone fitting the description. Jones is not convinced by the 19-year-old's account of why he's there, and soon discovers what he thinks might be a tool for breaking windows nearby. This gives Jones the grounds he needs to search him, and he begins running his hands down the man's legs, through the seams of his jacket and the turn-ups of his trousers.

As one suspect admitted to me – immediately after Jones had unloaded a spare watch and a wallet containing someone else's ID card from his jacket – Jones is "one of the good guys. Other officers, certain CID, unmarked cars, they're the worst."

Earlier this year, a police watchdog suggested there could be quite a few of these "other officers". A review by HMIC (Her Majesty's Inspectorate of Constabulary) found police had carried out a stop and search without reasonable grounds in 27% of cases. Under the law, the power should be used when based on specific and objective information that a person is in possession of a stolen or prohibited item. The home secretary Theresa May resolved that it was "time to get stop and search right", launching a public consultation that closed in September with 4,000 responses. Keenly aware of the fact that stop and search is a major issue for ethnic minority voters, May will announce the outcome before the end of the year.

Denise Richards, the chair of a stop-and-search scrutiny group for Brent police (where Jones works), believes the large response to the consultation indicates the widespread distrust and resentment created in communities by "negative" searches – those in which stolen or prohibited items are not found. "Stop and search doesn't just have an impact on the young person. It impacts on their family, their mothers, their fathers, their grandmas. It impacts on the community," she says.

Police officers arrest a suspect in a still from Ben Ferguson's video Photograph: Ben Ferguson

It's perhaps no coincidence that the Metropolitan police suggested we see how stop and search works in the London borough of Brent; it has a better record than most, with 21% of searches resulting in arrest, considerably more than the national average of 9%. To the general public, this figure makes stop and search look at best like a waste of police time and at worst like the result of incompetence. But there's a second, more disturbing reason why the power has become discredited: in England and Wales, black people are searched seven times as often as white. For Richards, the only explanation is institutional racism. "It's linked in with the perception that the police have of young black men. I don't have a problem with stop and search. I have a problem with the disproportionate number of young black men who are being searched."

The recent political attention has left many officers worried that the power might even be taken away. "I think stop and search is one of my best tools to prevent crime and solve it," says Jones, a 33-year-old who worked as an estate agent before joining the police. "If I'm going to a call with suspects on the premises, I'm looking for that person as I approach. I might not catch them in the house, but if I can catch them coming out of the house with stolen property, stop and search is the only way I can get my hands in their pockets."

Other officers from Wembley police station share Jones's view, recounting successes that range from the discovery of a meat cleaver strapped to a penis to a rock of crack cocaine the size of a bar of soap up someone's backside. Tales of unsuccessful searches are not so common, yet despite Brent's good record, officers are still statistically more likely to have a negative stop and search.

A few days later we accompany 23-year-old PC Rob Farriello on patrol in one of Brent's three large social housing estates, Church End. Farriello, who joined the force at 18, is on the look-out for two suspects described as mixed-race teenagers with large afros, who local residents have witnessed committing a street robbery. As we drive, he says that from his point of view even negative stop and searches are useful, as a deterrent, because gang members need to bear in mind that they could be frisked. "People you know are actively involved in drug dealing, in carrying weapons – they're the right people to search," he says. "If you don't find something, there's a good chance you've disrupted them from doing something."

Stop and search on the streets of Brent – a still from the video

Most people will be reassured to learn that the police have a pretty good idea who all the gang members are, as well as other people involved in crime. This information is kept on an extensive "CrimInt" (criminal intelligence) database, alongside updates on gang rivalries and what to look out for while on patrol. But a known face or previous convictions do not constitute reasonable grounds for a search, and it's here that the practical application of the power risks veering beyond the margins of legality. Putting aside the philosophical problems – Is the friend of a gang member also a gang member? Is a gang member a gang member when he's with his mother? – a negative search leaves those who have been frisked feeling like victims of harassment. The result, one resident says, is that people who would be quite useful sources of information put up walls of silence. On a more prosaic level, the "right" people know they are the "right" people; they are used to being searched and the boy (usually a boy) just gives his knife or drugs to someone less likely to get searched – a girl or a white friend – instead.

During our time with the Brent officers, we witness Jones and Farriello uncover drugs, weapons and stolen items from suspects; they also dodge left hooks, dress stab wounds, and do other things that seem quite noble and heroic. Yet in spite of all this, members of the public continue to peer into the police car as if there is a huge rubber hand on the roof giving every passer-by the middle finger. With every scowling face I'm reminded of a brilliant interview with a Bronx police chief called Tony Bouza in the 1977 documentary The Police Tapes. "You look at the average policeman … he goes to the police academy and he's told he's going to be helping people … and then he gets out there and he suddenly discovers he is bitterly resented and he's shocked … the policeman has great difficulty assimilating this knowledge and he becomes hardened. The insularity grows, the parochialism grows and they become an island." For any officer who might want to step off this island to go fishing, stop and search would act as quite a good rod.

During our time with the officers, the only white person we see searched is a 50-year-old man with mental health problems in the leafy northern part of the borough. Our days and nights are mainly spent responding to emergency calls on the built-up south side, where non-white faces are much more common. On a day off we meet Roy Croasdaile, a local resident and member of an independent police advisory group. He explains that the geography of policing is associated with inadequate housing, poor education, high rates of unemployment and cultural perceptions. Together these things become inextricably linked to the disproportionate number of young black men searched. He adds that quoting disproportionality figures – as Theresa May has done this conference season – is misleading.

"If you have people reporting crimes involving black people fighting black people, it's understandable that you should have police officers targeting black people under section 60," he says, referring to a provision of the Police and Public Order Act allowing officers to perform stop and search in exceptional circumstances in response to specific gang violence. But he makes a key distinction: "That's not the same as discrimination, when you don't have that kind of intelligence but you're still stopping lots of black people."Since 2011, when stop and search was found to have been a major source of discontent among those who took part in the riots, broad changes have occurred in the use of the power. Across London, Pace (Police and Criminal Evidence Act) searches for drugs, weapons and stolen items – which have a broad profile of offenders – have dropped from over half a million to 356,567. Brent has halved its figure from 33,331 to 16,556. According to Farriello and Jones's boss, Borough Detective Superintendent Simon Rose, "Officers have to be more selective, to use the power with more consideration and measure outcome rate not usage." Consideration, it seems, includes not searching people based on the colour of their skin, since the proportion of black people searched has also fallen since 2011.

Midnight. A 21-year-old Somali man is stopped near Church End after the Audi A3 hire car he is driving triggers Farriello's automatic number-plate reading system. The display says: "Involved in the supply of drugs". This on its own isn't enough to justify searching the driver, but a waft of weed smoke, along with flakes of cannabis on the car floor, torn off Rizla packets and broken cigarettes, give Farriello the reasonable grounds he needs.

"You're just picking on me right now. That's how it feels," sighs the driver, who was on his way to eat. The street search is negative so the hungry motorist is taken back to Harlesden police station, where a strip search also turns up nothing. Earlier that week I'd witnessed Jones tell two boys sharing a joint that Brent are too busy to arrest for cannabis, so later I ask Farriello whether the intelligence on this man's car meant he was treated differently. "No," says Farriello. "His previous convictions or anything he was known for had nothing to do with the reason he was strip searched," adding that "often a positive search comes down to luck."

Metropolitan police commissioner Bernard Hogan-Howe accepts that it is difficult to eliminate the "luck" factor in stop and search, and tells me that commanders of forces need to work on teaching their officers how to stand down in situations where they have got it wrong. One resident we meet, Kevin Noel, acquired a criminal record at the age of 18 when a negative search turned into a charge of assault on two police officers. He argues that people who live on housing estates that have become synonymous with crime get targeted by surveillance and disruption powers far more frequently than anyone else. "We're all just poor around here, but not everyone is a criminal. When officers use stop and search as a deterrent, we're going to see it as harassment."

Jones concedes that others members of the force might be less professional than him. "I guess when you get 30,000 officers you're going to get 30,000 different ways of doing something." He thinks the focus needs to be on conversion figures – the ratio of stops to arrests – for specific officers: an officer with a poor conversion rate should spend time alongside officers with a better record.

Back in Willesden, a police van arrives to collect the 19-year-old who Jones has now arrested. The suspect finishes his cigarette and climbs in. I manage to get a word with him before he gets in the cage. Whatever comes of the consultation on stop and search, the teenager says, it will take at least 10 years of good policing for everyone to have faith once again.

• This article was amended on 22 October 2013. An earlier version said that black people are searched seven times as often as white people without clarifying that those figures refer to England and Wales, not Brent.