Here is the story of a young woman who was abused in Rotherham. For six years, she suffered threats and beatings, using make-up to hide her injuries. She had no money, could not buy her own clothes and was not allowed to go out alone. She could not even go to the corner shop unaccompanied and was never permitted to use the phone. "Basically," she said, "I lived like a prisoner."

In a week when a report exposing the scale of abuse of children and teenagers in Rotherham has rightly caused shock waves, you will not have read about this young women or others like her. Most of the victims mentioned in Professor Alexis Jay's damning report were white and their abusers men of Pakistani origin, who subjected them to almost unimaginable sexual torture. It is a dreadful scandal, exposing the failure of one agency after another which should have stepped in to protect vulnerable young people.

Between 2002 and 2006, the council failed to act on three reports about child sexual abuse, while another report published last week revealed that South Yorkshire police went out of their way to try to "disprove" allegations of rape and sexual violence.

But it would be wrong to frame this solely as a story about race. The young woman I've just mentioned is Asian, and she suffered years of abuse after being brought from Pakistan and forced to marry her cousin. Some men whose parents came to this country from conservative areas of Pakistan have grown up with appalling attitudes towards women, sometimes but not always connected to strict interpretations of Islam. They have ended up abusing both white girls and women from their own community.

The problem is about power and gender as much as ethnicity, posing difficult questions for the Labour Party in towns and cities with sizeable Asian populations. From Yorkshire to some areas of London, the party has embraced self-appointed "community leaders" without asking questions about their attitudes to women, gay rights and equality. Many of these men have suffered discrimination – the BNP won three council seats in Rotherham at one point – and Labour has always seen itself as the champion of victims of racism. Up and down the country, it has found allies in the Asian community, even though some of them are small-scale entrepreneurs whose values seem more in tune with the Conservative Party. Along the way, Labour seems not to have noticed – or turned a blind eye – to a different equality problem.

Six or seven years ago, when I first visited Rotherham with my then partner Denis MacShane (who was the town's MP), I noticed that most of the British-Asian activists in the local party, and all the Asian councillors, were men. I didn't think this was peculiar to Rotherham but it made me uneasy; on one occasion, I refused to go to a wedding reception for a councillor's daughter when I discovered that men and women would be segregated. I don't think separation by gender is any more acceptable than separation by race, and the fact that some people on the left tolerate or even defend it is profoundly shocking.

What this meant, in practice, was that crucial voices were missed in discussions about the extent of both domestic and child abuse in Rotherham. As the Jay report points out, women from a Pakistani background felt there was "wholesale denial" of child sexual abuse within their community, despite 12 cases of forced marriage coming to light in 2005, many involving young girls. The report has no doubts about how this came about: "There was too much reliance by agencies on traditional community leaders such as elected members and imams being the primary conduit of communication with the Pakistani-heritage community."

Like other towns and cities with a sizeable ethnic-minority population, Rotherham has an Asian women's organisation which could have provided a very different view of what was going on in the community. A few years ago I met Zlakha Ahmed, founder of Apna Haq, an organisation which helps black, Asian and Muslim victims of domestic violence in South Yorkshire. From her, I first heard about a network of Asian taxi drivers who hunt down girls who have run away from abusive families; this is what happened to 17-year-old Shafilea Ahmed, who was murdered by her parents in Warrington. Her father, who is serving a life sentence, was a taxi driver.

The story of the Asian woman who is a prisoner in her own home appears on Apna Haq's website. It lists forms of domestic abuse including beatings and rape, which happen in every community, but also types of behaviour specific to British-Asian families, such as being threatened with deportation and being told that "white women are much better". It is a telling detail, debunking the widely held assumption that Asian men who abuse white girls behave better towards women in their own community. Of course, not all Asian men are abusers, but those who are despise all women; the difference is that their wives and daughters are subjected to forms of abuse associated with pernicious notions of "honour". The common factor, whether they are keeping wives in isolation or taking part in a gang rape, is control.