Blatant failures of political and police leadership contributed to the sexual exploitation of 1,400 children in Rotherham over a 16-year period, according to an uncompromising report published in the aftermath of allegations of gang rape and trafficking in the South Yorkshire town.

Written by Prof Alexis Jay, a former chief inspector of social work, the investigation concluded that the council knew as far back as 2005 of sexual exploitation being committed on a wide scale by mostly Asian men, yet failed to act.

This is the fourth report clearly identifying the problem of child sexual exploitation (CSE) in Rotherham. The first, commissioned by the Home Office back in 2002, contained "severe criticisms" of the police and local council for their indifference to what was happening under their noses. But instead of tackling the issue, senior police and council officers claimed the data in the report had been "fabricated or exaggerated", and subjected the report's author to "personal hostility," leading to "suspicions of collusion and cover up", said Jay.

Council and other officials sometimes thought youth workers were exaggerating the exploitation problem. Sometimes they were afraid of being accused of racism if they talked openly about the perpetrators in the town mostly being Pakistani taxi drivers.

Roger Stone, Rotherham's Labour council leader since 2003, said that he had stepped down with immediate effect following the publication of the Jay inquiry. "I believe it is only right that I, as leader, take responsibility on behalf of the council for the historic failings that are described so clearly in the report and it is my intention to do so," he said.

Jahangir Akhtar, the former deputy leader of the council, is accused in the report of naivety and potentially "ignoring a politically inconvenient truth" by insisting there was not a deep-rooted problem of Pakistani-heritage perpetrators targeting young white girls. Police told the inquiry that some influential Pakistani councillors in Rotherham acted as barriers to communication on grooming issues.

On a number of occasions, victims of sexual abuse were criminalised – arrested for being drunk – while their abusers continued to act with impunity. Vital evidence was ignored, Jay said, with police apparently trying to manipulate their figures for child sexual exploitation by removing from their monitoring process girls who were pregnant or had given birth, plus all looked after children in care.

Jay concluded that from 1997-2013, Rotherham's most vulnerable girls, some as young as 11, were raped by large numbers of men. Others were trafficked to other towns and cities in the north of England, abducted, beaten, and intimidated, with some children doused in petrol and threatened with being set alight if they told anyone what had happened.

No case involving Rotherham men came to court until November 2010 when five "sexual predators" were convicted of grooming three girls, two aged 13 and one 15, all under children's social care supervision, before using them for sex. In the past 12 months, 15 people have been prosecuted or charged with child sexual exploitation offences in Rotherham.

The victims were offered gifts, rides in cars, cigarettes, alcohol and cannabis. Sex took place in cars, bushes and the play areas of parks.

A mortgage adviser who drove a BMW and owned several properties promised to treat a 13-year old "like a princess". Another man pulled the hair of a 13-year old and called her a "white bitch" when she tried to reject his attempt to strip her.

Keith Vaz, chair of the home affairs select committee, which interviewed Rotherham council officials during its own inquiry, said: "When we took evidence, Rotherham council were in denial and Stone is right to step down. Others responsible should also be held to account."

Yet everyone else involved will keep their jobs, according to council chief executive Martin Kimber. He said he did not have the evidence to discipline any individuals working for the council despite the report saying: "Over the first 12 years covered by this inquiry, the collective failures of political and officer leadership were blatant."

In summer 2013 Vaz's select committee published its own report, which criticised the council and the police in Rotherham, particularly for the lack of prosecutions over a number of years. That report was prompted in part by an investigation by the Times reporter Andrew Norfolk, which alleged that Rotherham police and council had deliberately covered up CSE. Jay's report is particularly critical of the authorities' failure to engage properly with the 8,000-strong members of Rotherham's Pakistani-heritage community. Akhtar, deputy leader until he lost his seat in May, told Jay he had not understood the scale of the child exploitation problem in Rotherham until 2013. Jay writes: "He was one of the elected members who said they thought the criminal convictions in 2010 were 'a one-off, isolated case', and not an example of a more deep-rooted problem of Pakistani-heritage perpetrators targeting young white girls. This was at best naive, and at worst ignoring a politically inconvenient truth."

She found that attempts by senior people in the council and the police to downplay the ethnic dimensions of CSE in Rotherham were ill judged. There was also a failure to engage with women in the Pakistani community, she said, writing: "There was too much reliance by agencies on traditional community leaders such as elected members and imams as being the primary conduit of communication with the Pakistani-heritage community."

Other than two meetings in 2011, there had been no direct engagement with either men or woman from the Pakistani community about CSE over the past 15 years, she added.

The issue of race, regardless of ethnic group, should be tackled as an absolute priority if it is known to be a significant factor in the criminal activity of organised abuse in any local community, wrote Jay. She suggested councillors can play an effective role in this, "especially those representing the communities in question, but only if they act as facilitators of communication rather than barriers to it. One senior officer suggested that some influential Pakistani-heritage councillors in Rotherham had acted as barriers."

Following the publication of Jay's report on Tuesday calls also came for the head of Shaun Wright, who became the police and crime commissioner for South Yorkshire two years ago following a five-year spell in charge of children and youth services on Rotherham borough council.

Though he is not criticised by name in the report, he was responsible for child protection during a period in which Jay said police and council were well aware that there was a serious problem with child exploitation in Rotherham and yet failed to act.

Caven Vines, a Ukip councillor in Rotherham and a member of the police and crime panel in South Yorkshire, which scrutinises the PCC's activities, said he would be demanding Wright's resignation: "I think he should resign immediately and at the next meeting of the panel I will be standing up to say exactly that – I should hope the rest of the panel will join me. You can't defend the indefensible."

Colin Ross, the leader of the Lib Dem group on Sheffield City council, said: "Shaun Wright was the councillor in charge of children's services at Rotherham Council and also sat on the authority of South Yorkshire police when both organisations knew about the level of child sexual exploitation, but chose not do anything about it. It's difficult to see how local people can have confidence in him to continue as our police and crime commissioner."

Wright declined to be interviewed. His spokeswoman issued a statement: "The commissioner has previously apologised for the failure of Rotherham council while he was in its cabinet from 2005 to 2010. He repeats that apology today and he fully accepts that there was more that everyone at Rotherham council should have done to tackle this terrible crime.

"Since becoming police and crime commissioner he has repeatedly publicly made tackling child sexual exploitation his number one priority."

No 10 said: "The failings of local agencies exposed by this inquiry are appalling. We are determined that the lessons of past failures must be learned and that those who have exploited these children are brought to justice."

• This article was amended on 27 August 2014. An editing error resulted in a comment being misattributed to Keith Vaz in an earlier version.