Eric Pickles has been criticised for furthering mental health prejudices after he told a survivor of alleged child abuse to “adjust your medication” when she accused him of ignoring her.

The Communities and Local Government minister made the comment in a recorded confrontation with constituent Teresa Cooper earlier this month. Ms Cooper was one of at least six women who say they were drugged as teenagers at the Kendall House care home, Kent, in the late 1970s and early 1980s, whose children now have genetic defects.

To hear the recording, click on the video above

Download the new Independent Premium app Sharing the full story, not just the headlines

Confronting Mr Pickles, who is her constituency MP, she said: “Nothing has changed on the Kendall House abuse. Only you have ignored it. You have ignored it.” Pickles then interjected, shouting: “just, just, adjust your medication”.

The two met while at a wildlife event in Ongar, Essex, on 14 September and Ms Cooper recorded the conversation. Ms Cooper, 46, said: “I was shocked to be honest. He’s implying I’m mentally ill which is really rude of him because people do have mental health problems and they should not be spoken to that way.”

She added: “The Conservatives say ‘we need to change attitudes about mental health’ but clearly they need to start with Eric Pickles.”

Paul Farmer, chief executive of the mental health charity Mind, said: “It is disappointing that a minister would use language that can feed the prejudice around mental health. We would encourage everyone to really consider the impact of using mental health language in a way that could contribute to fuelling stigma.”

Mr Pickles’ adviser said that the comment was said in “the heat of the moment” and was his way of giving a blunt piece of advice. Mr Pickles had corresponded with Ms Cooper about some of her concerns but she claims she has been trying to speak to him in person for 16 years.

Mr Pickles said: “It was never my intention to insult Teresa Cooper. I was giving her a frank piece of advice in private.”

Marjorie Wallace, chief executive of mental health charity Sane, said: “It’s a very thoughtless phrase which could be very hurtful. It may be exaggerated out of context, but whatever the provocation, it’s a ghastly phrase which should never be used.”

The recording is published as the Conservatives begin their party conference under the shadow of the arrest of their vice-chairman Alan Lewis. The 75-year-old was arrested on Friday on suspicion of raping a teenage girl in the 1960s.

Senior members of the Conservative party last night accused police of having political motives in the timing of the arrest. A senior member of the parliamentary party said: “This appears to be yet another example of political and crass police action which follows a litany of others, including Hillsborough, Jean Charles de Menezes, Plebgate and the G20 incident.”

Greater Manchester Police said the arrest was “proportionate and necessary”, but one source familiar with police protocols said: “I’m very surprised at the decision to arrest, rather than just invite him in. Under Code G of the new guidelines, arrest should be the very last resort ... It does look a bit provocative. I’m not saying the police would have given the press the name, but the timing does invite suspicion.”