Lord Freud, the welfare reform minister, has offered a “full and unreserved apology” after Ed Miliband revealed that he told a fringe meeting at the recent Tory conference that some disabled people were “not worth” the minimum wage.

As the Labour party called for Freud to be sacked after he suggested that some people with mental disabilities could be paid as little as £2 an hour, the minister said it was “offensive” to suggest that anyone should be paid less than the minimum wage.

In a statement issued by the Department for Work and Pensions, Freud said: “I would like to offer a full and unreserved apology. I was foolish to accept the premise of the question. To be clear, all disabled people should be paid at least the minimum wage, without exception, and I accept that it is offensive to suggest anything else.

“I care passionately about disabled people. I am proud to have played a full part in a government that is fully committed to helping disabled people overcome the many barriers they face in finding employment. I am profoundly sorry for any offence I have caused to any disabled people.”

Freud issued his statement after Mencap called on him to consider his position and Downing Street distanced itself from the minister by saying there could be no exceptions to the minimum wage.

Esther McVey, a fellow DWP minister, said the remarks would “haunt” Freud. Speaking on the Daily Politics on BBC2, McVey said: “Those words will haunt him. I cannot justify those words. They were wrong. We have the minimum wage. Everybody has the minimum wage. We have done a lot to support people with disabilities.”

McVey spoke out after Miliband disclosed the remarks by Freud in answer to a question at the recent Tory conference in Birmingham about paying the minimum wage to disabled people. Around 100,000 disabled people are paid the minimum wage. The Labour leader quoted Freud as saying to his questioner at the fringe meeting: “You make a really good point about the disabled. There is a group where actually, as you say, they’re not worth the full wage.”

Asked by Miliband whether that was his view, the prime minister said: “No, absolutely not. Of course disabled people should be paid the minimum wage and the minimum wage under this government is going up and going up in real terms. It is now at £6.50. We will be presenting our evidence to the low pay commission, calling for another real-terms increase in the minimum wage.”

Miliband then quoted further from Freud, who added that he was looking at “whether there is something we can do if someone wants to work for £2 an hour”. Amid cries of “outrage” and “shame” from the Labour benches, Miliband said: “Surely someone holding those views can’t possibly stay in his government.”

Cameron, who was informed of Freud’s remarks shortly before prime minister’s questions, said: “Those are not the views of the government, they are not the views of anyone in the government. The minimum wage is paid to everybody, disabled people included.”

With his voice rising in anger, the prime minister added: “Let me tell you: I don’t need lectures from anyone about looking after disabled people. So I don’t want to hear any more of that. We pay the minimum wage, we are reforming disability benefits, we want to help disabled people in our country, we want to help more of them into work. And instead of casting aspersions why doesn’t he get back to talking about the economy.”

Miliband said: “I suggest, if he wants to protect the rights of disabled people, he reads very carefully what his welfare minister has said because they are not the words of someone who ought to be in charge of policy related to disabled people.

“In the dog days of this government the Conservative party is going back to its worst instincts – unfunded tax cuts, hitting the poorest hardest, now undermining the minimum wage. The nasty party is back.”

Freud is likely to argue in private that the full exchanges at the fringe meeting show he was trying to help people with mental illness into the workplace. He is also likely to say that he was suggesting that the universal credit system could help people with “fluctuating conditions”.

But Labour said after the Commons exchanges that Freud should be sacked. Dan Scorer, the head of policy at Mencap, told The World at One on BBC Radio 4: “He needs to very seriously consider his position after making these comments … It is a very serious proposition to put forward and one we fundamentally disagree with.”

But Freud was offered some support when David Scott, a Conservative councillor from Tunbridge Wells whose question prompted his remarks, said that the discussion was about helping vulnerable people into the workplace.

Scott told The World at One: “I was wanting to explore [with Freud] how to help some very vulnerable people in the community and to find ways that these individuals could actually get greater self-worth and be introduced to the workplace to help them … The sentiment is quite clearly that he was concerned: how do we help these individuals to enter the workplace so they can feel they are adding something and gaining worth.”

In the full text, released by the Labour party, Freud was quoted as saying to Scott at the fringe meeting: “You make a really good point about the disabled. Now I had not thought through, and we have not got a system for, you know, kind of going below the minimum wage.

“But we do have … you know, universal credit is really useful for people with the fluctuating conditions who can do some work – go up and down – because they can earn and get … and get, you know, bolstered through universal credit, and they can move that amount up and down.

“Now, there is a small … there is a group, and I know exactly who you mean, where actually as you say they’re not worth the full wage and actually I’m going to go and think about that particular issue, whether there is something we can do nationally, and without distorting the whole thing, which actually if someone wants to work for £2 an hour, and it’s working can we actually…”

Scott had said: “The other area I’m really concerned about is obviously the disabled. I have a number of mentally damaged individuals, who to be quite frank aren’t worth the minimum wage, but want to work. And we have been trying to support them in work, but you can’t find people who are willing to pay the minimum wage.

“We had a young man who was keen to do gardening; now the only way we managed to get him to work was actually setting up a company for him, because as a director in a company we didn’t have to pay the minimum wage, we could actually give him the earnings from that. But trying to maintain his support and allow him to work, which he wanted to do, so to stay with benefits, and stay with some way of managing to continue on in that way. And I think yes, those are marginal areas but they are critical of actually keeping people who want to work supported in that process. And it’s how do you deal with those sort of cases?”