David Cameron has been forced to disassociate himself from the welfare reform minister Lord Freud after Ed Miliband revealed that he told a fringe meeting at the recent Tory conference that some disabled people are “not worth” the minimum wage.

The prime minister said the comments by Freud, who suggested that some disabled people could be paid as low as £2 an hour, did not represent the view of the government. But Cameron showed anger when he alluded to his severely disabled late son, Ivan, as he told Miliband that he would take no lectures on disabled people.

The angry exchanges came as Miliband disclosed the remarks by Freud in answer to a question at the recent Tory conference in Birmingham about paying the minimum wage to disabled people. The Labour leader quoted Freud as saying to his questioner at the fringe meeting: “You make a really good point about the disabled. There is a group where actually, as you say, they’re not worth the full wage.”

Asked by Miliband whether that was his view, the prime minister said: “No, absolutely not. Of course disabled people should be paid the minimum wage and the minimum wage under this government is going up and going up in real terms. It is now at £6.50. We will be presenting our evidence to the low pay commission, calling for another real terms increase in the minimum wage.”

Miliband then quoted further from Lord Freud, who added that he was looking at “whether there is something we can do if someone wants to work for £2 an hour”. Amid cries of “outrage” and “shame” from the Labour benches, Miliband said: “Surely someone holding those views can’t possibly stay in his government.”

Cameron said: “Those are not the views of the government, they are not the views of anyone in the government. The minimum wage is paid to everybody, disabled people included.”

With his voice rising in anger, the prime minister added: “Let me tell you: I don’t need lectures from anyone about looking after disabled people. So I don’t want to hear any more of that. We pay the minimum wage, we are reforming disability benefits, we want to help disabled people in our country, we want to help more of them into work. And instead of casting aspersions why doesn’t he get back to talking about the economy.”

Miliband said: “I suggest, if he wants to protect the rights of disabled people, he reads very carefully what his welfare minister has said because they are not the words of someone who ought to be in charge of policy related to disabled people.

“In the dog days of this government the Conservative party is going back to its worst instincts – unfunded tax cuts, hitting the poorest hardest, now undermining the minimum wage. The nasty party is back.”

In the full text, released by the Labour party, Freud was quoted as saying: “You make a really good point about the disabled. Now I had not thought through, and we have not got a system for, you know, kind of going below the minimum wage.

“But we do have … you know, universal credit is really useful for people with the fluctuating conditions who can do some work – go up and down – because they can earn and get … and get, you know, bolstered through universal credit, and they can move that amount up and down.

“Now, there is a small … there is a group, and I know exactly who you mean, where actually as you say they’re not worth the full wage and actually I’m going to go and think about that particular issue, whether there is something we can do nationally, and without distorting the whole thing, which actually if someone wants to work for £2 an hour, and it’s working can we actually…”

Freud had been responding to a question from David Scott, a Tory councillor from Tunbridge Wells. Scott had said: “The other area I’m really concerned about is obviously the disabled. I have a number of mentally damaged individuals, who to be quite frank aren’t worth the minimum wage, but want to work. And we have been trying to support them in work, but you can’t find people who are willing to pay the minimum wage.

“We had a young man who was keen to do gardening; now the only way we managed to get him to work was actually setting up a company for him, because as a director in a company we didn’t have to pay the minimum wage, we could actually give him the earnings from that. But trying to maintain his support and allow him to work, which he wanted to do, so to stay with benefits, and stay with some way of managing to continue on in that way. And I think yes, those are marginal areas but they are critical of actually keeping people who want to work supported in that process. And it’s how do you deal with those sort of cases?”

After Freud’s response (above), there was this further exchange:

Scott: “They particularly want to work because it does add so much to their lives…”

Freud: “Yes.”

Scott: “…Being able to do something. And being employed in a job gives them so much self-esteem, but nobody is willing to pay that minimum wage. And then we’re supporting them massively financially, but we also want them to work, for their own self-esteem and everything else.”

The work and pensions minister, Esther McVey, told the Daily Politics on BBC2: “I cannot justify those comments. They were wrong. He will have to explain them.”

Freud is likely to argue that the full exchanges at the fringe meeting show he was trying to help people with mental illness into the workplace. He is also likely to argue that he was suggesting that the universal credit system could help people with “fluctuating conditions”.