The government's welfare reform minister has suggested lone parents, sickness claimants and other people on benefits are too comfortable not having to work for their income, saying they are able to "have a lifestyle" on the state.

In an interview with House Magazine, Lord Freud is reported to have said the benefits system is "dreadful" and discourages poor people from taking the risks he implied they should be willing to bear to change their circumstances.

"The incapacity benefits, the lone parents, the people who are self-employed for year after year and only earn hundreds of pounds or a few thousand pounds, the people waiting for their work ability assessment then not going to it – all kinds of areas where people are able to have a lifestyle off benefits and actually off conditionality," the Conservative peer said.

Liam Byrne, the shadow work and pensions secretary, said: "The nasty party is well and truly back. This government has comprehensively failed to get Britain back to work and frankly it's a disgrace that ministers now choose to kick people when they are down rather than even pretend to offer a helping hand."

Freud, a former journalist and investment banker, told the magazine that his background did not make him unable to understand the reality of living on benefits. "You don't have to be the corpse to go to the funeral, which is the implied criticism there," he said.

It is not the first time senior Tories have publicly criticised the culture of benefits, and the party has been accused of stirring public distrust by making dramatic claims about rich lifestyles and fraud.

A report by the University of Kent this week calculated that hundreds of thousands of people were put off claiming benefits because of perceived stigma, a problem the researchers also blamed on the media.

The report dismissed government claims that there were families receiving as much as £100,000 a year in housing benefit – it later emerged there were just five such families – and that people had been off work for more than a year with diarrhoea – it turned out they had serious bowel diseases.

Freud is in charge of one of the government's most risky and ambitious programmes: to reform the benefits system, the centrepiece of which is the new universal credit, which will replace several separate benefits and is designed to "make work pay" by encouraging people to take more paid work while still being able to claim state support.

The universal credit, due to go nationwide next autumn, has been the subject of a number of warnings recently from critics who fear it is still unworkable and that many claimants will not cope with the online system.