The work and pensions secretary, Iain Duncan Smith, has been warned by his own advisers that a vital element of his flagship bill to simplify the benefits system is "unworkable and unfair".

Plans to force part-time workers to seek greater hours of employment or risk losing their benefits have been condemned as unrealistic at a time of prolonged recession by the social security advisory committee, a Whitehall body in charge of monitoring the reforms.

Its concerns come amid growing controversy over Duncan Smith's proposed plans for a universal credit, which is due to replace the current range of welfare benefits with one monthly payment.

Earlier this month, David Cameron attempted to reshuffle Duncan Smith to the job of justice secretary as fears over the progress and costs of the universal credit circulated in parliament. However he was rebuffed by the minister, who is determined to see his reforms through.

Seventy organisations wrote to the Commons work and pensions select committee last week, raising a host of potential objections to the universal credit, including doubts about the ability of the government to successfully deliver the IT necessary to unify benefit payments or use real-time wage information to ensure that work always pays better than welfare.

Those working with the vulnerable said the insistence that the system be wholly internet-based will leave many unable to access benefits, and claim the government does not have a plan B.

There are also major concerns that the decision to pay out benefits just once a month will push vulnerable people into the hands of payday lenders as they struggle to budget.

The Observer has learned that the advisory committee, appointed by the government, has focused its criticisms on the pressures that will be brought to bear on Britain's burgeoning part-time workforce. It has told Duncan Smith that the vulnerable in part-time work are likely to be punished for economic circumstances not of their making.

The government's proposals would affect anyone earning less than they would in a 35-hour-a-week job on the minimum wage. Part-time workers could lose money for not applying sufficiently frequently for vacancies, not attending job interviews for a better paid job within 48 hours of being directed by a jobcentre, or not taking up a full-time job within 90 minutes from home – although the government says each case will be assessed on its merits.

In its response to Duncan Smith's white paper on universal credit, the committee condemned the plan as "unrealistic in the current economic climate", adding: "With high underemployment – currently over one million part-time workers in the UK want to work more hours – sanctioning clients who cannot increase their hours seems to be both unworkable and unfair.

"We cannot see how in-work sanctions can be policed and are concerned that customers working short hours may be penalised as a result of labour market conditions, rather than as a result of their response to the conditionality regime applicable to them."

Critics have also pointed out that many members of the part-time workforce would have to alter childcare arrangements in order to work longer hours, which may leave them worse rather than better off. It is expected that women, more of whom work part-time than men, would be disproportionately hit by the reforms.

The Children's Society has called for the government to provide more information on how the policy will work. Charities have also suggested that constant interviews and applications made elsewhere could adversely affect existing employer-employee relationships.

Last week it was revealed that the number of people in part-time work rose by 134,000 to 8.12 million, the highest level since 1992, when these figures began to be collected.

This figure includes 1.42 million people – a record number – who would like to work full-time but are unable to find such employment.

Dame Anne Begg MP, who chairs the influential cross-party work and pensions select committee, said Duncan Smith would face a grilling on the issue when he faces her members on Monday. She said: "The universal credit is about making work pay, so there's the carrot – we don't need the stick at a time of economic recession, when people would take on more work if they could".

Gavin Kelly, a former deputy chief of staff in Downing Street and now chief executive of independent thinktank the Resolution Foundation, which is set to launch a report on the impact of universal credit on the working poor, said: "Very little is known about how this will operate in practice, the numbers of people affected. This is likely to become an issue of mounting concern for working households already struggling just to get by."

Gillian Guy, chief executive of Citizens Advice, whose organisation receives government funding to offer advice on benefits and debt, said she feared her staff would struggle with the expected flood of people affected by the changes: "We are heading into a perfect storm. Given the times, this is a massive reform to throw into the mix."

A Department for Work and Pensions spokesman said: "Individuals will get support for the first time to increase their hours in work and will see a clear increase in their income when they do – instead of losing out as they do in the current system. However, we will not expect someone to work full time where caring responsibilities, or illness, means it isn't right for them.

"The welfare system needs to be reformed. The recent employment figures show the private sector continuing to create jobs so it is important that we are helping people not only to get into work, but who are already in work and would want to work more hours to do so."