Here's the moral dilemma that faced Paul Farmer, chief executive of the mental health charity Mind, last week: should he continue to sit on a government advisory panel, charged with scrutinising a policy that his charity believes to be inhumane? Or should he resign, publicising his anger at the coalition government's refusal to listen to the charity's concerns, and remove himself from the room where improvements are being discussed?

Farmer chose to leave the panel responsible for monitoring the functioning of the work capability assessment (WCA), the new fitness-for-work test that determines who is eligible for sickness benefits, frustrated that the government was not paying attention to the growing chorus of alarm over the reliability of the test.

His departure from the committee reflects the intensifying anger among charities such as Mind that represent people affected by the government's commitment to reassessing approximately 1.6 million recipients of incapacity benefit – which is being phased out – to see whether they are eligible for the new benefit, employment and support allowance.

Until now, charities have been voicing their concerns but expressing a desire to work with the government to get things right. Farmer's resignation marks a new, tougher stance.

Amid the fallout from his departure last week, there was despondency among campaigners over the government's failure to implement substantial improvements to a system that charities identified as "not fit for purpose" more than 18 months ago, but which is still being used to assess the fitness of 11,000 people a week.

Farmer's decision has been widely supported by other charities, which are also anxious about the consequences of pushing some of the country's most unwell and vulnerable people through a "flawed" WCA. There is growing concern about the rapidly rising number of people who appeal against judgments that they are fit for work (up to 50% of all those who go through the test) and of those who are successful in their appeal (around 40% of all those who appeal). At least 390,000 people have gone to appeal since 2009; tribunal courts have been forced to open on Saturdays and to increase staff by 30% since January 2010, to deal with the backlog. Appeals are costing the government around £50m a year, in addition to the £100m it is paying the IT company, Atos, to carry out the largely computer-led test.

"The DWP [Department for Work and Pensions] seems absolutely committed to pushing 11,000 people a week through a flawed system. That's the real problem for us," says Farmer. "That doesn't feel fair. I've moved from being puzzled about the reluctance to change, to being increasingly frustrated. I genuinely don't understand why the government doesn't just pause the process and reflect on why it's not working."

Although the government has implemented some improvements, he says: "We are not hearing anything that suggests that people's experiences have much changed." Mind was still seeing "extremely unwell people, who have only recently been discharged from psychiatric services, being found fit for work", he says. The charity's branch in Oxford has supported more than 100 people through the appeals process over the past year, 90% of whom have seen the decision that they were fit for work overturned.

Charities are anxious to be involved in government decision-making, so to step out of that conversation was a difficult choice. In his resignation letter, Farmer told Chris Grayling, the employment minister: "We have reached the point where we feel that the lack of progress in improving the system and the lack of willingness to consider more fundamental reform makes our involvement in this element of the process no longer tenable."

The DWP's version of events suggests that it was keen to push him off the committee even before he decided to leave. Farmer explains that officials had asked him to consider his position, concerned that a conflict of interest might be emerging, on the back of reports of imminent legal action by individuals with mental health problems against the government, over the fitness-for-work test.

In a statement, Grayling says: "I asked officials to ask Paul Farmer to step down after his charity became involved in legal action against the department. It's obviously not possible for someone involved in suing the department to also remain as an adviser to it."

Mind says there was confusion within the DWP, and stresses that the charity is not, at this stage, supporting any such legal action.

Farmer's decision to leave the four-person scrutiny panel, which includes two employers' occupational health experts and the new president of the British Medical Association, is backed by Richard Hawkes, chief executive of the disability charity Scope. "We share Mind's concerns about the work capability assessment," he says. "The huge number of successful appeals are a damning indictment of a test that isn't fit for purpose. There's little point in scrutiny if the government doesn't listen to it."

Sarah Lambert, head of policy at the National Autistic Society, says colleagues back Farmer's decision. "We share some of Mind's frustration. There are a lot of problems with the WCA; the system is not working. We would like the process to be slowed down while improvements are made."

David, 52, went through the WCA earlier this year, and was found to be fit to work despite having rheumatoid arthritis so severe that he has been unable to work for the past decade. He says: "Anyone who can make the wider public more aware of this issue should be supported. If I was in his [Farmer's] position, I would have done the same. There is no empathy at all in the system; it is all accusatory."

Even before Farmer's resignation, tension between charities and the government had been mounting. Grayling recently expressed his frustration with charities he had asked to advise the department on improving the system. Commenting in a parliamentary debate, he said that instead of offering tweaks to the process, they had come up with a "total transformation of the whole process including … a new scoring system and a new computer system", and had offered no evidence that their revised system would be any better.

Grayling says the DWP is committed to implementing improvements "to try to make this a better and fairer process". He adds: "The only alternative to this is to leave people on benefits for the rest of their lives, doing nothing." Farmer stresses that Mind is also in favour of helping people into work. "We are very strong believers in the idea that people with mental health problems can and should work where that is possible … but we also know that there are huge barriers to finding work," he says.

"If I was Chris Grayling, I would have looked at the appeal figures and decided to pause the process," he adds. "It feels like an inhumane system … If people feel that they have not been treated as human beings, then you are not getting off to a very good start."