The government's changes to the welfare system are having a "devastating" impact on hundreds of thousands of people with mental health problems and have driven some of the most vulnerable to try to take their own lives, according to charities and medical experts.

A letter published in Wednesday's Guardian brands the government's drive to reassess 1.5 million people receiving incapacity benefit as "deeply flawed" and warns that the rapid pace of change is having dire consequences.

"We've found that the prospect of incapacity benefit reassessment is causing huge amounts of distress and tragically there have already been cases where people have taken their own life following problems with changes to their benefits," the letter, signed by leading mental health charities and a senior consultant from the Royal College of Psychiatrists, states.

"We are hugely worried that the benefits system is heading in a direction which will put people with mental health problems under even more pressure and scrutiny, at a time when they are already being hit in other areas such as cuts to services."

In April the government began to send out 7,000 letters a week asking people to attend a "work capability assessment". It is now sending out more than 11,000 reassessment requests and the first interviews will take place this month.

Critics say the test is not sophisticated enough to identify the challenges faced by people with mental health problems and warn that the process is increasing the pressure on those already suffering high levels of anxiety and stress.

A recent study for the mental health charity Mind found that three-quarters of people it surveyed said the prospect of a work capability assessment had made their mental health worse and 51% said it had left them with suicidal thoughts.

The survey, which polled more than 300 people claiming incapacity benefit, found that 95% thought they would not be believed at their assessment. "It is extremely worrying that so many people are so concerned," said Paul Farmer, the charity's chief executive. "Many of these people have been out of work for many years and are frightened by the prospect of being reassessed for something that makes a massive difference to their lives."

Farmer signed the letter alongside Paul Jenkins, chief executive of Rethink Mental Illness, Professor Bob Grove, joint chief executive of the Centre for Mental Health, and Jed Boardman, consultant and senior lecturer in social psychiatry, of the Royal College of Psychiatrists.

The work capability assessment was introduced for new claimants of the employment support allowance in 2008 but critics are increasingly concerned that it will be used to reassess the first wave of incapacity benefit claimants from June.

Liz Woollard, 48, who suffers from depression and anxiety, says she failed the test despite two GPs, a psychiatrist and a senior nurse stating she was not able to work. Following a 54-minute assessment, Woollard says she was turned down for employment support allowance. The subsequent report stated she "did not appear to be trembling … sweating … or making rocking movements".

"It was a couple of weeks before Christmas and I had been out for a Christmas lunch with some friends and they made a lot of that … They did not have any sympathy or understanding of mental health issues. In that fortnight I had a major depressive episode that left me in bed for three days [but] they virtually brushed over that … In the written report they didn't mention that – they focused instead on the Christmas lunch I had managed to attend."

Woollard, 46, appealed but had to wait 11 months for her case to be heard and the decision overturned. She has now been told that she will have to be reassessed again.

Julie Tipping, an appeals officer for the charity Disability Solutions, said Woollard's case was not unusual. She said many people with mental health problems had had their benefits cancelled and appealed successfully, only to be told their case needed to be reassessed again. "This is having a devastating impact on people with mental health issues," said Tipping. "It is a constant reassessment process which is just absolutely relentless. It is almost like they want to assess you to death or reassess you until you can't face it any longer and drop out of the system altogether. It is like a deliberate grinding down process. It is devastating to see."

Tipping said that in the last year, two of her clients had made "real attempts" at suicide after a decision that they were fit for work. Both were taken to hospital and subsequently sectioned.

"These were really serious attempts, not cries for help, these were people who had just had enough and this was the final straw for them … do we really need to wait to such a stage where people are trying to throw themselves from a bridge before somebody listens to how chronically affected they are by their condition? Is that the kind of society we want to live in?"

The government says it is committed to ensuring the work capability assessment is "as fair and accurate as possible" adding that it has accepted the recommendations of an independent review by Professor Malcolm Harrington last year.

"[We] have already put in place his recommendation to create a network of 'mental health, intellectual and cognitive champions' in each … medical examination centre to spread best practice and build understanding of these conditions," said a spokesman for the Department for Work and Pensions.

"In consultation with charities including Mind, Professor Harrington is now looking at the way mental health is assessed in the work capability assessment and we look forward to receiving his recommendations later this year."