The government's focus on alleged fraud and overclaiming to justify cuts in disability benefits has caused an increase in resentment and abuse directed at disabled people, as they find themselves being labelled as scroungers, six of the country's biggest disability groups have warned.

Some of the charities say they are now regularly contacted by people who have been taunted on the street about supposedly faking their disability and are concerned the climate of suspicion could spill over into violence or other hate crimes.

While the charities speaking out – Scope, Mencap, Leonard Cheshire Disability, the National Autistic Society, Royal National Institute of Blind People (RNIB), and Disability Alliance – say inflammatory media coverage has played a role in this, they primarily blame ministers and civil servants for repeatedly highlighting the supposed mass abuse of the disability benefits system, much of which is unfounded.

At the same time, they say, the focus on "fairness for taxpayers" has fostered the notion that disabled people are a separate group who don't contribute.

Scope's regular polling of people with disabilities shows that in September two-thirds said they had experienced recent hostility or taunts, up from 41% four months before. In the last poll almost half said attitudes towards them had deteriorated in the past year.

Tom Madders, head of campaigns at the National Autistic Society, said: "The Department for Work and Pensions is certainly guilty of helping to drive this media narrative around benefits, portraying those who receive benefits as workshy scroungers or abusing a system that's really easy to cheat."

He added that ministers such as the work and pensions secretary, Iain Duncan Smith, were being "deeply irresponsible" in conflating Disability Living Allowance (DLA), which helps disabled people hold down jobs, and Employment and Support Allowance (ESA), a payment for those unable to work. This "scrounger rhetoric" was already having an impact on people's lives, Madders said, citing a woman who rang the charity to say a neighbour who formerly gave lifts to her autistic child had stopped doing so following press articles about disabled people receiving free cars under a government scheme.

Some disabled people say the climate is so hostile they avoid going out, or avoid using facilities such as designated parking bays if they "don't look disabled".

The government has committed to making significant cuts to disability benefits, including a 20% reduction in the DLA bill by 2015/16. Much of its public focus has been on alleged fraudulent claims or cutting benefits to those whose conditions have improved.

Charities point to a series of ministerial statements arguing that the "vast majority" of new ESA claimants are able to work, while the disabilities minister, Maria Miller, said last month that £600m of DLA was overpaid each year, not mentioning that a greater sum is saved by others not receiving what they are due.

This is "playing directly into a media narrative about the need to weed out scroungers," said Richard Hawkes, chief executive of Scope. "Our polling shows that this narrative has coincided with attitudes towards disabled people getting worse.

"Disabled people tell us that increasingly people don't believe that they are disabled and suddenly feel empowered to question their entitlement to support."

David Congdon, head of policy at Mencap, said the charity feared where this could lead. "We are concerned that this narrative of benefit scroungers or fakers connected to the welfare reform bill does risk stigmatising all people with a disability," he said. "The worry would be that this could lead to an increase in resentment against disabled people, and even an increase in hate crimes."

There was "an incredibly strong focus on benefit fraud within the DWP", said Guy Parckar, policy manager for Leonard Cheshire. "It is mentioned at all possible opportunities. Of course, whenever there is fraud you want that to be tackled, but there should be some serious thought given to the long-term impact that this has. There is the impact of potential hate crime, and issues around that."

Neil Coyle, head of policy for Disability Alliance, said his organisation was being told of increasing levels of verbal abuse, and worried this could lead to attacks.

"There's a lot of concern that the level of abuse and harassment goes unrecorded because it's seen almost as a norm. It seems to be growing as a result of a mis-perception of much more widespread abuse of benefits than actually exists. That's being fed by the DWP in their attempts to justify massive reductions in welfare expenditure."

A DWP spokeswoman said the department was committed to supporting disabled people but needed to "do more to change negative attitudes", and had begun a cross-government consultation on tackling discrimination.

She said: "Our welfare reforms are designed to restore integrity into the benefits system and to ensure that everyone who needs help and support receives it."

David Gillon from Chatham in Kent, said: "I think we've lost all the progress we made in the last 30 years in terms of acceptance." Gillon, whose chronic back condition forced him to give up a job with British Aerospace, recounts walking on crutches past a pub in the middle of the day and receiving shouts of: "We're going to report you to the DWP." He said: "When there's a bad article in the press, the next day you think, 'Do I really need to go out of the house?' We're being forced back into the attic, locked away from society."

Fazilet Hadi, head of inclusion for the RNIB, said she also felt the tone was set by ministers: "I think they should be more careful. At the moment it feels like the government is not on the side of disabled people. Most people don't have that much exposure to disabled people. They don't see us in the lifestyle pages, they don't see us in the fashion pages. The only reference they see is in these stories. And that's why the language is so important."

• This article was amended on 6 February 2012. The original referred to the Royal National Institute for the Blind. The name has been corrected.