The Channel 4 documentary series Benefits Street, focusing on people living in one of the poorest streets in Birmingham, has immediately reignited a topic that polarises Britain. Increasingly, those who claim benefits are being labelled as scroungers or scum. The reaction to the show on social media has included people calling for claimants to be "put down" or "shot".

Having worked in social housing for 25 years, I've met hundreds of people in Birmingham and the Midlands from many walks of life. I've met pensioners, young single mothers, women who have fled domestic violence, former offenders, people with disabilities and people in low-paid jobs. They have all needed state support to get by in life and a helping hand at a difficult time, and should not be labelled as scroungers. The programme reinforces a number of harmful stereotypes about people who use the welfare system. But let's try to set the facts straight.

Myth 1: In many households nobody has ever worked

In 2012, 18% of people in working-age households were unemployed, but in only 2% of cases had no one ever worked. Only 0.9% of households had two generations of a family that had never worked. When the Joseph Rowntree Foundation attempted to find families where three generations had never worked, it failed to find any.

Myth 2: Benefit fraud is common among claimants

It is estimated (for 2011-12) that just 0.8% of total benefit expenditure was overpaid as a result of fraud, challenging the common view that benefit expenditure is so high because of fraudsters and cheats. Far more relevant is the amount of benefits that goes unclaimed, which dwarfs the cost of that lost to fraud: in 2009-10, up to £12.3bn of the main six benefits went unclaimed. For housing benefit alone, government figures show that up to 6.9% of benefits went unclaimed in 2009-10.

Myth 3: People on housing benefit are unemployed

The proportion of housing benefit claimants who are in work is rising and fast approaching 1 million. These are the people who work but, because of low pay, still claim housing benefit because wages are not keeping up with the ever-increasing cost of living.

Around 90% of new housing benefits claimants are already in work – in a range of jobs. Research commissioned by Unison found more than half of local government staff are entitled to some form of in-work benefits. There will also be a significant percentage of health services workers in receipt of state benefits. Are these people scroungers, too?

It's time to put down the labels and challenge the kneejerk assumptions and stereotypes about people who claim benefits. We need to build fairer communities, where low earners are paid better wages, jobs and apprenticeships are created for young people and people living in areas of high unemployment are given the skills and support to lift them out of poverty.

Concentrating on creating sustainable, well-paid jobs, economic growth and affordable housing that really is affordable, would be far more profitable than constantly attacking those individuals and households for whom a benefit claim is the only way they can survive.

Chris Handy is chief executive of the Accord Group

This content is brought to you by Guardian Professional. Interested in housing? Join the housing network for more news, analysis and comment direct to you.