More than a million vulnerable people on low incomes are being driven deeper into poverty after being shunted into the private rental sector due to an acute shortage of social accommodation.

A report commissioned by the Nationwide Foundation, an independent charity, says that the shortfall in social housing has been met by a doubling in size of the private rented sector in the past 25 years.

But this has forced more households, many on benefits with dependent children or a disabled family member, to pay significantly more for unsuitable housing.

The shake-up of the benefits system – which has led to sanctions being imposed on people claiming universal credit who fail to attend meetings with job advisers or decline to participate in employment schemes – has had a dramatic effect on the attitudes of private landlords.

“Because of sanctions you’re more likely to fall into arrears and to be asked to leave because you are in arrears,” said the author of the report, Dr Julie Rugg, of the University of York’s centre for housing policy. She has spent 20 years studying the benefits system and its relationship with the housing sector.

“The welfare system change has created vulnerability,” Rugg said. “It didn’t used to be the case 10 years ago but it is now. People know the benefits system is tightening up but they might not realise that if you’re at the bottom end and receiving benefits then your situation can be pretty precarious indeed.”

Rugg’s report found that more than a third (38%) of the private rented sector now comprises low-income households who are classed as vulnerable.

And almost nine out of 10 of these – equivalent to 1.4 million households – are living either in poverty or in poor or overcrowded conditions.

The shortage of social housing stock means private landlords can charge more than housing associations, often for inferior accommodation.

“Generally speaking, people are paying an extra £25 a week because they are living in the private rented sector,” Rugg said. “It might not sound a lot but if your benefit income is £75 a week, £25 is quite a big chunk of money.

“We know from talking to people on benefits that after paying their tax and utilities and rent they might be looking at £30 a week to live on. If they are paying an extra £25 a week as a result of living in the private rented sector then that’s actually creating a level of destitution that’s quite frightening.”

Last week Theresa May announced £2bn to build new “affordable” homes in England. Under the plan, housing associations, councils and other organisations will be able to bid for the money to spend on new projects, starting from 2022.

But Leigh Pearce, chief executive of the Nationwide Foundation, said that the government needed to examine the role of the private rented sector, too.

“We need a fundamental rethink about who private renting is for and a comprehensive strategy to ensure it is fit for purpose, to ensure that everyone in this country has a home they can thrive in.

“This includes addressing the really important question about what is expected of the private rented sector, including who it can and should provide homes for, and how it sits alongside other housing tenures.”