Imagine two sets of people, both renting from private landlords. One is an Islington couple who have never worked. The other is an Oldham family with four children, where the working parent has just lost his or her job. The Islington couple currently receive £250 a week in housing benefit, while the Oldham family gets only £150.

Times are tough, and the government wants to save money. Which family should have its housing benefit cut? George Osborne has chosen the Oldham family. He is cutting its housing benefit to £96 a week, while allowing the Islington couple to continue to claim £250 a week for as long as they like.

It may sound like odd logic, but that is the reality of the £26,000 benefit cap. It takes no account of your employment history or family size. So a central London couple who have never worked are unaffected, because they currently receive less than £26,000 in benefits. But a large family – even in a cheap house – will be hit. That is not sensible.

The worst hit, of course, are large families in the south-east, where rents are higher. Even in Tolworth, described by the Evening Standard as the "scrag end of Kingston borough", a four bedroom house will give you little change from £400 a week. Cutting housing benefit to £100 a week – which is broadly what the cap means if you have four children – makes life impossible. After rent, council tax and utilities, a family with four children would have 62p per person per day to live on. That is physically impossible.

It is easy to say that people shouldn't have large families if they can't afford them. But most affected families could afford their children when they conceived them, and continued to be able to afford them – until they lost their jobs in what has proven to be the worst recession for more than a century. Should they now follow Greece and give up their children for adoption?

It is the effect on children in large families that has led Christian, Jewish and Muslim leaders to take a stand against the government over the weekend, speaking up for the poor in a way entirely consistent with their faiths.

The cap doesn't even hit the families the Daily Mail so dislikes – single parents with many children and many fathers who have never worked. Those families, by and large, are sufficiently dysfunctional to be in social housing, and so will not be hit – at least not much – by the reforms. Instead the people hit hardest are stable families previously in work on low to middle incomes – the really squeezed middle, if you like. They were not rich enough to buy a house, and not poor enough to qualify for social housing. As a result they pay a fortune to rent privately and are vulnerable to the cap.

Civil servants tell me they don't expect rents to fall – quite the reverse, as the market is buoyant. Nor do they expect families to migrate from the south-east to low-cost housing areas such as Merthyr Tydfil or Barrow. These are, in the main, people who want to work and will choose to stay in an area with good job prospects. Instead, they expect families to downsize. Children will end up sharing a room with multiple siblings, and parents will sleep on a sofa bed in the lounge. Clearly people can live like that, but frankly I thought that overcrowded tenements were something that Britain had left behind.

Britain is not poor. In only five years of our history have we ever been richer than we are today. The savings from the cap are very small – £270m. Yet we spend £53bn on welfare payments to people in the top half of the income distribution. Cutting their payments by one half of one per cent would be a much better way to save £270m.

Even better would be to allow more houses to be built in the south-east, over the objections of organisations such as the CPRE and the National Trust. Standard supply and demand tells us that more houses mean lower prices and lower rents. Lower rents mean lower housing benefit bills without making a single poor family suffer. If you crunch the numbers, you find that increasing the number of houses by 1.3% would cut the housing benefit bill by £270m. It would also get people back into work. Surely that is a better option.

• Follow Comment is free on Twitter @commentisfree