One of the few domestic political issues that occasionally manages to make itself heard over the Brexit cacophony is the rollout of universal credit. It is a huge and disruptive shake-up of the welfare system which will end up taking more than a decade from legislation to implementation. And it is no niche concern. One in three working-age households will be entitled to it once in place. Spending will top £70bn a year – roughly equivalent to what we spend on all primary and secondary education. And from April most benefits will have been frozen for the fourth year in a row.

We’d normally expect the chancellor to be focusing on this issue at this time of year. But the traditional spring budget has been moved to the autumn, and the spring statement – which Philip Hammond will give on Wednesday – is not intended to be a major fiscal event, so policy and spending announcements are unlikely.

Universal credit flaws leaving families in debt, campaign group says Read more

The UK’s total bill for working-age benefits is about £100bn a year. That’s one pound in every eight spent by the government; five times what we spend on social care; seven times what we spend on overseas aid; and more than twice what we spend on defence. It’s also over 50% more than we were spending just 20 years ago. Yet for all that spending, the system is hardly generous. One in three children still lives in poverty, according to a key official government measure. The generosity of most benefits has been cut in recent years. Stories of the difficulties faced by those needing to access disability and incapacity benefits abound.

We of course need to get the rollout of universal credit right. But doing so will not address the underlying challenges created by low pay, high housing costs and an apparently ever-increasing number needing support in the face of mental or physical illness.

Play Video 2:03 'Absurd and degrading': how universal credit can ruin lives – video

Even without the move to universal credit, the benefits system has changed fundamentally in recent decades. As well as growing, its structure and focus have changed almost beyond recognition. Most of the money now goes to households with someone in paid work. A system originally designed largely to support those out of work for a limited period has become one that supports the low-paid and the sick and disabled for long stretches of their lives. Equally, the contributory principle – by which receipt of benefits was supposed to be linked to previous payments through national insurance – exists only in the most residual form.

The first of the great challenges is simply low pay. Nearly 60% of benefits go to households where someone is in paid work but not earning enough to achieve an adequate standard of living. Despite this spending, two-thirds of working-age poverty occurs among working households. The big problem here is not just low pay, but the lack of progression. For too many people, low-paid work is not a stepping stone to better pay, it is a trap. The current benefit system has been effective in some ways, though. While earnings inequality has grown over the past 20 years, overall income inequality – which takes into account the growth of the tax credit system – has not grown among working households.

The second great challenge, all too familiar to millions of young families, is the cost of housing. In the benefits system this translates to the cost of housing benefit. This comes in at well over £20bn a year, double what it was in the early 2000s, as both owner-occupation and council housing have been replaced by the private-rented sector as the only option for many on modest incomes. Yet for all that money, the system hasn’t kept up with rising rents, and in most of the country won’t cover the full cost of renting even the cheapest properties. In fact, because of the absurd way entitlements are calculated, the housing benefit system is becoming a postcode lottery, with the gap between rents and entitlements growing wide in some areas but not others.

Finally, there is the challenge of the growing number of working-age people claiming incapacity and disability benefits – the former being paid to those unable to work because of mental or physical incapacity, the latter paid to those with disabilities irrespective of their work status. Spending on disability benefits has risen by a remarkable 50% since 2010, despite attempts to tighten up access. Those attempts have themselves created hardship and injustices.

One in four UK workers covered by national living wage underpaid Read more

Spending on incapacity benefits has stabilised but against a worrying trend. Benefits that, 20 years ago, were overwhelmingly paid to those over the age of 50 are now frequently relied upon by those in their 20s and 30s. A poorly educated 30-year-old is now twice as likely to be receiving incapacity benefits as a well-educated 60-year-old. The challenges in the labour market go well beyond those posed by poor health.

There is no magic bullet here. Neither universal credit nor a universal basic income nor “back to Beveridge” can overcome the inevitable trade-offs involved in designing a system where benefits are adequate and affordable, while not undermining work incentives. Universal credit will not move us very far. It is a huge project, but in the end it is fiddling at the edges of these trade-offs. Though a vast amount of energy will have been poured into making this one huge change, in truth, focus has been lost of what the problem was in the first place.

Ultimately, the solutions to this lie not just in the welfare system but depend on improved education, skills, wages, housing and health. Devising a comprehensive system that is fit for purpose is one of the greatest political challenges of our age. Philip Hammond will be grateful to have a few more months to resolve it.

• Paul Johnson is director of the Institute for Fiscal Studies