Summer holiday season is upon us, and one of the little sun-lounger fantasies I like to indulge, undoubtedly along with millions of others, is the idea of jacking it all in and buying a one-way ticket to somewhere drenched in sunshine all year round but so cheap you can make your savings stretch. Then I get back to the 9-5, and the September drizzle, and file it away until next summer.

But what if there’s a bit more substance to this daydream – at least to the notion of working less, if not giving it up altogether? The idea of a shorter working week may sound fantastical but it’s gaining traction on the left: last year the TUC backed a four-day week, and the shadow chancellor John McDonnell has commissioned a review into it.

It’s not actually that radical an idea, as I found out while making a programme on it for BBC Radio 4. The working week was once an 80-hour affair. As recently as 50 years ago, everyone assumed it would continue to get shorter; that as society became more prosperous, workers would enjoy the proceeds, not just in bigger pay cheques, but with more time off. University leisure studies departments flourished in the 60s and 70s as we supposedly stood poised to enter the age of recreation.

So how did the five-day, 40-or-so-hour week come to be seen as standard? Trade unions got weaker, but that’s far from the whole story, according to the historian Ben Hunnicutt. Around the turn of the 20th century, work evolved from being simply a means to an end for the vast majority, to an end to aspire to in itself. This was compounded by an explosion of consumerism in the 1920s. At first, this worked in favour of a shorter week: Henry Ford was one of the first manufacturers to move his factories from a six- to a five-day week, partly because he knew that in order to sell more cars, people needed the free time to enjoy them. But only up to a point: our reliance on consumption-fuelled economic growth put us on a treadmill that’s not so much work to live as work to buy.

There were a couple of points in history where we could have got back on to the path of shorter working time; both came at moments of economic crisis. President Franklin Roosevelt’s initial response to the Great Depression was to back a 30-hour working week, the idea being to share the dwindling pool of work among more people. But US manufacturers persuaded him to switch tack. In 1974, Ted Heath’s Conservative government temporarily imposed a three-day working week in energy-intensive sectors to conserve the coal supply.While the 1970s have a reputation for being an economic disaster zone, productivity rose by 5% in just two months as a result of the three-day week. But the Reagan-Thatcher consensus that followed cemented the idea of work as fundamental to growth; leisure as, at best, irrelevant. Ten years on from the financial crash, in an age of sluggish wage growth, it’s time to revisit it. The promise of work as something that can give our lives fulfilment and meaning hasn’t materialised. Three-quarters of British workers have little enthusiasm for their work, with one in five resentful and unhappy. One in eight live below the poverty line.

Work evolved from being simply a means to an end for the vast majority, to an end to aspire to in itself

There was a lot of talk about improving the quality of low-paid work during the Blair-Brown years. It’s a worthy objective, but doesn’t eliminate the uncomfortable truth that there are some jobs that aren’t the pinnacle of aspiration for most of those who do them. It’s a small step from acknowledging the naivety of believing work can be fulfilling for everyone, to thinking it would be healthy to reset our relationship with it.

There are two other reasons to reconsider the length of the working week. We already work a 32-hour week on average (pretty much a four-day week); it’s just that it’s predominantly women who work part-time. That’s because mothers are more likely than fathers to go part-time after having children because they earn less on average, further choking off their career progression. And so a 30-hour week could help close the gender pay gap by redistributing paid work from men to women, not to mention giving men more time to do some childcare and hoovering.

Many environmentalists argue that we shouldn’t simply be redistributing paid work, but embracing a produce less/consume less lifestyle because the planet can’t sustain more untrammelled growth. It’s intuitively appealing, but people working in low-paid, monotonous jobs most deserve to enjoy a cut in their hours – and it’s they who can least afford to take a pay cut, and who anyway have a comparatively tiny carbon footprint.

This is the paradox at the heart of the case for a shorter working week. The priority must be cutting hours for people working in today’s dockyards and mines – Amazon warehouses and call centres. People in well-paid, fulfilling jobs who already have a lot of autonomy over their work are not really the target.

But if the government were to impose a shorter working week without first tackling structural labour market inequalities, the latter would benefit to the exclusion of the former. France, with even lower rates of union membership than the UK, and where the government imposed a 35-hour working week in the late 90s, provides an instructive example. Managers got to bank the extra time in additional annual leave which is one reason middle-class Paris is deserted in August. Lower-paid workers saw much less of a reduction, as employers were allowed to redefine working time, and got more power to dictate their shift patterns.

So feel free to dream away on the lounger, but the reality for many middle-class professionals is that they can already afford to go on a four-day week, and have the power to negotiate it. A shorter working week for those who need it most implies a labour market where the low-paid have more rights and bargaining power. The rest, rightly, should expect to pay more as taxpayers and consumers for the services and goods they produce for us.

Analysis: The Working Week is on BBC Radio 4 at 8.30pm on Monday 22 July

• Sonia Sodha is the Observer’s leader writer