The biggest threat to our jobs is the rise of the robots, or so we’re told by report after report. And so an idea has taken root in our collective psyche that there may be a point in the not-too-distant future where the daily grind is a thing of the past. A future in which there’s a small number of jobs designing artificial intelligence and programming the robots, leaving the rest of us to spend our days at leisure.

Whether you see that prospect as delicious or frightening, this future-gazing is a dangerous distraction. First, it was industrialisation, then the computer revolution: predictions about the end of work have been made countless times. But history tells us that as technology has replaced certain jobs, others have been created. There’s no reason to think it will be any different this time.

Also, speculating about the future risks diverting our attention from what’s happening to our labour market now. If technology really were on the cusp of replacing work, we’d expect workplaces across the economy to be taking on a more futuristic sheen. But in some sectors, they are moving backwards. Far from investing in technology, some employers are adopting an old-fashioned approach to sweating their labour.

That’s what Peter Nolan, the director of the Centre for Sustainable Work and Employment Futures, who I spoke to for a Radio 4 programme on workers’ rights, sees happening in Leicester’s burgeoning textiles industry. Offshoring resulted in Britain’s garment industry experiencing a steady decline from the 1970s, as big retailers started to source from cheaper factories in the global south. In recent years, that has started to reverse, thanks to the rise of “fast fashion”. High street brands expect turnarounds of seven days or so from placing an order to taking delivery in a store, time scales that factories in Turkey and India simply can’t meet.

This revival has been hailed as a “huge renaissance” by the local enterprise partnership: there are now 20,000 jobs in apparel manufacturing in the East Midlands. But scratch the surface and beneath there sits a tale of quasi-Victorian labour exploitation. Nolan’s research has uncovered widespread flouting of minimum wage legislation across the industry. Many of the workers are migrant women, with limited English, who are denied basic employment rights. I spoke to a woman from Kenya, who has worked stitching clothes since moving to Britain 13 years ago, on the condition of anonymity. She told me that she was paid just £4.50 an hour and had no contract or regular hours. The intense nature of her work had given her chronic back problems, but she had no option but to continue working as she gets no sick pay. Her experience is far from atypical.

The garments these workers produce ultimately find their way into high street stores across the UK. But the sweatshop-style factories and workshops in Leicester are to be found all the way down a subcontracting chain that delegates responsibility for employment rights from company to company, until it ends up in the hands of the cowboys running these operations.

They deploy a number of practices to get round the law, such as paying workers for a 20-hour week at the minimum wage when they might be putting in closer to 60 hours or, if a company gets raided, simply winding it up and creating a new company with the same directors.

This is the underbelly of our labour market: illegal exploitation, plain and simple. But there are other legal means employers can use to sweat their labour. In a sector such as logistics, smart technology is not being used to replace workers altogether, but to make them increasingly resemble robots. Parcel delivery and warehouse workers find themselves directed along exact routes in the name of efficiency. Wrist-based devices allow bosses to track their every move, right down to how long they take for lavatory breaks and the speed with which they move a particular piece of stock in a warehouse or from the delivery van to someone’s front door.

This hints at a chilling future: not one where robots have replaced us altogether, but where algorithms have completely eroded worker autonomy, undermining the dignity of work and the sense of pride that people can take in a job well done.

The state must play a bigger role in combating these practices. The last 30 years have seen a transformation in employment rights, but what is happening in Leicester shows that for some workers, their rights are not worth the paper they are written on.

In the UK, it’s overwhelmingly up to individual workers to enforce their own rights by taking unscrupulous employers to employment tribunals. That was always a dubious assumption for vulnerable workers facing exploitation and intimidation, but the hefty tribunal fees introduced by the government in 2012 have effectively taken justice completely out of the reach of the low paid.

Just as bobbies visibly policing their beat are critical to preventing crime, we need a properly resourced network of workforce inspectors charged with checking that employers in high-risk industries are complying with employment law. This would not only be pro-worker, but pro-business: the state’s failure to enforce employment rights allows cowboys to undercut those companies committed to treating their workers well, dragging down standards across a whole industry.

But the state can only enforce minimum standards. It cannot legislate against companies using technology to turn workers into robots. Fifty years ago, trade unions fought for an improvement in working conditions that went above and beyond minimum standards.

But the union movement is a shadow of what it used to be: membership rates have dropped from one in two workers in the late 1970s to fewer than one in four today. Unions have increasingly become the preserve of better-paid, educated public service professionals and represent fewer than one in 10 of the lowest paid.

There is indeed, then, a dystopian threat to the labour market. But it’s not the rise of the robots: it’s the emergence of a two-tier labour market enabled by a weak state and an enfeebled union movement that results in workers being denied their legal rights and technology being used to further degrade low-skill work.

Who Speaks For the Workers? is on Radio 4 at 8.30pm on 26 June