Before he urged bosses last week to dish out pay rises, David Cameron should have met Martin. If nothing else, the encounter might have given him pause before that “Come on, chaps” routine. Because Martin could have told him of an entire world of work that the public hardly ever hears about – whose misery runs as deep as any life on those much-publicised zero-hours contracts. His story confirms a truth some frontbench politicians admit in private but none dare acknowledge in public: that something in the jobs market is fundamentally broken.

Martin isn’t his real name and, to prevent any reprisals, I shall obscure a few key details. In a curious way that fits his story. Martin, you see, is an unperson, in a workforce that is breeding unpeople by the hundreds of thousands.

He works at the Jacob’s biscuit factory in Liverpool, although he’s not employed by the firm. Instead, he works for Prime Time Recruitment, one of Britain’s biggest suppliers of contract labour. He does the same work as Jacob’s own staff, and often substitutes for their absences – but is on the minimum wage; they get paid over £2 an hour more. They’re on a rota, while he has to wait by his phone for each shift. His holiday and sick-pay entitlements are much worse, too. Jacob’s staff enjoy access to a workplace gym and medical centre; when Martin had an accident with a forklift truck, the medical centre turned him away, telling him to find his own doctor.

Martin never knows when he will be working: just last week, he was rung up in the morning to be told he had 45 minutes to his next shift. Nor does he know where he will be working: his agency regularly offers him work in a factory in Blackpool. As Prime Time pointed out to me on Monday, that’s within a contractual 25-mile radius as the crow flies. But as Martin demonstrates on Google Maps, it’s over 50 miles by road, and the travel costs mean that a six-hour shift would only earn him just over £3 an hour. When he turns down such impossible jobs, the agency then uses a legal loophole to chalk him up as unavailable for work and pays him nothing.

Martin’s phone, with its texts and calls from Prime Time, has more control over his life than he does. As a middle-aged father, he’s borrowed to cover food and fuel bills, and “I’ve not known when I’ll be able to pay anything back”. Holidays? Forget it. He points at one of two toddlers messing around on the living-room floor: her first Christmas was “nearly ruined” for lack of money.

What makes this different from zero-hours work is Martin isn’t directly employed by his workplace. But the effect is almost the same: the workers and their families are burdened with all the insecurity and powerlessness, while their employers enjoy the flexibility of labour on tap.

Add the estimated 1.2 million agency workers to the 600,000 on zero-hours contracts, and you have a shadow workforce of about 1.8 million unpeople, enjoying none of the security that should come with employment in a rich country. Thanks to official neglect, so much about these unpeople is unknown. We don’t even know how many employment agencies there are, let alone how many names they have on their books.

But we can say some things about such workers. They are never going to be helped by prime ministerial exhortations for a pay rise. As such, they form part of the answer to one of the major conundrums in Britain: what’s gone wrong in our labour market. This Wednesday will see the monthly ritual of a jobs report showing record employment, but meagre wage rises. This will be followed by Labour politicians attacking Tory austerity, and government spokespeople assuring us that brighter days are just ahead.

The shadow workforce tells us that all this is bunk. Its growth points the way to a new world in which big employers draw upon a pool of casual labour, stripped of most of their rights and bargaining power. It’s a pool that’s been growing fast since the banking crisis. Particularly heavy users of agency workers are logistics companies of the kind that deliver our online Christmas orders, and food manufacturers such as Jacob’s, which disclosed last year that at any one time up to 250 of the 900 staff at its Aintree factory could be agency. (Today the firm estimated the agency number at 70.) Last year, the company marked its centenary of operations in Liverpool. Generations of locals would have passed through there, and the royal seal still hangs over the door. But those social ties now count for nothing; the factory is becoming a hub in an extractive labour market.

The shadow workforce is spreading especially fast in those economies that never recovered from Thatcher’s great industrial wipeout. Much of the former steel town of Corby is now just agency land. And in Martin’s home of Liverpool, the city council found that of the 13,771 vacancies advertised in Merseyside, nearly half – 6,600 – were for temporary agency work. Its report, published last autumn, in effect punctures all the smug forecasts that the jobs market will soon turn a corner. What it tells you is that for people like Martin, in hollowed-out economies like his suburb of Liverpool, the market isn’t suddenly going to right itself. And until that happens – as Barry Kushner, a Labour councillor, points out – the housing association will continue to rack up arrears, and the waiting list for advice at the local Citizens Advice bureau will grow.

Sure, there are supply teachers enjoying the freedom that agency status brings. And one imagines that others do skip from temp job to fully fledged career. But elsewhere there are people like Martin, who have no choice. He’s holding out for a permanent staff position, yet can’t think of anyone who has made the leap. Nor can he sign on while looking for other work, because that would count as making himself voluntarily unemployed and lead to being deprived of jobseeker’s allowance. So he is stuck at the very bottom of the pile.

There are winners from Martin’s position, of course – just not him and his family, or us taxpayers. The workfare company A4e, which sent Martin to the factory, gets a fee from the taxpayer for placing a warm body in a job. Jacob’s, an arm of a £2bn business, gets cheap labour with none of the pesky overheads or obligations. Middleman Prime Time, part of a group with a £500m annual turnover, takes its cut and was until last August getting government funding on top. The losers are Martin who, for all his wit and diligence, faces years on the minimum wage; and us, who top up his poverty pay with benefits.

A few hours after Martin and I say goodbye, he texts me with a message for you, the readers. What you need to know about agency workers, he says, is, “they don’t live, they get by”.