Let's face it, we all live on Benefits Street. The Channel 4 series may be raising hackles to left and right, but I doubt if there is a person reading this column who is not "on the take" in some sense. We may work a bit, mostly obey the law and not look a total mess, but then we are not really poor. We can still be "on benefit", and some of us are rich because of it.

I was once driving down a country lane with a diplomat friend when we slowed to a crawl behind a tractor. My friend finally lost patience and blurted out that he could count "seven different subsidies" holding us up ahead. He listed them in a rage.

I proceeded to ask after his relocation allowances, subsidised school fees and index-linked pension. He retorted with my heating grants, tax deductions and charity reliefs. I asked which state-funded arts consultancy employed his daughter and which awareness-raising quango kept his son from occupying the family nest. We were about to deflect to the banker in a nearby Cotswold manor, snorting his quantitative easing, when the tractor turned off into a higher-level stewardship meadow to plan a subsidised windfarm. We bowled on down our own benefits lane.

We are all on the game; some of us are just smarter at concealing it. I have a book on my shelf by the Americans Mark Zepezauer and Arthur Naiman called Take the Rich Off Welfare. It glares down at me whenever I think of writing about poverty. It shows how well-heeled Americans, starting in the Reagan years, cornered the lion's share of public spending. They had capital depreciations, fiscal reliefs, muni bonds, fuel subsidies, bailouts, price supports, cultivated waste and tax frauds. It was called "wealthfare".

This was no leftwing tract. It merely pointed out that "wealthfare costs the American taxpayer some three and a half times the cost of welfare for the poor". The relentlessness of the rich lobbying Congress for tax breaks and subsidies meant "the US government today functions mostly as a huge Robin Hood in reverse". If there is money going begging, those who beg loudest get most.

By this book lay other revelations of fiscal outrage. There was The Sacred Cow (agriculture), Putting Patients Last (the NHS), The Terrible Truth about Lawyers (law), A Dinosaur in Whitehall (defence) and Plundering the Public Sector (consultants). The loftier the profession, the wilder the scams. Looming over them was JK Galbraith's pre-crash diatribe, The Economics of Innocent Fraud. After skimming this lot, I thought Benefits Street was like a meeting of the Mothers' Union.

Of course, many of these beneficiaries do serious jobs and rarely break the law. But then, they can get their hands on other people's money without stealing it. One in five British employees now works for the state. In addition to their pay, they have negotiated index-linked pensions that cost the taxpayer £9bn a year, beyond the dreams of those working for themselves.

This is merely the tip of the iceberg. Child benefits, agriculture subsidies, windfarm subsidies, house-buying subsidies, arts subsidies and tax avoidance schemes may satisfy the political and electoral goals of passing ministries, but then so do subsidies to the poor. Benefits Street's denizens may do drugs, door-to-door selling, odd jobs and pilfering. At least I know them. Running down the Guardian's interactive guide, Visualising Whitehall, I am not sure what is meant by "corporate development, change delivery, compliance strategy". They sound like upmarket benefits scrounging to me. I am sure Benefits Street's White Dee would say she was in human resources strategic delivery if she realised it held the key to George Osborne's wallet.

Whole industries are on white-collar Benefits Street. Higher education, despite fees, is a cross-subsidy from taxpayers (including the poor) to mostly middle-class students and professors, now to the tune of billions of pounds a year. The construction industry is a monument of public plutocracy. Having pocketed £9bn from the Olympics, it is now building Crossrail and hopes to win HS2 and Heathrow Three. Its builders are eating their way through Osborne's soaring subsidies. These projects are essentially tax transfers from poor to relatively (or very) rich.

"Austerity Osborne" claims to be cutting back on public sector jobs to boost private ones. He is shortening Benefits Street to lengthen Enterprise Alley. Public sector employment is falling overall, but the fall is in lower-paid local government jobs outside Osborne's control. His personal Benefits Street, known as central government, actually grew last year by 55,000 jobs. Austerity is always for the other guy.

Eight years ago, David Craig's Plundering the Public Sector calculated that 10 years of New Labour had seen £70bn vanish from taxes into management consultancy, PFI and IT fees, to no noticeable public gain. Most Whitehall IT projects had been fiascos, and there is a new one each week. The beneficiaries have been the rich: firms such as KPMG, Deloitte, PwC, Capita, Serco, McKinsey and others. Today's public accounts committee may howl about waste, but the stable is bare and the horses are over the horizon, laden with gold.

None of this approaches the greatest benefits scam of all, the accumulation of public and private debt incurred by today's citizens at the expense of future tax- and interest payers. Britain's public and private debt together runs to some 500% of GDP, by far the highest ratio in history. Such benefits to today's citizens – in pensions, housing, travel, lifestyle – are at the expense of tomorrow.

As Niall Ferguson said in The Great Degeneration, his 2012 Reith lectures revised in book form, equitable finances have long relied on trust between one generation and the next. He concludes: "The enormous inter-generational transfers implied by current fiscal policies [are] a shocking and perhaps unparalleled breach" in that trust. The young are entitled to see parents and grandparents as all on benefits at their expense. The pensions time-bomb is real.

Like all journalism, Benefits Street tells a partial truth. Its lesson is banal, that any group of people will live according to their means. At least Iain Duncan Smith is making the first ever serious effort at reform – crippled by computer failure by another sub-contractor beneficiary. But in fairness, Channel 4 should now go for somewhere far harder. It should move from Benefits Street to Subsidy Row, and dig out the serious scroungers.