What was the most amazing thing about last weekend's 24 Hour Panel People (highlights of which will be shown all week on BBC3 in the run-up to Friday's Comic Relief)? Was it David Walliams's incredible achievement of performing live in panel shows for a solid 24 hours – a feat of comic endurance not equalled since someone at the BBFC had to classify the DVD release of the complete Chucklevision? Or the generosity with which a host of Britain's best-loved comedians and Nicholas Parsons gave their time to make the event such a success? Or the speed with which the people at Dave subsequently put together a business plan for a 24-hour live rolling panel-show channel entirely fronted by Chinese children?

Well, I was there and I can tell you it was none of those things. The most amazing aspect of it, as a contributor, was the number of people bustling around with clipboards and headsets. Wherever I stood, unless actually on camera, dozens of them would immediately try to push past, politely but hurriedly, as if I'd obstinately positioned myself on the route of an air traffic controllers' fun run. "What can they possibly all be doing?" I thought irritably, forgetting temporarily that I lack the knowledge or power to self-televise.

It's an easy attitude to fall into, assuming that everyone else is perversely inconveniencing you, rather than having jobs or problems of their own – sitting in heavy traffic thinking: "Where are all these people going? Do they really need to? I'm late! They're getting in the way." In the case of this particular TV studio, I was the one who was getting in the way, and also having the gall to question the necessity and urgency of what I was getting in the way of: "Where are they going with the clipboards? Who are they talking to on the headsets? None of this makes any sense! All this process requires is people like me going in front of cameras and talking some shit."

That's precisely what David Cameron thinks about government. He simply can't understand what all the guys in headsets – the civil service – are up to. And he says it's not just him they're annoying – they're pushing past or obstructing the whole private sector. In an extraordinary speech to the Conservatives' spring conference last weekend, he called them the "enemies of enterprise". To him, they're the Klingons.

He said he was "taking on… the bureaucrats in government departments who concoct those ridiculous rules and regulations that make life impossible for small firms". On the face of it, this is simple crowd-pleasing stuff. It's easy to slag off the faceless bureaucrats, who supposedly waste our time and money with all their stupid rules. It's convenient to forget that bureaucrats, or civil servants as they're called when they're not being victimised, don't actually make rules, they just enforce them. Maybe, sometimes, they enforce them officiously. Maybe, sometimes, the processes they "concoct" for enforcing them are unnecessarily time-consuming. Maybe fewer of them could enforce the rules just as effectively. But they don't make the rules, Parliament does.

In seeking to blame the civil service for the rules as well as their enforcement, I think this speech is more sinister than Cameron's usual second-rate demagogy and I'm surprised it didn't attract greater attention. To me, these remarks are just as damaging as the prime minister's disparagement of multiculturalism, which rightly drew criticism, and a truer reflection of his political standpoint. Here he's breaking new ground for his evidence-averse Thatcherite ideological crusade.

The whole premise of this government, of its NHS policy, of the "big society", of the "free schools" initiative is that the public sector sucks. The private sector, according to the Tories, beats it for efficiency every time, can be just as compassionate and, at the top, "rewards enterprise". Meanwhile the top of the public sector merely "pays people more than the prime minister".

But in this speech Cameron takes the argument further. By labelling civil servants as enemies of business, he's trying to make them responsible, not just for the failings of the public sector, but also those of the private: "Every regulator, every official, every bureaucrat in government has got to understand that we cannot afford to keep loading costs on to business," he says. "If I have to pull these people into my office to argue this out myself and get them off the backs of business then believe me, I will do it."

He's always said that, when the state wastes money, it's because of the bureaucrats. Now he's also saying that, if private enterprise fails to grow, prosper or fill the gap that shrinking government creates, that's not a flaw in George Osborne's economic policy, that, too, is because of the bureaucrats. In short, whatever goes wrong is the bureaucrats' fault.

If he can get this to stick, it's a masterstroke. It's what Mao was doing when he declared war on sparrows or intellectuals. In difficult times, deft powermongers deliver up whipping boys for the disgruntled. By picking on civil servants, Cameron has made an excellent choice: they work for him, so it's hard for them to complain; they enforce government policies so if policies fail, he can blame the enforcement; yet if they succeed, he can keep the credit.

As a policy, however, it's meaningless. He can't act separately from bureaucrats, he has to act through them. Everything he does – every transparency initiative, every "big society" clarification document, every restructuring of the NHS or the welfare system, creates work for bureaucrats. He also said in the speech: "There's only one strategy for growth we can have now and that is rolling up our sleeves and doing everything possible to make it easier for businesses to grow", without acknowledging that it's the bureaucrats' sleeves he's talking about, not his own or those of his party faithful.

Cameron also doesn't realise, or is wilfully ignoring, how important our large and basically effective bureaucracy is to our place in the front rank of free nations. Without the civil service, acts of Parliament are only words and elections just millions of little slips of paper, like they are in Afghanistan. Civil servants don't merely oil the wheels, they're the axles that join them. Without them David Cameron and his policies would be no more a government than Ian Hislop sitting in a field being sarcastic would be an episode of Have I Got News For You.