Stumbling into the dining room of a Brecon hotel after a night at a literary festival, I saw that the only spare seat in the dining room was next to Michael Gove. Should I take it, I wondered.

There is a loud body of opinion that says it is better to sup with the devil than breakfast with Gove. He is the most reactionary politician in the land, apparently. He "obsessively" follows an "elitist agenda", which will take English schools back to the 1950s. No one in the coalition, not even George Osborne, arouses such hatred.

How ridiculous all that can seem when you look at what he has done. Gove is completing the education reforms Labour began. Andrew Adonis, David Blunkett and Estelle Morris wanted the best graduates to become teachers in comprehensives and so does he. Adonis wanted to give schools the power and responsibility to become strong institutions by granting them academy status. Gove is just extending the Adonis programme. You will have noticed that Stephen Twigg, the Labour education spokesman, rarely mounts an effective assault. He keeps quiet because he knows he will accept many of Gove's changes when and if he returns to office

Gove is well read and a good writer, virtues I admire. (His Celsius 7/7 was an elegant contribution to the pamphlet wars of the Bush years.) In person, his conversation is an education in itself, while his exquisite manners are a lesson to us all. Of course I talked to him. I'm a journalist. I will talk to anyone and I had no trouble greeting a Tory gentleman, who seemed the nearest British politics can offer to a renaissance man. He is an easy minister to admire until, that is, the moment you cross him.

Here is how the retaliation works. The gang around him treat any slight to their master as an affront. The lead comes from his special advisers Dominic Cummings and Henry de Zoete. Cummings is a piece of work. He is a political hack of such reputation that Andy Coulson tried to blackball him from working for the coalition. If a former editor of the News of the World, now awaiting trial, warned me that a potential employee was too unsavoury to touch, I would pay attention. Gove did not.

Cummings and de Zoete can call on the services of Paul Staines, author of the Guido Fawkes website. They also have Telegraph journalists, the Murdoch press and most of the rightwing blogosphere at their disposal.

The Gove gang does not simply wish to beat his critics in argument but humiliates them too. Two weeks ago, Suzanne Moore criticised Gove in the Guardian. Every slight must be punished and Toby Young of the Telegraph duly denounced Moore's "hysterical, ill-informed rant". So delighted was his fellow Telegraph pundit James Delingpole, he cried that Young had given "Suzanne Moore such a seeing-to, she'll be walking bow-legged for weeks", an insult so gross even Delingpole had to apologise.

Last week, the Observer's editor put the allegation to Cummings that he smeared Gove's enemies from behind the coward's cloak of anonymity on the @toryeducation Twitter account. Cummings did not directly deny it but instead boomed: "Take a Twitter detox because it's melting your brains, focus on what's important, stop behaving like eight-year-olds." He sounded more like a student politician who had had a lager too many than an adviser to a cabinet minister. Yet so unable were Gove's enforcers to see themselves as others see them that they leaked the private correspondence to a news website, convinced that it made them look good.

Inside the education department, life is little better. This week, we report on the case of a senior civil servant who received a large payoff after claiming to have been "marginalised, undermined and bullied" by Gove's people. Whenever officials raised awkward issues, the civil servant said, the question "What do you fucking mean?" echoed around the department, which, lest we forget, is meant to educate our children.

Conservative readers may protest that Gove has to fight rough because he is up against the resistance of the "education establishment". Any reader of Education, Education, Education, Adonis's account of the struggle for school reform in England, will be struck by how hard Adonis found it to shift established interests and complacent bureaucrats. But Adonis's evidence cannot help Gove. Like every decent minister, Adonis allowed his subordinates to speak to him freely. It's not only the long-standing civil servant who found you cannot do that with Gove. Tim Loughton, a Tory MP who was children's minister until Cameron sacked him in the last reshuffle, recalled that Gove would appear on rare occasions in his department like "Mr Grace from Grace Brothers and tell us we've all done terribly well and then disappear".

Although there are worse insults than being likened to a much-loved character in Are You Being Served?, the Gove gang went wild – yet again. An unnamed "source" told the Spectator that Loughton was a "disgraceful" hypocrite and "a lazy incompetent narcissist obsessed only with self-promotion". No one can say my colleagues on the Spectator suffer from an excess of political correctness, but even they were shocked by the violence of the attack.

The most shaming comparison for Gove is with Gordon Brown. The old schemer put on the show of being a civilised man while his minions, Charlie Whelan and Damian McBride, ran filthy campaigns against enemies real and imagined. Brown, too, could call on the services of journalists who had long ago lost what little independence of thought they possessed. But there is a difference. Whelan and McBride relied on old media. Their successors in Gove's education department can get their black propaganda out in public via anonymous social media accounts.

If life were fair, Brown's plotting and smearing would have done for him. Alas, they took him to Number 10. It was only when he was there that the electorate saw him for what he was. Will Gove have to wait until he is prime minister before the public sees through him too or will his comeuppance come sooner? I hear that an alarmed David Cameron can see that Gove is preparing a leadership bid and is keen to hear an answer to that question as quickly as he can.

Nick Cohen's book You Can't Read This won polemic of the year at the Political Books of the Year awards