Ten years after hunting was banned, the people in red coats are gathering for their Boxing Day meets as enthusiastically as ever. Thousands from all walks of life will take part in a form of trail hunting in which the pre-laid scent of a fox is followed by the hounds.

I can think of nothing better after a day of overeating and sitting on the sofa watching bad television than getting up early, dressing smartly and climbing aboard a good horse to hurtle over hedges. But I am under no illusion. In a world where most people prefer their encounters with nature to be confined to a factory-plucked turkey followed by an animated film about penguins, I know I am in a minority in wanting to confront the natural world in all its rugged reality – or something like it, since we now can’t be useful and actually set out to kill vermin.

Hunting is a niche sport in Britain. But I would argue that opposing hunting is an even more rarefied pursuit. That is because the hatred that consumes the antis has nothing to do with animal cruelty, which is minimal compared with the cruelty that is done to animals in the name of the mass production of cheap food.

No. The reason the antis are so unhappy that we are still out there, revelling in the great outdoors, has less to do with the welfare of foxes than with the fact that they still have to look at “toffs” in red coats. The fury of Labour MPs at the legal form of trail hunting that still continues shows that what they were really after 10 years ago was a ban on posh people in riding breeches. Back in 2004 , when the ban was being put through, I interviewed Peter Bradley, then Labour MP for The Wrekin, who admitted the hunting ban was about “class war” and fighting the landed gentry. He lost his seat in the election that followed because voters in his constituency did not want such a conflict. Most people still don’t.

And yet Labour remains obsessed. A few weeks ago the MP Paul Flynn called anyone planning to attend a Boxing Day meet a “drunken sadist”. Well, I’m teetotal so I view that remark as extremely defamatory. As for sadism, I’ve been thrown off and broken a range of bones over the years. I call that masochism, if anything.

The Birmingham MP Roger Godsiff, meanwhile, wants the Environment Secretary Liz Truss to prohibit the sale of lapel badges bearing hunt insignia. When leftwingers descend into this petty tomfoolery, you can see why Tony Blair called the ban a mistake and “not one of my finest policy moments”.

He knew that the mere mention of hunting revealed a nasty, divisive side to Labour. Its hysteria on the issue reminds us that the party has a mean streak, one that envies and resents and seeks to bring down those who are thriving or enjoying themselves too much.

Naturally, if you ask our predominantly urban population whether people should be allowed to pull foxes apart with dogs (and some anti-hunting polling has asked almost exactly that), most will say no. But ask what affects them most – the health service or the way foxes are controlled – and hunting will never come up. And yet Maria Eagle, Labour’s rural affairs spokeswoman, is flirting with groups such as the League against Cruel Sports as it calls for further laws prohibiting the gathering of hunting people.

Let us be realistic. Even if they ban the wearing of red coats in public, preposterous and illiberal as this will be, the world will still turn. About 45,000 people hunt. They represent a minority within a rural minority.

But many more people understand that the agenda behind the ban was never meant to stop with foxhunting. Some 600,000 people in the UK shoot, millions fish and many millions more love horse racing. They know that they could be next. If Labour seeks to harden the hunting ban, it is going to alienate millions of voters. Ed Miliband should remember that in 1997 Labour had more than 100 rural MPs, and could, with a straight face, almost begin to claim to be the party of the countryside. It will never again come close to that high-water mark if it reopens its class war on the hunting community.