Culling wildlife has a bad name right now. The government's badger cull has been a disaster on its own terms and a catastrophe by any sensible assessment. But Britain is divided, and not just over badgers. While the culling of badgers in the West Country is condemned by conservationists, in the Highlands wildlife charities are seeking to dispatch large numbers of another majestic native: red deer. And many nature-lovers would cheer an intensified cull of deer populations across the country. To philosophers of animal rights, "conservation culling" is an Orwellian atrocity – "speciesism" in action. And conservationists – and I include myself in this group – sometimes flounder in the face of such clear and principled thinking.

First, the badger cull. Owen Paterson, the environment secretary, has staked his career on a cull proving efficacious, safe and humane. His target was to kill 70% of badgers in the Somerset and Gloucestershire cull zones over six weeks. He failed. Even when Defra moved the goalposts on the badger population to create a lower margin for success, his marksmen dispatched 65% and 39% in Somerset and Gloucestershire respectively. This lack of efficiency is bitterly ironic because the cull was based on farmers' premise that boffins make bad killers. Farmers believed they could organise a more brutally efficient cull than scientists, whose lengthy turn-of-the-century cull concluded that it would reduce TB in cattle by 16% at best.

The badger has endured as Britain's largest surviving carnivorous animal when far more formidable creatures, such as wolves, have been driven to extinction, because it is superbly furtive (and hasn't seriously threatened human interests before bovine TB). It was never going to be easy to cull. But a significant cause of the cull's failure is a human one: the activists who spooked badgers and shooters and smashed up traps, despite an overbearing police presence to support the marksmen.

Vilified by Paterson and farmers (some of whom received horrible threats over recent months), the activists I met in the cull zone were peaceable, ascetic vegans, who sacrificed careers to sabotage fox hunting and now devoted nights to saving badgers. Many were ideologically opposed to dairy farming and pheasant shooting, driven by a fundamental belief that every animal has a right to life. Their success puts wildlife groups in a tricky position: charities opposed to the badger cull, from the Wildlife Trusts to the RSPB, continue to quietly cull wild animals – shooting foxes to stop them predating the nests of endangered birds, for instance. Some of their members are animal lovers who sabotaged the badger cull.

In Scotland, there is a fascinating inversion of the badger debate in southern England. Here, the people calling for a serious cull of a native mammal are conservationists while those shouting to save it are hunters who profit from traditional deer stalking. The Scottish parliament is considering whether it should compel landowners to cull more deer.

Contrary to popular perception, runs the pro-cull argument, wild Scotland is shaped by man and denuded of wildlife. Highland clearances and sheep farming were the initial cause of a tree armaggeddon but this land is now chewed to an ecological desert by a burgeoning deer population. Red deer density in Scotland is 100 times higher than in Sweden, with an estimated population of 450,000 (and 300,000 roe deer), compared with just 180,000 red deer in Germany. Scotland's deer population is up threefold since the 1970s, thanks to climate change but also the winter feeding of wild deer by stalking estates. Many of the fragments of ancient woodland that remain – Scotland's lost rainforests – are in poor condition.

Habitats including peat bog and montane scrub would benefit from fewer deer and so would hundreds of plants, birds and insects: capercaillie, juniper, aspen, woolly willow, wood warblers, chequered skippers. Fencing won't help because it puts more pressure on the unfenced areas; anyway, healthy woodland has some deer, just not the current inundation. Reintroducing top predators such as wolves might check the deer population more "naturally" but this remains a rewilding fantasy: as every big mammal knows to its cost, humans still struggle to tolerate any animal powerful enough to impinge upon our safety or immediate economic wellbeing.

Animal lovers question how it can be rational to condemn a badger cull but support the culling of other native mammals. It is probably easier to argue in support of killing non-native creatures – American mink, released from fur farms by animal rights activists, have decimated water vole populations, while non-native grey squirrels wipe out reds – but for proponents of animal rights this smacks of fascism. The arrival of "exotic" species is not actually analogous to human immigration because introduced animals are different species and cannot always live together as humans should try to do, but it is wise not to base culls on whether an animal is native or exotic. When do we decide that an animal imported by humans has a right to a wild life in Britain? Does the rabbit now qualify? The brown rat? Wild boar? Surely, however, it is incumbent upon us to preserve as great a diversity of wildlife as we can. Notions of what is "natural" may be artificial but we should still use science to determine how best to manage habitats for as many species as possible.

Many conservationists oppose the badger cull with arguments from the disputed mountain of scientific evidence. But in this case, ultimately, our decision about whether we support it must come down to an ethical question, which was actually posed to me by a cattle farmer: do we want to farm in a way that is dependent upon wiping out a large proportion of a native mammal?

A cull of deer is ethically different because it is motivated not by a desire to profit from livestock but by a scientific understanding of how we can best conserve the widest range of species, and enhance our environment. Most vocal opponents of a deer cull in Scotland are not animal rights activists but landowners who are, in effect, farming a wild animal for profit.

Wild animals do have a right to life but, rightly or wrongly, we are custodians of the natural world and must make difficult decisions when a deer's right to life comes into conflict with the right to life of an aspen sapling or a capercaillie.

• This article was amended on 4 December 2013 to clarify deer population figures.