It was an "isolated incident", US officials insisted. The murder of 16 Afghan civilians as they slept, Hillary Clinton declared, was the "inexplicable act" of one soldier. And as Barack Obama and David Cameron prepared to put a public gloss on an earlier end to Nato's "lead combat" mission in Afghanistan, the US secretary of state pledged to continue "protecting the Afghan people".

After a decade of ever more degraded Nato occupation, who could conceivably wish for such protection? The slaughter of innocents in Panjwai, nine of them children, follows the eruption of killings and protests after US troops burned copies of the Qur'an last month. That came soon after the exposure of video of US marines urinating on dead Afghans.

The evidence surrounding the Panjwai massacre is so far contradictory. If it was the work of a single gunman, he was likely to have been unhinged or motivated by perverted religious or racist hatred. But however extreme, it was certainly not an isolated incident.

As in Iraq, the killing and abuse of civilians by occupation forces has been an integral part of this dirty war from its earliest days. As it drags on, ever more outrages emerge. Last year, members of a US unit were convicted of killing Afghan civilians for entertainment, cutting off body parts as trophies and leaving weapons with the corpses to make it seem as if they were killed in combat.

Nor is such depravity just a US habit, of course. Last year a hungover British guardsman stabbed a 10-year-old boy in the kidneys for no reason. British soldiers are currently on trial for filming their abuse of Afghan children, while US WikiLeaks files record 21 separate incidents of British troops shooting dead or bombing Afghan civilians.

The line between deliberate and accidental killings is in any case a blurred one. As the US General Stanley McChrystal, former commander of Nato troops in Afghanistan, commented: "We have shot an amazing number of people, but to my knowledge, none has ever proven to be a threat."

When six British soldiers were killed in Helmand last week, taking Britain's 10-year military toll over 400, their deaths were treated by politicians and media alike as a national tragedy. Meanwhile tens of thousands of Afghans have been killed in the war launched by the US and Britain in Afghanistan, but even the names of the 16 Panjwai victims are largely unreported.

Last year was a record for civilian deaths in the Afghan war: 3,021 were reported killed by the UN, which blamed Nato and its Afghan allies for 410 of them – though Afghan human rights organisations insist that such tallies heavily understate the numbers killed by foreign troops, whose casualties are said routinely to be blamed on the Taliban or not reported at all.

Many civilians are killed in night raids or air attacks, such as the one that incinerated eight shepherd boys aged 6 to 18 in northern Afghanistan last month. Across the border in Pakistan, CIA "targeted" drone attacks have killed 2,300, including hundreds of civilians and 175 children – a massacre of another kind — with the collusion of Britain's GCHQ electronic spying centre.

Of course, the Afghanistan occupation is far from unique in its record of civilian suffering. The Iraq war was punctuated by occupation massacres from the start: Haditha, where 24 men, women and children were murdered in cold blood by US marines in 2005, the killing of 17 by Blackwater military contractors in 2007, and another dozen by a US Apache crew in Baghdad the same year are among the more notorious. The only soldier convicted in the Haditha case walked free last month with a "general discharge under honourable conditions".

And in Vietnam, hundreds of villagers were notoriously murdered by US soldiers in My Lai in 1968, among other bloodbaths. The same was true of Britain's colonial war against Malaya's communist guerrillas, where 24 villagers were slaughtered by British soldiers in Batang Kali in 1948 – their relatives are still seeking some justice 64 years later.

Massacres are common in wars, but they flow from the very nature of foreign occupations. Brutalised soldiers, pumped up with racial and cultural superiority, sent on imperial missions to subdue people they don't understand, take revenge for resistance, real or imagined, with terror and savagery.

That has been the story of the Afghan campaign: a decade-long intervention supposedly launched to crush terrorism that has itself spawned and fuelled terror across the region and beyond. This is a war that has failed in every one of its ever-shifting kaleidoscope of aims: from destroying the Taliban and al-Qaida, to bringing democracy and women's rights, to eradicating opium production.

The warnings of its opponents from the start have been gruesomely borne out. The Taliban control swaths of the country, Afghanistan is the opium capital of the world, women's rights are heading backwards, and the robber-baron Karzai government is reviled by its people.

Where is the "good war" now? Foreign troops are a central cause of the conflict, not its solution – as is well understood in both the Nato countries and Afghanistan itself. In Britain, 55% want troops withdrawn immediately; in the US 60% believe the war hasn't been worth fighting; in Afghanistan 87% of men in the south say Nato operations are bad for Afghans, 76% in the north.

Yet Cameron insists this "very good work" must go on. Despite the growing pressure to bring an end to a disastrous occupation, US demands on the Afghan government for a long-term "enduring presence" to save Nato's face are intensifying. But it's not going to be saved. There is no serious prospect of a change in the balance of forces before the end of 2014, when Nato forces are scheduled to end combat operations. With the US and Nato now committed to negotiation with the Taliban, the case for speeding up withdrawal has become overwhelming.

The best chance of preventing a return to civil war is an inclusive, negotiated settlement backed by the main neighbouring states. Spinning out the occupation to 2014 or beyond will only mean years more of massacres, dead soldiers and civilians and destabilisation of the region.

Like Iraq, the Afghanistan war has been a disastrous miscalculation for the western powers, which are having to learn the lessons of empire again and again. In the 21st century, more than ever, foreign military occupation will be resisted, paid for in blood – and rebound on those who try to impose it.

Twitter: @SeumasMilne