It has become a pub bore's cliché to argue that we will never prevail in Afghanistan because no foreign power ever has: not even the Russians, whose nine-year occupation cost the lives of 14,000 of their soldiers and 35,500 wounded, and which ended in humiliating retreat in 1989. Those Cassandras irritate Western leaders, whose response is to insist that it is different this time. "We are not an occupying army," Gordon Brown told the BBC on Friday. "It's not like previous interventions.... We are actually creating the conditions by which the Afghans themselves, and not an occupying army, can run their own affairs."

Ture, history does not repeat itself in every detail. Nato, for instance, would never adopt the scorched earth tactics of the Soviet Union, which led to the deaths of more than a million civilians and the creation of 7 million refugees. In 1987 the Soviets carpet-bombed and bulldozed Kandahar, Afghanistan's second city, reducing its pre-war population from 200,000 to 25,000. The underlying motive of the Western mission, furthermore – the destruction of al-Qa'ida – has the justification of self-defence and is legitimised by the UN.

Nevertheless, the spectre of Soviet-style failure hangs over the coalition as never before. President Obama, as he dithers over a troop surge, is in the same shoes as Mikhail Gorbachev when he came to power in March 1985. He, too, inherited a counter-insurgency that had stagnated – and opted for a surge to force the result. Troop levels rose to 108,800: almost precisely the number that the US will have in the field if Obama agrees to General McChrystal's request for another 40,000. It did neither Gorbachev nor the Afghans any good: 1985 turned out to be the bloodiest year of the whole war. "The more troops you bring the more trouble you will have here," Russia's ambassador to Afghanistan, Zamir Kabulov said recently. "If this trend is the rule, if you bring here 200,000 soldiers, all of Afghanistan will be under the Taliban." Kabulov, who served in Kabul as a junior diplomat in the 1980s, understands better than most that nothing unites the fractious Afghans more effectively than the presence of infidel soldiers on their holy soil.

Download the new Independent Premium app Sharing the full story, not just the headlines

The experience of resistance has so shaped Afghan identity over the centuries that it has been blended into their religion. "We are against war," a Taliban commander called Abdullah told me in 2007. "It creates nothing but widows and destruction. But jihad is different. It is our moral obligation to resist you foreigners. We will never stop fighting. At judgement day, Allah will not ask, 'What did you do for your country?' He will ask, 'Did you fight for your religion?'"

I interviewed Abdullah in the mountainous Chak district of Wardak province, a scant 30 miles south-west of Kabul, and the scene of fierce fighting with US forces this year. The Talibs still control it. As elsewhere, they are helped enormously by the local topography. Afghanistan is not Mesopotamia: the steep-sided valley from which Abdullah continues to operate is a cul-de-sac with only one road in and out. As the many decaying wrecks of tanks prove, the Russians also failed to clear and hold Chak. Here, as in Helmand, the insurgents' tactics and ambush points are not just similar but in many cases identical to the ones used against the Soviets, and even the 19th-century British.

Advances in western military technology do not faze the Taliban. The Soviets' best weapon was the fearsome Hind helicopter gunship, which the mujahedin nicknamed Shaitan-Arba, or "Satan's Chariot." They were practically invincible until the US supplied the Afghans with shoulder-held Stinger missiles in 1987. The Soviets went on to lose over 300 helicopters; 2009 saw the first loss of a coalition Chinook to enemy fire.

The Soviets, of course, did not see themselves as an occupying army any more than we do. They arrived in 1979 at the specific and repeated request of the Kabul government. They too had a policy, and based their exit strategy on it, as we have done. By 1986, under Soviet guidance, the local armed forces had been built up to an official strength of over 300,000. But there were serious doubts about the reliability of these allies, just as we have found (and as the recent murder of five British soldiers by an Afghan policeman under their charge amply illustrates). In the 1980s, new recruits deserted the Afghan Army almost as fast as the Russians could train them: 32,000 of them per year, at one stage.

The difference between Nato and Soviet interventions is not always apparent to ordinary Afghans. For many years the shopkeepers along Chicken Street, the main tourist bazaar in Kabul, sold carpets patterned with Hind gunships and BTRs, the distinctive eight-wheeled armoured vehicles whose carcasses still litter the countryside, testimony to the mujahedin's Pyrrhic victory. These days such souvenirs are decorated with Chinooks and armoured personnel carriers that look dispiritingly like British Vikings. Another decade, another invasion: it's all the same to the carpet-sellers.

It's all the same to the Taliban, too. The oral tradition is strong in Afghanistan, and one of its effects is a strange telescoping of time; for them, it is almost as if the Soviet invasion was yesterday, and the British were here last week. "My grandfather died fighting you people," Abdullah told me. "Fighting the British feels like unfinished business for many of us."

It sometimes feels like unfinished business for us as well. In 1929, the RAF's 39 Squadron was stationed at Risalpur in the border areas of Pakistan, from where it flew bombing missions over Waziristan against the elusive insurgent leader, the Fakir of Ipi. Today, 39 Squadron operates a dozen MQ-7 Reaper drones over the same area – albeit from computer screens on an airbase in Nevada – in a bid to kill or capture Osama bin Laden. Some analysts think that drones could turn the tide of this war, and even be part of the West's exit strategy by eventually allowing machines instead of men to police the border. But as the Russians and the British before them found to their cost, it is a mistake to put too much faith in weapons technology when fighting in a country like Afghanistan.

James Fergusson is the author of A Million Bullets – The Real Story of the British Army in Afghanistan