The west should have tried talking to the Taliban a decade ago, after they had just been toppled from power, the top British commander in Afghanistan has told the Guardian, barely a week after the latest attempt to bring the insurgent group to the negotiating table stuttered to a halt.

General Nick Carter, deputy commander of the Nato-led coalition, said Afghan forces would need western military and financial support for several years after western combat troops head home in 2014. And he said the Kabul government may have to accept that for some years it would have only shaky control over some remoter parts of the country.

Speaking exclusively to the Guardian, he said: "Back in 2002, the Taliban were on the run. I think that at that stage, if we had been very prescient, we might have spotted that a final political solution to what started in 2001, from our perspective, would have involved getting all Afghans to sit at the table and talk about their future,"

Acknowledging that it was "easy to be wise with the benefit of hindsight", Carter added: "The problems that we have been encountering over the period since then are essentially political problems, and political problems are only ever solved by people talking to each other."

But he believed the police and army had been shaped into sustainable institutions that were strong enough to protect a critical presidential election next year, and guarantee stability for the majority of the country after the western withdrawal.

The US and Afghan governments are pushing hard for negotiations to end a conflict that has dragged on for more than 12 years. But critics have long argued that the west could have struck a deal with moderate Taliban leaders after ousting the group from power in 2001, perhaps saving thousands of lives and billions of dollars.

One academic who studies the Taliban said the group tried to reach out to their own and the US governments until 2004, and would have made major compromises. "There would not have been too much negotiating to be done, even, in 2001 or 2002, because the Taliban's senior leadership made their approaches in a conciliatory manner, acknowledging the new order in the country," said Alex Strick von Linschoten, author of An Enemy We Created.

Today the insurgent group dominates swaths of the country, and seems ambivalent at best about negotiating ahead of the departure of foreign troops. Underlining how challenging efforts to broker peace talks are, the latest efforts collapsed in diplomatic farce last week.

The Taliban opened an office in Qatar which was meant to be a formal base for meetings and was welcomed by Washington, but the Afghan president, Hamid Karzai, shut down the process after Taliban spokesmen presented the villa as a de-facto embassy for a government-in-exile.

Carter said he was confident that Nato's handover of security to Afghan forces, finalised last week, would eventually bring the Taliban to the negotiating table.

"What the opponents of the Afghan government now realise is they are likely to be up against capable Afghan security forces who are going to be here in perpetuity and therefore that old adage that 'We have the clocks but the Taliban have the time', has now been reversed," he said.

"They are now up against security forces who have the time, and they are also Afghan forces … for those reasons I think that there is every chance people will realise that talking is the answer to this problem," added Carter, who previously served as the top Nato officer in southern Afghanistan, the Taliban's birthplace.

With a potential political solution far off, 2013 has been a year of heavy violence. Civilian casualties are rising and a string of high-profile attacks have hit Kabul including an assault on the airport and an audacious raid into the heart of the heavily fortified diplomatic and military zone.

These assaults were unlikely to let up before western troops left, as the Taliban tried to position for possible talks, and wage a campaign to claim credit for a the western exit, Carter said, although troops were actually leaving on a timetable set by the Afghan government and Nato in 2010.

"First of all, people like to negotiate from a position of strength, and secondly I think the opponents of Afghanistan would like to appear to compel the international community's withdrawal," Carter said. "I don't think it's surprising that we are seeing spectacular attacks in Kabul and a continuance of attacks elsewhere."

The strength of the insurgency meant Kabul would not control all of the country for some years to come, said Carter, who previously described Afghanistan's likely post-2014 situation as "stable instability".

"There will be parts of Afghanistan which will not necessarily be as closely linked to central government as others … there will therefore be some local political solutions which won't in any way threaten central government," Carter said. "That phenomenon may go on for a while."

Nato only began the buildup of Afghanistan's police and army in earnest in 2009, and the rapid pace of expansion towards a target of 350,000 meant the police and army would need help for years to come, particularly in highly skilled areas such as bomb disposal, medical evacuation and logistics.

"The security forces will need continuing development because they have been built very quickly," Carter said. One major roadblock for now was the lack of planes and helicopters, critical to dominating a large, mountainous country with poor and often deadly roads. "The plan to field the airforce … is designed to give them more capability every year, but probably not to be fully fielded until 2017-18."

But he said their institutions were now sustainable and overall he was optimistic about Afghanistan's future, as long as the US and its allies came through on promises of financial and military support.

"When you see how this country has developed in the last 10 to 12 years, it's a completely different place, and people have completely different expectations," he said. "We owe it to all of those who have invested a great deal of effort in what we have been doing here since 2001 to see it through."