In the midst of Barack Obama’s much-vaunted military surge against the Taliban in Afghanistan in 2010, Hayam Mohammed, an elder from Panjwai near the Pakistani border confronted an officer from the US 101st Airborne who had come into his village.

“You walk here during the day,” the elder told the soldier bitterly as the Observer listened. “But at night [the Taliban] come bringing night letters” – threats targeting those collaborating with foreign forces.

Afghanistan papers reveal US public were misled about unwinnable war Read more

That surge, which like so many other initiatives in Afghanistan’s long war was celebrated as a huge success, today serves only as a grim reminder of the deception and failure revealed in the explosive Afghanistan papers published by the Washington Post this week.

Comprising more than 600 interviews with key insiders collected confidentially by the Office of Special Inspector General for Reconstruction in Afghanistan [Sigar], and published after a three-year court battle, the trove has been compared in significance to the Pentagon Papers, the secret Department of Defense history of the Vietnam war leaked in 1971.

Like that secret history, the Afghanistan Papers’ accumulated oral history depicts a war mired in failure – in sharp contrast to the “misleading” story told to the US and British publics by officials in massaged figures and over-optimistic assessments.

But even if that deception has been the main focus of reporting, the hundreds of interviews – with senior generals and Afghan governors, with ambassadors, aid officials and policy advisers – also tell another story: how successive presidents from Bush through Obama to Donald Trump, publicly rejected “nation-building” but created a violent, corrupt and dysfunctional state only barely propped up by US arms.

They detail too how – like the Soviet Union before them – the US and its allies came so badly unstuck in Afghanistan through a combination of hubris and ignorance, and with a political leadership – both under Obama and Bush – more concerned with domestic politics than the impact of their decisions on Afghanistan.

Reading at times like an extended exercise in blame, the interviews trace a widespread desire among participants to nail down an original sin for the failures in Afghanistan; even to seek an exculpation for a conflict that has cost $1tn and the lives of tens of thousands over almost 20 years.

Facebook Twitter Pinterest Hayam Mohammed: ‘You walk here during the day.’ Photograph: Antonio Olmos/The Observer

From the shortcomings of the US military’s newly rewritten counter-insurgency doctrine – described by one interviewee as “colonial” in its conceit – to a tolerance of widespread corruption and “warlordism” blamed for fueling the resurgence of the Taliban, a key theme of many of those who spoke to Sigar was the lack of coherence in Washington’s approach to Afghanistan from the outset.

From the very beginning, as Richard Haas – the former US government policy coordinator for Afghanistan – confided in his interview, there was little enthusiasm in George W Bush’s White House for the country once the Taliban was driven out in 2001.

“I remember that we were around the table and there was the president [George W Bush]. The feeling was that you could put a lot into it and you wouldn’t get a lot out of it. I would not call it cynical, I would call it pessimistic about what the relationship [was] between investment and return in Afghanistan.”

Other interviewee subjects, however, place the initial failure even further back: with the original decision to elide the Taliban with Osama bin Laden’s al-Qaida in the wake of the September 11 attacks.

Among people like Jeffrey Eggers, a former US Navy Seal and staffer on the National Security Council in the George Bush and Obama administrations, this the key point.

“Why did we make the Taliban the enemy when we were attacked by al-Qaida? Why did we want to defeat the Taliban? Why did we think it was necessary to build a hyper-function[al] state to forgo the return of the Taliban?”

For Richard Boucher, assistant secretary of state for south and central Asian affairs between 2006 and 2009, the contradictions in the aims framed a mission that appeared founded on impossibility.

“If we think our exit strategy is to either beat the Taliban – which can’t be done given the local, regional, and cross border circumstances – or to establish an Afghan government that is capable of delivering good government to its citizens using American tools and methods, then we do not have an exit strategy because both of those are impossible.”

His summation, however, was even more scathing: “We did not know what we were doing.”

The lack of vision at the war’s start, which left the US and its allies racing to catch up as US attention turned quickly to the 2003 invasion of Iraq, was only exacerbated by the decisions that would belatedly get made in response to the rapidly resurgent Taliban in the middle of the last decade.

Facebook Twitter Pinterest American soldiers of the 101st Airborne blow up a compound suspected of containing IEDs in a village in Panjwai district in 2010. Photograph: Antonio Olmos/The Observer

Among the handful of Britons identified by name in the papers is Gen David Richards, the commander of Nato forces in Afghanistan between 2006 and 2007. “There was no coherent long-term strategy,” Richards told the Sigar interviewers, adding that the countries involved in the International Stabilisation Force [Isaf], under Nato command, had insufficient troops to do the job.

“Every nation was in charge of protecting an area larger than they had forces to do so with, and if you don’t have enough forces to protect what you’re trying to do, stabilization can’t be done.”

“No individual is to blame,” he added bleakly: “The system let everyone down.”

US lies and deception spelled out in Afghanistan papers' shocking detail Read more

But if there is one issue with the Afghanistan papers, however, it is that they represent the views of a selection of insiders who were at one time or another invested in the success of the war, often unable to step outside the sometimes arcane disagreements they were steeped in.

Austin Wright, an academic who specialises in Afghanistan at the University of Chicago, and has made a study of the declassified military records of the war, offers an outsider’s perspective from his reading of the interviews.

Wright notes that many in the media have picked up on the acknowledgment by the head of Sigar, John J Sopko, asserting the US public was “misled” consistently by official accounting of progress of the war. But he believes that focus misses the fact that many in leadership positions also misinterpreted their own data concerning the nature, intentions and capabilities of the Taliban at key points in the conflict in their desire to see signs of victory.

Wright points to US military and Nato surveys that indicated that at key points, including after their initial defeat, the Taliban simply withdrew either to regroup or instill a false sense of progress, echoing the observation made by village elder Hayam Mohammed at the height of the 2010 surge.

“The Taliban are strategic and patient,” says Wright. “They are also skillful at manipulating the information being seen in the field to maximise any political process.

“What you see – during the Obama era handover of leadership to Afghan forces for example – is a significant reduction in violence across large areas where there are foreign forces. But what the survey data also showed was a continuing Taliban presence, not necessarily fighting but not going away.

“Then, when you see the actual physical withdrawal of US troops, you see an increase in violence again. What that tells you is that they have been sitting it out to until the cost of re-intervention becomes too high.”

The sense of overdue re-evaluation is not confined to just those interviewed by Sigar but shared by many who served in senior roles in Afghanistan.

“I remember thinking at the time I was running around Helmand that we should be able to do this, whatever that thing was,” a former British diplomat who served in the region told the Observer.

Facebook Twitter Pinterest Presidents George W Bush and Hamid Karzai at a press conference in Kabul on 1 March 2006. Photograph: Rafiq Maqbool/AP

“But in retrospect, it was not realistic because while we were pissing into sand in Helmand you have to look at bigger picture.

“Looking back, [then-Afghan president Hamid] Karzai had already given up on the west when Bush went into Iraq.

“And by the time people realised that the Taliban were coming back it was too late.”

A very reasonable question was what would have happened if the international community had delivered the investment and committed the resources in 2003-04 that were delivered 2008-10.

The diplomat responds to a point made by Gen Dan McNeill, the former US and Nato commander in Afghanistan, Dan McNeill, who in his interview with Sigar complained that even before his first deployment in 2002 he could not find anyone able to give him a definition of what “winning meant”.

“If you can’t define ‘winnable’ how do you define if the war is lost. What is clear,” he added looking to the future, “is that there is going to be an ongoing low level of commitment for a long time to come in terms training and in terms of trying to prevent Afghanistan becoming a serious problem again.

“You can argue that you are preventing Afghanistan being used as an operational base for the really bad guys, but then there are plenty of other options for them. In terms of what we were trying to achieve does that mean loss?” He thinks for a moment.

“It probably does. Certainly we took a bloody nose.”