American experts in the field make two main points about intelligence gathering in Afghanistan. One is that in a complex and wholly alien cultural, ethnic and linguistic environment, such work is extremely difficult. Their other point is that the US and its allies must do much better if they are to have any chance of "winning" the Afghan campaign.

Intelligence weaknesses identified in this week's report for the Centre for a New American Security by Major General Michael Flynn and others will come as no surprise to operatives on the ground. Flynn's criticism that too much attention has focused on insurgents and not enough on understanding, protecting and persuading local people and leaders broadly echoes sentiments expressed by the new Afghan commander, General Stanley McChrystal.

Recent failures have lent urgency to fixing the problem. The year just ended was the Taliban's most successful since the 2001 invasion. The insurgents inflicted record casualties on allied forces, extended their influence into relatively settled provinces such as Kunduz, and effectively forced Barack Obama to sack his top commander and order a strategic review.

The political fiasco that followed fraud-tainted 2009 presidential elections, exacerbated by low voter turnout, was apparently not foreseen, and certainly not forestalled, by American intelligence – a lapse that recalled the US failure to foresee Hamas's election victory in Palestine in 2006. At the sharp end of the cloak and dagger scale, the CIA suffered a vicious blow when an alleged triple agent penetrated a forward operating base in Khost and killed seven operatives.

Not all Washington's problems in Afghanistan can be laid at the door of the intelligence community, any more than airline security breaches are exclusively its fault. Spooks also have a PR problem, in that the nature of their work prevents them from defending themselves in public. But Colonel John Agoglia, director of the counter-insurgency training centre at Camp Dubs outside Kabul, gave an insight into latest thinking on how to make the allied intelligence effort more fit for purpose – or, in a word, smarter.

"In Afghanistan we've been focused on counter-terrorism, not governance. That's stupid shit. We've been trying to implement a comprehensive integrated approach when we in the alliance don't know what that is. We're learning. The key is to to get the best match of capabilities to address three key issues: security, governance, development. You have to look at it in a holistic way. You have to change the mindset [concerning Afghanistan and the Afghan people]. It's about understanding, leading to respect, leading to trust. You need awareness, you need to take time to go out and learn."

In his recommendations, Flynn urged moves to "empower select teams of analysts [ie intelligence agents] to move between field elements, much like journalists, to visit collectors of information at the grassroots level [ie local sources] and carry that information back to the regional command." He also proposed regular district-level assessments on geographic rather than functional lines, unintentionally harking back to the system of British colonial administration.

Agoglia is already ahead of him as he describes new approaches taught at the counter-insurgency centre that emphasise the local, thereby tapping into accurate, real-time intelligence about the insurgency. In an interview last month he said:

"We're acting like police in many areas. We use the old British police mantra – 'look, listen, touch'. You've got to study the environment, understand the culture and the ethics, then listen to the population to gain information, in that order. And you've got to keep reviewing what you do. For example, ask yourself: have we alienated or attracted people this week? What have we done about corruption?"

An example of how not to proceed was provided by a Nato unit that decided to dig a well between two villages to help overcome seasonal water shortages, Agoglia said. Ownership of the new well became a matter of dispute between the two communities, leading to bloodshed. After a while, the well was abandoned and no one used it. Thanks to a local intelligence failure, well-meaning outsiders had made matters worse.

Building trust and mutual confidence between western troops and agencies and local people and leaders was the route to lasting success in Afghanistan, he suggested – meaning a stable, developing country that could educate and care for its people and did not pose a threat to others.

At the same time, you must know your enemy, the Brooklyn-born Agoglia said. "We try to isolate the hardline guys, win over the majority who are moderates. The insurgents we deal with two ways. We turn them or we kill them."