"I am sitting you next to Secretary Clinton at dinner. Say exactly what you think. If you don't, I never - ever - want to hear you criticise the policy again." So said Richard Holbrooke, the US Special Representative to the Af-Pak region, barely a week after assuming his new position under the Obama administration. He was talking to Rory Stewart, and Stewart told the anecdote on the Huffington Post after Holbrooke's sudden death in 2010.

Holbrooke, Stewart remembered, praised his acumen regarding Afghanistan, and listened to him, even though Stewart disapproved of the emerging policy of General Petraeus. To Holbrooke, Stewart was the expert who dared disagree, but whose disagreement still needed to be heard in the halls of power.

Stewart is widely considered an expert on Afghanistan. Currently a Member of Parliament in Britain, he sits on the influential Foreign Affairs Select Committee. Previously, he was the Ryan Family Professor of the Practice of Human Rights at the Harvard Kennedy School. Before that, in 2003-04, he worked in southern Iraq with the American administration. And prior to that, in 2002, he walked 6,000 miles - partly across Afghanistan. This last bit, his walk in Afghanistan, became the fulcrum of his 2004 book, The Places in Between, which was a bestseller in the UK and the USA. The website for his book declared that he survived his walk because of "his knowledge of Persian dialects and Muslim customs" and a grounded knowledge of the entire region.

Now, as an elected politician, Stewart has moved out of that nebulous region of policy experts into policy-makers, but his credentials in the field continue to dominate his public persona. A sure sign of his biographical and political heft was Ian Parker's recent profile in The New Yorker which frankly assessed his chances of becoming prime minister.

Parker notes that Stewart "speaks some Dari and no Pashto" and had only limited exposure to the country, having lived, on and off, in Kabul. Yet the very fact of his "walking" had transformed a recent college graduate with fantasies of becoming the next TE Lawrence (who tried to engineer the birth of a new Arabia during the First World War) or Wilfred Thesiger (who walked over the Empty Quarter of Arabia and became known for his sparse travelogues of the Middle East) into the "real thing". Unlike his heroes, however, Stewart's main competence was not in navigating the desert but knowing DC. He isn't the only person who has managed to merge a personal narrative implying site-specific knowledge, avowedly ethnographic in nature, with a deep engagement with the political and analytical clusters of the American and British military.

In July 2010, The New York Times reported on the popularity of Greg Mortenson's 2006 memoir Three Cups of Tea: One Man's Mission to Promote Peace … One Man's Mission to Fight Terrorism and Build Nations … One School at a Time among the US Military high-command. The report described General McChrystal and Admiral McMullen using the text as a guide to their civilian strategy in Pakistan. Mortenson's book quickly became required reading in military academies (the report hinted at the role played by the wives of senior military brass in promoting the title) and Mortenson has since spoken to the US Congress and testified in front of committees. Mortenson himself, though a selfless worker for the most disenfranchised of Pakistan's northwestern citizens, possesses no deep knowledge of the region's past or present and is avowedly "non-political" in his local role. Still, his personal story, his experiences and the work of his charity are now widely considered to be a blueprint for US strategy in the Af-Pak region.

Both Stewart and Mortenson illustrate one particular configuration of the relationship between knowledge and the American empire - the "non-expert" insider who can traverse that unknown terrain and, hence, become an "expert".

Even a cursory examination of the archive dealing with the American efforts in Iraq, Afghanistan and Pakistan demonstrates that there has been no related growth in specific scholarly knowledge about those sites of conflict. The knowledge of Arabic, Urdu or Pashto remains at extremely low levels in official corridors. There is, one can surmise simply from reading the back and forth sway of military and political policy in Afghanistan, very little advancement in understanding of either the text or context of that nation.

In America's imperial theatre, Stewart and Mortenson exemplify a singular notion of "expert". We can build, based on the profiles of other specimens - Robert D Kaplan, Fareed Zakaria, Robert Kagan - a picture of what the ideal type looks like from the official point of view. Such an "expert" is usually one who has not studied the region, and especially not in any academic capacity. As a result, they do not possess any significant knowledge of its languages, histories or cultures. They are often vetted by the market, having produced a bestselling book or secured a job as a journalist with a major newspaper. They are not necessarily tied to the "official" narratives or understandings, and can even be portrayed as being "a critic" of the official policy. In other words, this profile fits one who doesn't know enough.

At the same time there are greater claims, and greater efforts, towards satellite cameras and listening devices; drones which can hover for days; databases which can track all good Taliban and all bad Taliban. Yet who can decipher this data? When one considers the rise of "experts" such as Stewart or Mortenson against the growth of digitised data which remains elusive and overwhelming, one is left with a rather stark observation - that the American war effort prefers its human knowledge circumspect or circumscribed and its technical knowledge crudely totalised.

***

It wasn't always this way. In 1879, when the US Congress created the United States Geological Survey to chart and measure the American West, it simultaneously established the Bureau of American Ethnology. Funded directly by the Congress, this body was chartered to record the languages, habitations, folk-tales and oral histories of Native American tribes. It ought to be noted that these efforts to "know" occurred in direct relationship with the opening up of the American West to Eastern capital, labour and settlement.

The decade following the Second World War saw the creation of Area Studies departments across universities in America. These had an explicit charter to study those countries and regions which had remained "hidden" from American purview, but which were now considered the frontline in the emerging Cold War: China, India, Japan, the Middle East. Whether they were funded by the Department of State or Defense, or via external, "independent" sources such as the Social Sciences Research Council, the Ford Foundation or the Carnegie Mellon Foundation, conferences and publications were established to serve the interests of the American state. These developments saw the rise of the "Kremlinologist" and the "East Asianist", both within and outside the Academy.

If there was ever a situation in which linguistic, cultural and historical expertise were privileged in American foreign policy, then the period between the Vietnam War and the end of the Cold War fundamentally altered it. The academy grew critical of American foreign policy and tried to distance itself. At the same time, the activities of US academics became the subject of official scrutiny. Various scholars were investigated by the House Un-American Activities Committee for secret communist sympathies or for having "gone native". The realignment of expertise under Kissinger and, later, the Clinton Administration, eliminated those career foreign services officers who had lifelong attachments to the regions they covered. The rise of postcolonial and post-structuralist critiques of the relationship between power and knowledge further complicated the terrain.

This widening gulf between the corridors of power and the halls of academia came with the unintended consequence that, barring a few notable exceptions, any knowledgeable critique of American foreign policy gradually vanished. It is this vacuum that is filled by Stewart and Mortenson, who combine accessibility with a whiff of "on-the-ground" expertise. A very similar role is played, in a popular culture hungry for "authentic" voices from the conflicted sites, by the fictions of Khaled Hosseini or Daniyal Mueenuddin.

***

Yet these pundits are only part of the story. The more troubling aspect is the change from human expertise to technical knowledge. The Washington Post noted recently that the US Air Force is rolling out a satellite-based observation technology called "Gorgon Stare". A triumphalist quote described the programme thus: "Gorgon Stare will be looking at a whole city, so there will be no way for the adversary to know what we're looking at, and we can see everything."

This "everything" dominates most tech-based strategies which are regularly puffed in the media. Some mention databases of tribal affiliations and sympathies down to each inhabitant of a given street, neighbourhood, city and district. This database is then placed at the fingertips of US military personnel via their hand-held electronic devices, letting them bring up the dossier on each Afghan they encounter.

This peculiar urge to know and then unknow remains a central conundrum for all empires. A very similar teleology is visible in the history of the British Empire in India. The earlier colonisations were accompanied by a bevy of East India Company employees who assiduously studied languages, learnt the local customs and became - to use the term popularised by writer William Dalrymple - "White Mughals". That is, they "went native". But just as the colonial efforts to map and know India picked up steam - in the 1830s and 1840s - the company administration began to raise concerns that British officers were losing their loyalty to their own country.

Lord Ellenborough, who led the British invasion of Afghanistan, was famously sceptical of British officers such as Richard F Burton or James Outram, who were regarded with suspicion for being too good with languages, travelling in disguise among the natives. Before he became a renowned traveller and Orientalist, Burton served in Karachi and wrote the following regarding the conduct of his fellow officers: "The white man lives a life so distinct from the black, that hundreds of the former serve through what they call their 'terms of exile,' without once being present at a circumcision feast, a wedding, or a funeral".

The "mutiny" of 1857 fully cleaved this ruling elite from the ruled masses - as the British coloniser retreated from civic space, creating segregated communities, thoroughfares, and establishments. Linked to this withdrawal, however, was the most extensive and descriptive effort to count, catalogue and tabulate the vast populations of India. The 1870s and 1880s were, as in the case of the American West, decades of prodigious ethnographic output where geographies of caste, lineage, tribe, language, settlements were carefully and explicitly mapped through survey teams headed by colonial administrators and staffed by legions of local knowledge brokers. By the turn of the century, however, British high imperialism once again changed the character of knowledge gathering and the relationship of colonial power to the Indian landscape. The description gave way to the table.

One particularly pertinent example of this process is the geographical surveys and census of the North West Frontier Provinces. First conducted in 1904, and again, in 1910, they produced reams of maps, alongside came the Gazettes, which gathered lore, history, ethnographies. By 1930, this had progressed to the creation of databases - or Registers, as they were then called - on individual people.

In a register produced in December 1930, to give one quick example, titled - "List of leading Mullas on the border of the North West Frontier Province" - the following categories of information are listed: "Name", "Parentage", "Year of birth", "Caste or sect", "Residence", "Whether influential. If so with which tribes or sections", "Attitude towards Government as far as known", and "Remarks". Remarkably close in conception and execution to the databases maintained by the United States in Afghanistan, this Register shows the progression from the ethnographic narrative to the data table, as the instrumentalisation of political and colonial power began to converge explicitly into a brute-force stratagem.

Whether by the use of anthropologists and social scientists in the Human Terrain System or the reliance on the ethnographic "expert", the American empire has often held the British example as a template (most likely at the behest of other scholar-combatants like Niall Ferguson or Bernard Lewis). Implicit in their critique, as in that of Rory Stewart, is the express desire that America must do a better job at being an empire.

Even superficially this is, of course, a categorically illogical thing to assert. There is no better way to do empire. The condition of asserting political and military will over a distant population is one that cannot sustain itself in any modern, liberal society. The efforts to understand, will inevitably lead to the understanding that the people of Afghanistan or Pakistan or Iraq desire the power to make their own decisions - without the imposition of governments or militaries sanctioned and placed from afar.

The knowledge of languages and expertise will inevitably expose the lie that there is widespread support for unilateral military escalations. The hope of a civilisational mission (which sustained the likes of Lawrence or Burton in their critiques of the failure of the British to do empire better) does still glimmer in some eyes - those of Fouad Ajami or Thomas Friedman, or George W Bush.

This hope, being irrational and racist, actually requires blindness to the immediate and the real. Notice simply the befuddled faces of area experts when confronted by Tahrir Square. Notice simply that it isn't the masses in the street that confound but the lack of explicit violence from the masses and the lack of religiosity of the masses.

The appeal of the drone's eye is precisely that it does not see everything, because it carries no understanding of the things it records. The experts who are required to imagine Afghanistan or Pakistan traverse those spaces in a manner similar to the drones, on their own preprogrammed missions where every little thing becomes a target on which to pin their policies.

Manan Ahmed is a historian of Pakistan at Freie Universitat Berlin. He blogs at Chapati Mystery.