We steal secrets.” That’s how a recent CIA director summed up the agency’s mission. The CIA, like MI6 and similar bodies in other countries, exists to gather intelligence, meaning information of military or political value – and to do so clandestinely, because intelligence is more valuable if the other side thinks that it is still secret.

The purpose of intelligence gathering may be aggressive, to gain a military advantage, for example by attacking the enemy before his reinforcements arrive; or it may be defensive, as in protecting the population from terrorist attacks. And by enabling a better understanding of the way the other side thinks, intelligence may help to prevent conflict. Such, at any rate, is the theory. Yet history is littered with intelligence failures – the belief that Saddam Hussein possessed weapons of mass destruction (WMD) being a recent example. Even when intelligence is accurate, it can be discounted, misinterpreted, ignored, rejected or overlooked. The history of intelligence is also, at least in part, the history of stupidity.

In trying to assess whether or not Iraq did possess such weapons, Western intelligence services relied on unverifiable testimony from an informant who was only too aware of what he was expected to say. This was used to make the case for war. As is well known, the preparations for what would happen in Iraq after the war were woefully inadequate: so little thought had been given to Saddam’s capture, that he was initially interrogated by a CIA operative who could not speak Arabic, through an interpreter. Two months passed before the FBI came up with an Arabic-speaking interrogator. It eventually became obvious that in pretending to possess WMD, Saddam had been concerned to deter an attack from Iran, not the West.

Christopher Andrew, emeritus professor of modern and contemporary history at Cambridge, is the doyen of intelligence historians. He has long argued that intelligence is the “missing dimension” from history: that one cannot fully understand the past without an awareness of what went on in “the secret world”. His magisterial book is an attempt to supply that missing dimension. His stated aim is “to recover some of the lost history of global intelligence over three millennia, to show how it modifies current historiography, and to demonstrate its continuing relevance to intelligence in the 21st century”.

Taking the long view enables Andrew to see similarities between events in different epochs: to compare, for example, the medieval persecution of heretics with Stalin’s show trials; or the failure to anticipate the Japanese attack on Pearl Harbor in 1941 with the failure to anticipate the raid by the Dutch on the English fleet in the Medway in 1667 – “the most humiliating episode in the history of the Royal Navy”.

Despite extraordinary developments in technology, the principles of intelligence gathering remain largely the same. To take just one example, the technique of “enhanced interrogation” by waterboarding, which caused outrage when it was revealed in 2007 that it had been used by the CIA on al-Qaeda suspects, was discovered by officials of the Spanish Inquisition, and first used by American forces during the Spanish-American war of 1898.

The history of intelligence is one of the lessons learned and then forgotten. Elizabethan England was menaced by subversion from within and threats from without – above all, from Catholic Spain, the superpower of the 16th century. In response to such danger, and under the skilful direction of Francis Walsingham, espionage, counter-espionage, code-breaking and counter-subversion became better integrated into policymaking than had ever been achieved before. Yet this highly effective system of intelligence was dependent on Walsingham’s supervision, and quickly collapsed after his death in 1590.

During the Spanish wars, Walsingham saw the Queen or her chief minister Burghley almost every day. Though the information he brought was often unwelcome, Elizabeth recognised its value. In times of crisis, wise leaders have appreciated having intelligence professionals being free to “speak truth to power”. A recent “C” (head of MI6) defined his role as “telling the prime minister what the prime minister does not want to know”. Pitt the Younger received his intelligence chief at all hours, sometimes in his bedroom. Churchill went further, demanding to see raw intelligence himself. In contrast, Stalin did not just ignore a series of warnings about the imminent German invasion in 1941; he denounced many of those who provided them.

The man who brought news to Athens of military disaster at the hands of the Spartans in 413 BC was accused of bringing false information and tortured to make him admit that he was lying, until official messengers arrived to confirm his report.

Spying, it has been said, is the second-oldest profession – after prostitution. The history of espionage can be traced back to the Old Testament prophet Moses, who sent spies into Canaan. The Chinese general Sun Tzu (c544-496 BC) was the first writer that we know of to devote serious consideration to the subject. In The Art of War he identifies five types of spies, still recognisable today (“local spies”; “inward spies”; “converted spies”; “doomed spies”; “surviving spies”). “Secret operations are essential in war,” he wrote, and argued that the greatest achievement of a general was “to subdue the enemy without fighting”.

By contrast, the Ancient Greeks showed little interest in intelligence, relying instead on seers, oracles and portents. The Romans were not much better. In AD 9, three Roman legions were ambushed and slaughtered by Germanic tribesmen in the Teutoburg Forest, a defeat that ended Roman ambitions of extending their empire to the River Elbe. This catastrophe dramatically demonstrated the failings of Roman military intelligence. The Roman general Varus had been warned of the enemy’s intentions, but his contempt for “barbarians” blinded him to the danger. This mistake of underestimating one’s opponent is a recurring theme in the history of intelligence.

Soldiers have traditionally been suspicious of spying, considering it a form of cheating. Hitler himself, perhaps conscious of his lowly past as a corporal, declared that he found it “difficult to conceive that a genuine officer can be a sneaking spy”. “I abhor this dirty work,” one British diplomat wrote in 1785, “but when one is engaged to sweep chimneys one must black one’s fingers.”

In this country there has often been resistance to intelligence gathering as “underhand” and “contrary to all English practice”. In the early 1840s there was uproar when it became known that correspondence of the Italian nationalist Giuseppe Mazzini, then living in exile in London, was being intercepted by the authorities. The 1841-46 Conservative administration of Robert Peel was accused of bowing to pressure from the Austrian chancellor Prince Metternich, seen as the evil genius of reaction. Thomas Carlyle fulminated that letters were sacred, and that opening and reading them was worse than picking pockets. The Times condemned the practice, saying “it cannot be English, any more than masks, poisons, sword-sticks, secret signs and associations, and other dark inventions”. The protests led Peel to close down the Decryption Bureau and the Secret Office of the Post Office, not to reopen for the next 70 years, with the result that Britain entered the First World War lacking any official code-breaking capacity.

On taking office as US secretary of state in 1929, Henry L Stimson was shocked to be presented with decrypts of Japanese diplomatic messages. He declared that the practice of intercepting Japanese communications should “cease immediately”, stating that “it was a highly unethical thing for the government to do to be reading the messages sent to our ambassadorial guests from other countries”. He sacked the head of the American Cipher Bureau, Herbert Yardley; angry and resentful, and, like the retired MI5 officer Peter Wright – author of Spycatcher – lacking a pension, Yardley published his own sensational memoirs that revealed, among other things, that his bureau had been reading the Japanese ciphers. The Japanese changed their ciphers as a result.

Intelligence professionals have always had reason to be wary of politicians. In February 1917, publication of the Zimmermann telegram – a German offer to ally with Mexico in an attack on the United States – helped to persuade the American public to support war against Germany, but revealed to the Germans that the Allies were reading their diplomatic codes. Perhaps this was a reasonable trade-off; far worse were indiscretions by grandstanding cabinet members during a House of Commons debate in 1927, which made it obvious to the Russians that the British were reading their coded communications, prompting them to adopt the theoretically unbreakable “one-time pad” encryption system.

The Bolsheviks, having been spied on themselves by the Tsarist police, were keenly aware of the importance of spying. As soon as they took power they set up the Cheka, a ruthless organisation of secret police to combat counter-revolution and sabotage. Like Robespierre before him, Stalin imagined that the revolutionary administration was riddled with foreign spies, and saw counter-revolutionary plots where none existed. The Cheka, and its successor “organs of state security” fed on themselves: all three of Stalin’s security chiefs were executed. Andrew draws the moral: “In authoritarian regimes, particularly at times of crisis, conspiracy theory commonly degrades intelligence assessment.”

Soviet intelligence achieved some remarkable successes, notably in recruiting five young Englishmen who would later supply them with highly sensitive secrets; one of the “Cambridge Five”, Kim Philby, rose to become head of the anti-Soviet section of the Secret Intelligence Service (MI6), and was talked of as a future “C”. Yet the effectiveness of such spies was hampered by Soviet paranoia. During the Second World War, the Russians were deeply suspicious of their British allies: so convinced were they that the British were plotting against them that they came to believe that the “Cambridge Five” were double agents, on the grounds that they failed to supply evidence of non-existent British intelligence operations against the Soviet Union. According to the former head of the Joint Intelligence Committee Sir Percy Cradock, the chief weakness of Soviet intelligence was the “attempt to force an excellent supply of information… into an oversimplified framework of hostility and conspiracy theory”.

The greatest Soviet intelligence triumph in the final months of the war was to obtain plans of the atomic bomb. Stalin was unimpressed when Truman casually mentioned to him at the Potsdam Conference that “we had a new weapon of unusual destructive force” – which was hardly surprising, since Stalin had known about the bomb long before Truman did. There are many ironies in the history of intelligence. In 1944, for example, it was an open secret that the Allies were planning to launch a second front in Europe; the main question was where the invasion would come. The aim of the British Double-Cross (“XX”) operation was to convince the Germans that the Allies planned to attack the Pas-de-Calais, rather than the beaches of Normandy. Analysis of the intelligence received from one of the Germans’ best agents, valet to the British ambassador in Ankara, code-named Cicero, could have revealed the secret; but the Germans doubted the genuine intelligence from Cicero, while crediting the disinformation fed to them by their (largely imaginary) agents of the Double-Cross System.

In September 1940, after more than a year of sustained effort, the Americans succeeded in breaking the Japanese “Purple” ciphers. But inter-service rivalry led to a bureaucratic compromise for processing Japanese decrypts: the army was permitted to receive radio traffic intercepted on even days and the navy on odd days. Andrew departs from his normally dispassionate narrative to call this decision “absurd” and “crazy”. The resulting confusion helps to explain why the Americans were so unprepared for the attack on Pearl Harbor.

The rivalry between American intelligence departments continued into the Cold War. When Philby was informed that the Americans had broken the supposedly unbreakable Soviet codes, he was told that he could discuss this with colleagues in the FBI, but not with the CIA. Of course he discussed it with his KGB case officer instead.

The story of terrorism and counter-terrorism is much older than most people realise. Between 1883 and 1885, during the Fenian bombing campaign, more bombs exploded in London than in any of the IRA’s campaigns a century later. The Fenians even succeeded in blowing up the offices of the counter-terrorist police, the Special Irish Branch of the Met. In France, anarchists waged war against the state; in 1893 a bomb was detonated on the floor of the French parliament. Fortunately no one was seriously hurt, and after the furore had died down the president of the chamber, with remarkable sangfroid, announced that the debate would continue as normal.

Another comparatively unknown episode is that of German sabotage operations in America during the First World War. In July 1916, German saboteurs blew up a barge in New York harbour packed with ammunition awaiting shipment to the Allies. The massive explosion blew out most of the windows in Jersey City; the glass on thousands of windows from skyscrapers in Manhattan and Brooklyn cascaded into the streets; even the torch of the Statue of Liberty was damaged.

The attack on the World Trade Center on 11 September 2001 came as a shock to intelligence experts, although it should not have done; al-Qaeda had already launched several major attacks on American targets, including an earlier one on the World Trade Center itself. This was a failure of imagination as much as anything else: the CIA simply could not believe that “religious fanatics” were capable of carrying out such a sophisticated operation.

Religiously motivated terrorists are more dangerous than politically motivated terrorists because they are more likely to use weapons of mass destruction – and also because they are willing to sacrifice life on earth for paradise in heaven. As Osama Bin Laden put it, “We love death. The US loves life. That is the big difference between us.” Andrew concludes chillingly: “The question now is not whether some future group of (probably Islamist) terrorists will use WMD, but when they will do so.”

In arguing the need for a long-term perspective on intelligence, Andrew quotes Churchill: “The further backwards you look, the further forward you can see.” It seems obvious that there is peril ahead; but studying the past provides some hope that we may anticipate danger in the future.

Adam Sisman’s books include “John le Carré: the Biography” (Bloomsbury)

The Secret World: a History of Intelligence

Christopher Andrew

Allen Lane, 960pp, £35