The recent revelations by the whistleblower Edward Snowden were fascinating. But they - and all the reactions to them - had one enormous assumption at their heart. That the spies know what they are doing. It is a belief that has been central to much of the journalism about spying and spies over the past fifty years. That the anonymous figures in the intelligence world have a dark omniscience. That they know what's going on in ways that we don't. It doesn't matter whether you hate the spies and believe they are corroding democracy, or if you think they are the noble guardians of the state. In both cases the assumption is that the secret agents know more than we do. But the strange fact is that often when you look into the history of spies what you discover is something very different. It is not the story of men and women who have a better and deeper understanding of the world than we do. In fact in many cases it is the story of weirdos who have created a completely mad version of the world that they then impose on the rest of us. I want to tell some stories about MI5 - and the very strange people who worked there. They are often funny, sometimes rather sad - but always very odd. The stories also show how elites in Britain have used the aura of secret knowledge as a way of maintaining their power. But as their power waned the "secrets" became weirder and weirder. They were helped in this by another group who also felt their power was waning - journalists. And together the journalists and spies concocted a strange, dark world of treachery and deceit which bore very little relationship to what was really going on. And still doesn't.

PROLOGUE - SALISBURY PLAIN 1991 In January 1991, as the Gulf War began, MI5 became convinced they had discovered a secret Iraqi terror organisation based in Britain. They had found a list of thirty three Iraqis who were studying for PhDs in London. The list had been sent by the Iraq embassy in London to the Bank of England to ask the Bank not to freeze the grants the students lived on. The Bank sent the list to MI5 and the agents quickly realised that actually they were looking at something far worse - a nationwide Iraqi military terror cell. The reason they knew this was because the person who sent the list was the deputy military attache at the embassy. Immediately the police were told to swoop on the 33 "students" - and they were taken to a disused military camp at Rollestone in the middle of Salisbury plain and interned as prisoners of war. They were surrounded by two levels of high security razor wire and guarded by a hundred heavily armed soldiers. It was the first time anyone had been held like this in Britain since the Second World War.

You must enable javascript to play content

In fact the letter showed nothing of the kind. The Iraqi military attache was also in charge of administering student grants for Iraqis studying in Britain. Some of them did get funding from the Iraqi military - for studying things like the structure of polymers. But, as a British professor pointed out, if that same interpretation were applied to British science students, over half of them would be immediately re-classified as terrorists. Here is part of a programme made later that year about the absurdity of what happened. It shows how neither the detainees or their lawyers were even allowed to know what the evidence was that had led to them being imprisoned. The man who defends MI5 with such fervor will turn up later in this story - playing a very odd role. he is called Nigel West - but his real name is Rupert Allason. I've added on the news reports of the same Iraqis suddenly being released from the heavily fortified camp. But now everyone is referring to them as "students".

You must enable javascript to play content

An inquiry was held later that year into the scandal. It asked MI5 to produce its evidence. Other than the letter, the secret agents came up with nothing. They had imagined the whole thing. But they justified it by saying "It was best to err on the side of caution". NEARLY A HUNDRED YEARS EARLIER THE DAILY MAIL CIRCULATION DEPARTMENT 1906 William Le Queux was a popular novelist in the early part of the twentieth century. He was half French, half British and he wrote books with wonderful titles like Strange Tales of a Nihilist.

Le Queux had started off as a journalist on the Daily Mail - but then had travelled around Europe getting to know lots of famous and infamous people. But as he did so he became convinced that many of the European countries, but most of all Germany, envied Britain and wanted to get their hands on the wealth of the Empire. The trouble was that the British people didn't realise this. So Le Queux set out to warn them - above all by telling them that the Germans were sending spies to Britain to prepare for an invasion.

But the ruling classes in Britain laughed at Le Queux. They said it was just fiction - which it was. Plus he wasn't really British and he hadn't been to a proper school, he was far too vulgar and insistent in his patriotism. In short he was a bore. So Le Queux did what anyone in their right mind would do in such a situation. He turned to the Daily Mail. He wrote a gripping account of a future German invasion of Britain and took it to Lord Northcliffe who ran the Mail. It was called "The Invasion of 1910" and it described how the Germans landed in East Anglia and marched on London. Northcliffe loved it - but the Mail's circulation department said that many of the towns on Le Queux's invasion route didn't have many actual or potential Daily Mail readers in them. So Lord Northcliffe changed the route of the invasion to make sure that all the towns that were sacked and pillaged had lots of Daily Mail readers. Here is the map of the invasion as agreed with the circulation department.

The serialisation was an enormous success. The prime minister got up in the House of Commons and said Le Queux was "a pernicious scaremonger" and that the story was "calculated to alarm the more ignorant public opinion at home." Result. Then things started getting out of control. Thousands of Daily Mail readers sent Le Queux letters telling him that they had spotted people acting suspiciously - which meant they must be German spies. The letters were mirror images of what Le Queux had written in his books. But rather than making him suspicious, Le Queux decided that this proved that what he had written as fiction must actually be true. There was a gigantic German spy ring in Britain. Thousands of Daily Mail readers couldn't be wrong. The man whose job it was to uncover spies in Britain was very excited by all this. He was called Colonel Edmonds. He had a tiny budget and two assistants - and noone on the General Staff bothered with him. But now Col. Edmonds saw his chance. He teamed up with Le Queux and together they bombarded the Committee for Imperial Defence with the evidence from the Daily Mail readers. Edmonds said that the government should set up a "secret service bureau" to combat the threat. The head of the Committee - Lord Haldane - said this was ridiculous. But even he couldn't stand against the wave of spy fever that was sweeping the country. He gave in - and MI5 was set up - created in large part by the dreams of a socially excluded novelist, and the paranoid imaginings of the readers of the Daily Mail.

But the problem for MI5 was that the spy network didn't exist. The Germans did have some agents in Britain - but nothing like the 5000 that Le Queux had described. When war against Germany was declared in 1914 - MI5 immediately rounded up 21 alleged German spies and proudly announced they had broken the network. But a brilliant piece of historical research by the historian Nicholas Hiley has shown that this wasn't true. Hiley doesn't mince his words. Here are his conclusions (Kell and Holt Wilson were the director and deputy directors): "One of the most famous successes of the British Security Service was its great spy round-up of August 1914. The event is still celebrated by MI5, but a careful study of the recently-opened records show it to be a complete fabrication - MI5 created and perpetuated this remarkable lie. The great spy round-up of August 1914 never took place - as it was a complete fabrication designed to protect MO5(G) from the interference of politicians or bureaucrats. The claim made next day that all but one had been arrested was false, and its constant repetition by Kell and Holt-Wilson was a lie." In other words - MI5 had followed the shining example of William Le Queux and made it all up. But that didn't matter - because it made a great story, and journalists loved it. Even in 1997 the BBC made a breathless documentary - using the recently released files - about how in 1914 MI5 had brilliantly rounded up the Kaiser's spy network on the eve of the first world war. Aside from perpetuating a fiction, the film has two great moments - one is an interview with the grandson of the deputy head of MI5 who has an immortal line about his grandfather - "of course he was very private about MI5 - so the family knew nothing". And the end the programme has some wonderful stills of the party MI5 held to celebrate the end of the war - it's on their rooftop. Their faces are great.

You must enable javascript to play content

After the first world war MI5 declined in importance. But with the growing fears of communism in the 1920s and 30s a new threat emerged - not just communist agents from abroad, but British communists who might betray their own country. In many cases they came from the same upper classes as those running the secret services. And a strange dance began - of toffs suspecting toffs. But even then MI5 couldn't get it right. Take the case of Cecil Day Lewis - who was Daniel Day Lewis' father. Back in the 1930s he was a teacher at Cheltenham College - one of the great Victorian public schools. But, despite his job, Cecil was convinced that he was really a revolutionary. And in 1933 he decided to foment revolutionary action in Britain - by writing a poem. It was an epic he called "The Magnetic Mountain". He said his aim was to create "A violently revolutionary poem with abundant images (for example) of a barren, cancerous land led by 'getters not begetters', demanding 'It is now or never, the hour of the knife/ The break with the past, the major operation." Here is Cecil Day-Lewis looking both poetic and radical - alongside some of the poem - (you can see where Daniel Day Lewis gets it all from).

But Day-Lewis was disappointed by the lack of reaction. He admitted that the poem "did not create the slightest ripple of outrage amongst the guardians of Cheltenham." Even though the communist magazine - the Partisan Review - had said that it was "perhaps the most important revolutionary poem as yet written by an Englishman". And then MI5 noticed Cecil Day-Lewis. Not because of the poem - but because he had sent £5 as a donation to the headquarters of the Communist Party in London. So MI5 decided to put Day-Lewis under intense surveillance. The historian James Smith has written a wonderful book about how MI5 spent a lot of time covertly watching many upper class British writers between 1930 and 1960. It is a great book because what it records is a strange and confused dance of manners among different parts of the British elite. Smith describes how MI5 got the local police to spend weeks watching Day-Lewis' house and intercepting his post. But they found nothing suspicious. Their report said that: "Day-Lewis seldom wears a hat, and is not altogether of smart appearance in dress. He is a good singer. He has moved into his cottage after having considerable structural improvements done there." MI5 were completely incompetent. They didn't discover the poem that Day-Lewis hoped would help to bring about a communist uprising in Britain. And not only did they miss the poem - they didn't even realise he was a poet. All in all MI5 found nothing dangerous or revolutionary about Cecil Day-Lewis. It was humiliating. But they might have been right. James Smith describes how a few years later in 1940 Cecil Day Lewis was getting his mistress Rosamund Lehmann to pull strings in the British establishment so he could avoid getting called up to go and fight the fascists.

But in 1940 MI5 had its greatest success. It not only found a real German spy network in Britain - but managed to persuade many of the German agents to switch sides. It was called the Double-Cross system - and it is celebrated in histories of MI5 as a brilliant use of espionage. The German agents carried on spying for their masters in Berlin - sending back detailed reports. But the information was all fake, designed to mislead and confuse the Nazis. But something else happened to all the intelligence agencies during the war - MI6 as well as MI5. As they grew massively in size they became riddled with factions and infighting. And because all this happened behind a wall of secrecy, there was little to stop things becoming vicious and poisonous. The journalist Phillip Knightley has written a really good history of spies - called The Second Oldest Profession. In it he quotes an agent describing what happened during the war years: "The whole organisation was riddled with nepotism - dim, dreary people of utter unmemorability; sub-men who were doubled up with other sub-men to create an illusion of strength and only doubled the weakness; others made memorable only by poisonous, corrupt malevolence or crass, mulish stupidity; the whole run by a chain of command remarkable for its feebleness. The entire service was decrepit and incompetent." At the end of the war the new Labour government knew that something had to be done to sort out MI5. So they went and found Percy Sillitoe - who was running a sweet shop in Eastbourne

Sillitoe had retired after being Chief Constable of Glasgow - where he had become famous as the only policeman brave enough to take on the "Razor Gangs" in the eastern part of the city. The gangs had names like The Bingo Boys and The Baltic Fleet - and they terrorised Glasgow as they fought each other with hatchets, swords, open razors - and razor blades stitched into the brims of their hats. You can get a sense of Sillitoe from this short film where he shows the BBC a new kind of armoured car he has invented to stop criminals holding up vans carrying cash. He invented the security van. I very much like how he says he is "concerned for the little man". I've also added an odd bit from a BBC film about graphology where the expert - a "psycho-graphologist" - analyses Percy Sillitoe's signature, and compares it to J Edgar Hoover's signature. Hoover was Sillitoe's American counterpart.

You must enable javascript to play content

The government asked Sillitoe to come and sort out the chaos in MI5 - and he agreed. But he quickly found that it was a very odd place - all the insiders hated him, and they ridiculed him by speaking in Latin (which he didn't understand) in front of him. Plus they deliberately gave him the wrong papers when he went to see the Prime Minister. Sillitoe came back and told his wife - "I sometimes think I am working in a madhouse." But he realised that he was dealing with very much the same situation that he had found in the slums of Glasgow - different factions locked together in a strange, poisonous bubble. Here is a section of a very good film, made much later, about the successors to the razor gangs of Glasgow - the gangs that Sillitoe had tried to suppress in the 1930s. And you can see the similarity to the world of the spies - as one of the gang members puts it, "it's two ends of the same street at war with each other". I also love the pigeon-fancier who shows off the most high-security pigeon loft you have ever seen. He then reveals that he doesn't breed the pigeons for racing. Their job is to go and kidnap the pigeons from the other gangs.

You must enable javascript to play content

But before Sillitoe could do anything, it all went terribly wrong. Suddenly traitor after traitor was revealed in the very heart of the British establishment. It wasn't just pretentious radical poets who were a threat - it was spies, diplomats and nuclear scientists within the system itself who had been giving away secrets to the Russians. There was a high-flying diplomat called Donald McLean, a nuclear scientist at the heart of Britain's atomic bomb project called Klaus Fuchs, plus two of MI6's top agents - Guy Burgess and Kim Philby. One of MI5's main jobs was to find traitors - but the awful truth was that it had failed to spot any of them. Percy Sillitoe was booted out. But things got even worse. In 1964 MI5 were told that one of their own men had been a spy for the Russians. He was called Sir Anthony Blunt - and not only had he been high-up in MI5 - but he had gone on to work in Buckingham Palace looking after the Queen's art collection. And even worse than that he was the Queen Mother's cousin.

MI5 interrogated Sir Anthony and he calmly said that it was all true - he had been a traitor. MI5 was so embarrassed that they kept it all quiet, gave Blunt immunity from prosecution, and he carried on working at Buckingham Palace. The Daily Mail later said that the Royal Family had known all along anyway. That as far back as 1948 Sir Alan Lascelles - the most senior aide to the Royal family - had whispered "that's our Russian spy" to someone else as they passed Blunt in the palace. But that could have been a misinterpretation. Blunt had shocked the Queen Mother by telling her that he was an atheist - and she had immediately assumed that meant he must be a communist. Clever Queen Mother - wrong but right. Blunt had also become a bit of a TV star. Starting in the early 60s the BBC went to him regularly to take the viewers on a tour of the treasures of Buckingham Palace - a sort of early Fiona Bruce. Here is part of one programme from 1962 - two years before he was exposed as a traitor. Followed by a bit of another programme from 1972 - when a self-confessed KGB agent takes the viewers round Buckingham palace. Spot the difference. And at the end there is footage from 1979 - when Blunt was exposed as a traitor. It's from some rushes I found in the library. The press chasing Sir Anthony are straight out of a British movie. And I love the interviewer's obsession that it was Blunt's "homosexual leanings" that made him betray his country. Blunt, though, doesn't bat an eyelid. It's as though he is still talking about some painting.

You must enable javascript to play content

Then, in 1971, MI5 got another big shock to the system. Most of their opponents - Russian secret agents in Britain - were kicked out, leaving MI5 with little to do. The irony was that it happened as a result of one of their few successes. In August 1971 an ordinary London policeman arrested a man who was driving drunkenly down Tottenham Court Road. He turned out to be Oleg Lyalin who was a KGB agent. Lyalin spent a lot of his time buying socks in the West Midlands - pretending to be a member of the Soviet Trade delegation. But really he was spying. Lyalin panicked and offered to tell MI5 the names of all the Russian spies in Britain. In return he wanted to stay and live in Britain with his mistress. MI5 agreed - and the Home Secretary expelled 105 other members of the trade delegation, because Lyalin said they were spies. Here are the reports - plus a "News Special" which is an early example of the way TV journalism would report the hidden world of spying. It's got an anonymous British "research scientist" called "Jim Walker" who got caught up in all this - and has some great MI5 surveillance footage of Jim and his controller Viktor leaving information at a "dead letter drop." Plus a very good telephone non-interview with the British Ambassador in Moscow.

You must enable javascript to play content

But the problem for MI5 is that the expulsions pretty much destroyed the KGB presence in Britain. The historian Stephen Dorril who has written a series of brilliant detailed histories of the intelligence agencies says that a later KGB defector called Oleg Gordievsky admitted that "the London residency never recovered from the expulsions". Dorril also says that the British government and its civil servants were well aware of this, and they became deeply suspicious of claims from MI5 and its K Branch - whose job was to monitor foreign agents - that there was still a big Soviet threat in Britain: "Senior civil servants dealing with the intelligence community were therefore aware that K Branch claims about the penetration of British political life and the threat to security from Soviet bloc operations were generally exaggerated." The brutal fact was that by the early 1970s MI5 not only had very little to do - but also its political masters were beginning to question whether it might be seriously incompetent. Edward Heath - who had been Prime Minister when all this was happening - later got up in the House of Commons and said bluntly what he had discovered about MI5 officers: "They talked the most ridiculous nonsense, and their whole philosophy was ridiculous nonsense. If some of them were on the tube and saw someone reading the Daily Mirror they would say - 'Get after him, that man is dangerous, we must find out where he bought it.' " But those in charge in Britain also realised that there was nothing they could do to question or control the spies. The next prime minister in the 1970s - Harold Wilson - wrote a very serious book called The Governance of Britain full of long serious chapters. But when he got to chapter nine - about THE PRIME MINISTER AND NATIONAL SECURITY This is what it looked like.

There are two paragraphs explaining that the prime minister has ultimate responsibility for the security agencies. And it ends with two more that simply say this: "The prime minister is occasionally questioned on matters arising out of his responsibility. His answers may be regarded as uniformly uninformative. There is no further information that can usefully or properly be added before bringing this Chapter to an end." In response to these kind of doubts and attacks MI5 turned inwards. The problem for the MI5 men - stuck in their secret bubble - was that they just couldn't believe that their failure was due to them being useless at their job. Not only had they failed to find any of the traitors, but operation after operation had ended in failure. And they convinced themselves that this meant there had to have been another traitor lurking somewhere in their building - the MI5 HQ in Mayfair. They began a mad search for enemies inside the organisation itself - seeking to find more hidden traitors who could be used to explain why MI5 kept failing to do its job properly. It was the search for "Fifth Man" - to go with the other four already exposed, Burgess, McLean, Philby and Blunt

A small group of MI5 men went to their boss and said they wanted to investigate all the past failures looking for evidence of treachery. Their boss was called Sir Roger Hollis - and he said no. His argument was that operations often went wrong because of simple human failure, and to re-examine them on the basis that failure was evidence of treachery would tear the agency apart. Imagine what it would feel like he said to know you are being watched because a past operation you were involved with had gone wrong. "It's like the Gestapo" he said. So the small group of MI5 agents decided he must be the traitor. Here is a picture of Roger Hollis.

The small group in MI5 now became convinced that their organisation was not just penetrated by the Russians, it was actually run by a Soviet agent. They knew they had to get the truth out somehow even if it meant breaking the law. So they found a friendly journalist called Chapman Pincher and told him the hidden truth.

Here is Chapman Pincher being interviewed on the Wogan programme about what then happened. Up to this point Pincher had been the Defence correspondent on the Daily Express. He was successful for getting "scoops" from "inside sources" - although the historian EP Thompson said that really Chapman Pincher was: "A kind of official urinal in which ministers and intelligence and defence chiefs could stand patiently leaking." What the dissident MI5 agents now told Pincher was like super high-grade piss. Or, as he puts it in the Wogan interview, "it was like walking into an Aladdin's Cave". But what Pincher wrote was going to open the floodgates to a new kind of conspiracy journalism that still holds sway over large parts of the media imagination. Have a look at him and decide yourself - high grade toilet or investigative journalist? Or maybe often they are the same thing? I've also included Pincher being interviewed on the TV news reports as the scandal unfolded. Everyone tries to get in on the act. The BBC presenter quotes Kim Philby as saying that Hollis wasn't very good at his job. But the presenter says that this is "ambiguous" - and might be proof that Hollis really was a Soviet agent.

You must enable javascript to play content

The leading MI5 dissident who was leaking the information to Pincher was called Peter Wright. He was one of the most senior members of MI5 but he was also somewhat paranoid. To get a sense of Peter Wright and how he saw the world I have put together some bits of him being interviewed in the 1980s about another of his conspiracy theories. This was that the Prime Minister - Harold Wilson - had also been a Soviet agent. In Wright's mind much of the British establishment had been directly or indirectly taken over by the Soviet Union. He had no hard evidence for this - but he was driven by an underlying mind-set that was going to spread throughout much of the intelligence agencies - and journalism - over the next twenty years. This said that if you imagined the other side was doing something devilish and deceptive - then they probably were. It meant that in the dark world of intelligence, imagination was more powerful than obvious facts. Because if you couldn't find the evidence it proved how clever the enemy had been at covering their tracks. It was a fevered romantic view of the world that would both entrance the readers of newspapers - but would also lead the intelligence agencies into the disaster of the weapons of mass destruction in Iraq in 2003 Here is the grandaddy of that conviction - Peter Wright. The person called Angleton he refers to was an even odder American equivalent of Wright who was high up in the CIA - and who also was convinced Wilson was a Soviet agent. The tone of Wright's plaintive child like statement about Angleton - "he believed it - he did" tells you a great deal about the emotions driving these strange men in their spy-bubbles.

You must enable javascript to play content

But as in all organisations - egos started to come into play. Other MI5 agents started leaking other names to other journalists. Pincher's main rival was a writer called Nigel West. Nigel upped the stakes. He began to publish books and articles alleging that all sorts of other people had been traitors. Here he is on Nationwide in 1981 in full flow. He says that a man called Leo Long was a traitor, and then goes on to suggest that others - including even the former Governor of Uganda, Sir Andrew Cohen - might be traitors. It's worth looking closely at what Nigel West says about Sir Andrew Cohen - because it shows how weird this paranoid outpouring from the secret world was becoming. When he was an undergraduate at Cambridge in the 1930s Cohen had been a member of an intellectual society called The Apostles. So had two of the spies - Burgess and Blunt. The interviewer asks Nigel how he knows Sir Andrew might be a traitor. Nigel says: "I haven't named him (Sir Andrew) up to now because it's not known whether he was a Soviet agent. But I think it's worth saying that anybody, if you are talking about the Apostles, many of them were Soviet agents. And he would undoubtedly have been questioned since he rose to a very senior position in the Department of Overseas Development" That's it. But Nigel does have a fabulous haircut.

You must enable javascript to play content

In the early to mid 80s more and more names poured out - all accused of being KGB agents in the heart of the British establishment. One newspaper grouped them under headings "CONFESSED" - "PARTIALLY CONFESSED" - "UNRESOLVED" There was one great apology "Our list of MI5 spy suspects included Cedric Belfrage who MI5 officers said had made a partial confession and we said was dead. We are glad to make it clear he is alive, never made any confession and maintains he should not have been on the MI5 list at all." And Mrs Thatcher also got involved. Because it seemed to prove to her the thing she had believed all along - that the British establishment were weak, spineless and easily corruptible. She happily admitted in Parliament that Anthony Blunt had been a traitor. And here she is in 1986 merrily joining in with the latest accusation - that Lord Rothschild had been the 5th Man. It later turned out that he wasn't.

You must enable javascript to play content

It became farce. The journalists who had started the mole-hunt went to war. Nigel West wrote a whole book announcing that he had discovered that the 5th man wasn't really Hollis, but was actually Hollis' deputy. He was a man called Graham Mitchell who in his spare time was a grand master in correspondence chess. Apparently the dissidents in MI5 were convinced that the letters he sent his chess-friends were his way of contacting his Soviet controllers. The moves he typed out were actually secret codes that disguised his treachery. Here is one of Graham Mitchell's games that he played in 1950. You are looking at a complicated code, whether it was secret messages to the Russians has never been proved.

Another writer then found a letter in an old government file that had been written by Roger Hollis in the 1940s saying that the Russians shouldn't be trusted. Some journalists said that this proved he wasn't a traitor. But others said that Hollis had put the letter there deliberately so it could be found and throw MI5 off the scent. Here are the TV reports - both of the Graham Mitchell "revelation", and the Hollis letter. The leader of the pack - Chapman Pincher - still insists Hollis is the 5th man. Nigel West says he is innocent. But Nigel now has a very good late 80s haircut.

You must enable javascript to play content

Then another writer called W. J. West wrote a book saying that the 5th man was Hollis after all. W J West turns out to have been an ex-hippie whose early years were memorably captured in a semi-autobiographical novel by another ex-hippie - called "Ten Men". She describes a road trip across America as she desperately but unsuccessfully tries to shag him. Here's his book about the Fifth Man.

But then - in the midst of all these weirdos - a dissenting voice emerged. James Rusbridger had been a spy back in the 50s and 60s - and he now wrote a book called The Intelligence Game arguing that all this was rubbish - and that all the journalists had been conned by a crazy gang of right-wing nutters in MI5. Rusbridger said that the newspapers and TV were being used to promote the obsessive belief of MI5 officers that their failure to do anything worthwhile for a quarter of a century was the consequence of there being a Russian spy in MI5. They couldn't face the fact that they were completely useless and incompetent. At last a voice of sanity. But unfortunately James Rusbridger was then found dead in his garden shed - apparently the victim of an auto-erotic game that had gone wrong. He was naked apart from a rubber coat and a gas mask - and his feet and legs were attached to the wall by a complicated system of pulleys. Of course it might have been a fiendishly clever assassination. Or just another spy-world weirdo.

But this crazed witch hunt didn't harm MI5 at all. Quite the opposite - because together the spies and the journalists created an image in the public imagination of a dark world full of hidden treachery. The spy world became a fascinating other universe that was full of layer upon layer of deception, where the men who inhabited it spent their time trying to penetrate through the circles of falsehood to the inner sanctum of truth. It was an image that was powerfully helped by John Le Carre's novels - and his anti-hero George Smiley. Le Carre's novels were a clever piece of PR - because they appeared to be more gritty and realistic than the glamourised James Bond image. But it was just another layer of deception - because Smiley and his search for a hidden mole expressed powerfully the paranoid and unfounded fantasies of the dissident MI5 agents. But it was a world that was all made-up. Le Carre - who had himself been a spy - admitted this, and described what the true reality of the spy world was: "For a while you wondered whether the fools were pretending to be fools as some kind of deception, or whether there was a real efficient service somewhere else. Later in my fiction, I invented one. But alas the reality was the mediocrity. Ex-colonial policemen mingling with failed academics, failed lawyers, failed missionaries and failed debutantes gave our canteen the amorphous quality of an Old School outing on the Orient express. Everyone seemed to smell of failure."

But this new image couldn't conceal MI5's incompetence for long. Because at the very same time that everyone was talking excitedly about completely invented moles, MI5 missed the real moles at the heart of the intelligence services - even though they were completely obvious, and almost screaming to be noticed. Michael Bettaney worked in counter-espionage in MI5. He had been recruited when he was at Oxford university - where he had been an admirer of Adolf Hitler and had a habit of singing the Nazi Party anthem in local pubs. Here is Bettaney back then.

MI5 did a thorough check on him - called positive vetting - and decided he was fine. Perfect MI5 material. Bettaney was then sent off to Northern Ireland to fight terrorism where he was wounded by a car bomb. He then had a horrible experience. Hidden in a cupboard he had to watch in silence as one of his informants was shot through the kneecaps by other terrorists. Here is Bettaney later - after he had been working for MI5.

Bettaney came back to London a changed man. He decided that MI5 was both corrupt and incompetent. He started drinking heavily and told his colleagues loudly that he was no longer a fascist - but he had become a communist. So MI5 decided to promote him. He was positively vetted again - found to be perfect MI5 material, and sent to the Russian desk. Bettaney became more and more unstable. In October 1982 he was convicted of being drunk and disorderly. The next week he was convicted for fare-dodging. Finally MI5 did begin to notice - and two separate inquiries were set up to look into Bettaney's behaviour. But each was unaware of the other's existence. Neither of them noticed that he had been stealing a huge amount of MI5 top secret documents and stashing them at his home. Bettaney was only caught when he took some of the best of these secrets and tried to stuff them into the letter box of the Second Secretary of the Russian Embassy - Mr Gouk. This is a picture of Mr Gouk.

Mr Gouk was so confused by this that, instead of passing them on to the KGB, he went round to MI5 and gave them back, and told them where they had come from. MI5 arrested Bettaney and he was put on trial. The man who was in charge of the vetting of government employees - like Michael Bettaney - was then allowed to vet the members of the jury at Bettaney's trial. Luckily this time he got it right - and Bettaney was sent to prison on the Isle of Sheppey for 23 years. Here are some of the reports. Including Nigel West turning up yet again on Breakfast Time. Even Nigel is shocked by how MI5 didn't spot Bettaney. And he's having a bad hair day.

You must enable javascript to play content

The terrible truth that began to dawn in the 1980s was that MI5 - whose job it was to catch spies that threatened Britain - had never by its own devices caught a spy in its entire history. The case that really shocked Mrs Thatcher was the traitor Geoffrey Prime. In the 1970s he had worked at the top secret listening centre GCHQ and had been selling all it's secrets to the Russians.

And yet again it wasn't MI5 who uncovered his treachery - it was the local police in Cheltenham. In 1982 a policeman came to his house enquiring about his car - a rather distinct two-tone brown and white Mk IV Cortina - a which had been seen in the vicinity of an assault on a young girl. Prime told the policeman that he had been at home all day. But that evening he and his wife Rhona went for a drive to the top of Cleeve Hill. As they sat in the twilight Prime told Rhona that he was the man the police were looking for. And not only that, he was also a Russian spy. Here is part of a very powerful interview Rhona Prime gave to the BBC where she describes that day - and what she then did.

You must enable javascript to play content

Prime was a paedophile - and had used spying techniques to monitor the activities of thousands of young girls around Cheltenham. He had created a vast set of index cards which showed when the girls were most likely to be alone at home. He then went round to their houses in his two tone Cortina and sexually assaulted them. Despite this Prime had been positively vetted six times. Even the Russians got worried about his paedophile activities and seemed to want to dump him. In 1980 Prime had gone to Vienna to meet the KGB. Instead of meeting him secretly as they normally did, the Russians took him openly to the best restaurants where they knew Western intelligence agents would recognise them as KGB agents. But even then noone noticed them - or Prime. Prime's wife Rhona wrestled with her conscience - and in the end went to the police and told them everything about Prime. He was sent to jail for 35 years for spying and 3 years for the assaults on young girls - which says a lot about the priorities of the British establishment at that time.

You must enable javascript to play content

The cases of Bettaney and Prime revealed not only just how incompetent MI5 was - but also how sad and seedy the secret world of spies really was. But even in the midst of all this treachery - a surprising thing happened. Rhona Prime decided to stand by her husband. Here is Rhona describing how her deep christian beliefs gave her the strength to stand by her husband. She is very calm and composed, and somehow her dignity makes you realise just how odd the whole spy thing was. A strange hysteria driven by totally inadequate men - both agents and journalists - who were incapable of dealing with real human emotions like love and loyalty. Rhona talks about something else - unconditional love. Receiving unconditional love, she says, makes us whole and beautiful people because we are totally accepted. The very opposite of treachery.

You must enable javascript to play content

At the same time, one of the original traitors - Kim Philby - died in the Soviet Union. The BBC cameraman Phil Goodwin has given me the unedited rushes recording Philby's funeral in Moscow. He found it in the back of a cupboard in the BBC's Moscow office. It's an amazing record of a weird communist state funeral - held for an upper class Englishman in a Moscow graveyard in 1987. Standing all around are the faces of the Russian side of the spy world - and it is great to look at their faces, peeking out for a moment from their traditional secrecy. Then Philby's coffin arrives accompanied by a military band and members of the KGB holding all Philby's Soviet medals on orange cushions. It's an extraordinary scene. But also watch the woman with red hair. She is Philby's widow - Rufina - who had lived with him and helped him through alcoholism and depression. Watch what Rufina does. It's really moving. Love and loyalty breaking through again into this narrow, nasty world.

You must enable javascript to play content

And even Michael Bettaney found love. Marion Johnstone, who was a research scientist and also a communist, wrote to him in prison in 1985. She began to visit him - and they became engaged. But in 1995 there was a security scare that reawakened all the spy journalists on papers like the Mail - and made them huff and puff again. Marion was found to have taken some photos and made some drawings of the landscape on Sheppey around the prison and given them to Bettaney. The prison authorities confiscated them, and the journalists immediately said that this was part of an escape plan to spring the traitor from jail. Marion denied this - she insisted that because Bettaney was kept in solitary confinement she just wanted to show him how beautiful the landscape was outside. And she is right. The landscape around Bettaney's prison, Swaleside, is extraordinary and beautiful. A little while ago I managed to get onto Deadman's Island which is nearby on the river Swale. It is a moody place because it is where prisoners from a long time ago - the Napoleonic wars of the 1800s - were buried. They had been held on the "hulks", floating prisons off the coast of Sheppey. What makes the island so strange is that it is covered by water every high tide - and that washes away the mud and opens up the prisoners' graves. It means that the island is littered with human bones. The warden of Deadman's Island very kindly showed me round - and here he is showing me the open graves and the bones of prisoners, other kinds of traitors, from a very different war of long ago.

You must enable javascript to play content

But what really did for all of the intelligence agencies at the end of the eighties is that none of them predicted the collapse of communism. Mrs Thatcher's advisor - Charles Powell - summed up the extraordinary failure: "The biggest single failure of intelligence of that era was the failure of almost everybody to foresee the end of communism. It caught us completely on the hop. All that intelligence about their war-fighting capabilities was all very well, but it didn't tell us the one thing we needed to know - that it was all about to collapse. It was a colossal failure of the whole Western system of intelligence assessment and political judgement." But the real reason that the intelligence agencies didn't predict the collapse of the Soviet system was because many of the people at the top of the agencies couldn't believe it was true. Sir Percy Cradock was one of the most powerful figure in the British establishment. He was the chairman of the Joint Intelligence Committee - which co-ordinated the activities of MI5, MI6 and other intelligence groups. Even at the end of the eighties when everyone else was realising that the Soviet Union was collapsing, Sir Percy remained convinced that this was all a trick. That the Soviet Union was still aiming for communist domination of the world. Here is Sir Percy

Cradock - along with a number of others high up in the intelligence agencies - really believed that Gorbachev's reforms were just a cunning ruse to deceive the West. And - as Mark Urban has pointed out in his book UK Eyes Alpha - Sir Percy used his position to make sure that this view dominated the Joint Intelligence Committee. But as Urban also points out - Sir Percy and his allies had no secret evidence for this. They relied on what was pompously called "analysing open source data". Otherwise known as reading the newspapers and watching TV. Except they interpreted that data in a mad way - driven by their own fevered imaginings of a world completely possessed by infinite levels of deception. Mrs Thatcher realised this was bonkers - and she finally gave up on the spies. And that really should have been that for MI5. Except ten years later it was saved by the War on Terror - and since then MI5 has grown massively. But what no-one seems to know is whether MI5 has changed. For most of the twentieth century the combination of ineptitude and secrecy created an organisation that retreated more and more into a world of fictional conspiracies in order to disguise it's repeated failures. The question is whether the same is true today? Disasters like the total intelligence failure over the WMD in Iraq would suggest that nothing much had changed. But the trouble is there is no way we can ever find out. The spies live behind a wall of secrecy and when anyone tries to criticise them, the spies respond by saying that they have prevented attacks and saved us from terrible danger. But they can't show us the evidence because that is secret. It was recently revealed that back in the 1970s - at the height of the obsession with traitors - MI5 trained a specially bred group of gerbils to detect spies. Gerbils have a very acute sense of smell and they were used in interrogations to tell whether the suspects were releasing adrenaline - because that would show they were under stress and lying. Then they tried the gerbils to see if they could detect terrorists who were about to carry a bomb onto a plane. But the gerbils got confused because they couldn't tell the difference between the terrorists and ordinary people who were frightened of flying who were also pumping out adrenaline in their sweat. So the gerbils failed as well. Perhaps MI5 shouldn't have given up so easily. Maybe what we need is a better class of gerbil to find out the truth? But maybe we have them already - they're called journalists.