No one on the ground or in Pakistan's air defence spotted Area 51's latest toy as it kept watch on Osama bin Laden's compound in Abbottabad, Pakistan, on the night of the US raid that killed the Taliban leader.

Rather than one of the UFOs that the wilder fringes of the internet believe the military has stashed away at America's top-secret military site in Nevada, this "toy" was actually the latest Star Wars-type drone, or unmanned aerial vehicle (UAV), developed at the base whose existence Uncle Sam only barely admits.

Named the "Beast of Kandahar" after it was snapped at Kandahar air base in Afghanistan back in 2009, this stealthy grey batwing-shaped long-distance reconnaissance drone, officially known as the RQ-170 Sentinel, was a throwback to Area 51's golden age before the advent of the spy satellite put the spy plane out of business.

While for the post-X-Files generation Area 51 will always be associated with conspiracy theories from aliens to time machines, for journalists such as Annie Jacobsen its purpose was – and, indeed, is – to develop and test the latest kit from the military industrial complex that helps to maintain America's superpower status.

"Area 51 was the single most important Cold War facility as it was set up to push science faster and further than the Soviet Union," says Jacobsen, author of Area 51: An Uncensored History. "On one side of the road the Nevada test site was preparing for the Third World War, and on the Area 51 side they were trying to prevent it by developing air-surveillance technology.

"Today, it is doing the same job competing with whoever America sees as its enemy, whether North Korea, Iran or China."

Yet for Jacobsen one of the strangest things about Area 51 is that even in the age of WikiLeaks and Google Earth, she has "not seen anything ever leak out of Area 51", almost as if "the base is in a permanent state of lockdown".

Sitting on the edge of a dried-up lake bed cradled by mountains, only 90 miles or so from the fantasy world of downtown Las Vegas, it is perhaps not surprising that it is hard to separate the myth from the fact of Area 51, or Groom Lake, as old-timers have traditionally called it. Even "Area 51" sounds like an exercise in branding, as do its other names of Dreamland, Paradise Ranch or Homey Airport. And there is something sinisterly not-quite-real about the "use of deadly force authorised" signs that stand guard on the base's perimeter. Which can be found just inside the 4,687 square miles of the Nevada Test and Training Range and right next to the 1,350 square miles of the Nevada Test Site, where hundreds of nuclear weapons were exploded above and below ground until the test ban treaties of 1963 and 1996.

However, recent declassified documents have helped to shed light on the facts behind the myths of Area 51, from the U-2 spy plane missions that helped to unlock the secrets of the Soviet Union in the 1950s to the groundbreaking stealthy A12 that was obsolete before it even first served its country, and the record-breakingly fast recon plane the SR-71 Blackbird, which helped to spot North Vietnamese missile bases in the late 1960s and 1970s; and from the beginnings of stealth technology to the development of the F-117 stealth fighter itself, which was one of the few allied aircraft able to penetrate the air defences around downtown Baghdad and then bomb accurately. Also brought to light was the test flying of "acquired" Soviet MIG fighters in mock combat situations, which led to the founding of the Top Gun pilot programme made famous by the 1980s film of the same name.

The isolation of the base may be one good reason why the Beast of Kandahar was discovered on the opposite side of the world and not in Nevada. Along with the ability to time testing activities for when spy satellites have

already passed the base it also allows for the capacity to "go underground" to avoid prying eyes. Yet for Annie Jacobsen it is the "need to know" principles of the special-access "black" programmes that run out of the base that account for the cloak of secrecy that has been maintained even in the age satellite photography. These principles were developed from the "mother of all black projects", the Manhattan Project, which saw the creation of the world's first nuclear bomb during the Second World War.

And sometimes, like in a blockbuster movie, even the President doesn't need to know.

According to Jacobsen, during the 1994 enquiry into allegations of human radiation experiments during the Cold War, "certain records involving programmes at Area 51 were kept from President Clinton because he didn't have a need to know".

For ex-Area 51 radar man Thornton D Barnes, founder of the Area 51 veterans' association Roadrunners Internationale, the need to know meant that "everything was compartmentalised", with only a small number of highly screened workers allowed to remain permanently on the base. Everyone else had to fly in from Vegas to "prevent fresh faces showing up in the local communities and drawing attention to something that might be going on".

"Venturing into an area or showing an interest in something while lacking a need to know instantly banned an individual from the Area and the programme."

And that wasn't all.

"We were often away from our families Monday through Thursday and we couldn't tell them where we were or what we did.

"Even now we have retired, we don't ask each other questions about what we did unless we know that it has been declassified."

According to Bill Sweetman, editor-in-chief of defence technology for Aviation Week, simply "pulling out the plug" and "supergluing USB ports" has also helped to keep these black programmes disconnected from our interconnected world.

"Along with the traditional black techniques of the need to know and using a small number of slightly frightened people, these programmes have either now been disconnected from the internet or have never been connected to it in the first place, and the resulting 'air gap system' has prevented them from being compromised by Chinese hackers, like so many other programmes have been."

So successful are the secrecy techniques that it's clear that we are going to see more – not fewer – of these black programmes.

However, Sweetman believes that – along with strict secrecy – "black propaganda" helps to keep Area 51's secrets.

"There were certainly deliberate disinformation campaigns to generate a lot of noise about UFOs back in the 1950s and 1960s to cover secret flights of planes like the U-2, and then again in the late 1970s and early 1980s to link Area 51 to UFOs through 'fake' documents and eyewitness accounts of alien technology – and even alien bodies."

The goal, he believes, was originally to mask the rebuilding of the base and then to stop people "asking why is there this huge secret and expensive military base in the middle of the desert".

Annie Jacobsen has real fears about the "lack of respect for the constitution" that this level of secrecy can entail. The construction in 2007 of a new hangar twice the size of any other at Area 51 and another large hangar in 2010 meant that something was going on – but the public doesn't have a need to know.

It is possible to look at the Beast of Kandahar for clues as to what the new project might be, or even at the stealthy Special Forces Black Hawk helicopters that were also revealed or even compromised by the Bin Laden raid. The size of the hangars suggests that it could be early prototypes of the $4bn next-generation American bomber programme that are already being put through their paces down on Groom Lake.

Whatever it is, the Beast of Kandahar is only the shape of things to come.

'Area 51: An Uncensored History' by Annie Jacobson is published by Orion (£20)

A British Area 51?

If there is a British Area 51 then it is Area 51, says Nick Cook, former aviation correspondent of and now aerospace consultant for Jane's Defence Weekly and author of The Hunt for Zero Point. "Britain is just too small to have such a secret flight testing facility", he argues. "Boscombe Down is the closest thing we have to it, but the A303 runs right alongside so when a highly top secret US spy plane – that still hasn't been identified – crashed in 1994 it quickly made national headlines."