In a searingly hot afternoon at a campuslike new science park in Beer Sheva, southern Israel, I watched as a group of bright, geeky teenagers presented their graduation projects. Parents and uniformed army personnel milled around a windowless room packed with tables holding laptops, phones or other gadgets. There was excited chatter and a pungent smell of adolescent sweat.

This was a recent graduation ceremony for Magshimim (which roughly translates as “fulfilment”), the three-year after-school programme for 16 to 18-year-old students with exceptional computer coding and hacking skills. Magshimim serves as a feeder system for potential recruits to Unit 8200, the Israeli military’s legendary high-tech spy agency, considered by intelligence analysts to be one of the most formidable of its kind in the world. Unit 8200, or shmone matayim as it’s called in Hebrew, is the equivalent of America’s National Security Agency and the largest single military unit in the Israel Defence Forces.

It is also an elite institution whose graduates, after leaving service, can parlay their cutting-edge snooping and hacking skills into jobs in Israel, Silicon Valley or Boston’s high-tech corridor. The authors of Start-up Nation, the seminal 2009 book about Israel’s start-up culture, described 8200 and the Israeli military’s other elite units as “the nation’s equivalent of Harvard, Princeton and Yale”.

With a female IDF minder at my side, I listened as the teenagers described their projects. More than half were boys but there were girls too, and 8200 is open to both. Omer, 19, had designed a USB key that can suck information out of one computer and organise it on another: essentially, a hacking tool. “We made it appear like a keyboard so you can infiltrate any company in the world,” he told me. “It’s a proof of concept.”

Two 17-year-old boys, both named Lior (the IDF asked me not to use the students’ last names), had built a cellphone from scratch and programmed it to make and receive calls — until it had exploded in a power surge. “The project didn’t work as planned,” one of the pair explained.

Magshimim itself is difficult to get into. Funded by the Israeli state and the Rashi Foundation, a private outfit devoted to helping underprivileged youth, it targets gifted children in Israel’s poorer south and north. Applicants are admitted only after an online questionnaire, followed by a battery of more rigorous tests to gauge their abilities in programming, languages and thinking outside the box. (Another programme, Gvahim or “Heights” — targets children in central Israel, where wealth and opportunities are greater.) Of the 1,400 children who applied last year, about 500 got in; more than 2,000 have applied this year. “School can be boring for these kids — some fail,” Uri Rotem, one of the instructors, told me, “but they can do their best here, so they love it.”

Uri Rotem, instructor at Magshimim: 'School can be boring for these kids – but they can do their best here, so they love it'

Magshimim is not an automatic entry ticket to Unit 8200 but many of its students do their compulsory military service there: three years for Israeli boys, two years for girls. Most realise that in gaining entry to Magshimim they are stepping on to the fast track. “There were rumours, but they never told us it had anything to do with 8200,” Tal, a tall and earnest 18-year-old boy from Meitar, a small city of 10,000 north of Beer Sheva, told me. “They said at the end of the 11th grade, you will get invitations to interviews in certain places [in the IDF].”

If there is a beating heart to Israel’s high-tech security state — the spot on the Venn diagram where “cool” meets “creepy” — it is Unit 8200. In few other countries does the military establishment mingle so closely with academia and business, to all three sectors’ profit. Last year, Israel’s export of cyber security products — designed to protect companies, banks and governments from the growing “dark web” of hackers, fraudsters and snoopers — topped $6bn, exceeding Israeli exports of military hardware for the first time. Today Israel, with just eight million people, captures about 10 per cent of the global cyber security market, which is growing rapidly after high-profile hacks that in some cases — such as at Target, and Sony last year — have cost CEOs their jobs. Israel, with its vibrant start-up company culture, is already one of the world’s choicest targets for venture capital money.

$6bn Israeli cybersecurity exports last year

A few miles from where I attended the graduation ceremony, a new “advanced technologies park” is rising from the sandy soil of the Negev desert. It aims to cement those links and draw in investors from the wider world who want to benefit from Israel’s cyber expertise. The project combines an office park — whose tenants include Deutsche Telekom, IBM, Oracle, Lockheed Martin, EMC and PayPal — with Beer Sheva’s Ben-Gurion University and its Cyber Security Research Centre. By the end of the decade, Unit 8200 and the IDF’s other intelligence and technology units will have moved there, too.

But what does it say about a country that handpicks its best and brightest children and channels them into a spying unit?

In some ways, 8200 is Israel at its best and worst: a high-tech incubator that trains some of Israel’s smartest young people but effectively excludes minority Arabs — 20 per cent of Israel’s population — because so few do military service, which is compulsory for Jewish Israelis.

Tal, a student at Magshimim, the elite feeder programme for Unit 8200: 'They said at the end of the 11th grade you will get interviews to certain places [in the IDF]'

Unit 8200 also snoops on Palestinians living under Israeli occupation in the West Bank or naval and air blockade in the Gaza Strip, according to a whistle-blowing leak that created a stir last year. In an open letter in September 2014, published by Israel’s Yedioth Ahronoth newspaper and broadcast on Channel 10, a group of 43 serving and former 8200 reservists revealed what they said were coercive spying tactics being used on innocent Palestinians, including the collection of embarrassing sexual, financial or other information. One of the whistle-blowers, in a statement released along with the letter, described his “moment of shock” when watching The Lives of Others, the 2006 film about the Stasi’s pervasive spying in East Germany.

The furore has calmed since then, but I wanted to find out more about 8200. Officers in the unit are not allowed to discuss their service, even with relatives, and are limited — as I discovered — in what they can say after they leave it. I was, however, allowed to interview the Magshimim graduates. When I asked Tal what he wanted to do after military service, he answered somewhat overeagerly, as if I were interviewing him for a job: “I would like to form my own company, or join an existing company with an important role.”

Though forbidden from discussing it, many 8200 veterans are happy to drop the unit’s number freely in the corporate world as an elite calling card. It is fair to describe shmone matayim as one of Israel’s most powerful business brands. Gil Shwed, co-founder of Check Point, Israel’s largest cyber security company, was in 8200, as was Avi Hasson, Israel’s chief scientist, whose office dispenses risk-free government loans to technology start-ups. The 8200 alumni association, with more than 15,000 members, hosts networking events and community outreach programmes, including a start-up “accelerator” open to Arabs and ultra-Orthodox Jews, most of whom do not serve in the army. Team 8, a self-described cyber security “foundry” aimed at providing know-how for start-ups, was launched by former 8200 officers in Tel Aviv earlier this year, attracting Google’s Eric Schmidt as an investor. Isaac Herzog, head of the centre-left Zionist Union party, played up his past service in 8200 when campaigning to unseat Binyamin Netanyahu as prime minister in the recent election.

50% More than half of people in a recent poll said they were prepared to let the state monitor their online activity if it helped boost national security

But what does 8200 actually do? Israel, as Netanyahu never tires of saying, lives in a “bad neighbourhood” in the Middle East, surrounded by several countries it classifies as enemy states. This requires world-class hacking and artificial intelligence tools as warfare moves from conventional battlefields — land, sea and air — to include cyber terrain. This new theatre of operations needs both offensive and defensive tools. According to some media reports, which the IDF won’t confirm, the unit was responsible for the Stuxnet computer worm deployed in 2010 against Iran’s computers, including ones at its nuclear facilities.

Alongside countries such as Iran — which itself has formidable cyber capability — various non-state hackers also have Israel in their sights. Over the past three years, pro-Palestinian “hacktivists” grouped under the #OpIsrael banner have targeted Israeli government websites and public institutions’ computer systems. The 2013 attack fell on the eve of Holocaust Remembrance Day, and in that and subsequent attacks, some of the hackers have threatened to unleash an “electronic Holocaust”.

According to intelligence analysts, 8200’s remit is similar to that of the NSA or Britain’s Government Communications Headquarters, covering everything from analysis of information in the public domain to use of human operators and special signal intelligence. Its geographical remit is primarily outside Israel but it does include the Palestinian territories.

“Unit 8200 is probably the foremost technical intelligence agency in the world and stands on a par with the NSA in everything except scale,” Peter Roberts, senior research fellow at Britain’s Royal United Services Institute, told me. “They are highly focused on what they look at — certainly more focused than the NSA — and they conduct their operations with a degree of tenacity and passion that you don’t experience elsewhere.”

Apart from 8200, the IDF also has other technological and spying units with their own cadres of alumni in business: a large air force intelligence unit, C4I, its telecommunications, computer and information technology unit, and smaller intelligence units so secret that Israelis will not utter their names. And last month, the Israeli military announced it would be forming a new “cyber command” to combat new challenges in online warfare.

The culture of Unit 8200 resembles that of a start-up, according to former officers. Soldiers work in small groups, with limited resources, to crack challenges that — literally, in some cases — are life-and-death matters. Disruptive behaviour and challenges to authority are encouraged, even if this means defying senior officers. “In intelligence, you can’t work only by rules, you need to be open-minded,” said Rami Efrati, a former 8200 senior officer and serial entrepreneur who is on his third start-up, Firmitas Cyber Solutions. “We teach them how to work out of the box.”

Nir Lempert, chairman of the Unit 8200 alumni association and CEO of Mer Group: 'I took with me from this unit the understanding that we must fulfil the mission'

Nir Lempert, chairman of the Unit 8200 alumni association and CEO of MER Group, a mobile communications infrastructure company, said: “I think the best premise I took with me from this unit was the ability to manage big activities in uncertain situations with a lot of question marks regarding the environment — and the understanding that we must fulfil the mission.”

A growing focus in 8200, as in other spy agencies, is data mining, and specifically the ability to shift through mountains of information to find the one menacing email, or the recurring patterns that suggest something is awry. To get a clearer idea of the tools the unit uses in its work, one afternoon I went to Tel Aviv University to meet Oded Maimon, one of the world’s foremost experts on data mining and artificial intelligence — teaching computers to do not just what they have been told but to predict things that haven’t happened yet. Maimon has written 10 books and edited a 1,500-page tome called the Data Mining and Knowledge Discovery Handbook. Like other Israeli mathematics professors, he has worked for both the intelligence services and the private sector. In the past, he advised Verint, an Israeli-founded video-and-audio-monitoring company now based in Melville, New York. In 2008 he was awarded a medal by Mossad for services to the nation. He rarely gives interviews but he invited me to his office.

The first step in Israel’s intelligence work, he told me, was to obtain raw information. “This I won’t talk about,” he said, but went on to acknowledge that “8200 is very important here”. Once intelligence is gathered and organised into a database, an analyst needs to look for a common denominator. This is what big data experts call fusion: the ability to make sense of, for example, an object spotted from different angles by different means — maybe a drone in the air, a camera on the ground, or a listening device in a phone. Humans do this naturally, using their five senses and grasp of context, but computers have to be taught. One intelligence source might have identified somebody talking in a car on a phone while another, using a camera on a plane, identifies the same car. “You create a knowledge base,” Maimon said. “You now know not only that a person is in a vehicle but you have the information that his phone is interesting to you.”

Analysts can then apply data mining algorithms to this “knowledge base” — determining, for example, from a base of several million conversations, which two are relevant. Algorithms can also do what Maimon calls “data compression” — for instance, establish that a target makes calls every day at 7.30am and 4pm. This can then be matched with other intelligence. “Finding a modus operandi is important,” he said. Only at the end of this process is human intervention needed. The professor does not spell this out but presumably the options available might include an arrest, a drone strike or another military operation.

I asked Maimon about the “refuseniks” and last year’s protest letter. “I don’t want to comment. I don’t know the details,” he said. However, he added: “In general, one should be very careful. If I give you a knife, you can use it to cut your salad, but you can do other things with it, too.”

Gilad, former 8200 lieutenant: One of 43 serving and former 8200 reservists who last year signed a whistleblowing letter alleging coercive spying practices on Palestinians

About a week later, in another part of Tel Aviv University, I met Gilad, a 29-year-old philosophy student, one of the veterans of Unit 8200 who had signed the letter. He is affable and smart, the kind of young man who makes Israelis proud. He excelled in physics and maths in high school in northern Israel, and was drafted into 8200 in 2003, ranking as lieutenant by the time he left in 2009. “I felt like I was doing something important, something challenging, something I would learn from and something meaningful for my country,” he said.

Over time, though, Gilad became troubled by the intrusive methods being used against Palestinians in the West Bank and Gaza. The refuseniks say they were asked to gather information not only on people suspected of plotting to harm Israel but on their family members, neighbours and others who might supply information about them. This included information about medical conditions, financial problems and sexual orientation — a sensitive topic in deeply conservative Palestinian society. One of them said that during his training for 8200, he had been assigned to memorise different Arabic words for “gay”. Another said that soldiers would call one another over to listen when one of their targets was discussing a “funny” medical condition such as haemorrhoids.

From the protest, a picture emerged of bright young Israelis, still in their teens and twenties, making decisions that would affect the fate of Palestinians years older. “In a way, this power is intoxicating,” Gilad told me. “You get inside people’s lives and you laugh about their sexual habits or medical problems. And it shows how far it goes. It shows you how power can corrupt.”

Israel withdrew its military from much of the West Bank in the early 1990s and all of Gaza in 2005, but its forces can still enter Palestinian Authority-controlled areas for arrests or other security operations. Palestinians in both territories depend on permits to travel into Israel or Jerusalem, giving Israeli authorities, the protesters said, the ability to barter for information.

“It’s one thing to spy on Iranians or Syrians, another to spy on Palestinians, because they are subjects of Israel,” Gilad said. “It’s more like spying on your own citizens.”

When I mentioned the protest to RUSI’s Peter Roberts, he pointed out that rival spy agencies used similar tactics during the cold war. Russia, he said, still uses methods such as “honey traps” to ensnare targets. “The Israelis live in a different security environment from the rest of us,” he said.

Oded Maimon, data mining and artificial intelligence expert, Tel Aviv University: 'One should be very careful. If I give you a knife, you can use it to cut your salad, but you can do other things with it, too'

Gilad and his fellow protesters did things by the book: they remained anonymous. (Gilad would not allow me to print his surname or photograph his face.) They showed their testimony to the military censor before going public. (The censor approved publication, except for the use of the signees’ full names.)

Nevertheless, their exposé, which came barely a month after last summer’s Gaza war, caused anger, much of it directed at the whistle-blowers themselves. Some of it carried a tinge of class resentment: here were some of Israel’s most privileged youth turning on the country’s most respected institution: the military. Moshe Ya’alon, Israel’s hardline defence minister, said the refuseniks would be “treated like criminals”.

In fact, no charges were brought against them, although in January they received letters saying they were no longer reservists. Gilad is now writing his degree thesis on freedom of speech issues, and says he is disappointed that so little has emerged from the protest. “The thing that bothers us is that no one faced the content of what we said,” he said. “We didn’t say that Israel was a bad nation or Israelis are evil — we didn’t say that. [But] people thought it was a threat to Israel’s legitimate self-defence. They didn’t treat our criticism seriously.”

Less than a year later, the refuseniks’ protest is all but forgotten — but Israel’s military-industrial-cyber complex is moving from success to success. If the burgeoning cyber park in Beer Sheva develops as its backers wish, it could — as grand nation-building Israeli projects go — one day rival the building of Tel Aviv by early Zionists on the dunes north of Jaffa starting a century ago.

“Israel, at a national level, needs to be excellent in cyber,” Yoav Tzruya, a partner in JVP Cyber Labs, one of the office park’s first tenants, told me recently. “Unfortunately, we are getting attacked again and again — our banks, our critical infrastructure, our government.” JVP, a Jerusalem-based venture capital firm with $1.1bn under management, is hosting and incubating promising companies at the park in Beer Sheva, about half of which are founded from entrepreneurs with backgrounds in the IDF or related agencies.

Israel needs to be excellent in cyber. We are getting attacked again and again — our banks, our critical infrastructure, our government

One of JVP Cyber Labs’ early investments was CyActive, one of whose principals, Shlomi Boutnaru, spoke at the Magshimim graduation I attended. The company is at the cutting edge of cyber defence, with “predictive software” designed to anticipate hacking threats that do not exist yet. “The same way biologists try to predict what next year’s flu will be, we have programmes that can predict what today’s malware and threats will develop into,” Liran Tancman, CyActive co-founder, told me in an office so new it did not yet have all its furniture.

If Israel’s well-honed hacking, spying and cyber skills, developed in the military, can be deployed in the private sector to darker effect, few are taking notice. Privacy International, a human rights watchdog group, recently reported that two multinational companies with Israeli roots, Verint and Nice Systems, were supplying surveillance technology to repressive Central Asian countries, allowing “unchecked access to citizens’ telephone calls and internet activity on a mass, indiscriminate scale”. (In response to the report, Verint said that it only did business with countries with which Israel had commercial ties and in accordance with government regulations; Nice did not comment.)

The report passed almost unnoticed in Israel, where concerns for security trump demands for privacy. More than half of people in a recent University of Haifa poll said they were prepared to let the state monitor their online activity if it helped boost national security. By 2020, when 8200 and the IDF’s other technology and intelligence units will have moved to Beer Sheva, the links that already exist between the military, the academy and business will be visible in the city, which Israeli officials want to develop as an alternative to Tel Aviv and its sky-high property prices. Cyber, Netanyahu said recently, is “changing the face of the Negev”.

John Reed is the FT’s Jerusalem bureau chief

Photographs: Yaakov Israel