It is being described as the worst cyber hack of the US government in history – a huge personnel database of government employees pilfered by foreign spies. But yesterday it emerged that the vast trove of details stolen is likely to have included information people supplied to get security clearances – such as intimate details of their sex lives, drug use and finances.

All of this, it is feared, is now in the hands of a foreign power, which could use it to blackmail and extort officials and, ultimately, get hold of classified information.

The finger has been pointed at China (which has denied any role). But it is little wonder that cyber-spying was top of the agenda when President Obama sat down with Chinese officials in Washington this week. “We remain deeply concerned about Chinese government-sponsored cyber-enabled theft,” the US Treasury Secretary said of the wider problem. But it is not just America that has been hit. A man works on a security camera that was installed at Tiananmen Square in Beijing, November 1, 2013. Kim Kyung-Hoon/Reuters

The first detected breach of British government systems came in 2003, when an email purportedly from a Tibetan group was opened in the Foreign Office.

Hidden in a picture was a virus which allowed the hackers to get inside the network. In the following years, evidence that Britain was the target of a growing cyber espionage campaign increased.

It also became clear that not just government, but also businesses were being targeted for their valuable secrets.

In 2012, the then-head of MI5, Jonathan Evans, issued a warning about the scale of cyber espionage . In a speech he said that a “major London listed company” had lost a staggering £800m as a result of a “hostile state cyber attack”. It was a statement designed to shock – and it received extensive media coverage. But the victim was not identified. For the first time, the name of the company being referred to can now be revealed. A number of sources say that it was the mining giant Rio Tinto .

The company itself will not comment but the extractive industries have been a major target for Chinese spies as China has a vast appetite for raw materials to fuel its growth – including iron ore for the steel to build its skyscrapers, cars and for use in its factories. In 2009, Rio Tinto was engaged in negotiations which involved fixing iron ore prices with China over long periods for vast sums of money.

Renegotiation of these contracts was a game played for high stakes, notably as gaps opened up between the market price and the price that had been fixed. Rio Tinto staff were even arrested . China may have achieved its goal of lowering prices through a mixture of traditional means – such as pressuring staff – alongside cyber espionage at the crucial moment in order to ensure a “negotiated” shift to a pricing deal much more favourable to Beijing (although the final losses to Rio Tinto, experts say, may not quite have been as high as first estimated).

“There are now three certainties in life,” MI5’s then-head of cyber told me in 2013. “There is death. There are taxes. And there is a foreign intelligence system on your system.” despite this, companies are unwilling to discuss what has happened, partly out of embarrassment but also because they often still need to do business in China.

And it is not just the Chinese. In terms of volume of attacks on Western companies, they may be the largest player on the field but, as I was researching my book on computers and espionage, I found that one other country worried officials just as much.

The Chinese use big nets to trawl but are often sloppier and easier to spot. Russia’s hackers, by contrast, are more expert and operate below the radar.

Increasingly cyber espionage is combined with human espionage to research targets and work out whom to approach and how. One Russian espionage attack is said to have involved researching a business executive, leading to the conclusion that he was gay but not out of the closet. The hackers then sent him an email from a gay rights organisation which they suspected he would open since it looked as if it was sent personally to him. In fact it contained malicious computer code or malware. They then counted on the fact that, even if the executive did suspect it was malware, he would not be willing to go to his company’s IT department or security team for fear it would reveal his sexuality.

The Russian economy is heavily dependent on exporting energy, and the country’s intelligence services also appear to have prioritised this field. One major energy firm was told by the British government that it had something on its system.

Within a day, the company found that one computer was sending out a signal after a small number of individuals had been sent carefully researched emails. Forensics on the machine showed the malware had been in place for nine months. The company analysed the email, the target and the timing (correlating it with business transactions going on at the time). That left them 99 per cent sure that it was a state-sponsored attack by the Russians on behalf of their energy industry.

Russian spies are doing what they always did. The difference now is that they doing it in cyberspace. And of course, it would be naive to think that Britain is not carrying out cyber-espionage.

It, like the US, started spying over the internet as far back as the early 1990s.

British and American officials claim though that they do not carry out the kind of corporate espionage that others undertake (in which foreign companies are spied on to help their own companies). But they will spy on trade talks to support national economic “well-being” so the distinction is a subtle one that critics contest).

The White House had planned to confront China over cyber-espionage at a previous summit in 2013. “We were spring-loaded,” one former US intelligence chief said of that moment.

But hours before the summit opened, a top secret document was published which outlined America’s own offensive cyber operations. long with other material that was then emerging thanks to Edward Snowden, this changed the terms of debate and led people to ask who was really the most aggressive player in cyberspace. The plan to pressure Beijing fizzled out. Since then, however, the US has tried to increase the pressure. Britain has been less vocal.

The Snowden documents revealed the scale of the intelligence gathering machine that Britain and America have built to monitor the global flow of data. But what is rarely understood is that the same system of “bulk intercept” that is criticised for being “mass surveillance” is not just looking for people but also used to look for signs of cyber attack. This is done by looking for signatures of particular attacks and what kinds of stolen information are being moved around over the internet.

This gives the intelligence agencies a unique insight into what is happening but has also raised questions as to how far they go in informing and working with companies and the general public if they spot something bad happening to them. In the new world of cyber threats, spies may have to come out of their shadows a little more if they are still to protect us.

Intercept: The Secret History of Computers and Spies by Gordon Corera is published by Weidenfeld & Nicolson this week.