The deep web is a hive of illegal activity, rife with child pornography, drug deals and human trafficking.

But because it is 'buried' so deep it is considered out of the reach of mainstream search engines and many law enforcement agencies - until now.

The Defense Advanced Research Projects Agency (Darpa) has developed an engine dubbed Memex - a combination of memory and index - that not only scours content on this so-called dark net, but also identifies subtle patterns in activity.

Scroll down for video

The Defense Advanced Research Projects Agency (Darpa) has developed a search engine dubbed Memex - a combination of memory and index - that not only scours content on the so-called dark net, it identifies patterns and associations that may be too subtle to be recognised by human investigators

Memex was announced by Darpa last year and the agency recently gave Scientific American a preview of the software.

Darpa said: 'Today's web searches use a centralised, one-size-fits-all approach that searches the internet with the same set of tools for all queries.

'[But] common search practices miss information in the deep web - the parts of the web not indexed by standard commercial search engines - and ignore shared content across pages.'

Memex was designed to overcome these challenges by extending 'the reach of current search capabilities and quickly and thoroughly organise subsets of information based on individual interests.'

WHAT IS THE DARK NET? The dark net is a subsection of the deep web - the part of the internet that does not show up in searches or on social media. Most of the information on the web is far down on dynamically generated sites, unable to be found or seen by traditional search engines. This has been likened to dragging a net across the surface of the sea, missing much of the information in the depths. The dark net is used as a way of sharing information and trading goods, but the anonymous and encrypted nature of it has attracted large amounts of illegal activity. The Silk Road website, and its successor that were recently shut down, was used to sell drugs in exchange for Bitcoins. Other dark net sites let users share pornographic photographs, hacked information and credit card numbers. The Silk Road used an underground computer network known as the The Onion Router (TOR), which is a matrix of encrypted websites and servers that disguise the identity of users. It uses numerous layers of security and encryption, hiding the IP address and the activity of the user. Advertisement

It looks behind standard search results for patterns, links and similar behaviours.

The software scours all aspects of the web - including those hidden in the dark net - to create data maps that might reveal clues about illegal activity.

In particular, Darpa wants to use Memex to uncover human trafficking rings by searching for patterns in the number of online sex adverts being posted from certain regions, or porn sites featuring the same email addresses or phone numbers.

These patterns could reveal links that human investigators could miss, explained Scientific American.

'We're envisioning a new paradigm for search that would tailor indexed content, search results and interface tools to individual users and specific subject areas, and not the other way around,' said Chris White, Darpa program manager.

'By inventing better methods for interacting with and sharing information, we want to improve search for everybody and individualise access to information. Ease of use for non-programmers is essential.'

The Memex program gets its name from a hypothetical device described in 'As We May Think' - a 1945 article for The Atlantic Monthly.

It was written by Vannevar Bush, director of the US Office of Scientific Research and Development (OSRD) during World War II.

In the article, Memex was described as an analog computer that would supplement human memory.

It would store and automatically cross-reference all of the user's books, records and other information.

This cross-referencing, which Mr Bush called 'associative indexing', would let users quickly search large amounts of information, and gain insights from it.

Targeting the deep web is also an initiative being developed in the UK.

The Memex program gets its name from a hypothetical device described in a 1945 article for The Atlantic Monthly. In the article (pictured), Memex was described as an analog computer that would supplement human memory to store and automatically cross-reference all of the user's books, records and other information

In December, Prime Minister David Cameron (pictured) said a specialist unit was being set up to hunt down paedophiles using the dark net to share child pornography. The National Crime Agency and the Government Communications Headquarters (GCHQ) will use advanced technology to trace 'digital footprints' of criminals

In December, the UK government said a specialist unit was being set up to hunt down paedophiles using the dark net to share child pornography.

The National Crime Agency and the Government Communications Headquarters (GCHQ) will use advances in analysing images and communications to trace the 'digital footprints' left by the users who share them.

TRACKING CHILD ABUSE VIDEOS Facebook, Microsoft, Google, Twitter and Yahoo will use proposed digital fingerprints, or hash values, of thousands of known child sex abuse images to prevent them being shared. Google has also developed new technology that allows child abuse videos to be identified and blocked. Microsoft, Google and Mozilla have also said they will investigate ways of placing restrictions within their browsers to block people from accessing child pornography. Google and Microsoft have both also introduced a number of changes to their search functions globally. Google claims to have seen a five-fold reduction in the number of searches for child abuse images since these changes were made. However, these changes are thought to be partly responsible for driving much of the trade in child pornography into the dark net. Advertisement

Prime minister David Cameron said the new unit is aimed at 'shining a light on the web's darkest corners' as he announced a package of measures to tackle online child abuse.

The National Crime Agency estimates that around 20,000 people from the UK use secret or encrypted networks each day.

The dark net consists of a network of encoded websites that sit behind the publicly available websites and cannot be found with normal search engines.

It came to prominence in 2012 when the FBI made a series of raids on Silk Road - an online marketplace described as the 'eBay for illegal drugs'.

Figures compiled by the National Crime Agency suggest that use of the dark net rose by two thirds in 2012.

Hidden capabilities that let users email and host file storage through encrypted and anonymous networks are provided by services the The Onion Router (Tor).

Tor users currently represent 0.18 per cent of the total number of internet uses in the UK.

However, in a recent investigation, GCHQ and the NCA were able to track down a British man who had been maintaining chat rooms and websites in the Far East and Eastern Europe that were used to share child pornography around the world.

He was also offering advice to other paedophiles about how to hide their behaviour and was using software to keep himself anonymous.

Analysis by experts at GCHQ allowed them to trace the man and he was later sentenced to three years in prison for making and distributing indecent images of children.