The officer, Janis Dalins, has been developing the crawler as part of his PhD. “If a cop on the beat sees a dark web site, it lets this system know, it adds it to its database and keeps it updated," Mr Dalins says. “It’s definitely not to replace police. This is just a tool to put in an investigator's hands." The dark web is accessible only by using special software called Tor. It’s an underground internet that uses encryption to make users and creators of websites anonymous. That makes it attractive to criminal elements. In one example, Medicare card details were offered for sale on the dark web last year. In mid 2016, Mr Dalins and his Monash University PhD supervisor Dr Campbell Wilson fed a list of dark web URLs into a "crawler" – an automated program that scans websites.

They gave it two weeks to trawl the swamp. The "crawler" wasn’t programmed to seek out illegal activity, nor was it equipped with any tools for penetrating secure websites; neither skill was needed. Credit:MelisaTG / Flickr / Creative Commons On the dark web, crime wants to be found. About half of the 232,792 pages it scanned – only a small portion of the dark web - contained content of interest to police, including child exploitation material and sites advertising money laundering, identity theft and hacking services, as well as terrorism guides.

"We were polite and we respected basic website security," says Dr Wilson. "These sites want to be found. If you want to make money, you want to advertise your presence." Due to the huge number of pages, the team chose to manually sort only the top-level websites, categorising them and making notes about their content. Their findings were published in the Digital Investigation journal this month. The study was done as part of Mr Dalins' PhD and was not funded, nor initiated, by the AFP. But Mr Dalins has hopes of gaining funding for stage two of the project, which would feed his website database into an artificial intelligence program. The AI could then trawl the dark web and categorise websites automatically before alerting AFP officers when it comes across something potentially illegal.

But privacy advocates say it’s another step towards artificial intelligence (AI) policing and warrantless surveillance. "The powers of police, they are being really pushed," says Dr Monique Mann, an academic who studies the dark web. "In the context of child exploitation material maybe it’s acceptable, but I become concerned when we think of activists who use the dark web for quite legitimate purposes." Although it is not illegal, the potential for constant surveillance by artificial intelligence has privacy advocates very concerned. "Fundamentally it’s important that people have the ability to conceal their communications from interception from the government – we do have a human right to privacy," says Dr Mann, a Queensland University of Technology research fellow. Mr Dalins says he “tries not to be philosophical” about his crawler’s potential.