And a collaboration with the US Army explores "the optimal balance between the authority of the Autonomous Systems and the Human Operator". Supporters say weaponised AI lacks the ugly aggression exhibited by some human soldiers and will become more accurate at detecting threats, completing work too "dirty, dull, or dangerous" for mortals. Opponents – including 122 artificial intelligence researchers who wrote last week to Prime Minister Malcolm Turnbull – want lethal autonomous systems banned. "If developed, they will permit armed conflict to be fought at a scale greater than ever before, and at timescales faster than humans can comprehend," the researchers wrote. Next week, experts will meet at the United Nations in Geneva, Switzerland, for the first formal talks on autonomous weapons. Already, research into their deployment has revealed insights into the way humans think about deadly choices.

What counts as 'autonomous'? Jervis Bay is known for the dolphins playing in its clear waters year round. But, one weekend next year, the holiday spot on the NSW South Coast will be infiltrated by other creatures. Defence scientists and firms from the Five Eyes nations – US, UK, Canada, Australia and New Zealand – will converge there to trial underwater, surface and aerial machines. Part industry show, part military exercise, the "Autonomous Warrior" challenge will test robots that look nothing like humans but "assess intent by probing/interaction". It is part of the Defence Force's plan to create "game-changing" systems that "enable trusted and effective co-operation between humans and machines".

Australia already uses unmanned drones and mine-clearing robots. The ADF has trialled the IBM supercomputer Watson (better known for beating humans at the US quiz show Jeopardy) to select targets of "psychological operations" which project Australian influence abroad. And this year Defence Industry Minister Christopher Pyne pledged $50 million for a collaborative research centre on "trusted autonomous systems". But other countries are much more advanced. The US-based Northrop Grumman has an unmanned drone that can take off and refuel on its own. The Russian arms manufacturer Kalashnikov has said it is building an artificially intelligent machine gun. And the US military last year said it was a decade away from being able to build an independent machine that could decide whom to kill (although it did not plan to).

What counts as autonomous? The term can be as slippery as "intelligent"; it lacks a single scientific or mathematical definition. Engineers grade autonomy against a scale and ADF researchers are aiming for Level 4 plus: "May be human piloted but never needs to be." So far Australia has been "very cautious" about military AI, perhaps reflecting a "deep-seated fear of losing control of the system", says Malcolm Davis, a senior analyst at the Australian Strategic Policy Institute. Davis argues countries including China and Russia are in an autonomous arms race and Australia should develop the weapons, even if it does not deploy them outside of a "war for survival". "If we get caught up in legal and ethical wrangling while our adversaries race ahead, then we could be at a severe disadvantage," he says.

Workers assembling robots at a factory in Shenyang. Beijing hopes to dominate cutting-edge technologies like artificial intelligence. Credit:YUYANG LIU For now, only humans can make distinctions between combatants and civilians, between active fighters and those who are surrendering or incapacitated. But that could soon change, according to Sue Keay, chief operating officer of the Australian Centre for Robotic Vision, which is developing a robotics "road map" with Defence. Citing "crazy" progress in the field, Keay predicts robots may be able to pick out combatants "at least as well as humans". One machine advantage could come from something they lack: emotion.

Flesh-and-blood soldiers seek revenge and commit atrocities. They may shoot prematurely, fearing for their own safety. "Why are we always assuming that human judgment is better?" asks Ron Arkin, a robotics professor at the Georgia Institute of Technology who has worked with the US Defence Department. Arkin believes robots could one day conform to international humanitarian law better than humans, clearing battle zone buildings with more restraint and fewer deaths. "I believe firmly that technology can assist in helping to protect non-combatant lives better than we are currently doing." Who makes the decision to shoot?

DefendTex's Tempest drone can carry 80 rocket-propelled grenades or the same number of miniature drones. Credit:Alex Ellinghausen One Australian company is pushing autonomy from a flipped perspective: rather than empowering machines to choose and fire on targets, DefendTex is working on making them smart enough to decide when they should not pull the trigger even if a target has already been designated. At an army "innovation day" in Canberra this week, DefendTex was showing off one of its latest creations: the Tempest, a fearsome-looking drone that can carry 80 rocket-propelled grenades or the same number of miniature drones that the company also makes. Those mini-drones can either act as spies in the sky or as one-use precision munitions themselves, turning into a swarm that can overwhelm an enemy. Four Tempests working together and launching their mini-drones would present such a formidable sight they may not even have to use their munitions, says the firm's electronic combat systems engineer Damien Cahill. "What's your reaction if there are 320 of these things coming at you? Self-preservation becomes top of mind and while you are concentrating on survival and avoiding the drones, you are not firing on our troops and have therefore been effectively taken out of the battle without firing a shot."

At the same time the firm is using machine learning – and has made an application for funding to the government's trusted autonomy research centre – to develop autonomous capability to disarm. CEO Travis Reddy says an autonomous machine will follow exactly the same laws and rules of engagement as a person. They won't strike schools or hospitals for instance, he says. The purpose of autonomy is making sure the machine can carry out the military commander's intent on a shifting battlefield. That means the machine must be able to make some decisions on its own once it's out there. "If the machine is executing a completely pre-programmed path, it will result in mission failure because the battlefield is dynamic and things are changing," Reddy says. "It's about how do we take those changing parameters and execute the commander's intent without violating the rules of engagement?" But how far do we allow that independent decision making to go? Does it include killing without human involvement? Never, Reddy says.

"We never foresee a situation where a machine should be able to identify a target and initiate contact without a person being involved. This is the view of DefendTex," he says. Part of an autonomous drone tracking test at the US Joint Base, Cape Cod. Credit:New York Times But there are ways an autonomous machine could have more involvement in finding targets. Reddy explains that there is an adjustment to the oft-used "human in the loop" phrase – a variation known as "human on the loop". This means that instead of having a drone like a Predator that is entirely human controlled and takes a small team of people, you might have a largely independent drone that must only go to a human for clearance before firing. It can pick its own targets, again based on the rules of engagement and all relevant laws. That would mean one person could oversee several drones rather than needing several people per drone. DefendTex isn't currently doing this kind of work. Reddy acknowledges that "there's no denying that the use of autonomy, if done poorly, has the ability to raise concerns", which is why the company is "doing the flipside".

"Let's look at the advantages of autonomy – reducing and negating collateral damage rather than finding new targets. We can make these platforms more ethical." Most obviously that means an autonomous weapon such as explosive mini-drone deciding at the last moment not to strike. Even precision-guided munitions, once they are fired, generally will detonate on their target. But what if a child walks into the frame at the last moment or, on close inspection, a seemingly valid target turns out to be something entirely innocent? The idea of "trusted autonomy" goes both ways, he adds. An exhausted battlefield commander who's been awake for 36 hours can accidentally call in a strike on the wrong target. An autonomous machine could decide not to trust the human and could ignore, or at least question, the order. Is it all a slippery slope? Reddy says it is ultimately up to any military how it uses autonomy but if it can make the use of force more discriminating and precise, it is worth taking the step of handing over more responsibility to machines.

"We make sure it's only the combatants who are engaged," he said. "Life and liberty is protected. In our view those advantages far outweigh the disadvantages." 'No moral qualms against killing humans' Plenty of researchers are more pessimistic. Hundreds from Canada and Australia wrote to prime ministers Justin Trudeau and Malcolm Turnbull last week, calling for an international ban on "weaponised AI". "Autonomous weapons systems will be the perfect weapons of terror because they will have no moral qualms against killing humans," UNSW professor Toby Walsh told Fairfax Media. Walsh fears advances on robot swarm technology could lead to weapons of mass destruction. "They will allow us to scale warfare in a way that we've never been able to," he says.

The petitions, ahead of the first formal UN talks on a possible ban, oppose machines that do away with "meaningful human control". The UK has stipulated "operation of weapon systems will always be under human control". The US has said humans will always be "in the loop" to make the final decision. Major-General Kathryn Toohey, the army's head of land capability, says any autonomous system used by the ADF would follow the rules of armed conflict and adds that "the government's been quite clear that we will always have a human in the loop when it comes to armed UAVs [unmanned aerial vehicles]". She says the army is developing a robotics and autonomous systems strategy over the next 12 months. Ed Husic, federal Labor's spokesman on the digital economy and the future of work, says few governments worldwide were thinking enough about the impact of AI or about "boundary setting" and where the "no-go zones" should be.

"AI evangelists will hate this kind of talk but the reality is this: we need to ensure the tech remains a tool for good and a tool we always have control of." In the absence of global leadership, Australia should promote more thinking about it, he says. Meanwhile other Australian Defence research is exploring scenarios where machines might make up their own "minds". One 2017 study, "Human-Machine Integration, Legal & Ethical Decision Support", had as an objective to "identify the qualities needed for a fully or partly autonomous system to become sufficiently trusted by stakeholders to make or recommend life/death decisions". The Train Problem

In other words, look for a series of moral rules that could be programmed into a lethal robot. To do so, the military is twisting a 50-year-old philosophical thought experiment. Colin Wastell, a Macquarie University researcher specialising in decision making, has approached the ADF project by tweaking the classic Trolley Problem. The original asks people if they would pull a lever, diverting a trolley, or tram, from one set of tracks where it would crush five people to death to another, where only one person would die. Another version asks if it would be permissible to push onto the tracks a fat man whose body would jam the wheels and spare the others. In Wastell's version, an autonomous train can carry aid on a dangerous mission to a city ravaged by natural disaster. But the train would likely kill displaced people crossing the tracks. If 10 people were expected to be killed, how many lives would the aid need to save to justify the mission?

Before he puts scenarios to military personnel, Wastell posed this question to 80 to 100 university students of different ages and from various subject areas. "It was a shocker," he says of the response data. Answers ranged from one life saved to 20 million. It made no difference whether the people killed on the tracks were women or children. "We really haven't got guidance as to what number we should set for a machine making a decision," Wastell says. Even if a consensus emerges from experiments with defence force members, would that be enough?

Rob Sparrow, a professor of philosophy at Monash University, says it is no way to approach ethics. "If you're facing a moral dilemma, you can't escape that by just asking around, taking a bit of a poll amongst your friends," he says. Sparrow argues that even if machines lead to fewer deaths overall, their use would still not be justified, as fighting just wars goes beyond casualty numbers. "Sending a robot to kill someone expresses a profound disrespect for them," Sparrow says. With regular military technology, the instruction to fire can be traced back to an individual. With smart machines it becomes harder to apportion blame when something goes wrong. If a machine goes rogue or just makes a mistake and kills an innocent, whose fault is it? The battlefield commander, the software programmer, the manufacturer? It's a thorny legal question for governments that applies to self-driving cars as much as military drones.

Others worry leaders will be more likely to go to war if they do not have to send in troops and then watch coffins being unloaded at airbases. Or the machines could fall into the wrong hands. Greg Rowlands, a lieutenant colonel and project manager in the army, believes hacking may represent a "security nightmare". "ADF drones equipped with missiles could be 'compelled' to use those weapons against ADF force elements realising a form of forced autonomous fratricide," Rowlands wrote on a Defence blog. From drones to Skynet Driverless cars are already posing their own versions of the Trolley Problem. Credit:Christopher Goodney

The term "killer robot" brings to mind hellbent humanoids with red eyes, like the Skynet technology of the Terminator films. But many researchers say human-like intelligence is decades away, at least. "To the military's credit, they have a very realistic view of what can be achieved," says Jai Galliott, a defence analyst and ethicist at UNSW. Galliott believes more worrying are driverless cars, already posing their own versions of the trolley problem. And one of the most prominent voices against "killer robots", Elon Musk, is also the head of the driverless car pioneer Tesla. Loading

Galliott suggests the hype around autonomous weapons might be "very clever distraction on the part of multibillion-dollar corporations" such as Tesla. "My belief is that the AI Elon Musk is putting into his Tesla automobiles right now is probably going to kill a lot more people than any future 'killer robot' type AI might."