ELLIE works on a car engine assembly line. Despite her best efforts, she occasionally forgets to screw in one of the bolts, but a laser beam from the ceiling highlights the loose piece as the engine passes, reminding her to tighten it.

This imagined scenario could soon be reality. A robot called Watch-Bot can watch people work, learn the steps that make up the task, then remind people when they forget a step. In 24 trials watching humans at work in the office and kitchen, Watch-Bot was about 60 per cent accurate in shining a laser pointer at the missed task, such as returning milk to the fridge.

Ashutosh Saxena and Chenxia Wu, at Stanford University in California and Cornell University in New York, built Watch-Bot by mounting a depth-sensing camera on a tripod and connecting it to a computer and a laser pointer. The robot learns unaided, finding patterns in human movements it observes. “The good thing about unsupervised learning is you don’t need to annotate a lot of data, which means it’s cheaper,” says Yezhou Yang at the University of Maryland.

Watch-Bot could help people with daily activities – not just annoyances like taking your keys with you, but safety issues like turning the stove off. Saxena and Wu see their bot someday helping elderly people live independently.


“Watch-Bot could help people remember daily activities like remembering to turn the stove off”

Something like Watch-Bot could even be used in homeland security or the military, says Ronald Arkin at the Georgia Institute of Technology in Atlanta. “One of the problems is recognising hostility or hostile intent. Could these systems recognise what hostile intent was? If they could do this unsupervised, it would be very interesting.”

Saxena and Wu are not comfortable trusting their bot with tracking vital safety tasks until they can improve the accuracy. They also want to find out if the robot fills a need and whether people like it.

The beauty of Watch-Bot is its simplicity, says Saxena. “A robot with a camera is very easy to manufacture and ship. All a company needs to do is take that robot and let it sit there. Just by doing that, the robot can become useful over time.” After about a week of sitting in a workplace or home, Saxena thinks Watch-Bot will learn enough to give helpful reminders.

This article appeared in print under the headline “Bot that watches while you work can catch slip-ups”

Image credit: Andrey Rudakov/Bloomberg via Getty Images