Our Watch-Bot watches what a human is currently doing, and uses our unsupervised learning model to detect the human’s forgotten actions. Once a forgotten action detected (put-milk-backto-fridge in the example), it points out the related object (milk in the example) by the laser spot in the current scene. Credit: arXiv:1512.04208 [cs.RO]

(Tech Xplore)—At the International Conference on Robotics and Automation, researchers presented their work in creating Watch-Bot. Andrew Dalton in Engadget called it "a sort of robo-sentry."

Watch-Bot is designed to keep an eye on tasks in the home or office and remind you if one of those tasks is still not done—not with a beep, not with a soothing companion-like voice, but with a laser pointer to nab the object still needing attention.

Evan Ackerman in IEEE Spectrum said Watch-Bot can independently learn your household activity patterns in order to come up with its unfinished task reminders.

Core components of Watch-Bot are a 3D sensor (a Kinect, in this case), a camera that can pan and tilt, a laptop, and laser pointer, said IEEE Spectrum.

Parker Wilhelm in TechRadar said the Watch-Bot shines the laser pointer at the site of the error.

What kinds of tasks left undone will Watch-Bot catch?

You leave the milk out and good luck pouring it onto your cereal tomorrow morning. You forget to turn off your computer monitor. You forget to take yesterday's stew out of the microwave.

As described by Dalton in Engadget and Ackerman in IEEE Spectrum, it is apparent how impressive the powers of machine learning might be. Their research reveals actions based on probabilistic patterns observed.

Dalton wrote how Watch-Bot observed a week's worth of human activity in a kitchen and an office. During that time, it collected 458 videos. Half included a human deliberately "forgetting" to do something.

Ackerman rote that in its collecting 458 videos of normal activity, they were then annotated with 21 different actions and 23 types of objects. "In 222 of those videos, someone (deliberately, in the training case) forgot to do something."

Wilhelm said that the team tested the robot across eight offices and five kitchens, where it recorded data and established a baseline of normal behavior in these environments: preparing food, grabbing milk from the fridge.

Watch-Bot used its "unsupervised learning" algorithm to determine actions that were intentional and which ones were accidental. Unsupervised learning? Ackerman commented how "it figured that out on its own by using probabilistic learning models capable of detecting patterns and relations directly from the camera and Kinect data. This approach is based on what AI researchers call unsupervised learning."

The robot doesn't know what a microwave is, leave alone a neglected quiche. Fundamentally, said Ackerman, there's no semantic understanding of the scene. "All it's doing, using unsupervised learning algorithms, is tracking patterns to detect forgotten actions."

So how did the Watch-Bot do in picking up its neglected tasks? According to IEEE Spectrum, during testing, Watch-Bot was able to tell when humans forgot to do something (and successfully remind them) about 60 percent of the time.

Future directions from this research? It was a proof of concept of the underlying technology, which can be easily transferred to a variety of robots if equipped with an RGB-D sensor such as a Kinect and laser weapon, said Ackerman.

Wilhelm in TechRadar said "This could potentially include asbisting the memory of those struggling with neurodegenerative diseases like Alzheimer's, or increase safety in hazardous job sites where absent-mindedness carries dangerous consequences."

In the paper, "Watch-Bot: Unsupervised Learning for Reminding Humans of Forgotten Actions," submitted last year on arXiv server, they said, "Our simple setup can be easily deployed on any assistive robot. Our approach is based on a learning algorithm trained in a purely unsupervised setting, which does not require any human annotations. This makes our approach scalable and applicable to variant scenarios." The authors are Chenxia Wu, Jiemi Zhang, Bart Selman, Silvio Savarese, and Ashutosh Saxena.

Explore further Startup eyes industrial robotics payoff in random picking

More information: Watch-Bot: Unsupervised Learning for Reminding Humans of Forgotten Actions, arXiv:1512.04208 [cs.RO] Watch-Bot: Unsupervised Learning for Reminding Humans of Forgotten Actions, arXiv:1512.04208 [cs.RO] arxiv.org/abs/1512.04208 Abstract

We present a robotic system that watches a human using a Kinect v2 RGB-D sensor, detects what he forgot to do while performing an activity, and if necessary reminds the person using a laser pointer to point out the related object. Our simple setup can be easily deployed on any assistive robot.

Our approach is based on a learning algorithm trained in a purely unsupervised setting, which does not require any human annotations. This makes our approach scalable and applicable to variant scenarios. Our model learns the action/object co-occurrence and action temporal relations in the activity, and uses the learned rich relationships to infer the forgotten action and the related object. We show that our approach not only improves the unsupervised action segmentation and action cluster assignment performance, but also effectively detects the forgotten actions on a challenging human activity RGB-D video dataset. In robotic experiments, we show that our robot is able to remind people of forgotten actions successfully.

© 2016 Tech Xplore