Research

There is a large variety of objects and appliances in human environments, such as stoves, coffee dispensers, juice extractors, and so on. It is challenging to program a robot for each of these object types and for each of their instantiations.

In this work, we present a novel approach to manipulation planning based on the idea that many household objects share similarly-operated object parts.

We formulate the manipulation planning as a structured prediction problem and design a deep learning model that can handle large noise in the manipulation demonstrations and learns features from three different modalities: point-clouds, language and trajectory.







Robobarista: Object Part-based Transfer of Manipulation Trajectories from Crowd-sourcing in 3D Pointclouds

Jaeyong Sung, Seok Hyun Jin, Ashutosh Saxena

In International Symposium on Robotics Research (ISRR), 2015

[PDF] [arXiv] [Dataset] Jaeyong Sung, Seok Hyun Jin, Ashutosh SaxenaIn, 2015









Deep Multimodal Embedding: Manipulating Novel Objects with Point-clouds, Language, Trajectories

Jaeyong Sung, Ian Lenz, Ashutosh Saxena



Jaeyong Sung, Ian Lenz, Ashutosh Saxena







Robobarista project won Blue Sky Ideas Award!

Video As the PR2 robot stands in front of the object it has never seen before, the robot is given a natural language instruction (manual) and segmented point-cloud. Using our algorithm, the robot was even able to make a cup of latte.