What I’ve learned from studying Robotics in Japan for two weeks

Or, how to drink lots of sake and make an AI robot in under a week.

I’ve always desired to visit Japan since I was a child. I’ve always admired the culture, from their traditions to the cutting-edge technology. Moreover, right now one of my main interest is robotics. You can imagine how I reacted when I was accepted into the Tohoku Engineering Summer Program to study Robotics in Japan, at Tohoku University in Sendai.

Sendai.

Some colleagues of mine and I, along other 50 students from all over the world, had the opportunity to attend morning classes from various professors on many different fields of robotics, from molecular to space robotics. In the afternoon, we worked on different projects. I collaborated with the Space Robotics Lab, and the task of my team was to develop and build a mobile robot.

A picture of the campus.

The main engineering campus of Tohoku University is really amazing. A beautiful mixture of modern, well designed and minimalistic buildings in a green environment, full of trees and plants. I really enjoyed the exterior and interior architecture, that often gave a open space feeling, with glass walls and high ceilings, large common spaces, white and shiny interiors. I read some studies some time ago about the relation between university architecture and the overall performance of students, and I couldn’t help but notice how many of those areas gave a natural sense of peace and relax.

Japanese people are extremely nice and polite. Their organization is remarkable, and I loved living with them. The entire campus was very curated, from the cafeteria to the libraries. We had the opportunity to chat with the professors in a very friendly and informal way, a thing that I liked very much. They explained us how their system is more based on practice than pure theory. “We make the students work on a project as soon as possible, and ask them to make it work”, told us one professor while we were having lunch. This approach can give very good results, as happened with us. Facing a real world problem and trying to solve it with actual hardware and software makes you think about it in many different ways, highlighting the problems but motivating you to find effective solutions. Quoting Feyman, “What I cannot create, I do not understand”.

Detail of the Space Robotics Lab.

My favourite classes were the ones about Space Robotics given by prof. Yoshida. The idea of robotic space exploration is somehow romantic to me, as the desire for exploration that burns inside the human soul. I’ve had the opportunity to work with the Space Robotics Lab with a team of 6 students (3 from Sapienza University of Rome like me, 2 from Hong Kong University and one from Georgia University), to build a mobile robot that had remote control with a video stream of what the robot sees, but also had autonomous obstacle avoidance and navigation, and other features that were up to us. We had barely a week to work on it, spending the afternoon in the labs, but I believe we did a good job.

Our robot in all its beauty.

We worked with Lego Mindstorm, a kit from Lego that allows to build robots using sensors, motors and a microcontroller to control them. As a camera we used the Kinect, and the processing was made on a Raspberry Pi 3 and on a MacBook. Building the actual robot has been an harder task than we expected. We designed a simple differential drive robot, but building it with Lego pieces, trying to distribute the weight and balance it, but also make it stable enough to sustain the Kinect camera, a Raspberry Pi 3 and a power bank was not straightforward.

The actual communication between the Kinect and the MacBook was done using Processing, a language designed particularly for interactive computer graphics, from which I knew was very easy to use Kinect. Having the RGB and depth stream it was trivial to implement a remote control: the operator sees what the robot is seeing and simply commands it. We run several demos with obstacles and the results were satisfying.

A demo of the robot in action. From left to right: prof. Kazuya Yoshida, Norman Di Palo (myself), Matteo Colajanni.

Implementing autonomous navigation, obstacle avoidance and landmark detection and following was quite harder. I noticed a big flaw in the Kinect depth camera (or, at least, in our Kinect): it is completely blind up to one meter of distance (after all, it was not designed for these applications). Every obstacle in that range disappears, so the robot thinks the path is free. To fix this problem, we added ultrasonic sensors to the robot, that detected distances to obstacle with a good accuracy in that range. Based on those sensors information, I implemented a obstacle avoidance algorithm, that performed evasion maneuvers in the presence of obstacles, but until then the robot wandered randomly anyway. I added a computer vision algorithm that recognized and detected the coordinates (in the image) of the desired landmark (a red cone) based on color. The robot was then able to turn itself towards the landmark every couple of steps, while avoiding the obstacles.

Here’s a short video of autonomous navigation and landmark following.

The basic goals for the mobile robot behaviour were then achieved (not without troubles). But, since my major is focused on both Robotics and Artificial Intelligence, I decided to expand the software architecture by integrating more advanced deep learning vision models and natural language processing to give the robot commands in english language and make it understand further its environment.

I used the Tensorflow Object Detection model, trained on the famous COCO Dataset. This model generates bounding boxes for common objects that it is able to recognize. I added this output to the previous landmark-detection algorithm, so that the robot was then able to recognize and follow common objects, like laptops, chairs and people.

Example of object detection.

The overall behaviour was then controlled using simply text messages sent on Telegram, a famous messaging platform. I developed a chatbot that could receive text messages using the Python library telepot, and with the api.ai API I added language understanding, so the user could simply text the robot «go towards the table, please», and the robot will understand the request, look for a table and go towards it, while avoiding obstacles on the path.

Despite the different software parts of which the overall architecture was composed, the result was a easy-to-use robot that responded to simple commands in a intelligent way, combining techniques of robotics and AI.

Here’s a short video of the object recognition and language understanding capabilities.

I made a GitHub repository to share the code with my colleagues. You can check it out here, right now it’s still work in progress, probably in the near future I will add some stuff, especially a more detailed README.