MapNet: An Allocentric Spatial Memory for Mapping Environments

João F. Henriques, Andrea Vedaldi

Presented at the CVPR - IEEE Conference on Computer Vision and Pattern Recognition, 2018.









Active Vision Dataset results (trajectory and position heatmap on the left, orientation heatmap on the right).



SLAM (Simultaneous Localization And Mapping) is crucial for robotics, but traditional systems cannot improve by learning from data.

A major step towards this end was taken recently, with several works proposing deep networks for self-localization from video. However, these are necessarily short-term – they only address the localization component of SLAM. Tackling the problem from another direction, deep networks have been proposed that create internal representations of the environment (a “map”), but they assume perfect knowledge of ego-motion, and thus it is not known whether these are useful for localization.

In this work, we propose MapNet, a deep network that implements a full, yet surprisingly small SLAM pipeline, with a map representation that is built entirely online.

The main new component is an allocentric spatial memory, a memory that maps one-to-one to world-space coordinates. We show that this memory enjoys very efficient localization and registration, as a dual pair of convolution/deconvolution operators.

Unlike traditional SLAM systems, the map does not consist of a point cloud or geometric model, but semantic embeddings that encode knowledge about each particular place. We show some semantic properties of this map on simple synthetic mazes.

We also show that these map operations are useful for localization in real 3D environments, outperforming traditional SLAM systems and ego-motion-only networks on long-term localization from low-frame-rate video. We perform experiments with a dataset of hours of recorded gameplay of the classic game Doom, and with the very recent Active Vision Dataset, which consists of real images captured from a robot.





ResearchDoom results (trajectory and position heatmap on the left, orientation heatmap on the right).



Files

Paper

Slides

Source code (PyTorch reimplementation)

Reference