Practicing a task millions of times in just one hour

Replica can be loaded up in AI Habitat, a new open platform for embodied AI research. Facebook AI created AI Habitat to be the most powerful and flexible way for researchers to train and test AI bots in simulated living and working spaces. AI Habitat allows researchers to put a bot into a Replica environment, so it can learn to tackle different tasks, like "go check if my laptop is on my desk in the kitchen.” These chores are simple for humans, but for machines to master them, they must recognize objects, understand language, and navigate effectively. Today’s machines — like robotic vacuums, for example — can respond to commands, but they don’t understand and adapt to the world around them as people do. AI Habitat will help researchers develop bots that understand the physical world. But it is also an important research tool for creating next-gen AR experiences that begin to merge the physical and digital worlds. If we can teach an AI system to understand the physical space around you, we might one day be able to use it in combination with AR glasses. For example, it could help us to place your grandma's digital avatar in the seat next to you or to display digital reviews right next to a restaurant or store as you walk by.

Replica provides realistic 3D data, and AI Habitat provides simulation with speed and flexibility. While other simulation engines commonly run at 50 to 100 frames per second, AI Habitat runs at over 10,000 frames per second (multi-process on a single GPU). This enables researchers to test their bots much more quickly and effectively — an experiment that would take months on another simulator would take a few hours on Habitat. Facebook AI research intern Erik Wijmans, who is also a PhD student at Georgia Tech, and AI Resident Bhavana Jain used the system to do state-of-the-art research, training their bot with over a billion frames of experience. Using a rough estimate of how quickly people can look around and move in the real world, that would be the equivalent of more than 30 years of experience. A virtual bot can also bump into countless walls and make other mistakes as it learns, without any risk of doing real-world damage.

Facebook is now open-sourcing AI Habitat and releasing its Replica data set, so anyone in the community can build on it, try new approaches, compare their results, and learn from others’ work. (Technical details on Habitat are available here, and the Replica environments can be downloaded here.) This kind of open sharing of information between researchers at different companies and organizations has been key to recent advances in AI technologies like natural language understanding, computer vision, and embodiedQA, and researchers at Facebook AI and FRL believe the same will be true here.

To establish performance benchmarks that can be used by everyone in the field, Facebook AI also recently created the Habitat Challenge. The contest invited engineers and researchers from across the AI community to find the best way for bots to complete a particular navigation task in AI Habitat. “AI Habitat offers close to real-world experience for learning navigation,” says one of the challenge participants, Dmytro Bobrenko.