In the summer of 2018, a small Berkeley-based robotics startup received a challenge. Knapp, a major provider of warehouse logistics technologies, was on the hunt for a new AI-powered robotic arm that could pick as many types of items as possible. So every week, for eight weeks, it would send the startup a list of increasingly difficult items—opaque boxes, transparent boxes, pill packages, socks—that covered a range of products from its customers. The startup team would buy the items locally and then, within the week, send back a video of their robotic arm transferring the items from one gray bin to another.

By the end of the challenge, executives at Knapp were floored. They had challenged many startups over six or seven years with no success and expected the same outcome this time. Instead, in every video, the startup’s robotic arm transferred every item with perfect accuracy and production-ready speed.

“Every time, we expected that they would fail with the next product, because it became more and more tricky,” says Peter Puchwein, vice president of innovation at Knapp, which is headquartered in Austria. “But the point was they succeeded, and everything really worked. We've never seen this quality of AI before.”

KNAPP's Covariant-enabled robotic arm in a live warehouse environment in Berlin, Germany. Jannis Keil

Covariant has now come out of stealth mode and is announcing its work with Knapp today. Its algorithms have already been deployed on Knapp’s robots in two of Knapp’s customers’ warehouses. One, operated by the German electrical supplier Obeta, has been fully in production since September. The cofounders say Covariant is also close to striking another deal with an industrial robotics giant.

The news signifies a change in the state of AI-driven robotics. Such systems used to be limited to highly constrained academic environments. But now Covariant says its system can generalize to the complexity of the real world and is ready to take warehouse floors by storm.

There are two categories of tasks in warehouses: things that require legs, like moving boxes from the front to the back of the space, and things that require hands, like picking items up and placing them in the right place. Robots have been in warehouses for a long time, but their success has primarily been limited to automating the former type of work. “If you look at a modern warehouse, people actually rarely move,” says Peter Chen, cofounder and CEO of Covariant. “Moving stuff between the fixed points—that’s a problem that mechatronics is really great for.”

A robotic arm in Covariant's office Elena Zhukova

But automating the motions of hands requires more than just the right hardware. The technology must nimbly adapt to a wide variety of product shapes and sizes in ever-changing orientations. A traditional robotic arm can be programmed to execute the same precise movements again and again, but it will fail the moment it encounters any deviation. It needs AI to “see” and adjust, or it will have no hope of keeping up with its evolving surroundings. “It’s really the dexterity part that requires intelligence,” Chen says.

In the last few years research labs have made incredible advances in combining AI and robotics to achieve such dexterity, but bringing them into the real world has been a completely different story. Labs can get away with 60% or 70% accuracy; robots in production cannot. Even with 90% reliability, a robotic arm would be a “value-losing proposition,” says Pieter Abbeel, Covariant’s cofounder and chief scientist.

To truly pay back the investment, Abbeel and Chen estimate, a robot needs to be at least 99%—and maybe even 99.5%—accurate. Only then can it operate without much human intervention or risk slowing down a production line. But it wasn’t until the recent progress in deep learning, and in particular reinforcement learning, that this level of accuracy became possible.