Kindred, a San Francisco-based robotics and artificial intelligence startup, is bringing its human-assisted robotic arms to a big-name retailer. For the last six weeks, Kindred’s new production model robots, now called Kindred Sort, have been operating in a pilot program at a Gap warehouse, with plans to expand the fleet of robots to help the retailer’s full fulfillment network down the line.

These robots, guided by custom software but built by industrial robotics manufacturer Fanuc, are all operated with a blend of AI-assisted automation and human piloting. A group of six pilots now remotely operate the arms from Kindred’s new piloting center in Toronto. “Our whole premise is that we want to be in a physical body, interacting with the world to learn. We want to provide the intelligence from the cloud, and then provide human intelligence when needed remotely,” George Babu, Kindred’s chief product officer, tells me in an interview.

Kindred’s first commercial customer is the Gap, which is using robotic arms to sort products

Over time, algorithms should take over more and more complex tasks. But humans assist the robots along the way both to perform tasks the machine is incapable of performing today while generating data for the whole system to improve in the future, thanks to AI training techniques. “That’s the system we have put into production even at Gap,” Babu adds.

Right now, pilots are using Kindred’s software, built using the game engine Unity, alongside a standard keyboard and 3D mouse. Typically reserved for professional designers, the 3D mouse lets Kindred pilots send the robotic arm more easily in horizontal and vertical directions. Kindred pilots used to rely on VR headsets to operate the arm, but Babu says they found headsets to be uncomfortable at greater lengths of time. Kindred also considered using standard video game controllers, but it’s transitioned the system over to 3D mice because of how much more proficient the gadgets are.

Kindred has garnered interest in the past for its unique approach to AI. Created by D-Wave co-founder Geordie Rose and fellow former D-Wave employee Suzanne Gildert, Kindred wants to pave the path toward dynamic, human-like artificial intelligence, also known in the field as artificial general intelligence or “strong” AI. Yet only by giving this software a physical body does Rose and Gildert think that far-off milestone is actually achievable. Instead of focusing solely on research, Kindred has been putting resources toward a commercial goal: bringing automation to warehouse work. It’s using its learnings there to inform its AI research.

Related The next big leap in AI could come from warehouse robots

Rose and Gildert’s line of thinking is based on a theory known as embodied cognition. It’s been gaining traction in the academic community of late as AI experts team up with robot makers to apply cutting-edge software approaches to the physical realm. Now, the same techniques that have created DeepMind’s superhuman Go-playing machine — techniques like deep learning and reinforcement learning — are helping robots learn how to replicate human movement. By watching humans perform tasks and feeding that data into an AI system, roboticists are giving machines the ability to learn in much the same way a small child adapts to his or her body in the physical world.

To start, Kindred has focused on teaching its bots to move similar to the human arm and hand, which contain a large number of intricacies. The company’s robot arm right now is designed only to do the most tedious tasks a human warehouse worker might perform, which happens to be sorting a bundle of products by hand, grasping one item, scanning its barcode, and sending it on its way to the final destination.

But while even the smartest, most sophisticated software can perform specific tasks with extraordinary proficiency, machines are still terrible at maneuvering dynamic environments with ever-changing variables. To a piece of software, a fast-moving warehouse full of different products is a dizzying mess of data points. Complicating the problem is that robots, which are typically programmed to do only one very specific task, are terrible at even the most simple of human movements unless that movement involves extreme repetition. A robotic arm is used in the delicate task of fabricating nanometer-sized microchips, but the same machine would find it impossible to fish out a folded Gap T-shirt in a plastic bag from a bin of other products.

The problem is variability. Warehouse work may be mundane, but it changes too often to reliably employ robots for product picking and sorting. Products of different sizes, shapes, and textures are often bundled together, and they then need to be scanned and sorted. Human beings typically do this work by hand today, but Kindred’s model is designed to automate this in an unprecedented fashion. It uses humans for only part of the process, while letting the robots automate away the rest of the work.

Robots still need help doing simple human tasks like grasping objects

Right now, the most difficult part of the process for a robotic arm is grasping. In a warehouse setting, even for the Gap’s relatively homogenous product line, it’s still difficult to grab different articles of clothing, and even more complicated when shoes, belts, ties, and other garments and accessories are thrown in the mix. So a human pilot first grasps the item much like operating a toy claw machine, then lets go of the controls. The robot then automatically scans the barcode, and puts the product on a conveyor belt to be sorted further down in the fulfillment process.

“Mechanical Turk style, there are jobs to be done and resources to do the job. When they [human pilots] login, they become an available resource and get connected to a robot so they can help that robot,” Babu says. “When they’re done, they go back into the available queue and they get connected to a different robot. They’re a cognitive resource that’s available to any of our robots, whether that’s testing robots, prototype robots, or customer robots.” Kindred is now working on imbuing its robotic arms with even more human-like functions, including full-arm movement that is less strict and pre-programmed and more fluid and adaptable.

The company is charging the Gap, and plans to charge future commercial partners, not for individual robots, but for the robot’s time, as if a Kindred Sort were an ultra-productive employee of its own. “It’s pay-per-intelligent action. We’re not selling the robots, and it’s not a monthly lease either,” Babu says. “We’re trying to create the AI, so we price the use of the AI, and the hardware price is baked into that. Some tasks will be simple, so the price of that action is low. Some will be more complex, so the price will be higher — 10 cents, 20 cents, 30 cents.”

Babu says the complexity of a task is a product of a number of factors, including how long the action takes to perform, how many seconds the humans may need to assist the robot, and how much computing power is required. Kindred is also raising more venture funding, announcing today that it’s secured a $28 million investment led by Chinese media and web giant Tencent.

As for concerns that Kindred is actively looking to displace warehouse workers from their jobs, Babu says that the plan right now is not to eliminate jobs, but to move workers away from tedious, repetitive work and onto more challenging and rewarding tasks. “We had a lot of concern six months ago. Will people be afraid for their jobs?” Babu explains. “In practice, we’ve seen that people are sensitive to it — especially executives, they’re sensitive to scaring their workers. At the same time, they’re trying to tell investors, ‘Hey, we’re investing in the future, investing in supply chain, we want to take on Amazon.’”

Earlier this year, the company began a tour of the US to visit warehouses across the country and pitching Kindred Sort to both executives and workers, with the hope that firsthand experience of the gripping portion of the job the robot is designed to do will make it feel less intimidating. “The first thing is like, ‘Oh this is coming for my job.’ But not really... you won’t have this difficult job to do,” Babu says. “Then they’re like, ‘Ok, this is a cool robot,’ and they get really into it.”

Correction: An earlier version of this article stated that Kindred had only one customer with the Gap and was raising $20 million in venture capital funding. That is incorrect; Kindred has raised an additional $28 million, and it has pilot programs with a number of customers that it has yet to disclose.