Sure, Alphago—a Google computer that plays the game Go—beat Lee Sedol, the world's reigning master of the game. AI once again effortlessly outmaneuvered us poor bags of flesh. The machine revolution is nigh!

Except there's one crucial thing AlphaGo couldn't do: pick up those black and white Go stones and put them down on the board. A Google programmer had to do that.

“Maybe the hardest part is not playing the game but moving the pieces,” says Siddhartha Srinivasa, a roboticist at Carnegie Mellon University. He's only half kidding. Srinivasa is an expert in robot manipulation—the art of grabbing, holding, and using objects. And this, it turns out, is the real challenge for our emerging Skynet. Robots are increasingly able to understand the world, but they're terrible at handling it. If robots are really going to start helping us out in everyday life, they're going to have to get more than smart. They're going to have to get physical.

As an example, take a look at the Amazon Picking Challenge. In this contest, robots had to grab loose objects—like a package of Oreos or a rubber duck—and put them in a container. The winner took fully 20 minutes to grapple with a mere 10 items. “Like watching paint dry,” as one observer noted. The other teams did far worse; a toddler could have beaten them all.

The physical world defeats our bots because it's been designed by and for humans. We're masterful at dealing with mess and uncertainty. We intuitively grok the behavior of stacks of crap, things that roll over on their sides. Bots don't. “Just look at your own desk,” Srinivasa says. “It's filled with clutter, because humans are expert at dealing with clutter.”

Today's workplace robots—like the droids that move stuff around in Amazon warehouses or the robots that weld parts on automobile assembly lines—work in super-clean “structured environments” designed to accommodate their potent but narrow set of capabilities. In other words, they're mollycoddled. When they reach to pick something up, we make sure it's exactly where they expect it to be. And when uncertainty arises, humans have to step in. Mercedes-Benz has lately been replacing some robots with humans because customers increasingly want their cars customized—and robots can't rejigger auto trim on the fly.

So how can we give these robots a hand? One approach is “soft pneumatics,” designed to cushion a grab at everyday objects, says Oliver Brock, head of the Robotics and Biology Lab at the Technical University of Berlin (which won the Amazon Picking Challenge). Another would be better guidance algorithms for navigating the hard-to-predict physics of, say, piles of apples or stacks of pens.

But either of those angles will require gathering tons more data on such objects—“orders of magnitude more” than we have now, says Stefanie Tellex, a Brown University roboticist. She's trying to get all the academic labs around the world that use one popular two-handed robot—known as Baxter—to network the machines together, so they can learn from one another. (Which, yes, sounds a little Skynetish.)

Now, one note of caution: Do we want robots to be nimble enough to fold origami? Machines like that could take over nearly any manual-labor or service job from humans. But they'd also be our helpmates. As Srinivasa points out, millions of people struggle with mobility problems as a result of issues ranging from spinal-cord injuries to just sheer old age. Dexterous robots could help them feed and clothe themselves. “I think it's really important that we enable these people to have dignity of life,” he says. Nimble bots could do that.

Plus, they could finally slap down their own Go pieces. Or petulantly wipe them all off the board in frustration when some human beats them, someday.