As Gaga walked over to the rose-gold keyboard that was planked atop two robotic arms, the piano moved up to meet her hands as she started to play. The setup swayed as Gaga belted out Bowie's "Suffragette City." The mechanical arms moved in and out to the rhythm of the song, all while keeping the keyboard steady enough for the singer to control the keys.



To pull off that choreographed piano, team Gaga enlisted the help of Andy Robot, a Las Vegas-based roboticist and computer animator working with NASA's Jet Propulsion Laboratory. Robot (yes, that's his name) repurposed some software he had built to program and animate two industrial bots for the performance. He followed Gaga's vision for an instrument that had a new dimension –- it was alive. But the process that made it possible hadn't been tried in this context before. So when the robotic team ran into a problem during rehearsals, Robot's association with Brian Lim, head of JPL's Planetary Landing Testbed initiative, led to a solution that involved a small piece of rubber.

I spoke to Robot and Lim to know more about the robotic star and the rescue mission.



What kind of machines did you bring in for the robotic keyboard?



Andy Robot: We used ABB's robots. They're typically used in manufacturing. So you'll see these robots, hundreds of them, working in unison in an assembly line creating BMWs in car factories. We used these because they're very robust. They're meant to operate for 30 years at full speed, in the dark with no heat, with zero loss of accuracy. When you compare that type of robot to a homemade robot or a one-off bot, it has to be the robot doing mission-critical manufacturing routine across the world. We applied it to art at the Grammys because it's similar to a mission-critical process where it can't go down; it has to keep moving.

Lady Gaga plays her keyboard during rehearsal. [Gif from Intel's behind-the-scenes footage.]

How did these industrial robotic arms turn into a dancing instrument?



AR: The two robots moved in synchrony so that they didn't tear the keyboard apart. Imagine if one robot went too far to the left -- it would tear the keyboard. We used the song itself. Once [Gaga] had the music done, we timed the robots by animating them with the music and synchronizing them to move in perfect unison. We use inverse kinematics -- that's the basis of the skeletal system for any human or animal. For example, if you reach and grab your wrist, your hand moves your arm. You're not thinking about how your shoulder moves or your elbow bends, you're thinking about where your hand is going. We used that system to make the keyboard the parent of the two robots. We were able to animate the keyboard in the computer program and move it around and both the robots calculated their own skeletal system motions to follow the parent's movements.



So imagine if I took your palms and glued them to the bottom of the keyboard, I'd be able to move the keyboard and your arms would naturally calculate the rotation in each joint. The robots did the same using the IK system.



Industrial bots are incredibly efficient machines. But their power is often perceived as dangerous. Were there risks involved in bringing these bots to a new artistic environment?



AR: We have a great deal of experience in terms of artists working with robots and interacting with them. The first thing we do is a safety review. Although it looks dangerous, we calculate all the risks; we can never eliminate them, but we can minimize them. A robot never wants to hurt a person. It's very rare but any time someone is hurt by a robot, unfortunately, they're not following procedure. As robots become more prevalent, the common-sense component will make them safer in terms of human interaction. One of the reasons Gaga was standing was so that if there were some push against her she wouldn't be pushed away. Protecting the pop star and the adjacent personnel is the No. 1 priority -- that's with anything in show business.

"A robot never wants to hurt a person."

Since you were trying to adapt these robots to a new environment (an instrument, in this case), did you run into any specific problems leading up to the performance?



AR: During rehearsals we had some issues with the tension on the pink piano. There's no precedent for this so we had to come up with a new solution. Based on my work with JPL, I was able to get the help that was needed.



Brian Lim: At JPL, because we encounter so many problems with machines, what we're good at is recognizing what the stressors are. Because the robot positioning [on Andy's set up] was extremely precise, slight misalignment for any reason could generate extremely high and undesirable loads. The model keyboard Andy got for programming the robot was significantly softer and lighter than the actual keyboard used on the stage. So, when the final keyboard came in, it had more complex movements that caused the robots to fight each other, resulting in load errors that did not appear with the softer model of the keyboard.

How did you stabilize the bots?

BL: I suggested that the best way to solve the problem was to add compliance. Think of two very strong machines that are working together: Unless they're perfectly synchronized, which doesn't happen in real-world physics, you have to have compliance. So for Lady Gaga to be pounding her keyboard and not have a problem, the system needed flexibility so that the robots would not eat each other up. In the short time that they had to fix the problem, I suggested adding "squishiness" with a thin piece of rubber that gave some compliance between the robots and the keyboard. This allowed the robots just enough give and allowed them to work together without screaming, "Error!"

The rose gold robotic keyboard. [Image credit: LG]

Working in an environment like the Grammys, where we have six minutes to essentially set up a factory of robots that would typically takes weeks to install, is challenging. We had to roll in the robots that were in off position, calibrate them and bring them up to speed in under six minutes before we came back from the commercial and had the countdown from the director. You have to institute and build a robotic system that would be ready to perform in that time frame in front of millions of people on national television.It's heart-stopping. Plans go out the window when you have one band unloading and the next loading in. One is performing on left; the other is loading in on the right. Typically, loading at concerts takes hours. It's very tedious and precarious. We had checklists just like a pilot does in an airplane. And if we had problems we had disaster recovery plans in place.Another challenge was that stagecraft hasn't caught up with the technology. It goes back decades with workers who don't know what you're rolling in. They're used to rolling in props but they're not used to bringing in robots that are very sophisticated and finicky. In the coming years, you're going to see stagecraft become more and more equipped for robotics. People who have the capacity to look forward to it and embrace it and also work with robotics to teach them about stagecraft will be successful. I learned so much from old-school stagehands about how this stuff works.It's all about amplifying the artists' capabilities and enabling them to express themselves on an exponential level. The artist has to use robotics in a way that is authentic to their art form. If they're trying to do it for the sake of robots, it doesn't work. But when the artist truly uses robotics systems as a longer plank for them to step out on to express themselves, the audience appreciates that. It's about making the art better.