Tell me about Sketch-RNN, which is a recent Magenta experiment that lets you draw with a recurrent neural network—basically, you start drawing a pineapple and then Sketch-RNN takes over and completes it, over and over, in many different styles.

We were able to use a bunch of drawings done by people playing Pictionary against a machine-learning algorithm—this was [data from another Google AI drawing experiment made by Google Creative Lab,] Quick, Draw!

There are limits on the data. There’s only so much you’re going to get out of these tiny little 20-second drawings. But I think the work done by the main [Sketch-RNN] researcher, David Ha, was really beautiful. He basically trained a recurrent neural network to learn how to reproduce these drawings. He sort of forced the model to learn what’s important. The model wasn’t powerful enough to memorize the entire drawing. Because it can’t memorize all the strokes it’s seeing, its job is just to reproduce lots of cats or whatever, it’s forced to learn what’s important about cats—what are the shared aspects of cat drawings across millions of cat drawings? And so when you play with this model you can ask it to generate new cats out of thin air. It generates really interesting looking cats that look, I think, uncannily like how people would draw cats.

TensorFlow

I read that you’re working with Magenta to teach computers to tell jokes. What kind of jokes do computers generate? (That was not itself the first line of a joke.)

The project was very preliminary, very exploratory, asking the question: can we understand that component of joke telling which is about surprise? Especially punch-line-related jokes, and puns, there’s clearly a point where everything’s running along as normal, I think I know what’s going on with this sentence, and then, boom! Right? And also I think, intuitively, there’s a geometry to the punch line. It’s surprising if the building collapses on your head; [a punch line is] not that kind of surprise. It’s, like, oh, right, I get it! You know? And that sense of “I get it” is, I think, a kind of backtracking you’re forced to do to get it. So we were looking at particular kinds of machine-learning models that can generate these things called truth vectors that are trying to understand what’s happening semantically in a sentence and then, can we actively manipulate those to get a different effect?

And the kind of joke we were hearing about was … “The magician was so angry she pulled her hare out.” And the pun of hare and hair, and rabbit—you get it, right?

Yeah. But you have to know a lot about words and language to understand it.

Yeah, you have to know a lot. Not only did this model not tell any jokes, funny or not, but we didn’t actually get the code to converge.

What are you in the middle of trying to figure out with Magenta right now?

Trying to understand more of the long-term structure with music and also trying to branch out into another interesting question, which is: can we learn from the feedback, not from an artist, but from an audience?

This is looking at the artistic process as kind of iterative. The Beatles had 12 albums and every one of them was different. And they were all showing that these musicians are learning from feedback they’re getting from peers and from crowds, but also other things that are happening with other artists. They’re really tied in with culture. Artists are not static.

And this very simple idea: can you have someone making something with a generative model, putting it out there, but then taking advantage of the fact that the feedback they get? “Oh, that was good, that was bad.” That feedback that we get, the artist can learn from that in one way, but maybe the machine-learning model can learn from it as well, and say, “Oh, I see, here are all the people and here’s what they think of what I’m doing, and I have these parameters.” And we can set those parameters vis-à-vis the feedback, using reinforcement learning, and we’re working on that, too.

As I listen to music created with Magenta, I wonder: if you’re using data to train artificial intelligence, can the AI then create anything truly original, or will it just be derivative of what it’s been trained on, whether that’s Madonna songs or impressionist paintings, or both?

I think it depends on what we mean by original. I think it’s unlikely to me that a machine-learning algorithm is going to come along and generate some transformative new way of doing art. I think a person working with this technology might be able to do that. And I think we’re just so, so, so far from this AI having a sense of what the world is really like. Like it’s just so, so far away. At the same time, I think that a lot of art is original in another sense. Like, I do one more cool EDM song with the drop at the right place, that’s fun to dance to and is new, but maybe is not, like, creating a completely new genre. And I think that kind of creativity is really interesting anyway. That by and large most of what we do is sitting in a genre we kind of understand, and we’re trying new things, and that kind of creativity I think the AI that we have now can play a huge role in. It’s not reproducing the data set, right? It’s mixing things up.