Runway asked me to come in to their office for a few hours each week over the summer to work on some language-related prototypes. This post is a summary of what I got up to. I also talk a little bit about what I think is special about Runway.

Porting my first model

I’m an educator and my students are mostly artists and designers. A recent focus of my teaching is doing creative things at the intersection of machine learning and the language arts, mostly using the Python programming language. If you’ve ever taught programming before, you’ll know that one of the biggest challenges for instructors is just getting the damn software installed and working on your students’ machines. The challenge is even tougher when you’re teaching machine learning: there are dozens of machine learning frameworks, and hundreds of frameworks built on top of those frameworks, each of which potentially has multiple versions, mutually-incompatible dependencies, and/or arcane installation instructions. Even downloading and installing pre-trained models can be a hassle: services like Tensorflow Hub aim to make the process easier, but actually end up adding yet another library requirement on top of everything you’ve already installed.

One model that I frequently recommend to my students is Google’s Universal Sentence Encoder. (In class, I usually start off the discussion of sentence vectors by telling students just to average word vectors in a sentence together to get sentence vectors. This works fine for many applications, especially poetic applications, but the Universal Sentence Encoder is a handy step up from that technique.) But in order to use this model, you have to go through the rigamarole of installing Tensorflow, SentencePiece, etc. This makes it difficult to use it as a drop-in replacement for other techniques, especially in introductory classes and workshops.

So the first project I took on at Runway was to make what the Runway folks call a “port” of this model. A Runway port is usually just a wrapper around an existing model that extracts the parts of the underlying code that perform prediction and puts them together in a simple web API, using a Flask-like framework called the Runway Model SDK. The interface of this web API — including the data formats and data types — follow a set of conventions understood by the Runway desktop application. The “porting” process itself is easy, even for beginner Python programmers, and getting the basic version of the model running in Runway’s local development mode ended up taking less than an hour. (The one tricky part of porting the Universal Sentence Encoder model was including an extra step in the build instructions to download the model from Tensorflow Hub at image build-time. Thank you Anastasis for helping me figure this out!) The source code for my port is open source and freely available, if you want to take a look.

(The Runway Model SDK is, essentially, a common set of conventions that machine learning researchers and engineers can use to make their models easy to use and interoperable. This is, in my opinion, Runway’s most interesting and important innovation.)

Sentences and t-SNE

Runway makes the output from models immediately visible in a “Preview” panel, which makes the interface friendly in a way that notebooks and command-lines are not. (You don’t have to go poking and prodding through directories just to see what happened.) Many of the models most prominently featured in the application seem to be chosen specifically for their immediate visual impact. The application comes with a number of different Preview modes, which are selected automatically based on the type of data returned by the model.

My problem when porting the Universal Sentence Encoder was this: the Model SDK didn’t seem to have a good match for the kind of data that the model outputs. The best solution I could figure out was to make the model return two arrays: the first is an array of strings (the sentences from the input text) and the second is an array of vectors (the encodings from the model), where the strings and their corresponding vectors share indexes in their respective arrays.

Simple enough! But Runway doesn’t have a built-in preview mode for this kind of data type, like it does for images and text. This means that if you were to use the model right now, nothing will show up in the output panel at all, even though the model is successfully returning data. So another part of my work with Runway was to propose how the preview panel should work when displaying this kind of data. The obvious solution, of course, is to reduce the number of dimensions in the vectors from 512 to two (with an algorithm like t-SNE) then use those dimensions as X/Y coordinates when plotting the sentences on the screen.

So I patched together a JavaScript t-SNE library, a library for panning and zooming, and a bit of p5.js, to make something that looked like this: