visakanv-RNN — machine-generated husband chatter

Machine Learning can reproduce TEDtalks, Obama speeches and Death Metal band logos. Could it also replicate my husband’s ramblings?

About a month ago, I graduated from a 12-week coding bootcamp at General Assembly (SG). It was a wonderfully gruelling and intense process, and I now feel a tiny bit like Neo from the Matrix. It’s really cool to look at the source of things and be able to make sense of what’s going on.

Since then, between code katas and job interviews, I’ve had a lot of fun working on little side-projects. Here’s a story about one of my favourites.

It all began when I attended a Creative Coding meetup. I had just spent 12 weeks learning some rather practical skills, and I was eager to see how they could be used for mischief. The theme was Machine Learning.

There were 2 speakers, and their talks were fascinating. One of them, Rob Peart, wrote about how he used neural networks to generate death metal band logos:

Delightfully ominous. (Image: Rob Peart)

Sounded fun — I wanted to play!

Getting started (dry run)

I followed the installation instructions on Jeff Thompson’s tutorial, and referred to Assad Udin’s installation guide when I ran into trouble. Like Rob said, training on Shakespeare gets boring pretty quick. I didn’t really want to use images either, so I started looking around.

Finding other examples to learn from

I discovered two projects I really loved — samim’s TED-RNN (trained on over 4 million words!) and Obama-RNN (>700k words).

Now, it just so happens that my favourite human being in the world is 66.2% through a project to write a million words. That’s at least 662k words — good enough for what I wanted to do.

Getting started… for real this time

First, I exported the blog from Wordpress, getting a XML file with over 60,000 lines. Next, I spent about 2 hours cleaning it up. Removing the metadata was easy, removing formatting from within posts was harder. I got lots of regexp practice! Once that was done, it was training time. Training the Shakespeare data set took about an hour, so I wasn’t expecting this to take much longer… Boy, was I wrong. It took 5 full hours!

At long last…

Creating my first sample was super quick… and then I got a bunch of disappointing rubbish.

“Play I tencid a plistance of good how from a bit of tirely problems we imagine from that 4n.”

This didn’t look anything like the results from Obama-RNN… Time to actually read the documentation.

Digging into the details

In Karpathy’s article on RNN-effectiveness, he explains that setting the temperature changes how a sample is created.

Lower temperature will cause the model to make more likely, but also more boring and conservative predictions. Higher temperatures cause the model to take more chances and increase diversity of results, but at a cost of more mistakes.

That made a lot of sense. The samples below are 6000-character long chunks of text starting with “Let’s talk about procrastination”, at every temperature setting.

Displaying the information in a way that makes sense

I wanted a way to visualise what was happening so… word clouds! (No need to reinvent the wheel here, wordclouds.com is free and does exactly what it says on the tin.)