neural-storyteller

neural-storyteller is a recently published experiment by Ryan Kiros (University of Toronto). It combines recurrent neural networks (RNN), skip-thoughts vectors and other techniques to generate little story about images. Neural-storyteller’s outputs are creative and often comedic. It is open-source.

Experiment

This experiment started by running 5000 randomly selected web-images through neural-storyteller and experimenting with hyper-parameters. neural-storyteller comes with 2 pre-trained models: One trained on 14 million passages of romance novels, the other trained on Taylor Swift Lyrics. Inputs and outputs were manually filtered and recombined into two videos.

Generating Romance

Using Romantic Novel Model. Voices generated with a Text-to-Speech.

Generating Taylor Swift

Using Taylor Swift Model. Combined with a well known Swift instrumental.

How does it work?

Train a recurrent neural network (RNN) decoder on romance novels. Each passage from a novel is mapped to a skip-thought vector. Conditions RNN on skip-thought vector & generate the encoded passage. Train a visual-semantic embedding between COCO images and captions. Captions and images are mapped into a common vector space. After training, embed new images and retrieve captions.

A selection of generated stories