In this tutorial, we’ll see how to use Tensorflow to make some Recurrent Neural Networks (RNNs). We’ll start simple, and increase the complexity step by step.

I’ll assume that you already have some simple experience with Tensorflow. If not, check this Get Started tutorial: https://www.tensorflow.org/get_started/mnist/beginners

I will also assume that you know what an RNN is, along with LSTMs and mini-batch gradient descent.

Why make a tutorial about RNNs then? Because I found the official Tensorflow RNN tutorial to be too vague, with too much pseudo-code.

Here is the regression problem we will try to solve: Given some floats, output their sum. It’s a simple problem, not necessarily the fittest for RNNs, but it will provide a good example.

Step 1: 1 LSTM cell, no batch

We can easily generate some data with numpy: an array X of length input_length containing float vectors. Y is the sum of those vectors, of length input_length as well.

We have two variables, a weight matrix and a bias vector. We’ll add a non-linear operation. I chose tanh, but feel free to experiment and try something else.

Let’s use a LSTM cell for the moment, as well as a classic GradientDescentOptimizer. Other options are available, and the Tensorflow API allows you to swap cells and optimizers simply.

Step 2: 1 LSTM cell, mini-batch

Now we use mini-batch. For this, we need to add another dimension to inputs and target. We also have to communicate the size of the mini-batch to our LSTM cell through state. Tensorflow will handle the mini-batch process inside the LSTM for us. Pretty handy ✌️

We also have to change the way we get the data_dict during execution, and everything’s good.

Step 3: Truncated backpropagation with mini-batch

Now we can add truncated backpropagation. To keep it simple, we will only take the output form the last cell. We could take all the outputs and compare it to the cumulative sum as well, but we’re not going to do this right now.

This step is fairly simple: just add a for-loop in the graph.

Coming soon: Stacking LSTMs, other datasets, and a classification problem. Also: how to quickly use Tensorboard.

Hope you’ll find this useful ✌️