Deep Learning requires a lot of computations. It typically involves neural network(s) with many nodes, and every node has many connections — which must be updated constantly during the learning. In other words, at each layer of the network, hundreds and thousands of identical artificial neurons perform the same computation.

Therefore, the structure of a neural network fits very well with the kinds of computation that a GPU (Graphic Processing Unit) can efficiently perform — which are designed to compute, in parallel, the same instructions.

As the deep learning and AI fields have been moving extremely fast in the last few years, we’ve also seen the introduction of many deep learning frameworks. Deep learning frameworks are created with the goal to run deep learning systems efficiently on GPUs. They all rely on the concept of computational graphs — which define the order of computations that need to be performed.

What you have in these frameworks is a language that sets up the computational graph and an execution mechanism that’s different from the host language. Then the graph can be optimized and run, in parallel, in the target GPU.

In this post, I want to introduce to you the 5 frameworks that are the workhorses of deep learning development. They make it easier for data scientists and engineers to build deep learning solutions for complex problems and perform tasks of greater sophistication. These are just a small selection of a wide range of open-source frameworks, backed by different tech giants, which push each other to innovate faster.

1 — TensorFlow (Google)

TensorFlow was originally developed by researchers and engineers working on the Google Brain Team. Its purpose is geared towards deep neural networks and machine intelligence research. The library has officially been open-sourced on GitHub since late 2015. TensorFlow is extremely useful to do graph-based computations quickly. The flexible TensorFlow API can deploy models across multiple devices with its GPU-supported architecture.

In brief, the TensorFlow ecosystem has 3 main components:

The TensorFlow API, written in C++, contains the API to define the models and train the models with data. It also has a user-friendly Python interface.

TensorBoard is a visualization toolkit to help with analyzing, visualizing, and debugging TensorFlow graphs.

TensorFlow Serving is a flexible, high-performance serving system used to deploy pre-trained machine learning models in production. Also written in C++ and accessible with a Python interface, Serving is able to switch from old to new models instantaneously.

TensorFlow has been used widely in academic research and industrial applications. Some notable current uses include Deep Speech, RankBrain, SmartReply, and On-Device Computer Vision. You can check out some of the best official uses, research models, samples, and tutorials of TensorFlow at this GitHub repo.

Let’s take a look at a running example. Here I train a 2-layer ReLU network on random data with L2 loss on TensorFlow.

There are 2 main components of this code: defining computational graph and running this graph many times. While defining the graph, I create placeholders for input x, weights w1 and w2, and targets y. Then during the forward pass, I compute prediction for target y and the loss variable, which is the L2 distance between y and y_pred.

Lastly, I tell TensorFlow to compute loss of gradient with respect to w1 and w2. After getting done building the graph, I enter a session to run the graph. Here I create numpy arrays that will fill in the placeholders above, feeding them for x, y, w1, w2. In order to train the network, I run the graph over and over, using gradient to update the weights and getting numpy arrays for loss, grad_w1, and grad_w2.

Keras: The High-Level Wrapper

Deep Learning frameworks operate at 2 levels of abstractions: Low Level — where mathematical operations and neural network primitives are implemented (TensorFlow, Theano, PyTorch etc.) and High Level — where low level primitives are used to implement neural network abstractions, such as models and layers (Keras).

Keras is a wrapper over its backend libraries, which can be TensorFlow or Theano — meaning that if you’re using Keras with TensorFlow backend, you’re running TensorFlow code. Keras takes care a lot of the nitty-gritty details for you, as it’s geared towards neural network technology consumers and is well suited for those practicing data science. It allows for easy and fast prototyping, supports multiple neural network architectures, and runs seamlessly on CPU/GPU.

In this example doing similar neural network training as above, I first define the model object as a sequence of layers, then define the optimizer object. Next, I build the model, specify loss function, and train the model with a single ‘fit’ line.

2 — Theano (University of Montreal)

Theano is another Python library for fast numerical computation that can be run on the CPU or GPU. It is an open-source project developed by the Montreal Institute for Learning Algorithms group at University of Montreal. Some of its most prominent features include transparent use of GPU, tight integration with NumPy, efficient symbolic differentiation, speed / stability optimizations, and extensive unit testing.

Unfortunately, Youshua Bengio (head of MILA lab) announced in Nov. 2017 that they will no longer actively maintain or develop Theano. The reason is that most of the innovations Theano introduced across the years have now been adopted and perfected by other frameworks. If interested, you can still contribute to its open-source library.

Theano is similar in many ways to TensorFlow. So let’s take a look at another code example training neural networks using the same batch size and input/output dimensions:

I first define Theano symbolic variables (similar to TensorFlow placeholder). For the forward pass, I compute predictions and loss; for the backward pass, I compute gradients. Then I compile a function that computes loss, scores, and gradients from data and weights. Lastly, I run this function many times to train the network.