In the last chapter we learned that deep neural networks are often much harder to train than shallow neural networks. That's unfortunate, since we have good reason to believe that if we could train deep nets they'd be much more powerful than shallow nets. But while the news from the last chapter is discouraging, we won't let it stop us. In this chapter, we'll develop techniques which can be used to train deep networks, and apply them in practice. We'll also look at the broader picture, briefly reviewing recent progress on using deep nets for image recognition, speech recognition, and other applications. And we'll take a brief, speculative look at what the future may hold for neural nets, and for artificial intelligence.

The chapter is a long one. To help you navigate, let's take a tour. The sections are only loosely coupled, so provided you have some basic familiarity with neural nets, you can jump to whatever most interests you.

The main part of the chapter is an introduction to one of the most widely used types of deep network: deep convolutional networks. We'll work through a detailed example - code and all - of using convolutional nets to solve the problem of classifying handwritten digits from the MNIST data set:

We'll start our account of convolutional networks with the shallow networks used to attack this problem earlier in the book. Through many iterations we'll build up more and more powerful networks. As we go we'll explore many powerful techniques: convolutions, pooling, the use of GPUs to do far more training than we did with our shallow networks, the algorithmic expansion of our training data (to reduce overfitting), the use of the dropout technique (also to reduce overfitting), the use of ensembles of networks, and others. The result will be a system that offers near-human performance. Of the 10,000 MNIST test images - images not seen during training! - our system will classify 9,967 correctly. Here's a peek at the 33 images which are misclassified. Note that the correct classification is in the top right; our program's classification is in the bottom right:

Many of these are tough even for a human to classify. Consider, for example, the third image in the top row. To me it looks more like a "9" than an "8", which is the official classification. Our network also thinks it's a "9". This kind of "error" is at the very least understandable, and perhaps even commendable. We conclude our discussion of image recognition with a survey of some of the spectacular recent progress using networks (particularly convolutional nets) to do image recognition.

The remainder of the chapter discusses deep learning from a broader and less detailed perspective. We'll briefly survey other models of neural networks, such as recurrent neural nets and long short-term memory units, and how such models can be applied to problems in speech recognition, natural language processing, and other areas. And we'll speculate about the future of neural networks and deep learning, ranging from ideas like intention-driven user interfaces, to the role of deep learning in artificial intelligence.

The chapter builds on the earlier chapters in the book, making use of and integrating ideas such as backpropagation, regularization, the softmax function, and so on. However, to read the chapter you don't need to have worked in detail through all the earlier chapters. It will, however, help to have read Chapter 1, on the basics of neural networks. When I use concepts from Chapters 2 to 5, I provide links so you can familiarize yourself, if necessary.

It's worth noting what the chapter is not. It's not a tutorial on the latest and greatest neural networks libraries. Nor are we going to be training deep networks with dozens of layers to solve problems at the very leading edge. Rather, the focus is on understanding some of the core principles behind deep neural networks, and applying them in the simple, easy-to-understand context of the MNIST problem. Put another way: the chapter is not going to bring you right up to the frontier. Rather, the intent of this and earlier chapters is to focus on fundamentals, and so to prepare you to understand a wide range of current work.

In earlier chapters, we taught our neural networks to do a pretty good job recognizing images of handwritten digits:

We did this using networks in which adjacent network layers are fully connected to one another. That is, every neuron in the network is connected to every neuron in adjacent layers:

In particular, for each pixel in the input image, we encoded the pixel's intensity as the value for a corresponding neuron in the input layer. For the $28 \times 28$ pixel images we've been using, this means our network has $784$ ($= 28 \times 28$) input neurons. We then trained the network's weights and biases so that the network's output would - we hope! - correctly identify the input image: '0', '1', '2', ..., '8', or '9'.

Our earlier networks work pretty well: we've obtained a classification accuracy better than 98 percent, using training and test data from the MNIST handwritten digit data set. But upon reflection, it's strange to use networks with fully-connected layers to classify images. The reason is that such a network architecture does not take into account the spatial structure of the images. For instance, it treats input pixels which are far apart and close together on exactly the same footing. Such concepts of spatial structure must instead be inferred from the training data. But what if, instead of starting with a network architecture which is tabula rasa, we used an architecture which tries to take advantage of the spatial structure? In this section I describe convolutional neural networks* *The origins of convolutional neural networks go back to the 1970s. But the seminal paper establishing the modern subject of convolutional networks was a 1998 paper, "Gradient-based learning applied to document recognition", by Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. LeCun has since made an interesting remark on the terminology for convolutional nets: "The [biological] neural inspiration in models like convolutional nets is very tenuous. That's why I call them 'convolutional nets' not 'convolutional neural nets', and why we call the nodes 'units' and not 'neurons' ". Despite this remark, convolutional nets use many of the same ideas as the neural networks we've studied up to now: ideas such as backpropagation, gradient descent, regularization, non-linear activation functions, and so on. And so we will follow common practice, and consider them a type of neural network. I will use the terms "convolutional neural network" and "convolutional net(work)" interchangeably. I will also use the terms "[artificial] neuron" and "unit" interchangeably.. These networks use a special architecture which is particularly well-adapted to classify images. Using this architecture makes convolutional networks fast to train. This, in turn, helps us train deep, many-layer networks, which are very good at classifying images. Today, deep convolutional networks or some close variant are used in most neural networks for image recognition.

Convolutional neural networks use three basic ideas: local receptive fields, shared weights, and pooling. Let's look at each of these ideas in turn.

Local receptive fields: In the fully-connected layers shown earlier, the inputs were depicted as a vertical line of neurons. In a convolutional net, it'll help to think instead of the inputs as a $28 \times 28$ square of neurons, whose values correspond to the $28 \times 28$ pixel intensities we're using as inputs:

As per usual, we'll connect the input pixels to a layer of hidden neurons. But we won't connect every input pixel to every hidden neuron. Instead, we only make connections in small, localized regions of the input image.

To be more precise, each neuron in the first hidden layer will be connected to a small region of the input neurons, say, for example, a $5 \times 5$ region, corresponding to $25$ input pixels. So, for a particular hidden neuron, we might have connections that look like this:

That region in the input image is called the local receptive field for the hidden neuron. It's a little window on the input pixels. Each connection learns a weight. And the hidden neuron learns an overall bias as well. You can think of that particular hidden neuron as learning to analyze its particular local receptive field.

We then slide the local receptive field across the entire input image. For each local receptive field, there is a different hidden neuron in the first hidden layer. To illustrate this concretely, let's start with a local receptive field in the top-left corner:

Then we slide the local receptive field over by one pixel to the right (i.e., by one neuron), to connect to a second hidden neuron:

And so on, building up the first hidden layer. Note that if we have a $28 \times 28$ input image, and $5 \times 5$ local receptive fields, then there will be $24 \times 24$ neurons in the hidden layer. This is because we can only move the local receptive field $23$ neurons across (or $23$ neurons down), before colliding with the right-hand side (or bottom) of the input image.

I've shown the local receptive field being moved by one pixel at a time. In fact, sometimes a different stride length is used. For instance, we might move the local receptive field $2$ pixels to the right (or down), in which case we'd say a stride length of $2$ is used. In this chapter we'll mostly stick with stride length $1$, but it's worth knowing that people sometimes experiment with different stride lengths* *As was done in earlier chapters, if we're interested in trying different stride lengths then we can use validation data to pick out the stride length which gives the best performance. For more details, see the earlier discussion of how to choose hyper-parameters in a neural network. The same approach may also be used to choose the size of the local receptive field - there is, of course, nothing special about using a $5 \times 5$ local receptive field. In general, larger local receptive fields tend to be helpful when the input images are significantly larger than the $28 \times 28$ pixel MNIST images..

Shared weights and biases: I've said that each hidden neuron has a bias and $5 \times 5$ weights connected to its local receptive field. What I did not yet mention is that we're going to use the same weights and bias for each of the $24 \times 24$ hidden neurons. In other words, for the $j, k$th hidden neuron, the output is: \begin{eqnarray} \sigma\left(b + \sum_{l=0}^4 \sum_{m=0}^4 w_{l,m} a_{j+l, k+m} \right). \tag{125}\end{eqnarray} Here, $\sigma$ is the neural activation function - perhaps the sigmoid function we used in earlier chapters. $b$ is the shared value for the bias. $w_{l,m}$ is a $5 \times 5$ array of shared weights. And, finally, we use $a_{x, y}$ to denote the input activation at position $x, y$.

This means that all the neurons in the first hidden layer detect exactly the same feature* *I haven't precisely defined the notion of a feature. Informally, think of the feature detected by a hidden neuron as the kind of input pattern that will cause the neuron to activate: it might be an edge in the image, for instance, or maybe some other type of shape. , just at different locations in the input image. To see why this makes sense, suppose the weights and bias are such that the hidden neuron can pick out, say, a vertical edge in a particular local receptive field. That ability is also likely to be useful at other places in the image. And so it is useful to apply the same feature detector everywhere in the image. To put it in slightly more abstract terms, convolutional networks are well adapted to the translation invariance of images: move a picture of a cat (say) a little ways, and it's still an image of a cat* *In fact, for the MNIST digit classification problem we've been studying, the images are centered and size-normalized. So MNIST has less translation invariance than images found "in the wild", so to speak. Still, features like edges and corners are likely to be useful across much of the input space. .

For this reason, we sometimes call the map from the input layer to the hidden layer a feature map. We call the weights defining the feature map the shared weights. And we call the bias defining the feature map in this way the shared bias. The shared weights and bias are often said to define a kernel or filter. In the literature, people sometimes use these terms in slightly different ways, and for that reason I'm not going to be more precise; rather, in a moment, we'll look at some concrete examples.

The network structure I've described so far can detect just a single kind of localized feature. To do image recognition we'll need more than one feature map. And so a complete convolutional layer consists of several different feature maps:

In the example shown, there are $3$ feature maps. Each feature map is defined by a set of $5 \times 5$ shared weights, and a single shared bias. The result is that the network can detect $3$ different kinds of features, with each feature being detectable across the entire image.

I've shown just $3$ feature maps, to keep the diagram above simple. However, in practice convolutional networks may use more (and perhaps many more) feature maps. One of the early convolutional networks, LeNet-5, used $6$ feature maps, each associated to a $5 \times 5$ local receptive field, to recognize MNIST digits. So the example illustrated above is actually pretty close to LeNet-5. In the examples we develop later in the chapter we'll use convolutional layers with $20$ and $40$ feature maps. Let's take a quick peek at some of the features which are learned* *The feature maps illustrated come from the final convolutional network we train, see here.:

The $20$ images correspond to $20$ different feature maps (or filters, or kernels). Each map is represented as a $5 \times 5$ block image, corresponding to the $5 \times 5$ weights in the local receptive field. Whiter blocks mean a smaller (typically, more negative) weight, so the feature map responds less to corresponding input pixels. Darker blocks mean a larger weight, so the feature map responds more to the corresponding input pixels. Very roughly speaking, the images above show the type of features the convolutional layer responds to.

So what can we conclude from these feature maps? It's clear there is spatial structure here beyond what we'd expect at random: many of the features have clear sub-regions of light and dark. That shows our network really is learning things related to the spatial structure. However, beyond that, it's difficult to see what these feature detectors are learning. Certainly, we're not learning (say) the Gabor filters which have been used in many traditional approaches to image recognition. In fact, there's now a lot of work on better understanding the features learnt by convolutional networks. If you're interested in following up on that work, I suggest starting with the paper Visualizing and Understanding Convolutional Networks by Matthew Zeiler and Rob Fergus (2013).

A big advantage of sharing weights and biases is that it greatly reduces the number of parameters involved in a convolutional network. For each feature map we need $25 = 5 \times 5$ shared weights, plus a single shared bias. So each feature map requires $26$ parameters. If we have $20$ feature maps that's a total of $20 \times 26 = 520$ parameters defining the convolutional layer. By comparison, suppose we had a fully connected first layer, with $784 = 28 \times 28$ input neurons, and a relatively modest $30$ hidden neurons, as we used in many of the examples earlier in the book. That's a total of $784 \times 30$ weights, plus an extra $30$ biases, for a total of $23,550$ parameters. In other words, the fully-connected layer would have more than $40$ times as many parameters as the convolutional layer.

Of course, we can't really do a direct comparison between the number of parameters, since the two models are different in essential ways. But, intuitively, it seems likely that the use of translation invariance by the convolutional layer will reduce the number of parameters it needs to get the same performance as the fully-connected model. That, in turn, will result in faster training for the convolutional model, and, ultimately, will help us build deep networks using convolutional layers.

Incidentally, the name convolutional comes from the fact that the operation in Equation (125)\begin{eqnarray} \sigma\left(b + \sum_{l=0}^4 \sum_{m=0}^4 w_{l,m} a_{j+l, k+m} \right)

onumber\end{eqnarray} is sometimes known as a convolution. A little more precisely, people sometimes write that equation as $a^1 = \sigma(b + w * a^0)$, where $a^1$ denotes the set of output activations from one feature map, $a^0$ is the set of input activations, and $*$ is called a convolution operation. We're not going to make any deep use of the mathematics of convolutions, so you don't need to worry too much about this connection. But it's worth at least knowing where the name comes from.

Pooling layers: In addition to the convolutional layers just described, convolutional neural networks also contain pooling layers. Pooling layers are usually used immediately after convolutional layers. What the pooling layers do is simplify the information in the output from the convolutional layer.

In detail, a pooling layer takes each feature map* *The nomenclature is being used loosely here. In particular, I'm using "feature map" to mean not the function computed by the convolutional layer, but rather the activation of the hidden neurons output from the layer. This kind of mild abuse of nomenclature is pretty common in the research literature. output from the convolutional layer and prepares a condensed feature map. For instance, each unit in the pooling layer may summarize a region of (say) $2 \times 2$ neurons in the previous layer. As a concrete example, one common procedure for pooling is known as max-pooling. In max-pooling, a pooling unit simply outputs the maximum activation in the $2 \times 2$ input region, as illustrated in the following diagram:

Note that since we have $24 \times 24$ neurons output from the convolutional layer, after pooling we have $12 \times 12$ neurons.

As mentioned above, the convolutional layer usually involves more than a single feature map. We apply max-pooling to each feature map separately. So if there were three feature maps, the combined convolutional and max-pooling layers would look like:

We can think of max-pooling as a way for the network to ask whether a given feature is found anywhere in a region of the image. It then throws away the exact positional information. The intuition is that once a feature has been found, its exact location isn't as important as its rough location relative to other features. A big benefit is that there are many fewer pooled features, and so this helps reduce the number of parameters needed in later layers.

Max-pooling isn't the only technique used for pooling. Another common approach is known as L2 pooling. Here, instead of taking the maximum activation of a $2 \times 2$ region of neurons, we take the square root of the sum of the squares of the activations in the $2 \times 2$ region. While the details are different, the intuition is similar to max-pooling: L2 pooling is a way of condensing information from the convolutional layer. In practice, both techniques have been widely used. And sometimes people use other types of pooling operation. If you're really trying to optimize performance, you may use validation data to compare several different approaches to pooling, and choose the approach which works best. But we're not going to worry about that kind of detailed optimization.

Putting it all together: We can now put all these ideas together to form a complete convolutional neural network. It's similar to the architecture we were just looking at, but has the addition of a layer of $10$ output neurons, corresponding to the $10$ possible values for MNIST digits ('0', '1', '2', etc):

The network begins with $28 \times 28$ input neurons, which are used to encode the pixel intensities for the MNIST image. This is then followed by a convolutional layer using a $5 \times 5$ local receptive field and $3$ feature maps. The result is a layer of $3 \times 24 \times 24$ hidden feature neurons. The next step is a max-pooling layer, applied to $2 \times 2$ regions, across each of the $3$ feature maps. The result is a layer of $3 \times 12 \times 12$ hidden feature neurons.

The final layer of connections in the network is a fully-connected layer. That is, this layer connects every neuron from the max-pooled layer to every one of the $10$ output neurons. This fully-connected architecture is the same as we used in earlier chapters. Note, however, that in the diagram above, I've used a single arrow, for simplicity, rather than showing all the connections. Of course, you can easily imagine the connections.

This convolutional architecture is quite different to the architectures used in earlier chapters. But the overall picture is similar: a network made of many simple units, whose behaviors are determined by their weights and biases. And the overall goal is still the same: to use training data to train the network's weights and biases so that the network does a good job classifying input digits.

In particular, just as earlier in the book, we will train our network using stochastic gradient descent and backpropagation. This mostly proceeds in exactly the same way as in earlier chapters. However, we do need to make a few modifications to the backpropagation procedure. The reason is that our earlier derivation of backpropagation was for networks with fully-connected layers. Fortunately, it's straightforward to modify the derivation for convolutional and max-pooling layers. If you'd like to understand the details, then I invite you to work through the following problem. Be warned that the problem will take some time to work through, unless you've really internalized the earlier derivation of backpropagation (in which case it's easy).

We've now seen the core ideas behind convolutional neural networks. Let's look at how they work in practice, by implementing some convolutional networks, and applying them to the MNIST digit classification problem. The program we'll use to do this is called network3.py , and it's an improved version of the programs network.py and network2.py developed in earlier chapters* *Note also that network3.py incorporates ideas from the Theano library's documentation on convolutional neural nets (notably the implementation of LeNet-5), from Misha Denil's implementation of dropout, and from Chris Olah.. If you wish to follow along, the code is available on GitHub. Note that we'll work through the code for network3.py itself in the next section. In this section, we'll use network3.py as a library to build convolutional networks.

The programs network.py and network2.py were implemented using Python and the matrix library Numpy. Those programs worked from first principles, and got right down into the details of backpropagation, stochastic gradient descent, and so on. But now that we understand those details, for network3.py we're going to use a machine learning library known as Theano* *See Theano: A CPU and GPU Math Expression Compiler in Python, by James Bergstra, Olivier Breuleux, Frederic Bastien, Pascal Lamblin, Ravzan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio (2010). Theano is also the basis for the popular Pylearn2 and Keras neural networks libraries. Other popular neural nets libraries at the time of this writing include Caffe and Torch. . Using Theano makes it easy to implement backpropagation for convolutional neural networks, since it automatically computes all the mappings involved. Theano is also quite a bit faster than our earlier code (which was written to be easy to understand, not fast), and this makes it practical to train more complex networks. In particular, one great feature of Theano is that it can run code on either a CPU or, if available, a GPU. Running on a GPU provides a substantial speedup and, again, helps make it practical to train more complex networks.

If you wish to follow along, then you'll need to get Theano running on your system. To install Theano, follow the instructions at the project's homepage. The examples which follow were run using Theano 0.6* *As I release this chapter, the current version of Theano has changed to version 0.7. I've actually rerun the examples under Theano 0.7 and get extremely similar results to those reported in the text.. Some were run under Mac OS X Yosemite, with no GPU. Some were run on Ubuntu 14.04, with an NVIDIA GPU. And some of the experiments were run under both. To get network3.py running you'll need to set the GPU flag to either True or False (as appropriate) in the network3.py source. Beyond that, to get Theano up and running on a GPU you may find the instructions here helpful. There are also tutorials on the web, easily found using Google, which can help you get things working. If you don't have a GPU available locally, then you may wish to look into Amazon Web Services EC2 G2 spot instances. Note that even with a GPU the code will take some time to execute. Many of the experiments take from minutes to hours to run. On a CPU it may take days to run the most complex of the experiments. As in earlier chapters, I suggest setting things running, and continuing to read, occasionally coming back to check the output from the code. If you're using a CPU, you may wish to reduce the number of training epochs for the more complex experiments, or perhaps omit them entirely.

To get a baseline, we'll start with a shallow architecture using just a single hidden layer, containing $100$ hidden neurons. We'll train for $60$ epochs, using a learning rate of $\eta = 0.1$, a mini-batch size of $10$, and no regularization. Here we go* *Code for the experiments in this section may be found in this script. Note that the code in the script simply duplicates and parallels the discussion in this section.



Note also that throughout the section I've explicitly specified the number of training epochs. I've done this for clarity about how we're training. In practice, it's worth using early stopping, that is, tracking accuracy on the validation set, and stopping training when we are confident the validation accuracy has stopped improving.:

>>> import network3 >>> from network3 import Network >>> from network3 import ConvPoolLayer , FullyConnectedLayer , SoftmaxLayer >>> training_data , validation_data , test_data = network3 . load_data_shared () >>> mini_batch_size = 10 >>> net = Network ([ FullyConnectedLayer ( n_in = 784 , n_out = 100 ), SoftmaxLayer ( n_in = 100 , n_out = 10 )], mini_batch_size ) >>> net . SGD ( training_data , 60 , mini_batch_size , 0.1 , validation_data , test_data )

I obtained a best classification accuracy of $97.80$ percent. This is the classification accuracy on the test_data , evaluated at the training epoch where we get the best classification accuracy on the validation_data . Using the validation data to decide when to evaluate the test accuracy helps avoid overfitting to the test data (see this earlier discussion of the use of validation data). We will follow this practice below. Your results may vary slightly, since the network's weights and biases are randomly initialized* *In fact, in this experiment I actually did three separate runs training a network with this architecture. I then reported the test accuracy which corresponded to the best validation accuracy from any of the three runs. Using multiple runs helps reduce variation in results, which is useful when comparing many architectures, as we are doing. I've followed this procedure below, except where noted. In practice, it made little difference to the results obtained..

This $97.80$ percent accuracy is close to the $98.04$ percent accuracy obtained back in Chapter 3, using a similar network architecture and learning hyper-parameters. In particular, both examples used a shallow network, with a single hidden layer containing $100$ hidden neurons. Both also trained for $60$ epochs, used a mini-batch size of $10$, and a learning rate of $\eta = 0.1$.

There were, however, two differences in the earlier network. First, we regularized the earlier network, to help reduce the effects of overfitting. Regularizing the current network does improve the accuracies, but the gain is only small, and so we'll hold off worrying about regularization until later. Second, while the final layer in the earlier network used sigmoid activations and the cross-entropy cost function, the current network uses a softmax final layer, and the log-likelihood cost function. As explained in Chapter 3 this isn't a big change. I haven't made this switch for any particularly deep reason - mostly, I've done it because softmax plus log-likelihood cost is more common in modern image classification networks.

Can we do better than these results using a deeper network architecture?

Let's begin by inserting a convolutional layer, right at the beginning of the network. We'll use $5$ by $5$ local receptive fields, a stride length of $1$, and $20$ feature maps. We'll also insert a max-pooling layer, which combines the features using $2$ by $2$ pooling windows. So the overall network architecture looks much like the architecture discussed in the last section, but with an extra fully-connected layer:

In this architecture, we can think of the convolutional and pooling layers as learning about local spatial structure in the input training image, while the later, fully-connected layer learns at a more abstract level, integrating global information from across the entire image. This is a common pattern in convolutional neural networks.

Let's train such a network, and see how it performs* *I've continued to use a mini-batch size of $10$ here. In fact, as we discussed earlier it may be possible to speed up training using larger mini-batches. I've continued to use the same mini-batch size mostly for consistency with the experiments in earlier chapters.:

>>> net = Network ([ ConvPoolLayer ( image_shape = ( mini_batch_size , 1 , 28 , 28 ), filter_shape = ( 20 , 1 , 5 , 5 ), poolsize = ( 2 , 2 )), FullyConnectedLayer ( n_in = 20 * 12 * 12 , n_out = 100 ), SoftmaxLayer ( n_in = 100 , n_out = 10 )], mini_batch_size ) >>> net . SGD ( training_data , 60 , mini_batch_size , 0.1 , validation_data , test_data )

That gets us to $98.78$ percent accuracy, which is a considerable improvement over any of our previous results. Indeed, we've reduced our error rate by better than a third, which is a great improvement.

In specifying the network structure, I've treated the convolutional and pooling layers as a single layer. Whether they're regarded as separate layers or as a single layer is to some extent a matter of taste. network3.py treats them as a single layer because it makes the code for network3.py a little more compact. However, it is easy to modify network3.py so the layers can be specified separately, if desired.

What classification accuracy do you get if you omit the fully-connected layer, and just use the convolutional-pooling layer and softmax layer? Does the inclusion of the fully-connected layer help?

Can we improve on the $98.78$ percent classification accuracy?

Let's try inserting a second convolutional-pooling layer. We'll make the insertion between the existing convolutional-pooling layer and the fully-connected hidden layer. Again, we'll use a $5 \times 5$ local receptive field, and pool over $2 \times 2$ regions. Let's see what happens when we train using similar hyper-parameters to before:

>>> net = Network ([ ConvPoolLayer ( image_shape = ( mini_batch_size , 1 , 28 , 28 ), filter_shape = ( 20 , 1 , 5 , 5 ), poolsize = ( 2 , 2 )), ConvPoolLayer ( image_shape = ( mini_batch_size , 20 , 12 , 12 ), filter_shape = ( 40 , 20 , 5 , 5 ), poolsize = ( 2 , 2 )), FullyConnectedLayer ( n_in = 40 * 4 * 4 , n_out = 100 ), SoftmaxLayer ( n_in = 100 , n_out = 10 )], mini_batch_size ) >>> net . SGD ( training_data , 60 , mini_batch_size , 0.1 , validation_data , test_data )

Once again, we get an improvement: we're now at $99.06$ percent classification accuracy!

There's two natural questions to ask at this point. The first question is: what does it even mean to apply a second convolutional-pooling layer? In fact, you can think of the second convolutional-pooling layer as having as input $12 \times 12$ "images", whose "pixels" represent the presence (or absence) of particular localized features in the original input image. So you can think of this layer as having as input a version of the original input image. That version is abstracted and condensed, but still has a lot of spatial structure, and so it makes sense to use a second convolutional-pooling layer.

That's a satisfying point of view, but gives rise to a second question. The output from the previous layer involves $20$ separate feature maps, and so there are $20 \times 12 \times 12$ inputs to the second convolutional-pooling layer. It's as though we've got $20$ separate images input to the convolutional-pooling layer, not a single image, as was the case for the first convolutional-pooling layer. How should neurons in the second convolutional-pooling layer respond to these multiple input images? In fact, we'll allow each neuron in this layer to learn from all $20 \times 5 \times 5$ input neurons in its local receptive field. More informally: the feature detectors in the second convolutional-pooling layer have access to all the features from the previous layer, but only within their particular local receptive field* *This issue would have arisen in the first layer if the input images were in color. In that case we'd have 3 input features for each pixel, corresponding to red, green and blue channels in the input image. So we'd allow the feature detectors to have access to all color information, but only within a given local receptive field..

Using the tanh activation function Several times earlier in the book I've mentioned arguments that the tanh function may be a better activation function than the sigmoid function. We've never acted on those suggestions, since we were already making plenty of progress with the sigmoid. But now let's try some experiments with tanh as our activation function. Try training the network with tanh activations in the convolutional and fully-connected layers* *Note that you can pass activation_fn=tanh as a parameter to the ConvPoolLayer and FullyConnectedLayer classes. . Begin with the same hyper-parameters as for the sigmoid network, but train for $20$ epochs instead of $60$. How well does your network perform? What if you continue out to $60$ epochs? Try plotting the per-epoch validation accuracies for both tanh- and sigmoid-based networks, all the way out to $60$ epochs. If your results are similar to mine, you'll find the tanh networks train a little faster, but the final accuracies are very similar. Can you explain why the tanh network might train faster? Can you get a similar training speed with the sigmoid, perhaps by changing the learning rate, or doing some rescaling* *You may perhaps find inspiration in recalling that $\sigma(z) = (1+\tanh(z/2))/2$. ? Try a half-dozen iterations on the learning hyper-parameters or network architecture, searching for ways that tanh may be superior to the sigmoid. Note: This is an open-ended problem. Personally, I did not find much advantage in switching to tanh, although I haven't experimented exhaustively, and perhaps you may find a way. In any case, in a moment we will find an advantage in switching to the rectified linear activation function, and so we won't go any deeper into the use of tanh.

Using rectified linear units: The network we've developed at this point is actually a variant of one of the networks used in the seminal 1998 paper* *"Gradient-based learning applied to document recognition", by Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner (1998). There are many differences of detail, but broadly speaking our network is quite similar to the networks described in the paper. introducing the MNIST problem, a network known as LeNet-5. It's a good foundation for further experimentation, and for building up understanding and intuition. In particular, there are many ways we can vary the network in an attempt to improve our results.

As a beginning, let's change our neurons so that instead of using a sigmoid activation function, we use rectified linear units. That is, we'll use the activation function $f(z) \equiv \max(0, z)$. We'll train for $60$ epochs, with a learning rate of $\eta = 0.03$. I also found that it helps a little to use some l2 regularization, with regularization parameter $\lambda = 0.1$:

>>> from network3 import ReLU >>> net = Network ([ ConvPoolLayer ( image_shape = ( mini_batch_size , 1 , 28 , 28 ), filter_shape = ( 20 , 1 , 5 , 5 ), poolsize = ( 2 , 2 ), activation_fn = ReLU ), ConvPoolLayer ( image_shape = ( mini_batch_size , 20 , 12 , 12 ), filter_shape = ( 40 , 20 , 5 , 5 ), poolsize = ( 2 , 2 ), activation_fn = ReLU ), FullyConnectedLayer ( n_in = 40 * 4 * 4 , n_out = 100 , activation_fn = ReLU ), SoftmaxLayer ( n_in = 100 , n_out = 10 )], mini_batch_size ) >>> net . SGD ( training_data , 60 , mini_batch_size , 0.03 , validation_data , test_data , lmbda = 0.1 )

I obtained a classification accuracy of $99.23$ percent. It's a modest improvement over the sigmoid results ($99.06$). However, across all my experiments I found that networks based on rectified linear units consistently outperformed networks based on sigmoid activation functions. There appears to be a real gain in moving to rectified linear units for this problem.

What makes the rectified linear activation function better than the sigmoid or tanh functions? At present, we have a poor understanding of the answer to this question. Indeed, rectified linear units have only begun to be widely used in the past few years. The reason for that recent adoption is empirical: a few people tried rectified linear units, often on the basis of hunches or heuristic arguments* *A common justification is that $\max(0, z)$ doesn't saturate in the limit of large $z$, unlike sigmoid neurons, and this helps rectified linear units continue learning. The argument is fine, as far it goes, but it's hardly a detailed justification, more of a just-so story. Note that we discussed the problems with saturation back in Chapter 2.. They got good results classifying benchmark data sets, and the practice has spread. In an ideal world we'd have a theory telling us which activation function to pick for which application. But at present we're a long way from such a world. I should not be at all surprised if further major improvements can be obtained by an even better choice of activation function. And I also expect that in coming decades a powerful theory of activation functions will be developed. Today, we still have to rely on poorly understood rules of thumb and experience.

Expanding the training data: Another way we may hope to improve our results is by algorithmically expanding the training data. A simple way of expanding the training data is to displace each training image by a single pixel, either up one pixel, down one pixel, left one pixel, or right one pixel. We can do this by running the program expand_mnist.py from the shell prompt* *The code for expand_mnist.py is available here.:

$ python expand_mnist.py

Running this program takes the $50,000$ MNIST training images, and prepares an expanded training set, with $250,000$ training images. We can then use those training images to train our network. We'll use the same network as above, with rectified linear units. In my initial experiments I reduced the number of training epochs - this made sense, since we're training with $5$ times as much data. But, in fact, expanding the data turned out to considerably reduce the effect of overfitting. And so, after some experimentation, I eventually went back to training for $60$ epochs. In any case, let's train:

>>> expanded_training_data , _ , _ = network3 . load_data_shared ( "../data/mnist_expanded.pkl.gz" ) >>> net = Network ([ ConvPoolLayer ( image_shape = ( mini_batch_size , 1 , 28 , 28 ), filter_shape = ( 20 , 1 , 5 , 5 ), poolsize = ( 2 , 2 ), activation_fn = ReLU ), ConvPoolLayer ( image_shape = ( mini_batch_size , 20 , 12 , 12 ), filter_shape = ( 40 , 20 , 5 , 5 ), poolsize = ( 2 , 2 ), activation_fn = ReLU ), FullyConnectedLayer ( n_in = 40 * 4 * 4 , n_out = 100 , activation_fn = ReLU ), SoftmaxLayer ( n_in = 100 , n_out = 10 )], mini_batch_size ) >>> net . SGD ( expanded_training_data , 60 , mini_batch_size , 0.03 , validation_data , test_data , lmbda = 0.1 )

Using the expanded training data I obtained a $99.37$ percent training accuracy. So this almost trivial change gives a substantial improvement in classification accuracy. Indeed, as we discussed earlier this idea of algorithmically expanding the data can be taken further. Just to remind you of the flavour of some of the results in that earlier discussion: in 2003 Simard, Steinkraus and Platt* *Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis, by Patrice Simard, Dave Steinkraus, and John Platt (2003). improved their MNIST performance to $99.6$ percent using a neural network otherwise very similar to ours, using two convolutional-pooling layers, followed by a hidden fully-connected layer with $100$ neurons. There were a few differences of detail in their architecture - they didn't have the advantage of using rectified linear units, for instance - but the key to their improved performance was expanding the training data. They did this by rotating, translating, and skewing the MNIST training images. They also developed a process of "elastic distortion", a way of emulating the random oscillations hand muscles undergo when a person is writing. By combining all these processes they substantially increased the effective size of their training data, and that's how they achieved $99.6$ percent accuracy.

The idea of convolutional layers is to behave in an invariant way across images. It may seem surprising, then, that our network can learn more when all we've done is translate the input data. Can you explain why this is actually quite reasonable?

Inserting an extra fully-connected layer: Can we do even better? One possibility is to use exactly the same procedure as above, but to expand the size of the fully-connected layer. I tried with $300$ and $1,000$ neurons, obtaining results of $99.46$ and $99.43$ percent, respectively. That's interesting, but not really a convincing win over the earlier result ($99.37$ percent).

What about adding an extra fully-connected layer? Let's try inserting an extra fully-connected layer, so that we have two $100$-hidden neuron fully-connected layers:

>>> net = Network ([ ConvPoolLayer ( image_shape = ( mini_batch_size , 1 , 28 , 28 ), filter_shape = ( 20 , 1 , 5 , 5 ), poolsize = ( 2 , 2 ), activation_fn = ReLU ), ConvPoolLayer ( image_shape = ( mini_batch_size , 20 , 12 , 12 ), filter_shape = ( 40 , 20 , 5 , 5 ), poolsize = ( 2 , 2 ), activation_fn = ReLU ), FullyConnectedLayer ( n_in = 40 * 4 * 4 , n_out = 100 , activation_fn = ReLU ), FullyConnectedLayer ( n_in = 100 , n_out = 100 , activation_fn = ReLU ), SoftmaxLayer ( n_in = 100 , n_out = 10 )], mini_batch_size ) >>> net . SGD ( expanded_training_data , 60 , mini_batch_size , 0.03 , validation_data , test_data , lmbda = 0.1 )

Doing this, I obtained a test accuracy of $99.43$ percent. Again, the expanded net isn't helping so much. Running similar experiments with fully-connected layers containing $300$ and $1,000$ neurons yields results of $99.48$ and $99.47$ percent. That's encouraging, but still falls short of a really decisive win.

What's going on here? Is it that the expanded or extra fully-connected layers really don't help with MNIST? Or might it be that our network has the capacity to do better, but we're going about learning the wrong way? For instance, maybe we could use stronger regularization techniques to reduce the tendency to overfit. One possibility is the dropout technique introduced back in Chapter 3. Recall that the basic idea of dropout is to remove individual activations at random while training the network. This makes the model more robust to the loss of individual pieces of evidence, and thus less likely to rely on particular idiosyncracies of the training data. Let's try applying dropout to the final fully-connected layers:

>>> net = Network ([ ConvPoolLayer ( image_shape = ( mini_batch_size , 1 , 28 , 28 ), filter_shape = ( 20 , 1 , 5 , 5 ), poolsize = ( 2 , 2 ), activation_fn = ReLU ), ConvPoolLayer ( image_shape = ( mini_batch_size , 20 , 12 , 12 ), filter_shape = ( 40 , 20 , 5 , 5 ), poolsize = ( 2 , 2 ), activation_fn = ReLU ), FullyConnectedLayer ( n_in = 40 * 4 * 4 , n_out = 1000 , activation_fn = ReLU , p_dropout = 0.5 ), FullyConnectedLayer ( n_in = 1000 , n_out = 1000 , activation_fn = ReLU , p_dropout = 0.5 ), SoftmaxLayer ( n_in = 1000 , n_out = 10 , p_dropout = 0.5 )], mini_batch_size ) >>> net . SGD ( expanded_training_data , 40 , mini_batch_size , 0.03 , validation_data , test_data )

Using this, we obtain an accuracy of $99.60$ percent, which is a substantial improvement over our earlier results, especially our main benchmark, the network with $100$ hidden neurons, where we achieved $99.37$ percent.

There are two changes worth noting.

First, I reduced the number of training epochs to $40$: dropout reduced overfitting, and so we learned faster.

Second, the fully-connected hidden layers have $1,000$ neurons, not the $100$ used earlier. Of course, dropout effectively omits many of the neurons while training, so some expansion is to be expected. In fact, I tried experiments with both $300$ and $1,000$ hidden neurons, and obtained (very slightly) better validation performance with $1,000$ hidden neurons.

Using an ensemble of networks: An easy way to improve performance still further is to create several neural networks, and then get them to vote to determine the best classification. Suppose, for example, that we trained $5$ different neural networks using the prescription above, with each achieving accuracies near to $99.6$ percent. Even though the networks would all have similar accuracies, they might well make different errors, due to the different random initializations. It's plausible that taking a vote amongst our $5$ networks might yield a classification better than any individual network.

This sounds too good to be true, but this kind of ensembling is a common trick with both neural networks and other machine learning techniques. And it does in fact yield further improvements: we end up with $99.67$ percent accuracy. In other words, our ensemble of networks classifies all but $33$ of the $10,000$ test images correctly.

The remaining errors in the test set are shown below. The label in the top right is the correct classification, according to the MNIST data, while in the bottom right is the label output by our ensemble of nets:

It's worth looking through these in detail. The first two digits, a 6 and a 5, are genuine errors by our ensemble. However, they're also understandable errors, the kind a human could plausibly make. That 6 really does look a lot like a 0, and the 5 looks a lot like a 3. The third image, supposedly an 8, actually looks to me more like a 9. So I'm siding with the network ensemble here: I think it's done a better job than whoever originally drew the digit. On the other hand, the fourth image, the 6, really does seem to be classified badly by our networks.

And so on. In most cases our networks' choices seem at least plausible, and in some cases they've done a better job classifying than the original person did writing the digit. Overall, our networks offer exceptional performance, especially when you consider that they correctly classified 9,967 images which aren't shown. In that context, the few clear errors here seem quite understandable. Even a careful human makes the occasional mistake. And so I expect that only an extremely careful and methodical human would do much better. Our network is getting near to human performance.

Why we only applied dropout to the fully-connected layers: If you look carefully at the code above, you'll notice that we applied dropout only to the fully-connected section of the network, not to the convolutional layers. In principle we could apply a similar procedure to the convolutional layers. But, in fact, there's no need: the convolutional layers have considerable inbuilt resistance to overfitting. The reason is that the shared weights mean that convolutional filters are forced to learn from across the entire image. This makes them less likely to pick up on local idiosyncracies in the training data. And so there is less need to apply other regularizers, such as dropout.

Going further: It's possible to improve performance on MNIST still further. Rodrigo Benenson has compiled an informative summary page, showing progress over the years, with links to papers. Many of these papers use deep convolutional networks along lines similar to the networks we've been using. If you dig through the papers you'll find many interesting techniques, and you may enjoy implementing some of them. If you do so it's wise to start implementation with a simple network that can be trained quickly, which will help you more rapidly understand what is going on.

For the most part, I won't try to survey this recent work. But I can't resist making one exception. It's a 2010 paper by Cireșan, Meier, Gambardella, and Schmidhuber* *Deep, Big, Simple Neural Nets Excel on Handwritten Digit Recognition, by Dan Claudiu Cireșan, Ueli Meier, Luca Maria Gambardella, and Jürgen Schmidhuber (2010).. What I like about this paper is how simple it is. The network is a many-layer neural network, using only fully-connected layers (no convolutions). Their most successful network had hidden layers containing $2,500$, $2,000$, $1,500$, $1,000$, and $500$ neurons, respectively. They used ideas similar to Simard et al to expand their training data. But apart from that, they used few other tricks, including no convolutional layers: it was a plain, vanilla network, of the kind that, with enough patience, could have been trained in the 1980s (if the MNIST data set had existed), given enough computing power. They achieved a classification accuracy of $99.65$ percent, more or less the same as ours. The key was to use a very large, very deep network, and to use a GPU to speed up training. This let them train for many epochs. They also took advantage of their long training times to gradually decrease the learning rate from $10^{-3}$ to $10^{-6}$. It's a fun exercise to try to match these results using an architecture like theirs.

Why are we able to train? We saw in the last chapter that there are fundamental obstructions to training in deep, many-layer neural networks. In particular, we saw that the gradient tends to be quite unstable: as we move from the output layer to earlier layers the gradient tends to either vanish (the vanishing gradient problem) or explode (the exploding gradient problem). Since the gradient is the signal we use to train, this causes problems.

How have we avoided those results?

Of course, the answer is that we haven't avoided these results. Instead, we've done a few things that help us proceed anyway. In particular: (1) Using convolutional layers greatly reduces the number of parameters in those layers, making the learning problem much easier; (2) Using more powerful regularization techniques (notably dropout and convolutional layers) to reduce overfitting, which is otherwise more of a problem in more complex networks; (3) Using rectified linear units instead of sigmoid neurons, to speed up training - empirically, often by a factor of $3$-$5$; (4) Using GPUs and being willing to train for a long period of time. In particular, in our final experiments we trained for $40$ epochs using a data set $5$ times larger than the raw MNIST training data. Earlier in the book we mostly trained for $30$ epochs using just the raw training data. Combining factors (3) and (4) it's as though we've trained a factor perhaps $30$ times longer than before.

Your response may be "Is that it? Is that all we had to do to train deep networks? What's all the fuss about?"

Of course, we've used other ideas, too: making use of sufficiently large data sets (to help avoid overfitting); using the right cost function (to avoid a learning slowdown); using good weight initializations (also to avoid a learning slowdown, due to neuron saturation); algorithmically expanding the training data. We discussed these and other ideas in earlier chapters, and have for the most part been able to reuse these ideas with little comment in this chapter.

With that said, this really is a rather simple set of ideas. Simple, but powerful, when used in concert. Getting started with deep learning has turned out to be pretty easy!

How deep are these networks, anyway? Counting the convolutional-pooling layers as single layers, our final architecture has $4$ hidden layers. Does such a network really deserve to be called a deep network? Of course, $4$ hidden layers is many more than in the shallow networks we studied earlier. Most of those networks only had a single hidden layer, or occasionally $2$ hidden layers. On the other hand, as of 2015 state-of-the-art deep networks sometimes have dozens of hidden layers. I've occasionally heard people adopt a deeper-than-thou attitude, holding that if you're not keeping-up-with-the-Joneses in terms of number of hidden layers, then you're not really doing deep learning. I'm not sympathetic to this attitude, in part because it makes the definition of deep learning into something which depends upon the result-of-the-moment. The real breakthrough in deep learning was to realize that it's practical to go beyond the shallow $1$- and $2$-hidden layer networks that dominated work until the mid-2000s. That really was a significant breakthrough, opening up the exploration of much more expressive models. But beyond that, the number of layers is not of primary fundamental interest. Rather, the use of deeper networks is a tool to use to help achieve other goals - like better classification accuracies.

A word on procedure: In this section, we've smoothly moved from single hidden-layer shallow networks to many-layer convolutional networks. It all seemed so easy! We make a change and, for the most part, we get an improvement. If you start experimenting, I can guarantee things won't always be so smooth. The reason is that I've presented a cleaned-up narrative, omitting many experiments - including many failed experiments. This cleaned-up narrative will hopefully help you get clear on the basic ideas. But it also runs the risk of conveying an incomplete impression. Getting a good, working network can involve a lot of trial and error, and occasional frustration. In practice, you should expect to engage in quite a bit of experimentation. To speed that process up you may find it helpful to revisit Chapter 3's discussion of how to choose a neural network's hyper-parameters, and perhaps also to look at some of the further reading suggested in that section.

Alright, let's take a look at the code for our program, network3.py . Structurally, it's similar to network2.py , the program we developed in Chapter 3, although the details differ, due to the use of Theano. We'll start by looking at the FullyConnectedLayer class, which is similar to the layers studied earlier in the book. Here's the code (discussion below)* *Note added November 2016: several readers have noted that in the line initializing self.w , I set scale=np.sqrt(1.0/n_out) , when the arguments of Chapter 3 suggest a better initialization may be scale=np.sqrt(1.0/n_in) . This was simply a mistake on my part. In an ideal world I'd rerun all the examples in this chapter with the correct code. Still, I've moved on to other projects, so am going to let the error go.:

class FullyConnectedLayer ( object ): def __init__ ( self , n_in , n_out , activation_fn = sigmoid , p_dropout = 0.0 ): self . n_in = n_in self . n_out = n_out self . activation_fn = activation_fn self . p_dropout = p_dropout # Initialize weights and biases self . w = theano . shared ( np . asarray ( np . random . normal ( loc = 0.0 , scale = np . sqrt ( 1.0 / n_out ), size = ( n_in , n_out )), dtype = theano . config . floatX ), name = 'w' , borrow = True ) self . b = theano . shared ( np . asarray ( np . random . normal ( loc = 0.0 , scale = 1.0 , size = ( n_out ,)), dtype = theano . config . floatX ), name = 'b' , borrow = True ) self . params = [ self . w , self . b ] def set_inpt ( self , inpt , inpt_dropout , mini_batch_size ): self . inpt = inpt . reshape (( mini_batch_size , self . n_in )) self . output = self . activation_fn ( ( 1 - self . p_dropout ) * T . dot ( self . inpt , self . w ) + self . b ) self . y_out = T . argmax ( self . output , axis = 1 ) self . inpt_dropout = dropout_layer ( inpt_dropout . reshape (( mini_batch_size , self . n_in )), self . p_dropout ) self . output_dropout = self . activation_fn ( T . dot ( self . inpt_dropout , self . w ) + self . b ) def accuracy ( self , y ): "Return the accuracy for the mini-batch." return T . mean ( T . eq ( y , self . y_out ))

Much of the __init__ method is self-explanatory, but a few remarks may help clarify the code. As per usual, we randomly initialize the weights and biases as normal random variables with suitable standard deviations. The lines doing this look a little forbidding. However, most of the complication is just loading the weights and biases into what Theano calls shared variables. This ensures that these variables can be processed on the GPU, if one is available. We won't get too much into the details of this. If you're interested, you can dig into the Theano documentation. Note also that this weight and bias initialization is designed for the sigmoid activation function (as discussed earlier). Ideally, we'd initialize the weights and biases somewhat differently for activation functions such as the tanh and rectified linear function. This is discussed further in problems below. The __init__ method finishes with self.params = [self.w, self.b] . This is a handy way to bundle up all the learnable parameters associated to the layer. Later on, the Network.SGD method will use params attributes to figure out what variables in a Network instance can learn.

The set_inpt method is used to set the input to the layer, and to compute the corresponding output. I use the name inpt rather than input because input is a built-in function in Python, and messing with built-ins tends to cause unpredictable behavior and difficult-to-diagnose bugs. Note that we actually set the input in two separate ways: as self.inpt and self.inpt_dropout . This is done because during training we may want to use dropout. If that's the case then we want to remove a fraction self.p_dropout of the neurons. That's what the function dropout_layer in the second-last line of the set_inpt method is doing. So self.inpt_dropout and self.output_dropout are used during training, while self.inpt and self.output are used for all other purposes, e.g., evaluating accuracy on the validation and test data.

The ConvPoolLayer and SoftmaxLayer class definitions are similar to FullyConnectedLayer . Indeed, they're so close that I won't excerpt the code here. If you're interested you can look at the full listing for network3.py , later in this section.

However, a couple of minor differences of detail are worth mentioning. Most obviously, in both ConvPoolLayer and SoftmaxLayer we compute the output activations in the way appropriate to that layer type. Fortunately, Theano makes that easy, providing built-in operations to compute convolutions, max-pooling, and the softmax function.

Less obviously, when we introduced the softmax layer, we never discussed how to initialize the weights and biases. Elsewhere we've argued that for sigmoid layers we should initialize the weights using suitably parameterized normal random variables. But that heuristic argument was specific to sigmoid neurons (and, with some amendment, to tanh neurons). However, there's no particular reason the argument should apply to softmax layers. So there's no a priori reason to apply that initialization again. Rather than do that, I shall initialize all the weights and biases to be $0$. This is a rather ad hoc procedure, but works well enough in practice.

Okay, we've looked at all the layer classes. What about the Network class? Let's start by looking at the __init__ method:

class Network ( object ): def __init__ ( self , layers , mini_batch_size ): """Takes a list of `layers`, describing the network architecture, and a value for the `mini_batch_size` to be used during training by stochastic gradient descent. """ self . layers = layers self . mini_batch_size = mini_batch_size self . params = [ param for layer in self . layers for param in layer . params ] self . x = T . matrix ( "x" ) self . y = T . ivector ( "y" ) init_layer = self . layers [ 0 ] init_layer . set_inpt ( self . x , self . x , self . mini_batch_size ) for j in xrange ( 1 , len ( self . layers )): prev_layer , layer = self . layers [ j - 1 ], self . layers [ j ] layer . set_inpt ( prev_layer . output , prev_layer . output_dropout , self . mini_batch_size ) self . output = self . layers [ - 1 ] . output self . output_dropout = self . layers [ - 1 ] . output_dropout

Most of this is self-explanatory, or nearly so. The line self.params = [param for layer in ...] bundles up the parameters for each layer into a single list. As anticipated above, the Network.SGD method will use self.params to figure out what variables in the Network can learn. The lines self.x = T.matrix("x") and self.y = T.ivector("y") define Theano symbolic variables named x and y . These will be used to represent the input and desired output from the network.

Now, this isn't a Theano tutorial, and so we won't get too deeply into what it means that these are symbolic variables* *The Theano documentation provides a good introduction to Theano. And if you get stuck, you may find it helpful to look at one of the other tutorials available online. For instance, this tutorial covers many basics.. But the rough idea is that these represent mathematical variables, not explicit values. We can do all the usual things one would do with such variables: add, subtract, and multiply them, apply functions, and so on. Indeed, Theano provides many ways of manipulating such symbolic variables, doing things like convolutions, max-pooling, and so on. But the big win is the ability to do fast symbolic differentiation, using a very general form of the backpropagation algorithm. This is extremely useful for applying stochastic gradient descent to a wide variety of network architectures. In particular, the next few lines of code define symbolic outputs from the network. We start by setting the input to the initial layer, with the line

init_layer . set_inpt ( self . x , self . x , self . mini_batch_size )

Note that the inputs are set one mini-batch at a time, which is why the mini-batch size is there. Note also that we pass the input self.x in twice: this is because we may use the network in two different ways (with or without dropout). The for loop then propagates the symbolic variable self.x forward through the layers of the Network . This allows us to define the final output and output_dropout attributes, which symbolically represent the output from the Network .

Now that we've understood how a Network is initialized, let's look at how it is trained, using the SGD method. The code looks lengthy, but its structure is actually rather simple. Explanatory comments after the code.

def SGD ( self , training_data , epochs , mini_batch_size , eta , validation_data , test_data , lmbda = 0.0 ): """Train the network using mini-batch stochastic gradient descent.""" training_x , training_y = training_data validation_x , validation_y = validation_data test_x , test_y = test_data # compute number of minibatches for training, validation and testing num_training_batches = size ( training_data ) / mini_batch_size num_validation_batches = size ( validation_data ) / mini_batch_size num_test_batches = size ( test_data ) / mini_batch_size # define the (regularized) cost function, symbolic gradients, and updates l2_norm_squared = sum ([( layer . w ** 2 ) . sum () for layer in self . layers ]) cost = self . layers [ - 1 ] . cost ( self ) + \ 0.5 * lmbda * l2_norm_squared / num_training_batches grads = T . grad ( cost , self . params ) updates = [( param , param - eta * grad ) for param , grad in zip ( self . params , grads )] # define functions to train a mini-batch, and to compute the # accuracy in validation and test mini-batches. i = T . lscalar () # mini-batch index train_mb = theano . function ( [ i ], cost , updates = updates , givens = { self . x : training_x [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ], self . y : training_y [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ] }) validate_mb_accuracy = theano . function ( [ i ], self . layers [ - 1 ] . accuracy ( self . y ), givens = { self . x : validation_x [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ], self . y : validation_y [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ] }) test_mb_accuracy = theano . function ( [ i ], self . layers [ - 1 ] . accuracy ( self . y ), givens = { self . x : test_x [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ], self . y : test_y [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ] }) self . test_mb_predictions = theano . function ( [ i ], self . layers [ - 1 ] . y_out , givens = { self . x : test_x [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ] }) # Do the actual training best_validation_accuracy = 0.0 for epoch in xrange ( epochs ): for minibatch_index in xrange ( num_training_batches ): iteration = num_training_batches * epoch + minibatch_index if iteration print ( "Training mini-batch number {0}" . format ( iteration )) cost_ij = train_mb ( minibatch_index ) if ( iteration + 1 ) validation_accuracy = np . mean ( [ validate_mb_accuracy ( j ) for j in xrange ( num_validation_batches )]) print ( "Epoch {0}: validation accuracy {1:.2 epoch , validation_accuracy )) if validation_accuracy >= best_validation_accuracy : print ( "This is the best validation accuracy to date." ) best_validation_accuracy = validation_accuracy best_iteration = iteration if test_data : test_accuracy = np . mean ( [ test_mb_accuracy ( j ) for j in xrange ( num_test_batches )]) print ( 'The corresponding test accuracy is {0:.2 test_accuracy )) print ( "Finished training network." ) print ( "Best validation accuracy of {0:.2 best_validation_accuracy , best_iteration )) print ( "Corresponding test accuracy of {0:.2

The first few lines are straightforward, separating the datasets into $x$ and $y$ components, and computing the number of mini-batches used in each dataset. The next few lines are more interesting, and show some of what makes Theano fun to work with. Let's explicitly excerpt the lines here:

# define the (regularized) cost function, symbolic gradients, and updates l2_norm_squared = sum ([( layer . w ** 2 ) . sum () for layer in self . layers ]) cost = self . layers [ - 1 ] . cost ( self ) + \ 0.5 * lmbda * l2_norm_squared / num_training_batches grads = T . grad ( cost , self . params ) updates = [( param , param - eta * grad ) for param , grad in zip ( self . params , grads )]

In these lines we symbolically set up the regularized log-likelihood cost function, compute the corresponding derivatives in the gradient function, as well as the corresponding parameter updates. Theano lets us achieve all of this in just these few lines. The only thing hidden is that computing the cost involves a call to the cost method for the output layer; that code is elsewhere in network3.py . But that code is short and simple, anyway. With all these things defined, the stage is set to define the train_mb function, a Theano symbolic function which uses the updates to update the Network parameters, given a mini-batch index. Similarly, validate_mb_accuracy and test_mb_accuracy compute the accuracy of the Network on any given mini-batch of validation or test data. By averaging over these functions, we will be able to compute accuracies on the entire validation and test data sets.

The remainder of the SGD method is self-explanatory - we simply iterate over the epochs, repeatedly training the network on mini-batches of training data, and computing the validation and test accuracies.

Okay, we've now understood the most important pieces of code in network3.py . Let's take a brief look at the entire program. You don't need to read through this in detail, but you may enjoy glancing over it, and perhaps diving down into any pieces that strike your fancy. The best way to really understand it is, of course, by modifying it, adding extra features, or refactoring anything you think could be done more elegantly. After the code, there are some problems which contain a few starter suggestions for things to do. Here's the code* *Using Theano on a GPU can be a little tricky. In particular, it's easy to make the mistake of pulling data off the GPU, which can slow things down a lot. I've tried to avoid this. With that said, this code can certainly be sped up quite a bit further with careful optimization of Theano's configuration. See the Theano documentation for more details.:

"""network3.py ~~~~~~~~~~~~~~ A Theano-based program for training and running simple neural networks. Supports several layer types (fully connected, convolutional, max pooling, softmax), and activation functions (sigmoid, tanh, and rectified linear units, with more easily added). When run on a CPU, this program is much faster than network.py and network2.py. However, unlike network.py and network2.py it can also be run on a GPU, which makes it faster still. Because the code is based on Theano, the code is different in many ways from network.py and network2.py. However, where possible I have tried to maintain consistency with the earlier programs. In particular, the API is similar to network2.py. Note that I have focused on making the code simple, easily readable, and easily modifiable. It is not optimized, and omits many desirable features. This program incorporates ideas from the Theano documentation on convolutional neural nets (notably, http://deeplearning.net/tutorial/lenet.html ), from Misha Denil's implementation of dropout (https://github.com/mdenil/dropout ), and from Chris Olah (http://colah.github.io ). Written for Theano 0.6 and 0.7, needs some changes for more recent versions of Theano. """ #### Libraries # Standard library import cPickle import gzip # Third-party libraries import numpy as np import theano import theano.tensor as T from theano.tensor.nnet import conv from theano.tensor.nnet import softmax from theano.tensor import shared_randomstreams from theano.tensor.signal import downsample # Activation functions for neurons def linear ( z ): return z def ReLU ( z ): return T . maximum ( 0.0 , z ) from theano.tensor.nnet import sigmoid from theano.tensor import tanh #### Constants GPU = True if GPU : print "Trying to run under a GPU. If this is not desired, then modify " + \ "network3.py

to set the GPU flag to False." try : theano . config . device = 'gpu' except : pass # it's already set theano . config . floatX = 'float32' else : print "Running with a CPU. If this is not desired, then the modify " + \ "network3.py to set

the GPU flag to True." #### Load the MNIST data def load_data_shared ( filename = "../data/mnist.pkl.gz" ): f = gzip . open ( filename , 'rb' ) training_data , validation_data , test_data = cPickle . load ( f ) f . close () def shared ( data ): """Place the data into shared variables. This allows Theano to copy the data to the GPU, if one is available. """ shared_x = theano . shared ( np . asarray ( data [ 0 ], dtype = theano . config . floatX ), borrow = True ) shared_y = theano . shared ( np . asarray ( data [ 1 ], dtype = theano . config . floatX ), borrow = True ) return shared_x , T . cast ( shared_y , "int32" ) return [ shared ( training_data ), shared ( validation_data ), shared ( test_data )] #### Main class used to construct and train networks class Network ( object ): def __init__ ( self , layers , mini_batch_size ): """Takes a list of `layers`, describing the network architecture, and a value for the `mini_batch_size` to be used during training by stochastic gradient descent. """ self . layers = layers self . mini_batch_size = mini_batch_size self . params = [ param for layer in self . layers for param in layer . params ] self . x = T . matrix ( "x" ) self . y = T . ivector ( "y" ) init_layer = self . layers [ 0 ] init_layer . set_inpt ( self . x , self . x , self . mini_batch_size ) for j in xrange ( 1 , len ( self . layers )): prev_layer , layer = self . layers [ j - 1 ], self . layers [ j ] layer . set_inpt ( prev_layer . output , prev_layer . output_dropout , self . mini_batch_size ) self . output = self . layers [ - 1 ] . output self . output_dropout = self . layers [ - 1 ] . output_dropout def SGD ( self , training_data , epochs , mini_batch_size , eta , validation_data , test_data , lmbda = 0.0 ): """Train the network using mini-batch stochastic gradient descent.""" training_x , training_y = training_data validation_x , validation_y = validation_data test_x , test_y = test_data # compute number of minibatches for training, validation and testing num_training_batches = size ( training_data ) / mini_batch_size num_validation_batches = size ( validation_data ) / mini_batch_size num_test_batches = size ( test_data ) / mini_batch_size # define the (regularized) cost function, symbolic gradients, and updates l2_norm_squared = sum ([( layer . w ** 2 ) . sum () for layer in self . layers ]) cost = self . layers [ - 1 ] . cost ( self ) + \ 0.5 * lmbda * l2_norm_squared / num_training_batches grads = T . grad ( cost , self . params ) updates = [( param , param - eta * grad ) for param , grad in zip ( self . params , grads )] # define functions to train a mini-batch, and to compute the # accuracy in validation and test mini-batches. i = T . lscalar () # mini-batch index train_mb = theano . function ( [ i ], cost , updates = updates , givens = { self . x : training_x [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ], self . y : training_y [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ] }) validate_mb_accuracy = theano . function ( [ i ], self . layers [ - 1 ] . accuracy ( self . y ), givens = { self . x : validation_x [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ], self . y : validation_y [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ] }) test_mb_accuracy = theano . function ( [ i ], self . layers [ - 1 ] . accuracy ( self . y ), givens = { self . x : test_x [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ], self . y : test_y [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ] }) self . test_mb_predictions = theano . function ( [ i ], self . layers [ - 1 ] . y_out , givens = { self . x : test_x [ i * self . mini_batch_size : ( i + 1 ) * self . mini_batch_size ] }) # Do the actual training best_validation_accuracy = 0.0 for epoch in xrange ( epochs ): for minibatch_index in xrange ( num_training_batches ): iteration = num_training_batches * epoch + minibatch_index if iteration % 1000 == 0 : print ( "Training mini-batch number {0}" . format ( iteration )) cost_ij = train_mb ( minibatch_index ) if ( iteration + 1 ) % num_training_batches == 0 : validation_accuracy = np . mean ( [ validate_mb_accuracy ( j ) for j in xrange ( num_validation_batches )]) print ( "Epoch {0}: validation accuracy {1:.2%}" . format ( epoch , validation_accuracy )) if validation_accuracy >= best_validation_accuracy : print ( "This is the best validation accuracy to date." ) best_validation_accuracy = validation_accuracy best_iteration = iteration if test_data : test_accuracy = np . mean ( [ test_mb_accuracy ( j ) for j in xrange ( num_test_batches )]) print ( 'The corresponding test accuracy is {0:.2%}' . format ( test_accuracy )) print ( "Finished training network." ) print ( "Best validation accuracy of {0:.2%} obtained at iteration {1}" . format ( best_validation_accuracy , best_iteration )) print ( "Corresponding test accuracy of {0:.2%}" . format ( test_accuracy )) #### Define layer types class ConvPoolLayer ( object ): """Used to create a combination of a convolutional and a max-pooling layer. A more sophisticated implementation would separate the two, but for our purposes we'll always use them together, and it simplifies the code, so it makes sense to combine them. """ def __init__ ( self , filter_shape , image_shape , poolsize = ( 2 , 2 ), activation_fn = sigmoid ): """`filter_shape` is a tuple of length 4, whose entries are the number of filters, the number of input feature maps, the filter height, and the filter width. `image_shape` is a tuple of length 4, whose entries are the mini-batch size, the number of input feature maps, the image height, and the image width. `poolsize` is a tuple of length 2, whose entries are the y and x pooling sizes. """ self . filter_shape = filter_shape self . image_shape = image_shape self . poolsize = poolsize self . activation_fn = activation_fn # initialize weights and biases n_out = ( filter_shape [ 0 ] * np . prod ( filter_shape [ 2 :]) / np . prod ( poolsize )) self . w = theano . shared ( np . asarray ( np . random . normal ( loc = 0 , scale = np . sqrt ( 1.0 / n_out ), size = filter_shape ), dtype = theano . config . floatX ), borrow = True ) self . b = theano . shared ( np . asarray ( np . random . normal ( loc = 0 , scale = 1.0 , size = ( filter_shape [ 0 ],)), dtype = theano . config . floatX ), borrow = True ) self . params = [ self . w , self . b ] def set_inpt ( self , inpt , inpt_dropout , mini_batch_size ): self . inpt = inpt . reshape ( self . image_shape ) conv_out = conv . conv2d ( input = self . inpt , filters = self . w , filter_shape = self . filter_shape , image_shape = self . image_shape ) pooled_out = downsample . max_pool_2d ( input = conv_out , ds = self . poolsize , ignore_border = True ) self . output = self . activation_fn ( pooled_out + self . b . dimshuffle ( 'x' , 0 , 'x' , 'x' )) self . output_dropout = self . output # no dropout in the convolutional layers class FullyConnectedLayer ( object ): def __init__ ( self , n_in , n_out , activation_fn = sigmoid , p_dropout = 0.0 ): self . n_in = n_in self . n_out = n_out self . activation_fn = activation_fn self . p_dropout = p_dropout # Initialize weights and biases self . w = theano . shared ( np . asarray ( np . random . normal ( loc = 0.0 , scale = np . sqrt ( 1.0 / n_out ), size = ( n_in , n_out )), dtype = theano . config . floatX ), name = 'w' , borrow = True ) self . b = theano . shared ( np . asarray ( np . random . normal ( loc = 0.0 , scale = 1.0 , size = ( n_out ,)), dtype = theano . config . floatX ), name = 'b' , borrow = True ) self . params = [ self . w , self . b ] def set_inpt ( self , inpt , inpt_dropout , mini_batch_size ): self . inpt = inpt . reshape (( mini_batch_size , self . n_in )) self . output = self . activation_fn ( ( 1 - self . p_dropout ) * T . dot ( self . inpt , self . w ) + self . b ) self . y_out = T . argmax ( self . output , axis = 1 ) self . inpt_dropout = dropout_layer ( inpt_dropout . reshape (( mini_batch_size , self . n_in )), self . p_dropout ) self . output_dropout = self . activation_fn ( T . dot ( self . inpt_dropout , self . w ) + self . b ) def accuracy ( self , y ): "Return the accuracy for the mini-batch." return T . mean ( T . eq ( y , self . y_out )) class SoftmaxLayer ( object ): def __init__ ( self , n_in , n_out , p_dropout = 0.0 ): self . n_in = n_in self . n_out = n_out self . p_dropout = p_dropout # Initialize weights and biases self . w = theano . shared ( np . zeros (( n_in , n_out ), dtype = theano . config . floatX ), name = 'w' , borrow = True ) self . b = theano . shared ( np . zeros (( n_out ,), dtype = theano . config . floatX ), name = 'b' , borrow = True ) self . params = [ self . w , self . b ] def set_inpt ( self , inpt , inpt_dropout , mini_batch_size ): self . inpt = inpt . reshape (( mini_batch_size , self . n_in )) self . output = softmax (( 1 - self . p_dropout ) * T . dot ( self . inpt , self . w ) + self . b ) self . y_out = T . argmax ( self . output , axis = 1 ) self . inpt_dropout = dropout_layer ( inpt_dropout . reshape (( mini_batch_size , self . n_in )), self . p_dropout ) self . output_dropout = softmax ( T . dot ( self . inpt_dropout , self . w ) + self . b ) def cost ( self , net ): "Return the log-likelihood cost." return - T . mean ( T . log ( self . output_dropout )[ T . arange ( net . y . shape [ 0 ]), net . y ]) def accuracy ( self , y ): "Return the accuracy for the mini-batch." return T . mean ( T . eq ( y , self . y_out )) #### Miscellanea def size ( data ): "Return the size of the dataset `data`." return data [ 0 ] . get_value ( borrow = True ) . shape [ 0 ] def dropout_layer ( layer , p_dropout ): srng = shared_randomstreams . RandomStreams ( np . random . RandomState ( 0 ) . randint ( 999999 )) mask = srng . binomial ( n = 1 , p = 1 - p_dropout , size = layer . shape ) return layer * T . cast ( mask , theano . config . floatX )

At present, the SGD method requires the user to manually choose the number of epochs to train for. Earlier in the book we discussed an automated way of selecting the number of epochs to train for, known as early stopping. Modify network3.py to implement early stopping.

method requires the user to manually choose the number of epochs to train for. Earlier in the book we discussed an automated way of selecting the number of epochs to train for, known as early stopping. Modify to implement early stopping. Add a Network method to return the accuracy on an arbitrary data set.

method to return the accuracy on an arbitrary data set. Modify the SGD method to allow the learning rate $\eta$ to be a function of the epoch number. Hint: After working on this problem for a while, you may find it useful to see the discussion at this link.

method to allow the learning rate $\eta$ to be a function of the epoch number. Hint: After working on this problem for a while, you may find it useful to see the discussion at this link. Earlier in the chapter I described a technique for expanding the training data by applying (small) rotations, skewing, and translation. Modify network3.py to incorporate all these techniques. Note: Unless you have a tremendous amount of memory, it is not practical to explicitly generate the entire expanded data set. So you should consider alternate approaches.

to incorporate all these techniques. Note: Unless you have a tremendous amount of memory, it is not practical to explicitly generate the entire expanded data set. So you should consider alternate approaches. Add the ability to load and save networks to network3.py .

. A shortcoming of the current code is that it provides few diagnostic tools. Can you think of any diagnostics to add that would make it easier to understand to what extent a network is overfitting? Add them.

We've used the same initialization procedure for rectified linear units as for sigmoid (and tanh) neurons. Our argument for that initialization was specific to the sigmoid function. Consider a network made entirely of rectified linear units (including outputs). Show that rescaling all the weights in the network by a constant factor $c > 0$ simply rescales the outputs by a factor $c^{L-1}$, where $L$ is the number of layers. How does this change if the final layer is a softmax? What do you think of using the sigmoid initialization procedure for the rectified linear units? Can you think of a better initialization procedure? Note: This is a very open-ended problem, not something with a simple self-contained answer. Still, considering the problem will help you better understand networks containing rectified linear units.

Our analysis of the unstable gradient problem was for sigmoid neurons. How does the analysis change for networks made up of rectified linear units? Can you think of a good way of modifying such a network so it doesn't suffer from the unstable gradient problem? Note: The word good in the second part of this makes the problem a research problem. It's actually easy to think of ways of making such modifications. But I haven't investigated in enough depth to know of a really good technique.

In 1998, the year MNIST was introduced, it took weeks to train a state-of-the-art workstation to achieve accuracies substantially worse than those we can achieve using a GPU and less than an hour of training. Thus, MNIST is no longer a problem that pushes the limits of available technique; rather, the speed of training means that it is a problem good for teaching and learning purposes. Meanwhile, the focus of research has moved on, and modern work involves much more challenging image recognition problems. In this section, I briefly describe some recent work on image recognition using neural networks.

The section is different to most of the book. Through the book I've focused on ideas likely to be of lasting interest - ideas such as backpropagation, regularization, and convolutional networks. I've tried to avoid results which are fashionable as I write, but whose long-term value is unknown. In science, such results are more often than not ephemera which fade and have little lasting impact. Given this, a skeptic might say: "well, surely the recent progress in image recognition is an example of such ephemera? In another two or three years, things will have moved on. So surely these results are only of interest to a few specialists who want to compete at the absolute frontier? Why bother discussing it?"

Such a skeptic is right that some of the finer details of recent papers will gradually diminish in perceived importance. With that said, the past few years have seen extraordinary improvements using deep nets to attack extremely difficult image recognition tasks. Imagine a historian of science writing about computer vision in the year 2100. They will identify the years 2011 to 2015 (and probably a few years beyond) as a time of huge breakthroughs, driven by deep convolutional nets. That doesn't mean deep convolutional nets will still be used in 2100, much less detailed ideas such as dropout, rectified linear units, and so on. But it does mean that an important transition is taking place, right now, in the history of ideas. It's a bit like watching the discovery of the atom, or the invention of antibiotics: invention and discovery on a historic scale. And so while we won't dig down deep into details, it's worth getting some idea of the exciting discoveries currently being made.

The 2012 LRMD paper: Let me start with a 2012 paper* *Building high-level features using large scale unsupervised learning, by Quoc Le, Marc'Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg Corrado, Jeff Dean, and Andrew Ng (2012). Note that the detailed architecture of the network used in the paper differed in many details from the deep convolutional networks we've been studying. Broadly speaking, however, LRMD is based on many similar ideas. from a group of researchers from Stanford and Google. I'll refer to this paper as LRMD, after the last names of the first four authors. LRMD used a neural network to classify images from ImageNet, a very challenging image recognition problem. The 2011 ImageNet data that they used included 16 million full color images, in 20 thousand categories. The images were crawled from the open net, and classified by workers from Amazon's Mechanical Turk service. Here's a few ImageNet images* *These are from the 2014 dataset, which is somewhat changed from 2011. Qualitatively, however, the dataset is extremely similar. Details about ImageNet are available in the original ImageNet paper, ImageNet: a large-scale hierarchical image database, by Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei (2009).:

These are, respectively, in the categories for beading plane, brown root rot fungus, scalded milk, and the common roundworm. If you're looking for a challenge, I encourage you to visit ImageNet's list of hand tools, which distinguishes between beading planes, block planes, chamfer planes, and about a dozen other types of plane, amongst other categories. I don't know about you, but I cannot confidently distinguish between all these tool types. This is obviously a much more challenging image recognition task than MNIST! LRMD's network obtained a respectable $15.8$ percent accuracy for correctly classifying ImageNet images. That may not sound impressive, but it was a huge improvement over the previous best result of $9.3$ percent accuracy. That jump suggested that neural networks might offer a powerful approach to very challenging image recognition tasks, such as ImageNet.

The 2012 KSH paper: The work of LRMD was followed by a 2012 paper of Krizhevsky, Sutskever and Hinton (KSH)* *ImageNet classification with deep convolutional neural networks, by Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton (2012).. KSH trained and tested a deep convolutional neural network using a restricted subset of the ImageNet data. The subset they used came from a popular machine learning competition - the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC). Using a competition dataset gave them a good way of comparing their approach to other leading techniques. The ILSVRC-2012 training set contained about 1.2 million ImageNet images, drawn from 1,000 categories. The validation and test sets contained 50,000 and 150,000 images, respectively, drawn from the same 1,000 categories.

One difficulty in running the ILSVRC competition is that many ImageNet images contain multiple objects. Suppose an image shows a labrador retriever chasing a soccer ball. The so-called "correct" ImageNet classification of the image might be as a labrador retriever. Should an algorithm be penalized if it labels the image as a soccer ball? Because of this ambiguity, an algorithm was considered correct if the actual ImageNet classification was among the $5$ classifications the algorithm considered most likely. By this top-$5$ criterion, KSH's deep convolutional network achieved an accuracy of $84.7$ percent, vastly better than the next-best contest entry, which achieved an accuracy of $73.8$ percent. Using the more restrictive metric of getting the label exactly right, KSH's network achieved an accuracy of $63.3$ percent.

It's worth briefly describing KSH's network, since it has inspired much subsequent work. It's also, as we shall see, closely related to the networks we trained earlier in this chapter, albeit more elaborate. KSH used a deep convolutional neural network, trained on two GPUs. They used two GPUs because the particular type of GPU they were using (an NVIDIA GeForce GTX 580) didn't have enough on-chip memory to store their entire network. So they split the network into two parts, partitioned across the two GPUs.

The KSH network has $7$ layers of hidden neurons. The first $5$ hidden layers are convolutional layers (some with max-pooling), while the next $2$ layers are fully-connected layers. The output layer is a $1,000$-unit softmax layer, corresponding to the $1,000$ image classes. Here's a sketch of the network, taken from the KSH paper* *Thanks to Ilya Sutskever.. The details are explained below. Note that many layers are split into $2$ parts, corresponding to the $2$ GPUs.

The input layer contains $3 \times 224 \times 224$ neurons, representing the RGB values for a $224 \times 224$ image. Recall that, as mentioned earlier, ImageNet contains images of varying resolution. This poses a problem, since a neural network's input layer is usually of a fixed size. KSH dealt with this by rescaling each image so the shorter side had length $256$. They then cropped out a $256 \times 256$ area in the center of the rescaled image. Finally, KSH extracted random $224 \times 224$ subimages (and horizontal reflections) from the $256 \times 256$ images. They did this random cropping as a way of expanding the training data, and thus reducing overfitting. This is particularly helpful in a large network such as KSH's. It was these $224 \times 224$ images which were used as inputs to the network. In most cases the cropped image still contains the main object from the uncropped image.

Moving on to the hidden layers in KSH's network, the first hidden layer is a convolutional layer, with a max-pooling step. It uses local receptive fields of size $11 \times 11$, and a stride length of $4$ pixels. There are a total of $96$ feature maps. The feature maps are split into two groups of $48$ each, with the first $48$ feature maps residing on one GPU, and the second $48$ feature maps residing on the other GPU. The max-pooling in this and later layers is done in $3 \times 3$ regions, but the pooling regions are allowed to overlap, and are just $2$ pixels apart.

The second hidden layer is also a convolutional layer, with a max-pooling step. It uses $5 \times 5$ local receptive fields, and there's a total of $256$ feature maps, split into $128$ on each GPU. Note that the feature maps only use $48$ input channels, not the full $96$ output from the previous layer (as would usually be the case). This is because any single feature map only uses inputs from the same GPU. In this sense the network departs from the convolutional architecture we described earlier in the chapter, though obviously the basic idea is still the same.

The third, fourth and fifth hidden layers are convolutional layers, but unlike the previous layers, they do not involve max-pooling. Their respectives parameters are: (3) $384$ feature maps, with $3 \times 3$ local receptive fields, and $256$ input channels; (4) $384$ feature maps, with $3 \times 3$ local receptive fields, and $192$ input channels; and (5) $256$ feature maps, with $3 \times 3$ local receptive fields, and $192$ input channels. Note that the third layer involves some inter-GPU communication (as depicted in the figure) in order that the feature maps use all $256$ input channels.

The sixth and seventh hidden layers are fully-connected layers, with $4,096$ neurons in each layer.

The output layer is a $1,000$-unit softmax layer.

The KSH network takes advantage of many techniques. Instead of using the sigmoid or tanh activation functions, KSH use rectified linear units, which sped up training significantly. KSH's network had roughly 60 million learned parameters, and was thus, even with the large training set, susceptible to overfitting. To overcome this, they expanded the training set using the random cropping strategy we discussed above. They also further addressed overfitting by using a variant of l2 regularization, and dropout. The network itself was trained using momentum-based mini-batch stochastic gradient descent.

That's an overview of many of the core ideas in the KSH paper. I've omitted some details, for which you should look at the paper. You can also look at Alex Krizhevsky's cuda-convnet (and successors), which contains code implementing many of the ideas. A Theano-based implementation has also been developed* *Theano-based large-scale visual recognition with multiple GPUs, by Weiguang Ding, Ruoyan Wang, Fei Mao, and Graham Taylor (2014)., with the code available here. The code is recognizably along similar lines to that developed in this chapter, although the use of multiple GPUs complicates things somewhat. The Caffe neural nets framework also includes a version of the KSH network, see their Model Zoo for details.

The 2014 ILSVRC competition: Since 2012, rapid progress continues to be made. Consider the 2014 ILSVRC competition. As in 2012, it involved a training set of $1.2$ million images, in $1,000$ categories, and the figure of merit was whether the top $5$ predictions included the correct category. The winning team, based primarily at Google* *Going deeper with convolutions, by Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich (2014)., used a deep convolutional network with $22$ layers of neurons. They called their network GoogLeNet, as a homage to LeNet-5. GoogLeNet achieved a top-5 accuracy of $93.33$ percent, a giant improvement over the 2013 winner (Clarifai, with $88.3$ percent), and the 2012 winner (KSH, with $84.7$ percent).

Just how good is GoogLeNet's $93.33$ percent accuracy? In 2014 a team of researchers wrote a survey paper about the ILSVRC competition* *ImageNet large scale visual recognition challenge, by Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei (2014).. One of the questions they address is how well humans perform on ILSVRC. To do this, they built a system which lets humans classify ILSVRC images. As one of the authors, Andrej Karpathy, explains in an informative blog post, it was a lot of trouble to get the humans up to GoogLeNet's performance:

...the task of labeling images with 5 out of 1000 categories quickly turned out to be extremely challenging, even for some friends in the lab who have been working on ILSVRC and its classes for a while. First we thought we would put it up on [Amazon Mechanical Turk]. Then we thought we could recruit paid undergrads. Then I organized a labeling party of intense labeling effort only among the (expert labelers) in our lab. Then I developed a modified interface that used GoogLeNet predictions to prune the number of categories from 1000 to only about 100. It was still too hard - people kept missing categories and getting up to ranges of 13-15% error rates. In the end I realized that to get anywhere competitively close to GoogLeNet, it was most efficient if I sat down and went through the painfully long training process and the subsequent careful annotation process myself... The labeling happened at a rate of about 1 per minute, but this decreased over time... Some images are easily recognized, while some images (such as those of fine-grained breeds of dogs, birds, or monkeys) can require multiple minutes of concentrated effort. I became very good at identifying breeds of dogs... Based on the sample of images I worked on, the GoogLeNet classification error turned out to be 6.8%... My own error in the end turned out to be 5.1%, approximately 1.7% better.

In other words, an expert human, working painstakingly, was with great effort able to narrowly beat the deep neural network. In fact, Karpathy reports that a second human expert, trained on a smaller sample of images, was only able to attain a $12.0$ percent top-5 error rate, significantly below GoogLeNet's performance. About half the err