Neural Style Transfer

What is it?

Neural Style Transfer is a fun application and will improve your understanding of CNNs. In its essence, you try to generate a new image which combines the content of one image with the style of another image, say from a popular artist. Do you want to know how Picasso would have painted you? Go ahead and try it out for yourself with Neural Style Transfer!

Van Gogh’s Starry Night applied to a GIF.

How does Neural Style Transfer work?

In Neural Style Transfer, you start with a generated image G, which contains random pixel values, as shown below. Next, you define a content image C and a style image S, which you want to combine. Your goal is to adjust the pixel values in G so that G becomes similar to both C and S. To do so, you define the cost functions J(C) and J(S) and try to minimize both.

Random pixel values to start NST.

J(C) makes sure that G looks similar to the content in C. You know that CNNs learn to recognize lower-level features like edges in earlier hidden layers and more complex features like faces in later hidden layers. Pick a hidden layer in the middle of the CNN and run forward propagation with C and G. Next, you compare the activation values for both images and try to minimize the difference between the activation values through backpropagation.

Next, you also have to adjust the style of G to match the style in S. The key to minimize J(S) is to adjust the correlation between the channel activations in G to match S. You do that through calculating the Gram matrices for S and G. The Gram matrix calculates every possible pair of filter combinations. Next, you choose a layer in the middle of the CNN and run forward propagation again for S and G. J(S) minimizes the difference between the gram matrices through backpropagation and makes G look more similar to S.