The previous post in this series, Understanding Deep Dreams, explained what deep dreams are, and what they can be used for. In this second post you’ll learn how to create them, with a step by step guide.

Introduction

In August 2015, Google Research released the full source code they have been using to generate their pictures. My guide is strongly inspired by this one, which unfortunately didn’t work for many users. The following instructions will show how to virtualise Ubuntu 14.04.02 with VirtualBox, and how to run your very own deep dreams on it.

You can test this very technique using DeepDreamThis, a twitter bot that will create deep dreams out of the images you send to it. You can check a detailed guide on how to use it here. You can also check DeepForger, which uses neural networks to transfer artistic styles from a painting to a picture.

The Instructions

Virtualise Ubuntu 14.04.02 Install VirtualBox Download the Ubuntu 14.04.02 iso Create a new Virtual machine for “Linux / Ubuntu 64bit“. You’ll need at least 10Gb of space and at least 2048Mb of RAM In the Settings > Storage properties, add the downloaded iso of Ubuntu in the CD drive Run the virtual machine Install Ubuntu

Install Guest Additions From the VirtualBox windows: Device > “Insert Guest Additions CD image“ Install Restart the machine

Install build essentials sudo apt - get install build - essential sudo apt - get install linux - headers - ` uname - r `

Install caffe and deepdream dependencies sudo apt - get install - y curl libprotobuf - dev libleveldb - dev libsnappy - dev libopencv - dev libboost - all - dev libhdf5 - serial - dev protobuf - compiler gfortran libjpeg62 libfreeimage - dev libatlas - base - dev git python - dev python - pip libgoogle - glog - dev libbz2 - dev libxml2 - dev libxslt - dev libffi - dev libssl - dev libgflags - dev liblmdb - dev python - yaml imagemagick python - opencv sudo easy_install pillow

and dependencies Download caffe cd ~ git clone https : // github .com / BVLC / caffe .git

Install caffe python dependencies cd caffe cat python / requirements .txt | xargs - L 1 sudo pip install sudo ln - s / usr / include / python2 . 7 / / usr / local / include / python2 . 7 sudo ln - s / usr / local / lib / python2 . 7 / dist - packages / numpy / core / include / numpy / / usr / local / include / python2 . 7 / numpy

python dependencies Configure caffe cp Makefile .config .example Makefile .config nano Makefile .config Uncomment the line # CPU_ONLY := 1 Under PYTHON_INCLUDE , replace / usr / lib / python2 . 7 / dist - packages / numpy / core / include with / usr / local / lib / python2 . 7 / dist - packages / numpy / core / include

Compile caffe make pycaffe make all make test make runtest Be sure the output of make runtest doesn’t include any error make distribute

Make pycaffe available to python sudo cp distribute / lib / libcaffe .so distribute / python / caffe sudo cp - r distribute / python / caffe / usr / local / lib / python2 . 7 / dist - packages / caffe sudo nano / etc / ld .so .conf .d / caffe .conf LD_LIBRARY_PATH = $ LD_LIBRARY_PATH : ~ / caffe / distribute / python / caffe Write the full path to the caffe / distribute / lib / folder; for instance: / home / alanzucconi / caffe / distribute / lib / . Do not use ~ . sudo ldconfig You should also put these lines in the .bashrc file, so that they are executed when your computer starts (remember to use the path to your folders):

export LD_LIBRARY_PATH = $ LD_LIBRARY_PATH : ~ / caffe / distribute / python / caffe export PYTHONPATH = $ PYTHONPATH : ~ / caffe / distribute / python /

available to Download the GoogLeNet model cd ~ / caffe / models / bvlc_googlenet curl - O "http://dl.caffe.berkeleyvision.org/bvlc_googlenet.caffemodel"

Download deepdream cd ~ git clone https : // github .com / google / deepdream .git

The code In the deepdream folder create a file called dream .py Fill it with the code below cd ~ / deepdream sudo python dream .py



The code

dream.py # imports and basic notebook setup from cStringIO import StringIO import numpy as np import scipy.ndimage as nd import PIL.Image from IPython.display import clear_output, Image, display from google.protobuf import text_format import cv2 import caffe showimages = True def showarray(a, fmt='jpeg'): a = np.uint8(np.clip(a, 0, 255)) f = StringIO() PIL.Image.fromarray(a).save(f, fmt) global showimages if showimages: a = cv2.cvtColor(a, cv2.COLOR_BGR2RGB) cv2.imshow('image',a) cv2.waitKey(1) model_path = '../caffe/models/bvlc_googlenet/' # substitute your path here net_fn = model_path + 'deploy.prototxt' param_fn = model_path + 'bvlc_googlenet.caffemodel' # Patching model to be able to compute gradients. # Note that you can also manually add "force_backward: true" line to "deploy.prototxt". model = caffe.io.caffe_pb2.NetParameter() text_format.Merge(open(net_fn).read(), model) model.force_backward = True open('tmp.prototxt', 'w').write(str(model)) net = caffe.Classifier('tmp.prototxt', param_fn, mean = np.float32([104.0, 116.0, 122.0]), # ImageNet mean, training set dependent channel_swap = (2,1,0)) # the reference model has channels in BGR order instead of RGB # a couple of utility functions for converting to and from Caffe's input image layout def preprocess(net, img): return np.float32(np.rollaxis(img, 2)[::-1]) - net.transformer.mean['data'] def deprocess(net, img): return np.dstack((img + net.transformer.mean['data'])[::-1]) def make_step(net, step_size=1.5, end='inception_4c/output', jitter=32, clip=True): '''Basic gradient ascent step.''' src = net.blobs['data'] # input image is stored in Net's 'data' blob dst = net.blobs[end] ox, oy = np.random.randint(-jitter, jitter+1, 2) src.data[0] = np.roll(np.roll(src.data[0], ox, -1), oy, -2) # apply jitter shift net.forward(end=end) dst.diff[:] = dst.data # specify the optimization objective net.backward(start=end) g = src.diff[0] # apply normalized ascent step to the input image src.data[:] += step_size/np.abs(g).mean() * g src.data[0] = np.roll(np.roll(src.data[0], -ox, -1), -oy, -2) # unshift image if clip: bias = net.transformer.mean['data'] src.data[:] = np.clip(src.data, -bias, 255-bias) def deepdream(net, base_img, iter_n=10, octave_n=4, octave_scale=1.4, end='inception_4c/output', clip=True, **step_params): # prepare base images for all octaves octaves = [preprocess(net, base_img)] for i in xrange(octave_n-1): octaves.append(nd.zoom(octaves[-1], (1, 1.0/octave_scale,1.0/octave_scale), order=1)) src = net.blobs['data'] detail = np.zeros_like(octaves[-1]) # allocate image for network-produced details for octave, octave_base in enumerate(octaves[::-1]): h, w = octave_base.shape[-2:] if octave > 0: # upscale details from the previous octave h1, w1 = detail.shape[-2:] detail = nd.zoom(detail, (1, 1.0*h/h1,1.0*w/w1), order=1) src.reshape(1,3,h,w) # resize the network's input image size src.data[0] = octave_base+detail for i in xrange(iter_n): make_step(net, end=end, clip=clip, **step_params) # visualization vis = deprocess(net, src.data[0]) if not clip: # adjust image contrast if clipping is disabled vis = vis*(255.0/np.percentile(vis, 99.98)) showarray(vis) print octave, i, end, vis.shape clear_output(wait=True) # extract details produced on the current octave detail = src.data[0]-octave_base # returning the resulting image return deprocess(net, src.data[0]) img = np.float32(PIL.Image.open('sky1024px.jpg')) showarray(img) frame=deepdream(net, img) PIL.Image.fromarray(np.uint8(frame)).save("frames/a.jpg") frame=deepdream(net, img, end='inception_3b/5x5_reduce') PIL.Image.fromarray(np.uint8(frame)).save("frames/b.jpg") frame = img frame_i = 0 h, w = frame.shape[:2] s = 0.05 # scale coefficient for i in xrange(100): frame = deepdream(net, frame) PIL.Image.fromarray(np.uint8(frame)).save("frames/%04d.jpg"%frame_i) frame = nd.affine_transform(frame, [1-s,1-s,1], [h*s/2,w*s/2,0], order=1) frame_i += 1 cv2.destroyAllWindows() 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 # imports and basic notebook setup from cStringIO import StringIO import numpy as np import scipy . ndimage as nd import PIL . Image from IPython . display import clear_output , Image , display from google . protobuf import text_format import cv2 import caffe showimages = True def showarray ( a , fmt = 'jpeg' ) : a = np . uint8 ( np . clip ( a , 0 , 255 ) ) f = StringIO ( ) PIL . Image . fromarray ( a ) . save ( f , fmt ) global showimages if showimages : a = cv2 . cvtColor ( a , cv2 . COLOR_BGR2RGB ) cv2 . imshow ( 'image' , a ) cv2 . waitKey ( 1 ) model_path = '../caffe/models/bvlc_googlenet/' # substitute your path here net_fn = model_path + 'deploy.prototxt' param_fn = model_path + 'bvlc_googlenet.caffemodel' # Patching model to be able to compute gradients. # Note that you can also manually add "force_backward: true" line to "deploy.prototxt". model = caffe . io . caffe_pb2 . NetParameter ( ) text_format . Merge ( open ( net_fn ) . read ( ) , model ) model . force_backward = True open ( 'tmp.prototxt' , 'w' ) . write ( str ( model ) ) net = caffe . Classifier ( 'tmp.prototxt' , param_fn , mean = np . float32 ( [ 104.0 , 116.0 , 122.0 ] ) , # ImageNet mean, training set dependent channel_swap = ( 2 , 1 , 0 ) ) # the reference model has channels in BGR order instead of RGB # a couple of utility functions for converting to and from Caffe's input image layout def preprocess ( net , img ) : return np . float32 ( np . rollaxis ( img , 2 ) [ : : - 1 ] ) - net . transformer . mean [ 'data' ] def deprocess ( net , img ) : return np . dstack ( ( img + net . transformer . mean [ 'data' ] ) [ : : - 1 ] ) def make_step ( net , step_size = 1.5 , end = 'inception_4c/output' , jitter = 32 , clip = True ) : '''Basic gradient ascent step.''' src = net . blobs [ 'data' ] # input image is stored in Net's 'data' blob dst = net . blobs [ end ] ox , oy = np . random . randint ( - jitter , jitter + 1 , 2 ) src . data [ 0 ] = np . roll ( np . roll ( src . data [ 0 ] , ox , - 1 ) , oy , - 2 ) # apply jitter shift net . forward ( end = end ) dst . diff [ : ] = dst . data # specify the optimization objective net . backward ( start = end ) g = src . diff [ 0 ] # apply normalized ascent step to the input image src . data [ : ] += step_size / np . abs ( g ) . mean ( ) * g src . data [ 0 ] = np . roll ( np . roll ( src . data [ 0 ] , - ox , - 1 ) , - oy , - 2 ) # unshift image if clip : bias = net . transformer . mean [ 'data' ] src . data [ : ] = np . clip ( src . data , - bias , 255 - bias ) def deepdream ( net , base_img , iter_n = 10 , octave_n = 4 , octave_scale = 1.4 , end = 'inception_4c/output' , clip = True , ** step_params ) : # prepare base images for all octaves octaves = [ preprocess ( net , base_img ) ] for i in xrange ( octave_n - 1 ) : octaves . append ( nd . zoom ( octaves [ - 1 ] , ( 1 , 1.0 / octave_scale , 1.0 / octave_scale ) , order = 1 ) ) src = net . blobs [ 'data' ] detail = np . zeros_like ( octaves [ - 1 ] ) # allocate image for network-produced details for octave , octave_base in enumerate ( octaves [ : : - 1 ] ) : h , w = octave_base . shape [ - 2 : ] if octave > 0 : # upscale details from the previous octave h1 , w1 = detail . shape [ - 2 : ] detail = nd . zoom ( detail , ( 1 , 1.0 * h / h1 , 1.0 * w / w1 ) , order = 1 ) src . reshape ( 1 , 3 , h , w ) # resize the network's input image size src . data [ 0 ] = octave_base + detail for i in xrange ( iter_n ) : make_step ( net , end = end , clip = clip , ** step_params ) # visualization vis = deprocess ( net , src . data [ 0 ] ) if not clip : # adjust image contrast if clipping is disabled vis = vis * ( 255.0 / np . percentile ( vis , 99.98 ) ) showarray ( vis ) print octave , i , end , vis . shape clear_output ( wait = True ) # extract details produced on the current octave detail = src . data [ 0 ] - octave_base # returning the resulting image return deprocess ( net , src . data [ 0 ] ) img = np . float32 ( PIL . Image . open ( 'sky1024px.jpg' ) ) showarray ( img ) frame = deepdream ( net , img ) PIL . Image . fromarray ( np . uint8 ( frame ) ) . save ( "frames/a.jpg" ) frame = deepdream ( net , img , end = 'inception_3b/5x5_reduce' ) PIL . Image . fromarray ( np . uint8 ( frame ) ) . save ( "frames/b.jpg" ) frame = img frame_i = 0 h , w = frame . shape [ : 2 ] s = 0.05 # scale coefficient for i in xrange ( 100 ) : frame = deepdream ( net , frame ) PIL . Image . fromarray ( np . uint8 ( frame ) ) . save ( "frames/%04d.jpg" % frame_i ) frame = nd . affine_transform ( frame , [ 1 - s , 1 - s , 1 ] , [ h * s / 2 , w * s / 2 , 0 ] , order = 1 ) frame_i += 1 cv2 . destroyAllWindows ( )

deepdream

end

net . blobs . keys ( )

uses thefunction to generate different dreams. Theparameter is used to determine which layer we want to analyse. You can check which layers can be analysed by printing

I’ve created a repository of all the layers you can query. Find an effect you like, and use the name of the file (without the extension) and the end parameter. Just remember to replace any __ (double underscore) with a / (Unix slash).

Conclusion

This post explained how to use the machine learning framework Caffe to generate deep dreams. Using a virtualised environment, however, might not be the best option. Neural networks require a lot of computational power, which is often unavailable in a virtual machine. For best performances, you should run deep dreams using a GPU.

This tutorial referred to the first, original tutorial released by Google Research. Other techniques have been evolved in the past months, including algorithms that works on videos.

Other resources

Part 1. Understanding Deep Dreams

Part 2. Generating Deep Dreams

deepdream: the GIT project used by Google Research;

Caffe: the framework used to implement neural networks. Valid alternatives are Theano and Torch;

Running the deep dream: an alternative tutorial to run deep dreams;

neural-style: a Torch variation of deep dreams, used to transfer artistic styles;

neural-doodle: a tool that allows to draw in the style of a particular artist.