This tutorial shows how to download videos from YouTube and to process their frames with Python; I have used this technique to create game barcode, an image created by sorting the colours in each frame of a particular video. You can see some of most intriguing here:

FEZ (video, 2000px)

Journey (video, 2000px)

Super Mario World (video, 2000px)

Fallout 4 (video, 2000px)

This tutorial is divided in four parts:

Introduction

The idea for this tutorial came to me after reading a post on Reddit titled The average color of every frame of a given movie. I immediately wanted to apply this technique to games. The main reason is that games, compared to movie, have much smoother transitions. This allows to see how colours change very clearly. On top of that, I wanted to explore this concept further, by applying the techniques I’ve discussed in The Incredibly Challenging Task of Sorting Colours. I tested my hypothesis on a short clip from a game I am working on, and the result was quite stunning.

Another important reason why I wanted to use games rather than movies, is that working with the latter brings you dangerously close to the neutral zone of piracy and copyright infringement. Switching to games, however, you are forced to used a particular Let’s Play video. All the game barcodes in this post have been generated from Let’s Plays I have found on YouTube. I have no affiliation with their respective creators. You can see all the game barcodes here.

Step 1. Processing a video

Let’s start with a top-down approach. Each movie is processed by the process_video function. Frames are extracted using the library OpenCV, then passed through process_frame which returns the list of sorted colours within that frame.

import cv2 def process_video (input_movie, size=(2000,100)): colours = [] # Takes the frames of the video cap = cv2.VideoCapture(input_movie) while cap.isOpened(): flag, frame = cap.read() if not flag: continue # Processes the frame colours_frame = process_frame(frame, size[1]) colours.append(colours_frame) # Generates the final picture generate_pic(colours, size) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 import cv2 def process_video ( input_movie , size = ( 2000 , 100 ) ) : colours = [ ] # Takes the frames of the video cap = cv2 . VideoCapture ( input_movie ) while cap . isOpened ( ) : flag , frame = cap . read ( ) if not flag : continue # Processes the frame colours_frame = process_frame ( frame , size [ 1 ] ) colours . append ( colours_frame ) # Generates the final picture generate_pic ( colours , size )

Finally, all the sorted colours are given to generate_pic which joints them together to create the final picture.

Step 2. Processing a frame

Processing a frame is relatively straightforward. First, the frame is reduced in size to save time and memory. Then, is it reshaped into an array and sorted. For this process I have chosen to sort the colours in the HSV space, since it is more consistent across different frames. In the last step, the sorted array is converted back into an image and interpolated to the requested size using cv2.resize.

import numpy as np import colorsys def process_frame (frame, height=100): # Resize and put in a single line image = resize_image(frame) image = cv2.cvtColor(image, cv2.COLOR_RGB2HSV) image = image.reshape((image.shape[0] * image.shape[1], 1, 3)) # Sort the pixels sorted_idx = np.lexsort( (image[:,0,2], image[:,0,1], image[:,0,0] ) ) image = image[sorted_idx] # Resize into a column image_column = cv2.resize(image, (1, height), interpolation=cv2.INTER_AREA) return image_column 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 import numpy as np import colorsys def process_frame ( frame , height = 100 ) : # Resize and put in a single line image = resize_image ( frame ) image = cv2 . cvtColor ( image , cv2 . COLOR_RGB2HSV ) image = image . reshape ( ( image . shape [ 0 ] * image . shape [ 1 ] , 1 , 3 ) ) # Sort the pixels sorted_idx = np . lexsort ( ( image [ : , 0 , 2 ] , image [ : , 0 , 1 ] , image [ : , 0 , 0 ] ) ) image = image [ sorted_idx ] # Resize into a column image_column = cv2 . resize ( image , ( 1 , height ) , interpolation = cv2 . INTER_AREA ) return image_column

The function to resize the original frame is this one:

def resize_image (image, size=100): # Resize it h, w, _ = image.shape w_new = int(size * w / max(w, h) ) h_new = int(size * h / max(w, h) ) image = cv2.resize(image, (w_new, h_new)); return image 1 2 3 4 5 6 7 8 def resize_image ( image , size = 100 ) : # Resize it h , w , _ = image . shape w_new = int ( size * w / max ( w , h ) ) h_new = int ( size * h / max ( w , h ) ) image = cv2 . resize ( image , ( w_new , h_new ) ) ; return image

Step 3. Creating the barcode

The final step to create the image is to join all the sorted columns together. We also convert the image back to RGB.

def generate_pic (colours, size): # Generates the picture height = size[1] img = np.zeros((height,len(colours),3), np.uint8) # Puts the colours in the image for x in range(0, len(colours)): for y in range(0, height): img[y,x,:] = colours[x][y,0] # Converts back to RGB img = cv2.cvtColor(img, cv2.COLOR_HSV2RGB) cv2.imwrite("barcode_full.png", img) 1 2 3 4 5 6 7 8 9 10 11 12 13 def generate_pic ( colours , size ) : # Generates the picture height = size [ 1 ] img = np . zeros ( ( height , len ( colours ) , 3 ) , np . uint8 ) # Puts the colours in the image for x in range ( 0 , len ( colours ) ) : for y in range ( 0 , height ) : img [ y , x , : ] = colours [ x ] [ y , 0 ] # Converts back to RGB img = cv2 . cvtColor ( img , cv2 . COLOR_HSV2RGB ) cv2 . imwrite ( "barcode_full.png" , img )

If you want to resize the image to a more manageable size, you can use this code:

image_1 = cv2.resize(img, size, interpolation=cv2.INTER_AREA) cv2.imwrite("barcode_%d.png" % size[0], image_1) 1 2 image_1 = cv2 . resize ( img , size , interpolation = cv2 . INTER_AREA ) cv2 . imwrite ( "barcode_%d.png" % size [ 0 ] , image_1 )

Step 4. Downloading from YouTube

Since many Let’s Play are on YouTube, it makes sense to invest some time to download them directly. I have used pafy, a Python library which allows to query YouTube.

# Download the video video = pafy.new('https://www.youtube.com/watch?v=Re5NFApk9dQ') resolution = video.getbestvideo(preftype="mp4") input_movie = resolution.download(quiet=False) # Process it process_video(input_movie) os.remove(input_movie) 1 2 3 4 5 6 7 8 # Download the video video = pafy . new ( 'https://www.youtube.com/watch?v=Re5NFApk9dQ' ) resolution = video . getbestvideo ( preftype = "mp4" ) input_movie = resolution . download ( quiet = False ) # Process it process_video ( input_movie ) os . remove ( input_movie )

If you work is derivative (i.e.: if you are using someone else video), don’t forget to credit them.

Conclusion

If you are curious to see this technique applied to movies, there is a nice selection here.

Other resources