This tutorial will cover the theory and practice of creating deepfakes: videos in which faces have been swapped using Machine Learning and Deep Neural Networks. If you are interested in learning more about this novel technique, this is the course for you.

After a theoretical introduction, this course will focus on how to make the most out of the popular applications FakeApp and faceswap; most of the deepfakes you will find online (such as the ones featuring Nicolas Cage) have been created using them.

You can read all the posts in this series here:

Introduction

Face detection has been a major research subject in the early 2000s. Almost twenty years later, this problem is basically solved and face detection is available as a library in most programming languages. Even face-swap technology is nothing new, and has been around for a few years.

In his article from 2016, Face Swap using OpenCV, author Satya Mallick showed how to swap faces programmatically, warping and colour correcting Ted Cruz’s face to fit Donald Trump (below).

When applied correctly, this technique is uncannily good at swapping faces. But it has a major disadvantage: it only works on pre-existing pictures. It cannot, for instance, morph Donald Trump’s face to match the expression of Ted Cruz.

That has changed in late 2017, when a new approach to face-swap has appeared on Reddit. Such a breakthrough relies on neural networks, computational models that are loosely inspired by the way real brains process information. This novel technique allows generating so-called deepfakes, which actually morph a person’s face to mimic someone else’s features, although preserving the original facial expression.

When used properly, this technique allows the creation of photorealistic videos at an incredibly low cost. The finale of Rogue One, for instance, featured a digital version of Princess Leia; a very expensive scene which required the expertise of many people. Below, you can see a comparison between the original scene and another one recreated using Deep Learning.

Creating Deepfakes

At the moment there are two main applications used to create deepfakes: FakeApp and faceswap. Regardless of which one you will use, the process is mostly the same, and requires three steps: extraction, training and creation.

Extraction

The deep- in deepfakes comes from the fact that this face-swap technology uses Deep Learning. If you are familiar with the concept, you should know that deep learning often requires large amounts of data. Without hundreds (if not thousands!) of face pictures, you will not be able to create a deepfake video.

A way to get around this is to collect a number of video clips which feature the people you want to face-swap. The extraction process refers to the process of extracting all frames from these video clips, identifying the faces and aligning them.

The alignment is critical, since the neural network that performs the face swap requires all faces to have the same size (usually 256×256 pixels) and features aligned. Detecting and aligning faces is a problem that is considered mostly solved, and is done by most applications very efficiently.

Training

Training is a technical term borrowed from Machine Learning. In this case, it refers to the process which allows a neural network to convert a face into another. Although it takes several hours, the training phase needs to be done only once. Once completed, it can convert a face from person A into person B.

This is the most obscure part of the entire process, and I have dedicated two posts to explain how it works from a technical point of view: An Introduction to Neural Networks and Autoencoders and Understanding the Technology Behind DeepFakes). If you really want to create photorealistic deepfakes, a basic understanding of the process that generates them is necessary.

Creation

Once the training is complete, it is finally time to create a deepfake. Starting from a video, all frames are extracted and all faces are aligned. Then, each one is converted using the trained neural network. The final step is to merge the converted face back into the original frame. While this sounds like an easy task, it is actually where most face-swap applications go wrong.

The creation process is the only one which does not use any Machine Learning. The algorithm to stitch a face back onto an image is hard-coded, and lacks the flexibility to detect mistakes.

Also, each frame is processed independently; there is no temporal correlation between them, meaning that the final video might have some flickering. This is the part where more research is needed. If you are using faceswap instead of FakeApp, have a look at df which tries to improve the creation process.

Conclusion

Deep Learning has made photorealistic face-swap not just possible, but also accessible. This technique is still in its infancy and many more improvements are expected to happen in the next few years.

In the meantime, places like the FakeApp forum or the fakeapp GitHub page are where most of the technical discussion around deepfakes is currently taking place. The community around deepfakes is constantly exploring new approaches, and developers are often very willing to share their creations. This is the case of user ZeroCool22 which created a deepfake video of Jimmy Fallon interviewing himself.

Another interesting reading on the subject is Exploring DeepFakes.

It cannot be denied that deepfakes have finally shown the world a practical application of Deep Learning. However, this very technique has often been used without the explicit consent of the people involved. While this is unlikely to be an issue with videos such as the ones shown in this article, the same cannot be said when it is used to create pornographic content. This is why, before showing how to create deepfakes, the next lecture in this online course will focus entirely on the legal and ethical issues of deepfakes.

You can read all the posts in this series here:

A special thanks goes to Christos Sfetsios and David King, who gave me access to the machine I have used to create the deepfakes used in this tutorial.

👾 This course was made possible thanks to the support of patrons on Patreon

🥇 $25+ 🥈 $10+ 🥉 $8+ 🥉 $5+ 💵 $1+ Dry Cactus

Nicolas Gascon

Aakash Sastry

Binary Soul

Antonio Cervantes

Ariel Ahrens

Sean

Magnopus

Skippy

Takeshi Mike King

Há»¯u Lá»™c

John Faulkenbury

Jashan Chittesh

Piotr Podziemski

Dexter AndrÃ© Osiander

Perren Spence-Pearse

Roberta Mota

Arcadely

Keith Miller

Adam Boyne

Craig Bowler

Bryan Duggan

Shawn Asmussen

Dries De Smet

Druelbozo

TrojanFighter

Shohei Takei

1011101

Glenn Jones

Jean Georges

Adam Frisby

David Peterson

Daniel Farrow

Minseck Cho

Zac Dixon

Nathan Grotticelli

Jessie Thomson

Theo Lagendijk

Ben Beagley

Michael Cook

Anthony Duquette

Brad Hammond

Danny Huynh

zhouhua

SaharaX

TraceYang

Brandon Tay

Joakim Westgren

Rohit Gupta

David Erosa

Jonathan Hopkins

Michael Romaszewicz

Omar Espinosa

Kevin Comerford

Shimon Vainer

Fernando Monroy

benfei

Lawrence

Hammad Bashir

Al-Rashid Jamalul

Kieran Belkus

Yin Liu

A. Moore

fangeles

Pixeye

Juan Trelles Trabucco

JoneyZ

Kyle Phillips

SUPER315

Abel

Magnus Paulson Erga

HyperDigital

Joan Llobera

Alb PÃ©rez-Bermejo

FruityFusion

Mike Wuetherick

Luke Kaalim

Thomas Ingram

unizen

rizal ardianto

William Wilson

Stefano Gandolfo

Stephen

Lukasz Pasek

fredqin

Arielle Kim

Jack Riddle

Yangtze River

åŸ¹åŸº æœ±

Josquin Z

OccupiedVR

shangmingyou

Jordan Jackson

Rick Palmer

James Neal

Yaser Saqib

Apoc

Shane Nilsson Jarrod Phillips

Julian Heinken

FranÃ§ois ChÃ©nÃ© Moritz Voss

Jonathan Wright

Jason Adams

Jordan Carroll

Juwenw Sun

Aaron Eastburn

TomÃ¡Å¡ Koza

Chris Sorrell

rustinlee

obscuresteel

TxCx

Ciro

Elan Feingold

Eric Cloutier

Wilco Weggeman

Mario Gutierrez

FELIX

Jaewon Jung

Anthony Rosenbaum

Justin Loudermilk

Ryan Schake

Mike Corsaro

Sebastian KÄ™dra

Tom

MÃ¡rton HelÃ©nyi

CatchCo

Hans Palm

mars

Matei Giurgiu

Zephyr Zhang

Jules Stevenson

Edouard Philippe

Dave Cowling

Andy De Meyer

Phil Gosch

Dave Chenell

JeanD

Kit Matthews

Klemen BrajkoviÄ

Liam Brady

Andrew Johnson

Gil Damoiseaux

Jungchul Ha

J

Jeremy Zeler

Elijah Cauley

Jonadab Oomon

Jake OBrien

tmabs

Gordon Chapman

Kevin Lee Jr.

Colin Schultz

Brad Weiers

Gabor Tornyos

Marcel Medak

Anders Ã˜rum Pedersen

Dirk Van Welden

Thomas Key

Yotam Harris

John Bullock

Justin Lindsey

Nick Maurer

Ray

Markus Arning

Bruno Di Pentima

Tuboy Thers

Daniel Saulnier

Jacob Herold

Ashley James

Lukas Lundberg

Jeremy Griffith

Tom Campbell

Luca Prasso

Luis Jose Quintana

Manjit Bedi

Fernando Urquijo SÃ¡nchez

starportx@sohu.co

SpaceToad

Jose Miguel Casas Pagan

Edward del Villar

Garrett Stevens

Atle MÃ¦land

Leonardo Marques

Maddox Max

Gopeekrishnan Sobhana Wariar

Devyn Cyphers

Martin Hedlund

Yaser S

Fox Buchele

Francis Ge

mscottmcbee

Gamoga Notyag

Rolly

Ted Milker

Michal Szczepaniak

Ian Thompson

Kevin Doyon

Ian Sterling

Miikka Harjuntausta

kc austin

Martin Leissler

Alex Chih Chung Lin

Diego Sarmentero

Andy Gittins

Neil Blakey-Milner

Victoria Mitchell

Louis Hong

mirko jugurdzija

Derek Reynolds

Mitu Khandaker

nathan witkowski

Daniel Plemmons

Nicoll Hunt

Kerry

Joe

Jonny Ree

Kevin Webb

Tim Vladislav Chetrusca

Shaders Laboratory

Peter Cardwell-Gardner

Robin Baumgarten

Andreas Wilcox

Terkel Gjervig Nielsen

Stella Cannefax

Iruken

Martin Eigel

Ryan

Chris

Pierce McBride

Shadow Darkwell

Alo Mis

Jim

clownbaby

Christopher Pereira

Steve

Demi

Iain

Wonky Kim

Karl Goodloe

Daniel Cassidy

Alessandro Salani

Doug Snook

Marco Elizondo

Edd Toomey

Hotgates

Brian Nicolucci

Bryan Cinman

Jan Ambroziak

Alice Ruppert

Josh Leong

parag ponkshe

Martin PraÅ¾Ã¡k

ClÃ©ment

Clement

Gavin Rich

Kevin Gregory Agwaze

HÃ¥kon NessjÃ¸en

James Id

Warren Moore

devMidgard

jp

Jason Burton

Ivan Herrera Maciel

Noaksey

Daniel Sanchez

Jon Kimbel

AndrÃ© GrÃ¶schel

friuns

Manish Gupta

David Proffer

Pavel

Marek Majcher

Arthur Cousseau

Orlando Batista

CptBonex

Jakub Barszczewski

Arlin

Matheus

Dom Camus

Aubrey Hesselgren

Kemp

Roger Fretwell

Cary Miller

Matthew Senne

Chris Butler

C Paterson

Nic DanielOthers:

NachoB

b

Stewart

Chris

Yaz Khabiri

Abisaid Fernandez de Lara

valryon

Thomas Schmall

Patrick Peluse

Goran Lalic