In this Unreal Engine 4 tutorial, you will learn how to use post process materials to create a cel shader.

Developing realistic-looking games is Unreal Engine 4 is easy due to physically based rendering. This model approximates how light interacts with materials, resulting in realistic images. However, if you want to develop a game with a stylized look, you need to explore other techniques.

One of the ways to achieve a stylized look is to use cel shading (also known as toon shading). This technique mimics the shading typically used in cartoons and anime. You can see examples of this in games such as Jet Set Radio, The Legend of Zelda: The Wind Waker and Gravity Rush.

In this tutorial, you will learn how to:

Create and use a post process material

Create a cel shader

Isolate the cel shader to specific meshes

Control the color bands using lookup tables

Note: This tutorial assumes you already know the basics of using Unreal Engine. If you are new to Unreal Engine, you should go through our 10-part Unreal Engine for Beginners tutorial series first.

Getting Started

Download the starter project and unzip it. Navigate to the project folder and open CelShader.uproject. You will see the following scene:

This is the character you will be cel shading. Before you start, you should know what cel shading is.

What is Cel Shading?

Cel shading is when you render something using multiple bands of color rather than a continuous gradient.

Below is an example of cel shading in The Legend of Zelda: Breath of the Wild. Note that only the character has cel shading while the background does not.

In this image, there are three bands. One for shadows, one for midtones and one for highlights.

A common misconception is that if something has outlines, it is cel shaded. An example of this is Borderlands. Although this game has a stylized look, it is not cel shaded. You can see this in the image below. Notice that the character’s shading does not utilize bands of color.

Even though outlines are not cel shading, it is common to combine them together. This helps an image look like it was drawn or inked. You will see this a lot in anime-styled games such as Guilty Gear Xrd and Dragon Ball FighterZ.

In the next section, you will learn how you can implement cel shading.

Cel Shading Method

The most common method is to compare the surface direction (known as the normal) and the light direction. By calculating the dot product between the normal and light direction, you will get a value between -1 and 1.

A value of -1 means the surface and light are facing opposite directions. 0 means they are perpendicular to each other. 1 means they are facing the same direction.

By thresholding the dot product, you can create multiple bands. For example, you can assign a darker color if the dot product is higher than -0.8. If the dot product is lower than -0.8, assign a light color. This will create a two band cel shader.

The limitation with this method is that other lights cannot affect cel shaded objects. Also, objects can not cast shadows on cel shaded objects.

To fix this, you need to use a different method. Instead of calculating the dot product, you calculate how lit a surface is. You can then use this value during thresholding instead of the dot product.

Now that you know what a cel shader is and how it works, it’s time to create one.

Creating the Cel Shader

The cel shading in this tutorial is a post process effect. Post processing allows you to alter the image after the engine has finished rendering it. Common uses for post processing are depth of field, motion blur and bloom.

To create your own post process effect, you need to use a post process material. Navigate to the Materials folder and create a new Material. Rename it to PP_CelShader and then open it.

To convert a material to a post process material, you need to change its domain. Go to the Details panel and change Material Domain to Post Process.

The first step in creating the cel shader is to calculate how lit each pixel is. We’ll call this the lighting buffer.

Calculating the Lighting Buffer

When Unreal renders an image to the screen, it stores passes to buffers. To calculate the lighting buffer, you will need to access two of these buffers:

Post Process Input: Once Unreal has performed lighting and post processing, it will store the image to this buffer. This is what Unreal will display to the player if you do not perform further post processing. Diffuse Color: This is the scene without any lighting and post processing. It will contain the diffuse colors of everything on-screen.

To access these buffers, you need to use the SceneTexture node. Create one and with it selected, go to the Details panel. To access the Post Process Input buffer, change Scene Texture Id to PostProcessInput0.

To access Diffuse Color, create another SceneTexture node. Change its Scene Texture Id to DiffuseColor.

The lighting buffer should only contain grayscale values (which describe how lit something is). This means you do not need the color information from both buffers. To discard the color, connect the Color output of both SceneTexture nodes to a Desaturation. This will completely desaturate both buffers.

To calculate the lighting buffer, simply divide SceneTexture:PostProcessInput0 by SceneTexture:DiffuseColor. Order is important here!

Afterwards, use a Clamp so the values stay within the 0 to 1 range. This makes it easier to perform thresholding since you know your possible values.

Here is a visualization of the lighting buffer:

As you can see, the lit areas are closer to white and the unlit areas are closer to black.

Next, you will use the lighting buffer to create a threshold.

Creating a Threshold

For this cel shader, any pixel with a value greater than 0.5 will use the normal diffuse color. Pixels with a value less than 0.5 will use the diffuse color at half brightness.

First, create an If node. This will let you compare two values. You can then specify different outputs depending on the comparison result.

Next, connect the Clamp to the A input. Afterwards, create a Constant with a value of 0.5 and plug it into the B input.

Note: You can change the value of the B input to change the threshold.

To get the colors, create a SceneTexture and set its Scene Texture Id to Diffuse Color. Afterwards, multiply Color by 0.5 to get the diffuse at half brightness.

Finally, connect everything like so:

Summary:

Desaturation will convert Post Process Input and Diffuse Color to grayscale images Divide will divide Post Process Input by Diffuse Color. This will give you the lighting buffer. The Clamp will keep the values within the 0 to 1 range The If will output the normal diffuse color if the lighting value is greater than 0.5. If it is less than 0.5 it will output the diffuse color at half brightness.

Now that you have your cel shader, you need to apply it to the scene.

Using Post Process Materials

To use post process materials, you need to create a Post Process Volume. These are commonly used to control post process effects such as white balance, saturation and contrast.

Click Apply and then go back to the main editor. To create a Post Process Volume, go to the Modes panel and select the Volumes category. Afterwards, drag a Post Process Volume into the Viewport to create one.

Now you need to tell the Post Process Volume to use the cel shader. With the Post Process Volume selected, go to the Details panel. Afterwards, locate Rendering Features\Post Process Materials and click the + icon. This will add a new entry to the array.

Next, click the Choose drop-down and select Asset Reference.

This will allow you to select a material. Click the None drop-down and select PP_CelShader.

By default, Post Process Volumes will only take affect if you are inside of it. However, in this case, you want it to affect the entire world. To do this, scroll down to Post Process Volume Settings and enable Infinite Extent (Unbound).

Now that the cel shader is being applied to the entire image, you will see this:

"Hang on, that doesn’t look like the cel shader you showed earlier!"

The main reason it looks different is because the engine is applying the cel shader after tonemapping. To fix this, you need to tell the engine to apply the cel shader before tonemapping.

Applying Cel Shading Before Tonemapping

Before displaying an image to the player, Unreal performs a process called tonemapping. One of the reasons for tonemapping is to make the image look more natural. It does this by taking an input color and then uses a curve to shift it to a new value.

Below are two images of the Post Process Input buffer. One before tonemapping and one after tonemapping:

As you can see, the highlights before tonemapping are too bright. However, after tonemapping, the bright areas are softer.

Tonemapping is fine in most cases. However, since you are using the Post Process Input buffer in a calculation, you need its original values.

Open PP_CelShader and make sure you do not have anything selected. Afterwards, go to the Details panel and locate the Post Process Material section. Set Blendable Location to Before Tonemapping.

Click Apply and then go back to the main editor. The colors are looking a lot better now!

In the next section, you will learn how to only apply cel shading to specific objects.

Isolating the Cel Shader

To isolate post process effects, you need to use a feature called Custom Depth. Like Post Process Input and Diffuse Color, this is a buffer you can use in post process materials.

Before you learn what Custom Depth is, you should know what the Scene Depth buffer is. Scene Depth stores how far each pixel is from the camera plane. Here is a visualization of Scene Depth:

Custom Depth stores the same information but only for meshes you specify. Here is a visualization of it with the viking rendered to Custom Depth:

By comparing Scene Depth against Custom Depth, you can isolate effects. If Scene Depth is less than Custom Depth, use the normal image. If Scene Depth is greater than Custom Depth, use the cel shaded image.

The first step is to render the viking to Custom Depth.

Using Custom Depth

Go to the World Outliner and select SK_Viking. Afterwards, go to the Details panel and locate the Rendering section. Next, enable Render CustomDepth Pass.

Next, you need to perform the depth comparison. Open PP_CelShader and create the following setup:

Note: The Mask (R) nodes are Component Masks. These allow you to convert multi-channel data into scalars. The reason you need to mask Scene Depth and Custom Depth is because the If node only accepts scalars for the A and B inputs.

Afterwards, connect the output of the cel shading network to A > B. Finally connect the output of the If you just made into Emissive Color.

Now, only meshes that rendered to Custom Depth will have cel shading.

Click Apply and then go back to the main editor. You will see that only the viking has cel shading now.

The cel shader is working great but it’s a bit simple. What if you want to have more than two bands? What if you want to create a softer transition between bands? You can accomplish this by using lookup tables (LUT).

What is a Lookup Table?

When you were young, you learned what multiplication is. However, a young mind might not have been able to perform these calculations. Instead of calculating, you might have used a multiplication table to "look up" the answers.

This is essentially what a LUT is. It is an array of values (usually precomputed) which you can access with inputs. In the case with the multiplication table, the inputs are the multiplier and multiplicand.

In the context of this cel shader, a LUT is a texture with some sort of gradient. Here are four examples of what a LUT could look like:

Currently, the way you are calculating the shadow color is by multiplying the diffuse color by 0.5. Instead of multiplying by a constant 0.5, you will use a value from the LUT. By doing this, you can control the number of bands and their transitions. You can get an idea of how the shading would look by the appearance of the LUT.

Before you use a LUT, you need to change some of its texture settings.

Changing LUT Settings

Navigate to the Textures folder and open T_Lut_01. This is what the LUT looks like:

The first setting you need to change is sRGB. When rendering, Unreal will convert any textures with sRGB enabled to linear color. Basically, this makes it easier for Unreal to perform rendering calculations.

The sRGB setting is good for textures which describe appearance. However, textures such as normal maps and LUTs hold values meant for math calculations. As such, Unreal should assume their values are already correct. By disabling sRGB, Unreal will not perform the conversion to linear color.

To do this, uncheck the sRGB checkbox. You can find this setting under the Texture section.

The next setting you need to change is how the texture tiles. Since you are not displaying this texture, there is no need to tile it. Furthermore, leaving tiling enabled will introduce issues when sampling from the edges. For example, if you were to sample a pixel from the left edge, it will try to blend to the right edge because of tiling.

To disable tiling, change X-axis Tiling Method to Clamp. Do the same for Y-axis Tiling Method.

That’s all for the settings. Now you need to use the LUT in the post process material.

Using LUTs

Close T_Lut_01 and open PP_CelShader. First, delete the highlighted nodes:

Next, create a Texture Sample and change its Texture to T_Lut_01. This LUT will create three bands with a soft transition.

If you remember, LUTs take inputs to determine which value to output. In this case, you will use the lighting buffer as the input.

To do this, connect the Clamp to the UVs of the Texture Sample.

This works because the lighting buffer and texture coordinates are in the range of 0 to 1. For example, if a pixel from the lighting buffer is 0.5, the LUT will output the pixel value from the middle of the texture.

Next, you need to multiply the diffuse color with the LUT. To do this, recreate the following setup:

The reason for the Append is to convert the Texture Sample’s output to a four-channel vector. You need to do this because you cannot multiply a three-channel vector by a four-channel vector (SceneTexture).

Finally, connect everything like so:

Now, instead of multiplying diffuse color by a constant, you are multiplying by a value from the LUT. This controls how many bands there are and their transitions (depending on the LUT). The lighting buffer determines which value the LUT will output.

Click Apply and then close PP_CelShader. The shading will now have three bands with a softer transition between bands.

Below is a comparison of what alternative LUTs would look like. These LUTs are also included in the project.

Where to Go From Here?

You can download the completed project here.

As you can see, post process materials are very powerful. They allow you to create a lot of realistic and stylized effects. If you’d like to learn more about post processing, check out the Post Process Effects documentation.

If there are any effects you’d like to me cover, let me know in the comments below!