Outlining 3d geometry in games is nothing new. The simplest effect, the hull outline, goes back almost to the dawn of 3d games. It is quite easy to make. Just take a copy of object you want to outline, extend the vertices outward along their normals, and then invert the normals of the entire model.

There are other post-process methods to generating outlines, such as edge detection algorithms(sorbel) or generating the outline by sampling neighboring pixels in screen space from a full-screen buffer. While these methods have their advantages and disadvantages, the Outline Hull method is likely to be the fastest among them all, making it suited to mobile games.

Opening the example scene, ToonOutlinePost, we can already see that it doesn’t look quite right.

Artifacts due to the pushed outline clipping with the mesh

Although the character is more or less outlined, there are a lot of artifacts in the interior section of the mesh. We would much rather have only the silhouette of the character visible. To do this, we will need to utilize the Stencil Buffer to mask out the main character mesh so the outlines are not drawn where the character is dawn. Let’s look at our existing OutlineHullEffect asset, which is a type of RenderData which our UniversalRenderPipelineAsset uses to define the custom rendering.

As you can see, there is already something here called “Outline”, which is rendering “After Rendering Opaques”. It is rendering everything in the Layer Mask “Postprocessing”, of which our characters are a part of. Let’s add a new Render Feature of “Render Objects” type and name it “Characters”, and order it before the Outline effect.

This will render our characters(and anything in the post processing layer) with the stencil value of 1 so that we can mask the outline out. I also change the default layer mask to not render the Postprocess layer. If we didn’t do this, our characters would be rendered twice!

Make sure we’re not rendering our characters twice

Great, now we change our Outline effect to mask out the pixels that are not covered by our characters.

We should now have a silhouetted character.

Looking better.

It looks better, but there are still some issues. If you look at the bottom of the sword and fur around his shoulders, we can see places where the outline isn’t showing up. This is because the model has normals that are split, or have a hard edge, thus leaving a gap whenever the outline shader pushes the verticies outward. This is tricky to fix because, while we can change the normals calculation to not have split normals(set smoothing angle to 180), that will effect the lighting.

Changing the normals calculation from within Unity.

One way to fix this would be to have two different models- one with smooth normals for rendering the outline, and another with split normals for rending the character. Another possibility would be to use the split normals version, but encode the smoothed normals as vertex colors in the model. For my purposes, I just just duplicated the model, set it’s normals to be smooth in the importer, and assigned it a different layer.