Table of Contents

The Basis of Tangent Space

The problem is with how tangent space normal maps work. Specifically, how that tangent space (or tangent basis) part of tangent space normal maps work. I’m not going to go too deep into the technical aspects of what they’re doing. I’ve done that a little already elsewhere, as have others, but in short tangent space is what determines the orientation of the direction in a normal map is supposed to represent. What actual direction “forward”, “up” and “right” in the normal map actually is. That orientation is roughly the orientation of the texture UVs and the vertex normals.

The more important thing to understand is the program being used to bake a normal map from a high poly mesh to a low poly mesh needs to calculate its tangent space exactly the same way as the program the normal map is being used in. Because different programs may use tangent space differently, normal maps baked for a mesh aren’t guaranteed to be universal between applications. Even for applications that use the same tangent space!

Recognizing Your Enemy

Flipping Out Over an Obvious Error

The most common error is the “OpenGL” vs “Direct3D” issue. This is also called “Y+” and “Y-” normal maps respectively. I’m including this here just for completeness. It’s likely most people who come across this already know about it, or at least find a way to fix it. Because normal maps generated in one orientation are basically unusable if used in an application designed to use the other.

Direct3D orientation normal maps in Unity, which expects OpenGL orientation

This can be recognized if the lighting seems to be inverted, or coming from the wrong side, but only for some light directions. Certain angles or faces might look totally fine until you move the light.

This one is fairly straight forward. Most applications that bake out normal maps have a nice big button or drop down someplace to switch between OpenGL / Direct3D or have a Y orientation setting. Figure out which orientation your final application uses and make sure you’re baking to that. Unity is OpenGL, Y+. Unreal is Direct3D, Y-. They can be easily distinguished by just looking at the normal map texture itself. If the look like they’re being lit from “above” in the green channel, then they’re OpenGL orientation. If from below then it’s Direct3D.

sRGBeing Funny

Another common mistake is using a texture that’s marked as being an sRGB texture. This changes how the GPU samples a texture, telling it to transform the color values from sRGB space, the one people are used to picking color values in, to linear color space. You don’t want this for normal maps. In Unity if you set a texture’s type to “Normal map” it automatically disables this option, but you can also turn off the “sRGB (Color Texture)” option and it’ll work properly as a normal map. At least in Unity 2017.1 and newer.

Normal map set to use the Default texture type with sRGB enabled

The way to recognize this one is normal maps will be heavily shifted in one direction. In this model almost every face looks like it’s facing toward the light. Obviously it won’t always end up that way. With a more detailed normal map all the parts will be there and be reacting to light almost correctly, but just shifted at an angle. It can sometimes look like the entire mesh’s surface is twisted.

Too Different to be Normal

Another common issue is using different mesh normals for baking and the in game asset. Just like the tangents, these need to match exactly for things to work. This one can be common if artists aren’t exporting their meshes with proper vertex normals. This means any application that opens the mesh will have to calculate them on their own. There’s no one right way to do this as it depends on the mesh, so make sure you (or your artists) are exporting with the final normals!

Normal maps baked on a smoothed mesh, displayed on a hard edged mesh

Normal maps baked on a hard edged mesh, displayed on a smoothed mesh

This is another one that’s generally pretty easy to notice. If your model is supposed to look like it has some sharp edges, but the shading is curving away from those edges, it means you have a normal map that was baked with a smoothed normal but being rendered on a mesh with a hard edge. If the shading is curving towards the edge it means it was baked with a hard edge and rendering on a soft edge. Really both can be true on the same mesh since you can choose what edges are smooth or hard, and even to what degree. But basically you’re guaranteed for them to be wrong if you let another application guess at what was intended. The above model is also an extreme case and it’s often not that obvious as large parts of the model may look totally normal. But generally if you have a few edges that look especially odd it’s probably due to a vertex normal mismatch.

MikkTSpace vs Autodesk

Update: The below is true for 3ds Max 2020 and earlier. For 3ds Max 2021 Autodesk added proper support for baking MikkTSpace! See the application list at the end of the article for more information.

A more subtle issue is if you’re baking normal maps from 3ds Max or Maya, they’ll always be wrong for Unity. They’ll also be wrong for Unreal too, or really any other program but the program you baked them in. 3ds Max normal maps won’t work in Maya or vice versa either! Each of these applications calculate tangent space in their own unique ways that does not match other applications. They’ll look perfect when used for themselves, and if you’re generating normal maps for use in those programs, by all means bake out your normal maps using them. Almost nothing else will bake proper normal maps for those applications, so you kind of have to.

The only solution for this case is do not bake your normal maps out using these applications if you intend to use them elsewhere. If you need something free, use xNormal. It’s an industry standard tool for doing this. If you’re using Substance elsewhere in your content pipeline, use that.

3ds Max baked normal map used in Unity

The easiest way to notice this as being the problem is if some otherwise flat faces have some slight dimpling or curvature to them. This can be very subtle in some cases, especially on organic objects. So this is getting into the issues that go unnoticed far more often than not.

Many applications outside of Autodesk have moved to use a specific way of calculating tangent space which is referred to as MikkTSpace. Both xNormal and Substance, as well as Blender, Unity, Unreal, Godot, Marmoset, Houdini, Modo and several others, use MikkTSpace by default or at least support it in some way, greatly increasing their compatibility. Basically over the last decade it’s basically become the industry standard for anywhere tangent space normal maps are regularly used. Supposedly 3ds Max will add support for it at some point, but it’s unclear if they plan on letting you bake normal maps using MikkTSpace, or just support viewing / rendering with them.

Mesh Exports & Substance Painter

The other issue with 3ds Max and Maya is if you export meshes from them you have the option to export with “tangents and binormals”. These won’t be in proper MikkTSpace tangents, but based. Some applications will override these and recalculate a correct MikkTSpace by default (like xNormal, Unity & Unreal), but other applications will use the mesh’s tangents as they are if they exist and will only calculate MikkTSpace if the mesh doesn’t have any tangents. Substance Painter for example will use the tangents as they are on the mesh if they exist, which means Painter will produce normal maps that look very similar to those baked by 3ds Max! Similar, but not exactly the same, so if you import them back into 3ds Max they won’t look quite as good as those baked in Max.

MikkTSpace vs MikkTSpace?

The next problem is there’s more than one version of MikkTSpace! I know! It wouldn’t be a standard if it wasn’t confusing in some way. MikkTSpace defines both how the data should be stored on the mesh per vertex, and how the data is used when rendering. All MikkTSpace applications will calculate identical per vertex mesh data. But there are two different ways to use that data when rendering. MikkTSpace defines how to calculate a tangent vector and sign per vertex based on the UV orientation. But when using tangent space the bitangent is recalculated either per vertex or per pixel. That too needs to be the same for both baking and viewing.

xNormal and Substance both have an option for baking normal maps using either method. These options are listed as “compute tangent space per fragment” in Substance and “compute binormal in the pixel shader” in xNormal. You don’t want that on if you’re baking for Unity’s built in rendering paths. But you possibly do want it for the SRPs, depending on which one you use!

xNormal “pixel shader binormals” in Unity’s built in forward renderer

This is visually similar to using a completely different tangent space, but will usually be much more subtle. Some faces may look like the normal is slightly twisted rather than having a large dimple in it. The solution here is straight forward, don’t check that option if you’re baking for Unity’s built-in rendering paths. Do check it if you’re baking for Unreal or Unity’s HDRP.

There’s a lot of stuff out there that says that Blender and Unity are identical, but this has in fact was never true! Blender uses MikkTSpace & OpenGL orientation, like Unity, but uses per fragment binomals like Unreal.

The good news is while currently the LWRP, URP, and Shader Graph shaders all use per vertex, the HDRP’s built-in shaders use per pixel. In a future update to Shader Graph (7.2.0) it should be switching to per pixel, and URP’s built-in shaders sometime after that. And the default rendering paths can be modified with custom shaders if one chooses that route.

Triangulate, Triangulate, Triangulate!

The last one I’m going to go into is different triangulation. If you export your meshes without pre-triangulating them, there’s no guarantee other applications that take that model will triangulate it the same way. This also breaks normal maps! So make sure you triangulate your meshes, either as part of the export settings, or using a triangulate modifier.

Substance Painter baked normal maps in Unity on a mesh exported from 3ds Max as quads

This will present itself very similarly to using a totally wrong tangent space, like normal maps baked from 3ds Max. But generally it’s a slightly sharper almost crease rather than a soft dimple. In this mesh you can see an X shape showing up on the outside faces. This one can be tricky and annoying when dealing with a content pipeline where multiple people may touch a model through multiple applications as it’s often very useful to not triangulate a mesh and keep nice clean looking polygons. For that case you’ll either just need to deal with that loss of convenience, or make sure the last tool in the pipeline before the model gets exported for the game is also used to generate the mesh used to bake the normals. Like the tangent space, and the vertex normals (which are part of the tangent space), the triangulation also needs to exactly match.