This tutorial follow Tessellation and combines it with vertex displacement to add more detail to geometry, on top of normal mapping.

This tutorial is made with Unity 2017.1.0.

Repositioning Vertices

Meshes are usually made up of triangles, which are always flat. The illusion of curvature is added via vertex normals. Normal maps can be used to add the illusion of more surface irregularities, smaller than individual mesh triangles. Beyond that, parallax mapping makes it possible to fake surface displacement. But all these approaches are illusions. The most robust way to make a surface more complex is by simply using more smaller triangles. Smaller triangles means that we have more vertices, enough to describe all surface details that we want. Unfortunately, that would result in much larger meshes, requiring more storage space, CPU and GPU memory, and memory bandwidth. Tessellation is a way around this problem, because it allows us to generated more triangles on the GPU when needed. This means the GPU has to do more work, but we can limit that to when it's really needed.

Cutting up existing triangles and interpolating the vertex data isn't enough to add more details. That just gives us more triangles that describe the same flat surface. We have to introduce new data, adjusting the triangle's vertices somehow.

A straightforward way to add more detail is to adjust the vertices of a mesh via a displacement map. The map is used to move vertices up or down, like a height field can be used to turn a flat terrain mesh into an actual landscape. This tutorial will cover how to do that.

Hijacking Parallax Mapping To displace vertices, we need a displacement map. Although our Tessellation Shader doesn't have a property for such a map, it does have a parallax map that we used in the Parallax tutorial. The parallax map is really a displacement map, it's just that we used it to fake displacement. We can use the same map for actual displacement too. Let's say that a shader can decide to use true vertex displacement instead of parallax mapping, simply by defining VERTEX_DISPLACEMENT_INSTEAD_OF_PARALLAX. If that macro is defined and we have a parallax map, then we must make sure that the parallax code doesn't get included, replacing it with proper vertex displacement code. To do this, undefine _PARALLAX_MAP and define a convenient VERTEX_DISPLACEMENT macro in My Lighting Input, when we have a parallax map and should use vertex displacement. #include "UnityPBSLighting.cginc" #include "AutoLight.cginc" #if defined(_PARALLAX_MAP) && defined(VERTEX_DISPLACEMENT_INSTEAD_OF_PARALLAX) #undef _PARALLAX_MAP #define VERTEX_DISPLACEMENT 1 #endif Let's also create macro aliases for the _ParallaxMap and _ParallaxStrength variables, so we can use _DisplacementMap and _DisplacementStrength instead. This makes it easier to get rid of the parallax code and switch to proper displacement properties, in case you like to do that later. #if defined(_PARALLAX_MAP) && defined(VERTEX_DISPLACEMENT_INSTEAD_OF_PARALLAX) #undef _PARALLAX_MAP #define VERTEX_DISPLACEMENT 1 #define _DisplacementMap _ParallaxMap #define _DisplacementStrength _ParallaxStrength #endif As we won't used both parallax mapping and tessellation at the same time, we can get rid of all definitions related to parallax in the CGINCLUDE block of Tessellation Shader. Instead, we only have to define VERTEX_DISPLACEMENT_INSTEAD_OF_PARALLAX. CGINCLUDE #define BINORMAL_PER_FRAGMENT #define FOG_DISTANCE // #define PARALLAX_BIAS 0 // #define PARALLAX_OFFSET_LIMITING // #define PARALLAX_RAYMARCHING_STEPS 10 // #define PARALLAX_RAYMARCHING_INTERPOLATE // #define PARALLAX_RAYMARCHING_SEARCH_STEPS 3 // #define PARALLAX_FUNCTION ParallaxRaymarching // #define PARALLAX_SUPPORT_SCALED_DYNAMIC_BATCHING #define VERTEX_DISPLACEMENT_INSTEAD_OF_PARALLAX ENDCG We'll perform vertex displacement in object space. To allow for a decent amount of displacement, increase the maximum strength from 0.1 to 1. _ParallaxStrength ("Parallax Strength", Range(0, 1 )) = 0 Parallax map with strength set to 1.

Changing the Vertex Position Displacing vertices has to be done in the vertex program of My Lighting, before we use the vertex position for anything else. This means that if we want to support scaling and offsetting the displacement map like all other maps, we have to transform the texture coordinates before this point. So let's move the TRANSFORM_TEX lines before the first time the vertex position is used. InterpolatorsVertex MyVertexProgram (VertexData v) { InterpolatorsVertex i; UNITY_INITIALIZE_OUTPUT(InterpolatorsVertex, i); UNITY_SETUP_INSTANCE_ID(v); UNITY_TRANSFER_INSTANCE_ID(v, i); i.uv.xy = TRANSFORM_TEX(v.uv, _MainTex); i.uv.zw = TRANSFORM_TEX(v.uv, _DetailTex); i.pos = UnityObjectToClipPos(v.vertex); i.worldPos.xyz = mul(unity_ObjectToWorld, v.vertex); #if FOG_DEPTH i.worldPos.w = i.pos.z; #endif i.normal = UnityObjectToWorldNormal(v.normal); #if defined(BINORMAL_PER_FRAGMENT) i.tangent = float4(UnityObjectToWorldDir(v.tangent.xyz), v.tangent.w); #else i.tangent = UnityObjectToWorldDir(v.tangent.xyz); i.binormal = CreateBinormal(i.normal, i.tangent, v.tangent.w); #endif // i.uv.xy = TRANSFORM_TEX(v.uv, _MainTex); // i.uv.zw = TRANSFORM_TEX(v.uv, _DetailTex); … } Once we have the final texture coordinates, we can sample the displacement map. This works the same as sampling the map for parallax mapping, so we'll use its green texture channel. However, because we're not doing this in the fragment program, there are no screen-space derivatives available, so the GPU cannot determine which mipmap level to use. We cannot use tex2D , instead we have to use tex2Dlod to specify an explicit mipmap level. This is done by supplying two additional texture coordinates, the third being an unused 3D coordinate and the fourth specifying the mip level. We'll just use 0 for both, effectively using no mipmaps. i.uv.xy = TRANSFORM_TEX(v.uv, _MainTex); i.uv.zw = TRANSFORM_TEX(v.uv, _DetailTex); #if VERTEX_DISPLACEMENT float displacement = tex2Dlod(_DisplacementMap, float4(i.uv.xy, 0, 0)).g; #endif Like we do for parallax mapping, let's interpret the map so a value of 0.5 means no change, making it possible to move vertices both up and down. After that, factor in the displacement strength so we can control how much the vertices get moved in object space. #if VERTEX_DISPLACEMENT float displacement = tex2Dlod(_DisplacementMap, float4(i.uv.xy, 0, 0)).g; displacement = (displacement - 0.5) * _DisplacementStrength; #endif If we were working with a default height field, then we'd just have to add the displacement to the vertex Y position at this point. float displacement = tex2Dlod(_DisplacementMap, float4(i.uv.xy, 0, 0)).g; displacement = (displacement - 0.5) * _DisplacementStrength; v.vertex.y += displacement; Quad vertices displaced along Y. When applying this approach to a quad, the result will look like a mess of triangles that is still flat. That's because the quad is aligned with the XY plane in object space. If we want to perturb its otherwise flat surface, we have to adjust its Z coordinates instead. In general, a positive displacement should move vertices upward, from the point of view of the mesh. But not all meshes are planes. In the case of a sphere, it makes sense for displacement to move vertices outward. So in general it makes the most sense to displace along the vertex normal. // v.vertex.y += displacement; v.vertex.xyz += v.normal * displacement; Because we're using tessellation, the normal vectors of new vertices have been created via interpolation. So they're only guaranteed to be of unit length when all the vertex normals have the same orientation. To guarantee that we get unit-length normal vectors in general, we should normalize them before using them for displacement. v.normal = normalize(v.normal); v.vertex.xyz += v.normal * displacement; Displacement along normal vector.

Using Enough Triangles How much triangles are needed to support the desired detail level? It depends. Our displacement map at full strength produces quite a large change, so we need quite some triangles to make it look good. But we don't want to use more triangles than we need, so we should use the Edge tessellation mode instead of the Uniform mode. Variable tessellation of a quad. When in Edge mode, tessellation is controlled by both the Edge Length property and the view distance. So how many triangles get used can vary a lot. A quad by itself contains only two triangles. We'd need a significant amount of tessellation to get something better than a low-poly jagged plane. We can help tessellation a lot by using a base mesh that has more triangles. For example, Unity's default plane mesh consists of 10×10 quads. Using that instead of a quad prevents complete degeneration and can also produce much higher vertex resolution than a quad, if needed. Variable tessellation of a plane. Using a plane instead of a quad allows for a more fine-tuned tessellation, which makes it easier to achieve a visually uniform triangle density from all view angles. Shallow view angle, uniform triangle density. However, this still does not guarantee that all triangles end up with the same visual size. The tessellated triangles are only about the same size before their vertices are displaced. If a triangle's vertices end up displaced by different amount, it will become stretched along the normal vectors. Usually, vertex displacement isn't as extreme as the example that we use in this tutorial. If you're using it for a terrain mesh, the regular mesh should have sufficient resolution to represent the large features of the terrain. If you use a flat mesh as the basis for your terrain, consider displacing the original vertices before determining the tessellation factors, so coarse elevation gets taken into account when tessellating.