This tutorial covers how to add support for tessellation to a custom shader. It uses the Flat and Wireframe Shading tutorial as a basis.

This tutorial is made with Unity 2017.1.0.

Hulls and Domains

Tessellation is the art of cutting things into smaller parts. In our case, we're going to subdivide triangles so we end up with smaller triangles that cover the same space. This makes it possible to add more details to geometry, though in this tutorial we'll focus on the tessellation process itself.

The GPU is capable of splitting up triangles fed to it for rendering. It does this for various reasons, for example when part of a triangle ends up clipped. We cannot control that, but there's also a tessellation stage that we are allowed to configure. This stage sits in between the vertex and the fragment shader stages. But it's not as simple as adding just one other program to our shader. We're going to need a hull program and domain program.

Shading with tessellation.

Creating a Tessellation Shader The first step is to create a shader that has tessellation enabled. Let's put the code that we'll need in its own file, MyTessellation.cginc, with its own include guard. #if !defined(TESSELLATION_INCLUDED) #define TESSELLATION_INCLUDED #endif To clearly see that triangles get subdivided, we'll make use of the Flat Wireframe Shader. Duplicate that shader, rename it to Tessellation Shader and adjust its menu name. Shader "Custom/Tessellation" { … } The minimum shader target level when using tessellation is 4.6. If we don't set this manually, Unity will issue a warning and automatically use that level. We're going to add tessellation stages to the forward base and additive passes, plus the deferred pass. Also include MyTessellation in those passes, after MyFlatWireframe. #pragma target 4.6 … #include "MyFlatWireframe.cginc" #include "MyTessellation.cginc" What about the shadow pass? It is also possible to use tessellation when rendering shadows, but we won't do that in this tutorial. Create a material that relies on this shader and add a quad to the scene that uses it. I made the material gray so it isn't too bright, like the Flat Wireframe material. A quad. We'll use this quad to test our tessellation shader. Note that is consists of two isosceles right triangles. The short edges have length 1, while the long diagonal edges have length √2.

Hull Shaders Like geometry shaders, the tessellation stage is flexible and can work triangles, quads, or isolines. We have to tell it what surface it has to work with and feed it the necessary data. This is the job of the hull program. Add a program for this to MyTessellation , beginning with a void function that does nothing. void MyHullProgram () {} The hull program operates on a surface patch, which is passed to it as an argument. We have to add an InputPatch parameter to make this possible. void MyHullProgram ( InputPatch patch ) {} A patch is a collection of mesh vertices. Like we did for the stream parameter of the geometry function, we have to specify the data format of the vertices. We'll use the VertexData struct for now. void MyHullProgram (InputPatch <VertexData> patch) {} Shouldn't it be InputPatch<InterpolatorsVertex> ? As the hull stage comes after the vertex stage, logically the hull function's input type must match the vertex function's output type. This is true, but we'll ignore this fact for now. As we're working with triangles, each patch will contain three vertices. This amount has to be specified as a second template parameter for InputPatch . void MyHullProgram (InputPatch<VertexData , 3 > patch) {} The job of the hull program is to pass the required vertex data to the tessellation stage. Although it is fed an entire patch, the function should output only a single vertex at a time. It will get invoked once per vertex in the patch, with an additional argument that specifies which control point (vertex) it should work with. The parameter is an unsigned integer with the SV_OutputControlPointID semantic. void MyHullProgram ( InputPatch<VertexData, 3> patch , uint id : SV_OutputControlPointID ) {} Simply index the patch as if it were an array and return the desired element. VertexData MyHullProgram ( InputPatch<VertexData, 3> patch, uint id : SV_OutputControlPointID ) { return patch[id]; } This looks like a functional program, so let's add a compiler directive to use it as a hull shader. Do this for all three shader passes that are involved. #pragma vertex MyVertexProgram #pragma fragment MyFragmentProgram #pragma hull MyHullProgram #pragma geometry MyGeometryProgram This will produce a few compiler errors, complaining that we haven't configured our hull shader correctly. Like the geometry function, it needs attributes to configure it. First, we have to explicitly tell it that it's working with triangles. That's done via the UNITY_domain attribute, with tri as an argument. [UNITY_domain("tri")] VertexData MyHullProgram … That's not enough. We also have to explicitly specify that we're outputting three control points per patch, one for each of the triangle's corners. [UNITY_domain("tri")] [UNITY_outputcontrolpoints(3)] VertexData MyHullProgram … When the GPU will create new triangles, it needs to know whether we want them defined clockwise or counterclockwise. Like all other triangles in Unity, they should be clockwise. This is controlled via the UNITY_outputtopology attribute. Its argument should be triangle_cw. [UNITY_domain("tri")] [UNITY_outputcontrolpoints(3)] [UNITY_outputtopology("triangle_cw")] VertexData MyHullProgram … The GPU also needs to be told how it should cut up the patch, via the UNITY_partitioning attribute. There are a few different partitioning methods, which we'll investigate later. For now, just use the integer mode. [UNITY_domain("tri")] [UNITY_outputcontrolpoints(3)] [UNITY_outputtopology("triangle_cw")] [UNITY_partitioning("integer")] VertexData MyHullProgram … Besides the partitioning method, the GPU also has to know into how many parts the patch should be cut. This isn't a constant value, it can vary per patch. We have to provide a function to evaluate this, known as a patch constant function. Let's just assume we have such a function, named MyPatchConstantFunction. [UNITY_domain("tri")] [UNITY_outputcontrolpoints(3)] [UNITY_outputtopology("triangle_cw")] [UNITY_partitioning("integer")] [UNITY_patchconstantfunc("MyPatchConstantFunction")] VertexData MyHullProgram …

Patch Constant Functions How a patch is to be subdivided is a property of the patch. This means that the patch constant function is only invoked once per patch, not once per control point. That's why it's referred to as a constant function, being constant across the entire patch. Effectively, this function is a sub-stage operating in parallel with MyHullProgram . Inside a hull shader. To determine how to subdivide a triangle, the GPU uses four tessellation factors. Each edge of the triangle patch gets a factor. There's also a factor for the inside of the triangle. The three edge vectors have to be passed along as a float array with the SV_TessFactor semantic. The inside factor uses the SV_InsideTessFactor semantic. Let's create a struct for that. struct TessellationFactors { float edge[3] : SV_TessFactor; float inside : SV_InsideTessFactor; } ; The patch constant function takes a patch as an input parameter and outputs the tessellation factors. Let's now create this missing function. Simply have it set all factors to 1. This will instruct the tessellation stage to not subdivide the patch. TessellationFactors MyPatchConstantFunction (InputPatch<VertexData, 3> patch) { TessellationFactors f; f.edge[0] = 1; f.edge[1] = 1; f.edge[2] = 1; f.inside = 1; return f; }

Domain Shaders At this point, the shader compiler will complain that a shader cannot have a tessellation control shader without a tessellation evaluation shader. The hull shader is only part of what we need to get tessellation working. Once the tessellation stage has determined how the patch should be subdivided, it's up to the geometry shader to evaluate the result and generate the vertices of the final triangles. So let's create a function for our domain shader, again starting with a stub. void MyDomainProgram () {} Both the hull and domain shader act on the same domain, which is a triangle. We signal this again via the UNITY_domain attribute. [UNITY_domain("tri")] void MyDomainProgram () {} The domain program is fed the tessellation factors that were used, as well as the original patch, which is of type OutputPatch in this case. [UNITY_domain("tri")] void MyDomainProgram ( TessellationFactors factors, OutputPatch<VertexData, 3> patch ) {} While the tessellation stage determines how the patch should be subdivided, it doesn't generated any new vertices. Instead, it comes up with barycentric coordinates for those vertices. It's up to the domain shader to use those coordinates to derive the final vertices. To make this possible, the domain function is invoked once per vertex and is provided the barycentric coordinates for it. They have the SV_DomainLocation semantic. [UNITY_domain("tri")] void MyDomainProgram ( TessellationFactors factors, OutputPatch<VertexData, 3> patch , float3 barycentricCoordinates : SV_DomainLocation ) {} Inside the function, we have to generate the final vertex data. [UNITY_domain("tri")] void MyDomainProgram ( TessellationFactors factors, OutputPatch<VertexData, 3> patch, float3 barycentricCoordinates : SV_DomainLocation ) { VertexData data; } To find the position of this vertex, we have to interpolate across the original triangle domain, using the barycentric coordinates. The X, Y, and Z coordinates determine the weights of the first, second, and third control points. VertexData data; data.vertex = patch[0].vertex * barycentricCoordinates.x + patch[1].vertex * barycentricCoordinates.y + patch[2].vertex * barycentricCoordinates.z ; We have to interpolate all vertex data in the same way. Let's define a convenient macro for that, which can be used for all vector sizes. // data.vertex = // patch[0].vertex * barycentricCoordinates.x + // patch[1].vertex * barycentricCoordinates.y + // patch[2].vertex * barycentricCoordinates.z; #define MY_DOMAIN_PROGRAM_INTERPOLATE(fieldName) data.fieldName = \ patch[0].fieldName * barycentricCoordinates.x + \ patch[1].fieldName * barycentricCoordinates.y + \ patch[2].fieldName * barycentricCoordinates.z; MY_DOMAIN_PROGRAM_INTERPOLATE(vertex) Besides the position, also interpolate the normal, tangent, and all UV coordinates. MY_DOMAIN_PROGRAM_INTERPOLATE(vertex) MY_DOMAIN_PROGRAM_INTERPOLATE(normal) MY_DOMAIN_PROGRAM_INTERPOLATE(tangent) MY_DOMAIN_PROGRAM_INTERPOLATE(uv) MY_DOMAIN_PROGRAM_INTERPOLATE(uv1) MY_DOMAIN_PROGRAM_INTERPOLATE(uv2) The only thing that we do not interpolate are instance IDs. As Unity does not support GPU instancing and tessellation at the same time, there's no point in copying this ID. To prevent compiler errors, remove the multi-compile directives from the three shader passes. This wil also remove the instancing option from the shader's GUI. // #pragma multi_compile_instancing // #pragma instancing_options lodfade force_same_maxcount_for_gl Is it possible to use instancing and tessellation together? At the moment, that's not the case. Keep in mind that GPU instancing is useful when rendering the same object many times. As tessellation is expensive and about adding details, they're usually not a good combination. If you want to have many instances of something that should use tessellation up close, you could use a LOD group. Have LOD 0 use a non-instanced tessellated material, while all other LOD levels use an instanced non-tessellated material. We now have a new vertex, which will be send to either the geometry program or the interpolator after this stage. But these programs expect InterpolatorsVertex data, not VertexData . To solve this, we have the domain shader take over the responsibilities of the original vertex program. This is done by invoking MyVertexProgram inside it—like any other function—and return its result. [UNITY_domain("tri")] InterpolatorsVertex MyDomainProgram ( TessellationFactors factors, OutputPatch<VertexData, 3> patch, float3 barycentricCoordinates : SV_DomainLocation ) { … return MyVertexProgram(data); } Now we can add the domain shader to our three shader passes, but we'll still get errors. #pragma hull MyHullProgram #pragma domain MyDomainProgram