Tutorial: How to dynamically tessellate a planar mesh

Intro

Hello! I've been hard at work on my snow deformation shader, and this forum has been a huge help to me in getting it working. By far the trickiest part of getting what I have so far was the dynamic tessellation, and with so few tessellation tutorials out there, I figured I'd try and give back to the community by sharing how to dynamically tessellate an object. So, without further ado, let's get into it!

What is Tessellation?

Tessellation in relation to graphics, is the process of subdividing shapes into smaller versions of that shape. This can be used to enhance detail through increased geometry, or to make distortions and modifications to the mesh look more appealing or believable. The reason I say shapes, is because DX11 allows for three distinct shapes: triangles, quads, or isolines. For this tutorial, we'll be working with triangles.

How does the Tessellation Pipeline function?

The tessellation pipeline is separated into three stages: hull, tessellation, and domain.

Hull Program

Invoked once per patch (polygon), the hull stage takes one input and produces two output. Its input are the control points that make up a patch. Control points are essentially the original vertices that define the shape. For the output, it returns the control points as well as calculated tessellation factors. To return the two outputs, the hull program is actually made up of two separate functions that operate in parallel.

Tessellation Program

Thankfully, what would be the most challenging part of tessellation is handled for us internally. This stage takes in the control points and tessellation factors produced by the hull program and slices up the patch according to the defined domain (triangle, quad, isoline).

Domain Program

The purpose of the domain program is to take the subdivided points the tessellation program has produced, and turn them into vertices that can be rendered. To do this, it takes in the control points from the hull program, its tessellation factors, and, last but not least, the tessellation points.

Part 0 - Shader Set Up

We'll start from a standard unlit shader. We're going to leave most of the things the same. So begin by removing all fog related code, and adding in "#pragma target 4.6" with the other "#pragma"'s. While we're at it, change the name of the vertex and fragment programs to "VertexProgram" and "FragmentProgram" and update their pragma's accordingly.

Part 1 - Vertex and Fragment

Let's start with the vertex and fragment programs. The tessellation pipeline is actually sandwiched between these two programs, so they mostly stay the same. However, because we end up with more vertices than we started with, the vertex program will need to be called twice. To do this, we'll define two separate vertex programs. First however, let's define our structs. And, for the purpose of this tutorial, we'll keep them as simple as possible.

Code (csharp):

1. struct appdata
2. {
3.    float4 vertex : POSITION;
4.    float2 uv : TEXCOORD0;
5. };
6.  
7. struct v2f
8. {
9.    float2 uv : TEXCOORD0;
10.    float4 vertex : SV_POSITION;
11. };

Next up will be the vertex and fragment programs.

Code (csharp):

1. v2f VertexProgram (appdata v)
2. {
3.    v2f o;
4.    o.vertex = UnityObjectToClipPos(v.vertex);
5.    o.uv = TRANSFORM_TEX(v.uv, _MainTex);
6.    return o;
7. }
8.  
9. fixed4 FragmentProgram (v2f i) : SV_Target
10. {
11.    fixed4 col = tex2D(_MainTex, i.uv);
12.    return col;
13. }

Very basic so far. Now, if you'll recall, I mentioned that we'll need two vertex programs. We'll need to create a new struct, a new function, and change the vert pragma. First, lets change the pragma.

Code (csharp):

1. #pragma vert TessellationVertexProgram

Okay, with that out of the way let's move onto the struct. This struct will represent the control points and will be fed into the hull program. It will be almost identical to appdata.

Code (csharp):

1. struct ControlPoint
2. {
3.    float4 vertex : INTERNALTESSPOS;
4.    float2 uv : TEXCOORD0;
5. };

Now for our new vertex program. Because we're using this vertex program to pass along the vertices as control points to the hull program, nothing needs to change between the two, so we'll simply copy over their values.

Code (csharp):

1. ControlPoint TessellationVertexProgram(appdata v)
2. {
3.    ControlPoint p;
4.    p.vertex = v.vertex;
5.    p.uv = v.uv;
6.    return p;
7. }

Part 2 - Hull

So, we have our pre-tessellation vertex program and post-tessellation vertex program set up as well as our fragment program. Let's dive into the nitty-gritty of tessellation! If you recall, I had mentioned that the hull program was split up into two separate functions executed in parallel. The reason for this is because the hull program has two outputs: the control points, and the tessellation factors. Let's get the control point output out of the way.

Before sharing the code, I want to mention a few things. The hull program takes inputs as a patch. This means multiple control points (vertices). We'll need to define how many control points we're expecting, as well as the domain (triangle, quad, isoline). The last obvious thing we'll need to define, is what the other half of the hull program is called so they both can be executed in parallel. To define all these, we'll use attributes before the function definition.

Code (csharp):

1. [UNITY_domain("tri")]
2. [UNITY_outputcontrolpoints(3)]
3. [UNITY_outputtopology("triangle_cw")]
4. [UNITY_partitioning("integer")]
5. [UNITY_patchconstantfunc("PatchConstantFunction")]
6. ControlPoint HullProgram(InputPatch<ControlPoint, 3> patch, uint id : SV_OutputControlPointID)
7. {
8.    return patch[id];
9. }

Several things to note. There are two attributes I did not mention. output topology refers to the winding of the triangles. In our case, because we want the faces to face us, they will wind clockwise. The partitioning refers to how the tessellation factors will be interpreted. In our case, we want the full 1 to 64 range, so we'll partition them as integers. Lastly, because we are only returning one control point at a time, we need to specify which one in the patch we are working with, so we also take in an ID as well as a patch.

So, now we need to implement our other half of the hull program; this is referred to as the patch constant function. As you'll note, that's also what we named it. As this function will be returning the tessellation factors, let's talk about those. Because we're dealing with triangles we need to define 4 separate factors: one for each edge, and one for the inside. As these numbers vary, so do the patterns and complexity. We will create a struct for the tessellation factors so the tessellation program can get them as a single package.

Code (csharp):

1. struct TessellationFactors
2. {
3.    float edge[3] : SV_TessFactor;
4.    float inside : SV_InsideTessFactor;
5. };

Before implementing the patch constant function, we should do a few more things to prepare. For the simpleness of this tutorial, we'll be working with uniform tessellation factors. So we'll want a slider so we can control the level of tessellation we're working with. Add this line to the properties section of the shader.

Code (csharp):

1. _Uniform ("Uniform Tessellation", Range(1, 64)) = 1

Next, we want to get set for dynamic tessellation. We'll be dynamically tessellating based off of a texture. But you can alternatively tessellate based off of vertex color, or even vertex position. Anyway, for the texture, we'll need an additional texture slot in the properties.

Code (csharp):

1. _TessMap ("Tessellation Map", 2D) = "black" {}

Let's go ahead and initialize those two properties within the subshader too.

Code (csharp):

1. sampler2D _TessMap;
2. float _Uniform;

Okay, now we have everything we need to implement the patch constant function! This time I'll post the code first and then explain afterwards. Be sure to put this code before the HullProgram function.

Code (csharp):

1. TessellationFactors PatchConstantFunction(InputPatch<ControlPoint, 3> patch)
2. {
3.    float p0factor = tex2Dlod(_TessMap, float4(patch[0].uv.x, patch[0].uv.y, 0, 0)).r;
4.    float p1factor = tex2Dlod(_TessMap, float4(patch[1].uv.x, patch[1].uv.y, 0, 0)).r;
5.    float p2factor = tex2Dlod(_TessMap, float4(patch[2].uv.x, patch[2].uv.y, 0, 0)).r;
6.    float factor = (p0factor + p1factor + p2factor);
7.    TessellationFactors f;
8.    f.edge[0] = factor > 0.0 ? _Uniform : 1.0;
9.    f.edge[1] = factor > 0.0 ? _Uniform : 1.0;
10.    f.edge[2] = factor > 0.0 ? _Uniform : 1.0;
11.    f.inside = factor > 0.0 ? _Uniform : 1.0;
12.    return f;
13. }

As we can see, the function takes in a patch. From the patch, we use the control point (vertex) UV coordinates to sample our tessellation map. For this tutorial, we'll be sampling the red values of the texture to determine if we should tessellate. If any of the red values have a value greater than 0 (meaning there is some red at one of the three UV coordinates), then all 4 factors are given our uniform tessellation value. Otherwise, they are given a value of 1.0, telling the tessellation program not to tessellate.

Part 3 - Domain

We are now on the final piece of the puzzle: the domain program. Because the domain program produces more vertices than we started with, we'll need to process these new vertices through our vertex program. And that means constructing appdata objects ourselves. In our case, we'll need the vertex positions and UV coordinates for all our new points.

Unfortunately, what we get from the tessellation program is just barycentric coordinates for each new point. So we'll have to do a bit of leg work. Thankfully it's not too bad. All we need to do is use the barycentric coordinates as weights to interpolate our original control points. Doing this for all the control point fields gives us a complete appdata object that we can now feed into our vertex program.

Code (csharp):

1. [UNITY_domain("tri")]
2. v2f DomainProgram(TessellationFactors factors,
3.          OutputPatch<ControlPoint, 3> patch,
4.          float3 barycentricCoordinates : SV_DomainLocation)
5. {
6.    appdata data;
7.  
8.    data.vertex = patch[0].vertex * barycentricCoordinates.x +
9.          patch[1].vertex * barycentricCoordinates.y +
10.          patch[2].vertex * barycentricCoordinates.z;
11.    data.uv = patch[0].uv * barycentricCoordinates.x +
12.          patch[1].uv * barycentricCoordinates.y +
13.          patch[2].uv * barycentricCoordinates.z;
14.  
15.    return VertexProgram(data);
16. }

With everything now in place, all that's left is to get out pragmas in order.

Code (csharp):

1. #pragma vertex TessellationVertexProgram
2. #pragma fragment FragmentProgram
3. #pragma hull HullProgram
4. #pragma domain DomainProgram
5. #pragma target 4.6

Part 4 - Using the Shader

Now that our shader is done, you're probably going to want to test it out! To do that we need to do several things. First off, let's create a plane and put it in your scene (I recommend you use a higher poly plane - that way you can get more definition out of the tessellation). Next we need a material. Create one and select the shader to be the one we just created. You should now see two slots for textures that are appropriately named. Fill in the "Texture" slot with whatever you like. Next, you'll need a tessellation map. Go into your favorite image editing program and create a black image with some red areas (or use my tessellation map I created in the example section). Plug that into the "Tessellation Map" slot. All that's left is to apply the material to the plane and crank up that "Uniform Tessellation"!

Conclusion

This was a pretty basic implementation of dynamic tessellation, but hopefully it gets you started so you can modify it to suit your needs. This was my first tutorial I've ever done, so if something wasn't explained particularly well or you have questions, please let me know!

Example





Resources

CatlikeCoding Tessellation Tutorial

Tessellation Stages

Unity Surface Shader Tessellation

Barycentric Coordinates

 

Great tutorial, thanks! I'm saving this for later use

 

Glad I could help out!

 

Very useful!

 

Hey! First of all: thank you for this tutorial, everything it's incredibly well explained and I was able to follow along.
I have a problem though, and I hope not to disturb after almost two years from the posting of this:
I am using Unity 2020.1.2 and I am having an issue...the tessellation is applied to all polygons, despite of what the red level of the TessMap is. I can even not assign a texture at all, and tessellation will be applied the same.

I already checked for errors or typos in my code, but I can't seem to find any.
For some reason in the PatchConstantFunction the factor is always greater than 0...any idea why that might be happening?

 

I have read catlikecoding's tutorial on tessellation as well and stumbled upon his usages of return types/transfer structs and the presence of a

#pragma geometry

which appears to stem from his previous tutorial but which isn't explained.

You appear to have make clear the usage of return types/transfer structs and I have them like this as well but my shader outputs "nothing": no color, no pink, no error. It was a working vertex, fragment shader and I like to only (evenly) subdivide triangles of a mesh this shader is applied upon.

Code (CSharp):

1. Shader "Custom/MyShader" {
2.     Properties{
3.         // omitted here
4.     }
5.     SubShader{
6.         Tags { "Queue" = "Transparent" "RenderType" = "Transparent" }
7.        
8.         Blend SrcAlpha OneMinusSrcAlpha
9.         LOD 200
10.         Cull Off
11.  
12.         Pass {
13.             CGPROGRAM
14.             #include "UnityCG.cginc"
15.             #include "UnityLightingCommon.cginc"
16.             #pragma target 5.0
17.             #pragma multi_compile_lightpass
18.             #pragma multi_compile _ UNITY_HDR_ON
19.             #pragma vertex vert
20.             #pragma hull hull
21.             #pragma domain domain
22.             /*#pragma geometry geom*/
23.             #pragma fragment frag
24.  
25.  
26.             /* custom properties variables */
27.  
28.  
29.             struct VertexData {
30.                 float4 vertex : POSITION;
31.                 float3 normal : NORMAL;
32.                 float2 uv : TEXCOORD0;
33.             };
34.  
35.             struct TessellationData {
36.                 float4 vertex : INTERNALTESSPOS;
37.                 float3 normal : NORMAL;
38.                 float2 uv : TEXCOORD0;
39.             };
40.  
41.             struct GeometryData {
42.                 float4 vertex : SV_POSITION;
43.                 float3 normal : NORMAL;
44.                 float2 uv : TEXCOORD0;
45.                 /*
46.                     additional custom data calculated in vertexProgram
47.                 */
48.             };
49.  
50.             TessellationData vert(VertexData v) {
51.                 TessellationData p;
52.                 p.vertex = v.vertex;
53.                 p.normal = v.normal;
54.                 p.uv = v.uv;
55.                 return p;
56.             }
57.  
58.             struct TessellationFactors {
59.                 float edge[3] : SV_TessFactor;
60.                 float inside : SV_InsideTessFactor;
61.             };
62.  
63.             float _TessellationUniform = 64.0;
64.  
65.             TessellationFactors patchFunc(InputPatch<TessellationData, 3> patch) {
66.                 TessellationFactors f;
67.                 f.edge[0] = _TessellationUniform;
68.                 f.edge[1] = _TessellationUniform;
69.                 f.edge[2] = _TessellationUniform;
70.                 f.inside = _TessellationUniform;
71.                 return f;
72.             }
73.  
74.             [UNITY_domain("tri")]
75.             [UNITY_outputcontrolpoints(3)]
76.             [UNITY_outputtopology("triangle_cw")]
77.             [UNITY_partitioning("integer")]
78.             [UNITY_patchconstantfunc("patchFunc")]
79.             TessellationData hull(InputPatch<TessellationData, 3> patch, uint id : SV_OutputControlPointID) {
80.                 return patch[id];
81.             }
82.  
83.             inline GeometryData vertexProgram(VertexData tessellated) {
84.                 GeometryData toFrag;
85.                 toFrag.vertex = UnityObjectToClipPos(tessellated.vertex);
86.                 toFrag.normal = tessellated.normal;
87.                 toFrag.uv = tessellated.uv;
88.                 /*
89.                     custom calculations filling additional custom data in GeometryData, based upon toFrag.normal and custom properties
90.                 */
91.                 return toFrag;
92.             }
93.  
94.             #define DOMAIN_INTERPOLATE(fieldName) data.fieldName = \
95.                 patch[0].fieldName * barycentricCoordinates.x + \
96.                 patch[1].fieldName * barycentricCoordinates.y + \
97.                 patch[2].fieldName * barycentricCoordinates.z;
98.  
99.             [UNITY_domain("tri")]
100.             GeometryData domain(TessellationFactors factors, OutputPatch<TessellationData, 3> patch, float3 barycentricCoordinates : SV_DomainLocation) {
101.                 VertexData data;
102.                 DOMAIN_INTERPOLATE(vertex)
103.                 DOMAIN_INTERPOLATE(normal)
104.                 DOMAIN_INTERPOLATE(uv)
105.                 return vertexProgram(data);
106.             }
107.  
108.             /*[maxvertexcount(3)]
109.             void geom(triangle GeometryData i[3], inout TriangleStream<GeometryData> stream) {
110.                 stream.Append(i[0]);
111.                 stream.Append(i[1]);
112.                 stream.Append(i[2]);
113.             }*/
114.  
115.             float4 frag(GeometryData s) : SV_TARGET {
116.                 float4 c;
117.                 /*
118.                     custom calculations based upon additional custom data in GeometryData
119.                 */
120.                 return c;
121.             }
122.             ENDCG
123.         }
124.     }
125. }

Note the domain shader's argument 2 and 3 are switched in a working tessellating shader i purchased from the asset store (but which does much more than i intend to achieve here).

This is the code which surrounds my vertex and fragment code and which could be the bare minimum to add tessellation to such shaders. Unfortunately, like writte above, it ouputs nothing.

What can I do to get it apply my vertex and fragment code to a (evenly) tesselated triangular mesh?

 

the issue of my code is:

_TessellationUniform

is no property, it is only a "global" variable which appear to not exist in HLSL.

 

I'm not sure I understand what you want to say here. "_TessellationUniform" is a variable declared in line 63 of the code you just posted above. So it does exist in HLSL. Yes, there is no ShaderLab property for this variable which you don't need. Those property definitions at the top are only there to get neat UI support in the inpector of the material. You can still set that "_TessellationUniform" variable from C# using SetFloat.

Please create your own thread for your own problem. This is a tutorial thread