Tesselation with custom data in vertex shader: compile error

In fewer words:
If I try to pass some custom data from vertex function, Unity doesn't want to compile my surface shader with tesselation.

[HR][/HR]

Details:
I'm trying to create a complex DX11 shader.
On the one hand, it needs to perform tesselated displacement. On the other hand, it needs to calculate some custom stuff in vertex shader, which is then used in fragment shader.
And I'm stuck at some point. Unity just doesn't want to compile the shader and that's it!

To isolate the problem, I've wrote the simple test shader (tesselation + displace):

Code (csharp):

1.  
2. Shader "Simple Tessellation" {
3. Properties {
4.     _EdgeLength ("Edge length", Float) = 4
5.     _PhongTess ("Phong Strengh", Float) = 0.5
6. }
7.  
8. SubShader {
9.     Tags { "RenderType"="Opaque" }
10.     LOD 300
11.    
12.     CGPROGRAM
13.         #pragma target 5.0
14.         #pragma surface surf Lambert addshadow fullforwardshadows vertex:vert tessellate:tessEdge tessphong:_PhongTess nolightmap
15.         #include "Tessellation.cginc"
16.        
17.         float
18.             _EdgeLength,
19.             _PhongTess
20.         ;
21.        
22.         float4 tessEdge (appdata_tan v0, appdata_tan v1, appdata_tan v2) {
23.             return UnityEdgeLengthBasedTess (v0.vertex, v1.vertex, v2.vertex, _EdgeLength);
24.         }
25.        
26.         struct Input {
27.             fixed3 someCustomVector;
28.         };
29.        
30.         void vert (inout appdata_tan v)
31.         {
32.             v.vertex.xyz += normalize(v.normal) * 0.1f;
33.         }
34.        
35.         void surf (Input IN, inout SurfaceOutput o) {
36.             o.Albedo = 1.0f;
37.         }
38.     ENDCG
39. }
40. }
41.  

It works fine, just as expected.


But as soon as I try to pass some data from vertex to fragment function...

Code (csharp):

1.  
2. ...
3.         void vert (inout appdata_tan v, out Input o) // added output
4.         {
5.             UNITY_INITIALIZE_OUTPUT(Input, o); // and initialized it
6.             v.vertex.xyz += normalize(v.normal) * 0.1f;
7.         }
8. ...
9.  

..., my shader is unable to compile with the following warnings in console:

The question is obvious: what am I doing wrong and how do I fix it?

 

Any clue? Anyone?
Please...

 

I also have this problem. Does it have to do with the vertex shader actually being the domain shader when using tessellation?

 

I'm not that skilled to understand how all that new domain and hull shaders work. So for now I didn't even try to write them by myself (I'm using surface shader approach).
But I've sent a bug and Unity confirmed it is a bug. So... looks like we've got nothing more then just wait for it to be fixed.

 

Same problem here (6 months later still no fix?), doing a triplanar surface shader that makes use of custom vert data (working fine), once tessellation enabled I get the same error on the vertex modifier function:

 

AFAIK you cannot pass data from vertex program to surf program with tesselation. I read somewhere that you have to make a CG shader with a hull shader .... not sure how this works.

as most stuff in unity don't try to push it too much because it will break... like DX11 shader with custom lighting models.

 

I wrote to the support maybe a week after opening this thread, even attached a test scene to the bug report.
Don't remember if they even answered. But if they did, it was something like: "thank you for your interest in DX11 features, we'll take a look and will try to find out a best solution... bla-bla-bla"

There's still nothing fixed and therefore tesselation (as well as ANY other DX11 stuff) is unuseable.
I think, it's Unity's policy: make an illusion that they have a lot of features in their engine... which in fact just don't work for anything more then just a presentation of the feature.
This way they can tell their engine could be used for AAA-gaming. While it couldn't.

UPD: and since you're the 2nd person trying to use tesselation for the past 6 months... well... maybe, they're right with their policy. Looks like nobody here cares about DX11. Everyone see Unity only as the engine for smartphones.

 

bump

 

Well, it's quite depressing to read so as I just encountered the exact same problem!

This was posted more than a year ago and isn't there any fix?

 

Bump

 

bump

 

https://forum.unity3d.com/threads/a...hout-verts-out-parameter.432333/#post-2798796

TLDR: Surface shaders with tessellation don't support custom vertex data, but you can modify the generated shader code to make it work.

 

This does seem to be a limitation of the pre-processing that Unity does to convert Surface Shader code into final usable shaders. It still exists as a limitation in Unity 5.5.0f3.

I have found it is possible to get the surface shader to compile without errors, with a custom data item in the custom appdata structure, and the vertex shader filling in that custom value. But when you run the scene, the object with the surface shader applied disappears. The final code that Unity is creating must be throwing an error on the graphics card when it runs.

As a workaround, if you can spare it, I found it is possible to force custom data through the COLOR variable of the appdata_full structure, or through your own custom appdata structure (if it only contains kosher variables -- i.e. ones with existing semantic pre-defines after them).

Here is a short example (NOTE: I pieced this together from my working shader, so the code below may not actually compile and run -- but the technique does work):

Code (CSharp):

1. CGPROGRAM
2.     #pragma surface surf StandardSpecular tessellate:tessFixed vertex:vert alpha:fade
3.     #pragma target 4.6
4.  
5.     // **** Tesselation ****
6.     float TesselationLevel;
7.     float4 tessFixed ()
8.     {
9.         return TesselationLevel;
10.     }
11.  
12.     // *** Vertex ***
13.     struct appdata
14.     {
15.         float4 vertex : POSITION;
16.         float4 tangent : TANGENT;
17.         float3 normal : NORMAL;
18.         float2 texcoord : TEXCOORD0;
19.  
20.         // we will use this to pass custom data to the surface function
21.         fixed4 color : COLOR;
22.     };
23.  
24.     struct Input
25.     {
26.         // It is necessary to explicitly define the color variable here, as otherwise
27.         // it will not be visible to the surface shader
28.         float4 color : COLOR;
29.     };
30.  
31.     void vert (inout appdata v)
32.     {
33.         // as a simple example, calculate a custom set of UVs and pass them to the
34.         // surface shader using the COLOR component
35.         v.color = float4 (_Time.y, 0.5 * _Time.y, 0.0, 1.0);
36.     }
37.  
38.     // *** Surface ***
39.     void surf (Input IN, inout SurfaceOutputStandardSpecular o)
40.     {
41.         // retrieve custom UV coordinates from the COLOR component
42.         float2 customUVs = IN.color.xy;
43.  
44.         // ... do something with the custom data ...
45.     }
46. ENDCG

This is not ideal, but is a possible workaround for the pre-processor limitations.

I have tried the same trick with the TANGENT component, but it is evidently overwritten as part of the tesselation process. I believe the same is true for the NORMAL and TEXCOORDn components.

 

@Quiet-Pixel Thanks man! That's a good news. At least something, to be correct.
Have you tried using multiple COLOR semantics? I.e., COLOR0, COLOR1 etc.
Not in front of my working PC and still have Unity 5.3.6 anyway - so can't check it for myself right now.

 

No, I haven't tried the other COLOR semantics, but it might just work.

 

Any interpolator with : COLOR0 (or 1, or just COLOR) on it will just have the vertex colors copied to it. I asked support about this a while ago, because in previous experiences I've been able to pack additional fixed data into COLOR1, but apparently that is a DX9 specific artifact that doesn't really exist on newer platforms. The color semantics on these platforms actually just map to another TEXCOORD, so for convenience, when you write COLOR on an input structure, Unity just copies the vertex color in there and makes it another TEXCOORD in the compiled output.

As for using the standard shader with tessellation, it's pretty much a dead end if you need to do anything with the vertices except pass color. There are a lot of 'unfinished' areas of the standard shader like this (geometry shaders as well), which is a shame because the surface shader system is really nice to avoid having to write a ton of lighting code.

When I added tessellation to MegaSplat, I ended up reverse engineering all of Unity's lighting (and all of it's optional lighting features), which was a real pain, and I'll likely never know if all of it's correct. There isn't just some "LightThisPixel" function you can call, but rather a mess of Macros which have assumptions about naming conventions and structures that make them nearly unusable from other code. (Seriously, just let me pass the values to your macro instead of assuming I have a specific structure named v). I mean, shaders are a bit of a black art, but it would be much nicer if either the surface shader system completed it's abstractions of the shader pipeline for us, or if the lighting system was easily useable by non-surface shader.

 

I feel the same, man. Totally get it.

But seems like the only thing we could do right now is just wait. Hoping that UT will fix it. Some Day. Maybe.
It's frustrating, yeah. I'm glad I'm in mobile gamedev right now, where all of these next-gen-graphics ussues simply don't exist (but where are a plenty of others).

The only thing that brings me a hope for brighter future is that now (since v5) Unity is on the right track. Each new version finally brings us what we've been begging for so long. So maybe some day they'll finally make surface shaders actually work. In all the scenarios, and not only in those that UT have planned surface shaders for.
But I don't really expect it will happen sooner then a year or two.

So, I think, for now we're stuck with "avoid surface shaders system completely or deal with it's bugs" dilemma.
You can also do what some people recommend: to generate a main code via surface shader and then manually fix it, saving as a "low-lever vert&frag" shader. But you can imagine how much of a pain it becomes after each engine upgrade. Considering that, in some cases it will actually be easier to stick to one of the two options above.

 

It appears that only the COLOR semantic is allowed. Adding a variable to the appdata structure and using the COLOR1 semantic causes a shader compile error:

Vertex program 'tessvert_surf': unknown input semantics COLOR/1

My guess is that only the sematics listed in the appdata_full structure are supported. Of those, only the COLOR semantic does not get overwritten by the tessellation process. So, at best we have 4 fixed type channels to transfer custom data from the vertex function to the pixel function.

@jbooth -- based on what you wrote I thought maybe the trick was working because I was ending up with a d3d9 shader. I added #pragma only_renderers d3d11 to the shader and it still worked. Were you saying that it should not work for d3d11?

 

No, basically, COLOR0 is only a thing on DX9, on all other platforms, it gets turned into a regular TEXCOORD#. The only thing it does is add a:

o.color = v.color;

at the end of your shader. You can see this by looking at the compiled output.