Ignoring some triangles in a vertex shader

Hi,
I'm displaying data from Kinect using 3D mesh (each mesh vertex is updated every frame with position/color values). The problem is that Kinect sensor is not perfect and depth values of some pixels are unknown - their position is being set to (0,0,0). So the question is - how can I ignore (in a shader) all triangles, that have one or more zero vertex?

The effect I want to achieve is similar to what RGBDToolkit guys are doing:


Thanks in advance,

 

I thought that if you set all the vertices of a single triangle to be identical, then GPUs still discard the triangle from the pipeline, one of the core optimizations from the old fixed-function / OpenGL days?

 

This is a common optimisation but it still will process verts. I don't really see much relevance other than an optimisation - it will not answer the OP's problem from feb 2013

 

It was a top hit on google, and the title "ignoring some triangles in a vertex shader" is still very important today .

e.g. today I'm doing some animation in vertex shader that requires discarding large amounts of triangles for the visual effect (think: exploding objects procedurally).

(I had a bug where some triangles were rendering even after I (thought) I had set all their verts to (0,0,0), so I was double-checking if there was anything special in Unity around triangle dropping, and was surprised by the response from Daniel. It occured to me reading this thread that maybe the technique I'd used without thinking - degenerating a triangle - was a bad habit that maybe can't be trusted any more for some subtle reason!)

 

Just set output position to NaN

 

Degenerate triangles are fine, and are indeed skipped by the GPU for rasterization (ie, the fragment shader won’t run on them), but all other shader stages still will. Also a vertex shader has no knowledge of triangles, only of individual vertices. Each vertex may be used by one or multiple triangles. It’s possible if you’re setting other non-zero vertices of a triangle to also be zero, you’re affecting a shared vertex and what you’re seeing is those triangles stretching and the one you intended to hide is indeed being hidden.

If you want to skip triangles that have one vertex that is “bad”, then you can modify the mesh from script, or use a geometry shader or ...

... that. Any triangle with a NaN vertex position will not be rendered, just like a degenerate triangle.

So, how do you set a NaN? You can't just do

o.pos = NaN;

, there's no NaN constant in HLSL or GLSL. However you can create it with

0.0/0.0

or

sqrt(-1)

.

Code (csharp):

1. v2f vert (appdata v)
2. {
3.     v2f o;
4.     o.pos = UnityObjectToClipPos(v.vertex);
5.     if (v.vertex.xyz == float3(0,0,0))
6.         o.pos.w = 0.0 / 0.0;
7. }

Both will produce warnings on your shader though. These can be safely ignored, though it should be noted that some platforms may optimize the NaNs away (old mobile GPUs mainly at this point). If the warnings or old GLES 2.0 platforms are a problem for you, use a c# script to set a global NaN shader value and use that instead.

c#

Code (csharp):

1. Shader.SetGlobalFloat("_NaN", System.Single.NaN);

hlsl

Code (csharp):

1. float _NaN;
2.  
3. v2f vert(appdata v)
4. {
5.     v2f o;
6.     o.pos = UnityObjectToClipPos(v.vertex);
7.     if (v.vertex.xyz == float3(0,0,0))
8.         o.pos.w = _NaN;
9. }

 

Anybody knows how to do the same trick in shader graph?

 

This seems to work, unless the unseen triangles are placed somewhere in the scene. I intentionally put 0/0 for the fourth component of a float4 variable and multiplied it with an identity matirx - there would be a fancier way of doing correct transformation, but it might not matter for discarding triangles, I think. And using a comparison, I culled out triangles under 0. Please let me know if I did something wrong or if there is any better way.

 

For the original question, couldn't you just set SV_ClipDistance to be negative?

https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-semantics

Not sure if that's better, the same or worse as dividing by zero.

 

SV_ClipDistance

is output from the fragment shader, meaning the shader still has to run before the pixel gets rejected.

A NaN triangle position prevents the entire triangle from drawing at all before the fragment shader is even run. And it works to cull the triangle before raster even if the fragment shader is doing other things that might cause the GPU to normally switch to post-fragment culling (like

clip()

,

discard

,

SV_Depth

, or

SV_ClipDistance

).

So, no, it is not a better solution. Plus you'd need some way to detect in the fragment shader if one vertex of the triangle was in a bad spot. That can be done by having a single float value you set to 1.0 if the vertex is in a bad position, and checking if the value in the fragment is != 0.0, but it's not a perfect solution and you may get some phantom pixels showing still where the value does interpolate to 0.0 around the opposite edge of the triangle, especially when using MSAA.

 

That's not correct. SV_ClipDistance is not output from pixel shader, it's output from vertex shader. The docs clearly say fragments won't be invoked for SV_ClipDistances less than 0:

"Primitive setup only invokes rasterization on pixels for which the interpolated plane distance(s) are >= 0"

The reflections used in adam demo use SV_ClipDistance in vertex shader to discard vertices for reflection:

https://github.com/keijiro/AdamPlan...laneReflection/Shared/UnityStandardCore.cginc

 

Ah, you're right! I guess I read that wrong.

In that case, yes, it'd probably work pretty well at culling triangles if you set the plane to 0.0 on valid vertices and -1.0 on bad vertices.

I'm not sure it'd be any faster though, as the NaN test is something the GPU is doing all of the time anyway, and it works on GPUs that don't support

SV_ClipDistance

which isn't available on mobile until GLES 3.2, and there are a lot of GLES 3.1 devices still out there. And if we're talking in the context of the original post, there were still a lot of DX9 class GPUs around on the desktop in 2013. But that's less of an issue today since Unity doesn't even officially support anything under DX11 anymore.

 

It should be noted that Section 2.3.4.1 of the OpenGL ES 3.2 Spec states that

So to me it seems that outputting `NaN` as vertex output might actually be undefined behavior, though it works on those platforms I have tested this so far.

 

It is absolutely undefined behavior, but not because of a lack of NaN support.

Basically half of the OpenGL spec is technically "optional" for various reasons, even if 99% of all GPUs implement the vast majority of the spec. I also don't know of a single GLES 2.0 or better GPU that doesn't implement IEEE 754 for 32 bit floats. Where the weirdness is in is with

mediump

and especially

lowp

values which can have wildly different implementations between GPUs, if they're actually implemented. Though most if not all implementations of

mediump

these days use IEEE 754 half precision floats which also support NaNs. And I think PowerVR were the last ones to still implement

lowp

at all, and even they stopped implementing that with their first GLES 3.0 class GPUs back in 2012.

But my original post on it did comment about how NaNs will sometimes be expunged from compiled shaders on GLES 2.0 devices, hence my recommendation to pass a NaN via c#.

Use of

SV_ClipDistance

on 3.2 or desktop GPUs as suggested by @SunnySunshine does seem to be a more "correct" path, or at least an explicitly defined behavior. Though under certain extreme scenarios this could conceivably result in some ghost fragments still being rasterized along the edge of a triangle where only one vertex had an

SV_ClipDistance

value of -1 if the interpolated clip distance ends up equal to zero, which the NaN technique would not.

But yes, it does make the assumption that the GPU's rasterizer doesn't convert any NaNs to zero, which some GPUs absolutely could as there's no strict spec for how rasterization and triangle clipping is implemented. At which point you'd end up with the triangles stretching to the center of the screen. I just don't know of any, and it's highly unlikely to be implemented like that on anything made in the last decade.

 

Code (CSharp):

1. Setting any vertex position to NaN is a good way to cull a triangle in the Vertex Shader.
2. Depending on the GPU, this can be achieved by writing 0/0 or asfloat(0x7fc00000).

Tip number 42:
https://interplayoflight.wordpress.com/2022/01/22/shader-tips-and-tricks/

 

You can suppress NaN warning by this code in hlsl.

Code (CSharp):

1. #pragma warning(disable : 4008)

4008 is WARN_FLOAT_DIVISION_BY_ZERO.

 

Very helpful and clear: thanks! But weird results on my side: degenerating triangles obviously works on a machine, and not at all on the other, everything else being the same (code, Unity version, etc). While the "suppression" of some triangles generates more FPS on one machine, it has strictly no effect on the second, while the profiler confirms that the code is GPU bound here. Any idea or suggestion?

 

You should use a GPU profiler to see what the bottleneck is. If both machines are windows PCs, Microsoft PIX is great, and NVIDIA has NSight.

Generating NaNs is equivalent to generating a triangle that's too small to be seen, far off screen, has 2 points colinear, etc. The vertex shader still has to run. The rasterizer still needs to reject it. It's still going to take up space in the vertex cache. All that will happen is that it won't produce fragments, so the pixel shader won't run.

Any of those other things could be your bottleneck. More likely, this shader isn't your bottleneck at all.

 

Thanks a lot; I saw PIX on some occasions, but never did look into it, so that reminder is welcome (and I didn't know about NSight). I definitely agree (thus my post) that the rasterizer/pixel stage not running shall generate savings, which is confirmed on one of the two PCs. Got to find out what happens on the second one (bit older but the GPU is a Quadro P4000, quite fat)...

 

As suggested in the post above, using

Code (csharp):

1. o.vertex = asfloat(0x7fc00000);

Is same as setting to NaN (0/0) and does not generate a warning, I'd recommend that instead of suppressing the warning for your whole shader. But maybe I'm just paranoid haha. I use this trick to cull triangles myself.

 

It's absolutely wild how much work the gpu does every frame when you really start digging into the pipeline..! It's just, double checking every triangle every time you render a mesh to make sure it's not degenerate?! Wild!