How to (really) force forward rendering in shader?

Sorry in advance for the long post... I've posted this in Answers also, but I think it's more suited here.

First some background:

I've been trying a lot with a Surface Shader that:

• Moves its vertices.
• Uses a custom lighting model for specularity.
• Uses a first pass to draw color and write depth.
• Uses a second pass to draw alpha-blended and depth-testing.
• Is all forward rendered because of the alpha blended second pass.

Problem: when enabled Deferred mode in Unity, it seems the depth is being rendered with vertices in their original positions, so that when "real" vertices actually are behind those "imaginary" vertices, they are not being drawn (or at least it looks like this is the behaviour).

I haven't written a _PrePass lighting model function because, if it doesn't exist, it will just render in forward mode, which is what I want, according to the manual.

For narrowing the problem down, I've tried with example shaders from doc, which I think are proven enough to be reliable. Haven't found any that moves vertices and also has a view dependant lighting model (specular), so I combined a couple into this one:

Shader "SimpleSpec"
{
Properties
{
_MainTex ("Base (RGB) Trans (A)", 2D) = "white" {}
_SpecColor ("Specular Color", Color) = (0.5, 0.5, 0.5, 0)
}

SubShader
{
Tags { "RenderType"="Opaque"}

Cull Back
ZWrite On

CGPROGRAM
#pragma surface surf BlinnPhong2 vertex:vert

// tried with exclude_pathrepass, it just ignores _PrePass function
//#pragma surface surf BlinnPhong2 vertex:vert exclude_pathrepass

struct Input
{
float2 uv_MainTex;
};

void vert(inout appdata_full v)
{
v.vertex.xyz += v.normal * sin(_Time.y * 2) * 0.005;
}

inline fixed4 LightingBlinnPhong2 (SurfaceOutput s, fixed3 lightDir, half3 viewDir, fixed atten)
{
half3 h = normalize (lightDir + viewDir);
fixed diff = max (0, dot (s.Normal, lightDir));
float nh = max (0, dot (s.Normal, h));
float spec = pow (nh, s.Specular*128.0) * s.Gloss;
fixed4 c;
c.rgb = (s.Albedo * _LightColor0.rgb * diff + _LightColor0.rgb * _SpecColor.rgb * spec) * (atten * 2);
c.a = s.Alpha + _LightColor0.a * _SpecColor.a * spec * atten;
return c;
}

inline fixed4 LightingBlinnPhong2_PrePass (SurfaceOutput s, half4 light)
{
fixed spec = light.a * s.Gloss;
fixed4 c;
c.rgb = (s.Albedo * light.rgb + light.rgb * _SpecColor.rgb * spec);
c.a = s.Alpha + spec * _SpecColor.a;
return c;
}

sampler2D _MainTex;

void surf (Input IN, inout SurfaceOutput o) {
o.Albedo = tex2D (_MainTex, IN.uv_MainTex).rgb;
o.Gloss = 1;
o.Specular = _SpecColor;
}
ENDCG
}
}

This shader looks ok while the _PrePass function exists, but presents the described problem otherwise.

So, my final question is: how should I force my vertex-moving-plus-specular shader to forward mode, while still being able to use deferred in the rest of my project?

PS: It works perfectly fine with forward mode under Windows, and with any render path under Mac OS X, so I suspect this is a bug, but I would like to know if anyone has encountered this problem before and/or your opinions.

 

If you displace the verts position in a vertex shader you need to add a custom render type to the Camera-DepthTexture or Camera-DepthTextureNormal shader.

In your shader you will then use this custom render type in the tag. For example...

Tags { "RenderType"="MyCustomType"}

The reason is that in deferred render the mesh gets rendered a second time writing the depth and normals into a render texture which will be used later for the deferred lighting. The shader used to do this is the render type. If you just have opaque as your render type then the standard opaque shader will be use which will not displace you verts like in your original shader.

You can find the Camera-DepthTexture or Camera-DepthTextureNormal shader from the Unity default shaders download (cant remember where exactly).

You need to add you render type to a copy of these shaders and then add them to the resources folder where Unity will use your modified copies instead of the default ones.

 

Great, thank you! Understanding that, I tried using "RenderType"="Transparent", since anyway for my shader the second pass was for the alpha-blended part of the shader -- it's a hair shader. Combined with "Queue"="Transparent" to solve a shadow-see-thru problem (because having shadows is not a priority). And finally it looks correct in Deferred mode. Thank you!

Next time I need a shader that moves vertices and is opaque, I'll try to add a new render type as you suggested. For now I think it's more correct (and easier) to have it as a transparent render type.

Nonetheless, I still don't understand why it was being drawn correctly in Mac OS X with the wrong render type. I guess OpenGL works slightly different with depth buffers...

Anyway, thanks again scrawk!

 

Finally, found my fix here
I have a custom lighting model (ToonLighting) that was not working with alpha test.

At first I just needed the alpha test because the texture had cuts (in alpha). I got it working with:
Tags{ "Queue" = "AlphaTest" "IgnoreProjector" = "True" "RenderType" = "TransparentCutout" }
#pragma surface surf ToonRamp alphatest:_Cutoff addshadow
inline half4 LightingToonRamp (SurfaceOutput s, half3 lightDir, half atten)
...c.a = s.Alpha;
void surf (Input IN, inout SurfaceOutput o)
...o.Alpha = c.a;

But then I needed to do some custom clipping (plane) and changed the alpha in surf:
o.Alpha = c.a * visible;

I suspect the lighting prepass didn't know about this. (happened regardles my use of exclude_pathrepass)
And got some shading artifacts in the transparent areas.

The solution was to use the RenderType=Transparent
Tags{ "Queue" = "AlphaTest" "IgnoreProjector" = "True" "RenderType" = "Transparent" }

Looks good now.
I hope this is a good setup for all platforms - works on my computer

Much thanks!

[EDIT]
Nope, false alarm! Still not perfect
I get some shadow artifacts on the opaque geometry now.
I'll keep you updated of the progress.

[EDIT2]
Apparently these are from the ambient occlusion script on the camera. This never ends!

 

just for the record:

not transparent ("RenderType" = "TransparentCutout")

AO artifacts ("RenderType" = "Transparent")


no AO ("RenderType" = "Transparent") - GOOD