Question Issue with (probably old) Deferred Decals - using Command buffers / Shaders.

So, for context my game project is pretty old now (6+ years), so it's in the Built-in render pipeline, and when I wanted to implement decals I recall I found an old Deferred decal example and I adapted it (I know the code isn't great, I'm sorry).
I am mostly a designer, I play around with shaders, making cool things, but a lot of the more technical aspects often are pretty confusing to me.
However, I added emission and the option to use sprites with set sprite subdivisions for ease of use. It all worked fairly well:


The system works with a deferred decal Script that adds every decal to a static list when enabled which is then rendered by a Decal Manager.
Each decal sets up a Material property block with the required shader and when the Manager calls it, it gets drawn (I think this is all working fine).

Here's the more important bits of the MANAGER,
(I am still not sure why if I add a BuiltinRenderTextureType.GBuffer3, supposedly for emmission, it instead renders the normal map as emissive. This is all arcane to me, but removing the GBuffer3, and keeping the cameraTarget works... probably)

Code (CSharp):

1.  
2. public Mesh Mesh;
3.         public Material Material;
4.         public Material NormMaterial;
5.         public bool DoCull;
6.  
7.         struct CameraInfo {
8.             public CommandBuffer CommandBuffer;
9.             public CullingGroup CullingGroup;
10.         }
11.  
12.         private Dictionary<Camera, CameraInfo> m_cmdBuffer = new Dictionary<Camera, CameraInfo>();
13.         private RenderTargetIdentifier[] m_targets;
14.         private RenderTargetIdentifier[] mNorm_targets;
15.         private BoundingSphere[] m_boundingSpheres = new BoundingSphere[128];
16.  
17.         public static DeferredDecalManager Instance;
18.  
19.         void OnEnable() {
20.  
21.             if (Instance == null)
22.             {
23.                 Instance = this;
24.             }
25.             Camera.onPreRender += OnRender;
26.  
27.             m_targets = new RenderTargetIdentifier[] {
28.                 BuiltinRenderTextureType.GBuffer0, BuiltinRenderTextureType.GBuffer1, BuiltinRenderTextureType.CameraTarget
29.             };
30.  
31.             mNorm_targets = new RenderTargetIdentifier[] {
32.                 BuiltinRenderTextureType.GBuffer0, BuiltinRenderTextureType.GBuffer1, BuiltinRenderTextureType.GBuffer2,   BuiltinRenderTextureType.CameraTarget
33.             };
34.         }
35.  
36.         void Update() {
37.  
38.             int DecalCounts = DeferredDecal.Decals.Count + DeferredDecal.NormalDecals.Count;
39.             while (DecalCounts >= m_boundingSpheres.Length) {
40.                 Array.Resize(ref m_boundingSpheres, m_boundingSpheres.Length * 2);
41.             }
42.  
43.             for (int i = 0; i < DeferredDecal.Decals.Count; i++) {
44.                 var decal = DeferredDecal.Decals[i];
45.                 m_boundingSpheres[i].position = decal.transform.position;
46.                 m_boundingSpheres[i].radius = (0.5f * decal.transform.localScale).magnitude;
47.             }
48.  
49.             for (int i = 0; i < DeferredDecal.NormalDecals.Count; i++)
50.             {
51.                 var decal = DeferredDecal.NormalDecals[i];
52.                 m_boundingSpheres[i].position = decal.transform.position;
53.                 m_boundingSpheres[i].radius = (0.5f * decal.transform.localScale).magnitude;
54.             }
55.         }
56.  
57.         void OnRender(Camera cam) {
58.             if (Material == null) {
59.                 return;
60.             }
61.  
62.             CameraInfo camInfo;
63.             if (!m_cmdBuffer.TryGetValue(cam, out camInfo)) {
64.                 camInfo = new CameraInfo();
65.                 camInfo.CommandBuffer = new CommandBuffer();
66.                 cam.AddCommandBuffer(CameraEvent.BeforeReflections, camInfo.CommandBuffer);
67.  
68.                 camInfo.CullingGroup = new CullingGroup();
69.                 camInfo.CullingGroup.targetCamera = cam;
70.                 camInfo.CullingGroup.SetBoundingSpheres(m_boundingSpheres);
71.  
72.                 m_cmdBuffer[cam] = camInfo;
73.             }
74.  
75.             var buffer = camInfo.CommandBuffer;
76.        
77.             int count = DeferredDecal.Decals.Count;
78.             int nmcount = DeferredDecal.NormalDecals.Count;
79.  
80.             camInfo.CullingGroup.SetBoundingSphereCount(count + nmcount);
81.  
82.             buffer.Clear();
83.  
84.             buffer.SetRenderTarget(m_targets, BuiltinRenderTextureType.CameraTarget);
85.  
86.        
87.  
88.             if (count > 0)
89.             {
90.                 for (int i = 0; i < DeferredDecal.Decals.Count; ++i)
91.                 {
92.                     var decal = DeferredDecal.Decals[i];
93.  
94.                     if ( camInfo.CullingGroup.IsVisible(i) || !DoCull)
95.                     {
96.                         decal.Draw(Material, Mesh, buffer);
97.                     }
98.                 }
99.             }
100.  
101.             buffer.SetRenderTarget(mNorm_targets, BuiltinRenderTextureType.CameraTarget);
102.  
103.             if (nmcount > 0)
104.             {
105.                 for (int i = 0; i < DeferredDecal.NormalDecals.Count; ++i)
106.                 {
107.                     var decal = DeferredDecal.NormalDecals[i];
108.  
109.                     if (camInfo.CullingGroup.IsVisible(i) || !DoCull)
110.                     {
111.                         decal.Draw(NormMaterial, Mesh, buffer);
112.                     }
113.                 }
114.             }
115.         }
116.         }

As you can see, later I got the idea of adding normal maps to the system, so there's actually two lists of decals, a BASIC decal and a NORMALMAP decal. Because it needed a different different shader and material block, so I made it so that the script would conditionally apply the required material.
In each decal:

Code (CSharp):

1.  
2. public void Draw(Material material, Mesh mesh, CommandBuffer cmdBuffer)
3. {
4.             if (SpriteTexture != null)
5.             {
6.                 Vector4 subtextureRect =  new Vector4( SpriteTexture.rect.width / SpriteTexture.texture.width, SpriteTexture.rect.height / SpriteTexture.texture.height, SpriteTexture.rect.min.x/ SpriteTexture.texture.width, SpriteTexture.rect.min.y / SpriteTexture.texture.height);
7.  
8.                 m_block.SetTexture("_MainTex", SpriteTexture.texture);
9.                 m_block.SetVector("_MainTex_ST", subtextureRect);
10.                 if (Normal)
11.                 {
12.                     m_block.SetTexture("_Normal", Normal);
13.                     m_block.SetFloat("_Bump", Bump);
14.                 }
15.                 m_block.SetColor("_Color", Color);
16.                 m_block.SetFloat("_Glossiness", Smoothness);
17.                 m_block.SetFloat("_Metallic", Metallic);
18.                 m_block.SetFloat("_AngleCutout", AngleCutout);
19.                 m_block.SetFloat("_Emission", Emission);
20.  
21.                 cmdBuffer.DrawMesh(mesh, transform.localToWorldMatrix, material, 0, 0, m_block);
22.             }
23.         }

Here I started to run into some issues, because it seemed like I was redefining the material outs to be used in the G-BUFFERS but I think I managed to figure most of that out, however some more problems appeared:

• The normal maps, usually in tangent space applied in the default orientation and didn't change with the decal's rotation.
• Probably connected to above, the normal mapped decals don't seem to capture reflection probes or SSR properly.
• The normal in these decals fully overwrote the world existing normals, no matter what I did (still not sure what to do about this one, but it's not a priority).

To tackle the above issues, I read some tutorials and determined I needed to change the original normals from the map to WORLD SPACE, and then back to tangent space to get them to display properly. along with some swizzling of the textures to match the world. I am not quite sure this assumption is correct anymore.

Here's the shader (Not sure if it's too much to post the whole thing since it's pretty big, but I'm really not sure where the error-s could be):

Code (CSharp):

1. Shader "Custom/DecalNormal" {
2.     Properties{
3.         _MainTex("Albedo (RGB)", 2D) = "white" {}
4.     }
5.  
6.         SubShader
7.     {
8.  
9.         Pass{
10.         ZWrite Off
11.  
12. // SNIPPING THE PRAGMAS AND ALL THAT
13.  
14.     sampler2D _MainTex;
15.     uniform float4 _MainTex_ST;
16.     sampler2D _Normal;
17.  
18.     half _Glossiness;
19.     half _Metallic;
20.     half _AngleCutout;
21.     half _Emission;
22.     half _Bump;
23.  
24.     float4 _Color;
25.  
26.     sampler2D _CameraDepthTexture;
27.     sampler2D _CameraGBufferTexture2;
28.  
29.     // vertex-to-fragment interpolation data
30.     struct v2f_surf {
31.         float4 pos : SV_POSITION;
32.         half2 uv : TEXCOORD0;
33.  
34.         half3 orientation : TEXCOORD1;
35.         half3 worldPos : TEXCOORD2;
36.         half3 viewDir : TEXCOORD3;
37.         half4 screenUV : TEXCOORD4;
38.    
39.         half3 sh : TEXCOORD5; // SH
40.         half3 ray : TEXCOORD6;
41.  
42.         half3x3 worldToTangent : TEXCOORD7;
43.  
44.         // ARE THESE TOO MANY TEXCOORDS!?
45.     };
46.  
47.     // vertex shader
48.     v2f_surf vert_surf(appdata_full v) {
49.         v2f_surf o;
50.  
51.         o.pos = UnityObjectToClipPos(v.vertex);
52.         o.screenUV = ComputeScreenPos(o.pos);
53.         o.uv =  v.texcoord.xy * _MainTex_ST.xy + _MainTex_ST.zw;
54.  
55.         float3 worldPos = mul(unity_ObjectToWorld, v.vertex).xyz;
56.         fixed3 upObj = UnityObjectToWorldNormal(float3(0.0f, 1.0f, 0)).xyz;
57.  
58.         o.worldPos = worldPos;
59.         o.orientation = upObj;
60.         o.ray = UnityObjectToViewPos(float4(v.vertex.xyz, 1)).xyz * float3(-1, -1, 1);
61.  
62.         //MATRIX TO TRANSFORM BACK TO TANGENT SPACE
63.         half3 wNormal = mul((float3x3)unity_ObjectToWorld, SCALED_NORMAL);
64.         half3 wTangent = UnityObjectToWorldDir(v.tangent.xyz);
65.         half tangentSign = v.tangent.w * unity_WorldTransformParams.w;
66.         half3 wBitangent = cross(wNormal, wTangent) * tangentSign;
67.         o.worldToTangent = half3x3(wTangent, wBitangent, wNormal);
68.    
69.         float3 viewDirForLight = UnityWorldSpaceViewDir(worldPos);
70.         o.viewDir = viewDirForLight;    
71.         o.sh = ShadeSH9 (float4 (wNormal, 1.0));
72.  
73.         return o;
74.     }
75.  
76. //Arcane function which does things.
77. inline void LightingStandard_Deferred_Broken(SurfaceOutputStandard s, UnityGI gi, out half4 outDiffuseOcclusion, out half4 outSpecSmoothness)
78.     {
79.         half oneMinusReflectivity;
80.         half3 specColor;
81.         s.Albedo = DiffuseAndSpecularFromMetallic(s.Albedo, s.Metallic,specColor, oneMinusReflectivity);
82.  
83.         outDiffuseOcclusion =  half4(s.Albedo, s.Occlusion);
84.         outSpecSmoothness = half4(specColor, s.Smoothness);
85.     }
86.  
87.     // fragment shader
88.     void frag_surf(v2f_surf IN,
89.         out half4 outDiffuse : SV_Target0,
90.         out half4 outSpecSmoothness : SV_Target1,
91.         out half4 outNormal : SV_Target2,
92.         out half4 outEmission : SV_Target3) {
93.  
94.         // prepare and unpack data
95.         IN.ray = IN.ray * (_ProjectionParams.z / IN.ray.z);
96.         float2 uv = IN.screenUV.xy / IN.screenUV.w;
97.         // read depth and reconstruct world position
98.         float depth = SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture, uv);
99.         depth = Linear01Depth(depth);
100.         float4 vpos = float4(IN.ray * depth, 1);
101.         float3 wpos = mul(unity_CameraToWorld, vpos).xyz;
102.         float3 opos = mul(unity_WorldToObject, float4(wpos, 1)).xyz;
103.  
104.         clip(float3(0.5, 0.5, 0.5) - abs(opos.xyz));
105.  
106.         float4 wnormalRAW = tex2D(_CameraGBufferTexture2, uv);
107.         float3 wnormal = wnormalRAW.rgb * 2.0 - 1.0;
108.         // for flipped decals
109.         clip(abs(dot(wnormal, IN.orientation)) - _AngleCutout);
110.  
111.         float3 worldViewDir = normalize(UnityWorldSpaceViewDir(IN.worldPos));
112.  
113.         SurfaceOutputStandard o = (SurfaceOutputStandard)0;
114.  
115.         float4 texAlbedo = tex2Dbias(_MainTex, float4((opos.xz *_MainTex_ST.xy ) + (_MainTex_ST.zw) + (_MainTex_ST.xy*0.5) , 0.0, -1.0));
116.         float4 tNorm = tex2Dbias(_Normal, float4((opos.xz * _MainTex_ST.xy) + (_MainTex_ST.zw) + (_MainTex_ST.xy * 0.5), 0.0, -1.0));
117.         float3 scaledNorm = UnpackScaleNormal(tNorm,_Bump);
118.  
119.         // R IS object X;
120.         // G IS object Z;
121.         // B IS object Y;
122.         // FROM TANGENT TO WORLD, BUT SWIZZLED, this seems to work.
123.         float3 worldNorm = mul(unity_ObjectToWorld , scaledNorm.xzy);
124.         // BACK TO TANGENT SPACE TO USE.
125.         float3 tanNorm = normalize(mul(IN.worldToTangent , worldNorm));
126.  
127.         clip(texAlbedo.a - _Color.a);
128.  
129.         //SET VARIABLES
130.         o.Albedo = texAlbedo * _Color.rgb;
131.         o.Smoothness = _Glossiness;
132.         o.Alpha = texAlbedo.a ;
133.         o.Metallic = _Metallic;
134.  
135.         half atten = 1;
136.  
137.         // Setup lighting environment I DONT QUITE UNDERSTAND
138.         UnityGI gi;
139.         UNITY_INITIALIZE_OUTPUT(UnityGI, gi);
140.         gi.indirect.diffuse = 0;
141.         gi.indirect.specular = 0;
142.         gi.light.color = 0;
143.         gi.light.dir = half3(0,1,0);
144.         gi.light.ndotl = 0;
145.         UnityGIInput giInput;
146.         UNITY_INITIALIZE_OUTPUT(UnityGIInput, giInput);
147.         giInput.light = gi.light;
148.         giInput.worldPos = IN.worldPos;
149.         giInput.worldViewDir = worldViewDir;
150.         giInput.atten = atten;
151. #if UNITY_SHOULD_SAMPLE_SH
152.         giInput.ambient = IN.sh;
153. #else
154.         giInput.ambient.rgb = 0.0;
155. #endif
156.         giInput.probeHDR[0] = unity_SpecCube0_HDR;
157.         giInput.probeHDR[1] = unity_SpecCube1_HDR;
158.  
159. #if UNITY_SPECCUBE_BLENDING || UNITY_SPECCUBE_BOX_PROJECTION
160.         giInput.boxMin[0] = unity_SpecCube0_BoxMin; // .w holds lerp value for blending
161. #endif
162.  
163. #if UNITY_SPECCUBE_BOX_PROJECTION
164.         giInput.boxMax[0] = unity_SpecCube0_BoxMax;
165.         giInput.probePosition[0] = unity_SpecCube0_ProbePosition;
166.         giInput.boxMax[1] = unity_SpecCube1_BoxMax;
167.         giInput.boxMin[1] = unity_SpecCube1_BoxMin;
168.         giInput.probePosition[1] = unity_SpecCube1_ProbePosition;
169. #endif
170.         LightingStandard_GI(o, giInput, gi);
171.  
172.         //Throw all the settings them at the scary function
173.         LightingStandard_Deferred_Broken(o, gi, outDiffuse, outSpecSmoothness);
174.         outNormal = float4(tanNorm, 1);
175.         outEmission = float4(_Emission * o.Albedo,1);
176.  
177. #ifndef UNITY_HDR_ON
178.         outEmission.rgb = exp2(-outEmission.rgb);
179. #endif
180.         UNITY_OPAQUE_ALPHA(outDiffuse.a);
181.     }
182.  
183.     ENDCG
184.     }
185.     }
186. }

After a lot of testing, this is what I have:


I am not sure if this is perfect but it looks mostly consistent with a regular material with normal map:

However, the reflection / ambient issue still remains. The reflections on a mirror-like surface are clearly off:
I can notice sometimes that the decals DO reflect the scene but in a weird angle.

you can see the comparison below, all of them are the same texture, all white, full gloss, 0 metallic.

It's clear that the normal ones are doing something, but the only one receiving the light and the reflections correctly is the FLAT DECAL... but why!?

SO! since I've been at it for weeks now and I haven't been able to correct them, I thought maybe I'd ask for some help.

To be honest I'm happy with it just displaying normals somewhat effectively, but I really wish it was perfect...
SO! Does anyone have any pointers as to how I could fix this?

And above all, THANK YOU if anyone takes the time to go through this.

 

UPDATE!! I was playing around, checking the differences between the BASE SHADER and the NORMAL MAP shader, and it seems the issue is in the usage of the LightingStandard_Deferred_Broken method.

the original makes the NORMAL calculations in the function:

Code (CSharp):

1.  
2. inline void LightingStandard_Deferred_Broken(SurfaceOutputStandard s, half3 viewDir, UnityGI gi, out half4 outDiffuseOcclusion, out half4 outSpecSmoothness, out half4 outNormal)
3.     {
4.         half oneMinusReflectivity;
5.         half3 specColor;
6.         s.Albedo = DiffuseAndSpecularFromMetallic(s.Albedo, s.Metallic,specColor, oneMinusReflectivity);
7.  
8.         outDiffuseOcclusion = half4(s.Albedo, s.Occlusion);
9.         outSpecSmoothness = half4(specColor, s.Smoothness);
10.         outNormal = half4(s.Normal * 0.5 + 0.5, 1);
11.     }

However, it seems like this is simply not doing anything.
If I remove all my normal map calculations and allow it to return that the result is consistent, but of course I lose all normals from the maps:


HOWEVER, if outNormal is overwritten in ANY WAY it seemingly breaks. And I am not sure how I can have access to the existing world normal to blend. So it seems that in some way the normal stores the object transformation?.

BUT GOOD NEWS (?), the thing that I did find in my playing around, is that seemingly these decals work in WORLD SPACE, so there was no need to transform the world space normals back into tangent space normals. And no need to work tangents, bitangents and all of that noise.
SO I CAN DO THIS!!

 outNormal = half4(normalize(mul(unity_WorldToObject,worldNorm)) * 0.5 + 0.5); 

AAND I can get the correct NORMALS, LIGHTS AND REFLECTIONS!! *


*MAYBE correct, there's some weird blurring going on in some reflections, and the lights seem slightly off, but it is PROGRESS!!

I still haven't found out how to BLEND the existing normals with the map normals, as I can't seem to get access to them... and I haven't found a way to recreate them... though I may be missing something very obvious.

In any case maybe this will be useful for someone. I will continue polishing this, and if anyone can help it will be greatly appreciated.