[Solved] Outline custom pass example, how to exclude occluded objects?

I've been trying to recreate my former highlight fx system in the HDRP, and used outline custom example from the manual.

However, outline are drawn on top of everything. Is there a way to draw it only if the object is visible?

 

@antoinel_unity I've tried this example https://github.com/alelievr/HDRP-Cu...ection/Shaders/02_Selection_Fullscreen.shader and I can't make inner / outer color to hide correctly behind objects;

It is still visible behind other objects even with _BehindFactor 0;
Is depth actually broken?

This is what I've tried:

Code (CSharp):

1. float4 FullScreenPass(Varyings varyings) : SV_Target
2.     {
3.         float depth = LoadCameraDepth(varyings.positionCS.xy);
4.         PositionInputs posInput = GetPositionInput(varyings.positionCS.xy, _ScreenSize.zw, depth, UNITY_MATRIX_I_VP, UNITY_MATRIX_V);
5.         float3 viewDirection = GetWorldSpaceNormalizeViewDir(posInput.positionWS);
6.      
7.         float d = LoadCustomDepth(posInput.positionSS);
8.         float db = LoadCameraDepth(posInput.positionSS);
9.      
10.         float4 c = LoadCustomColor(posInput.positionSS);
11.  
12.         float obj = c.a;
13.      
14.         uint offset = 5;
15.      
16.         int sampleCount = min( 2 * pow(2, _SamplePrecision ), MAXSAMPLES ) ;
17.      
18.         float4 outline = float4(0,0,0,0);
19.      
20.         float2 uvOffsetPerPixel = 1.0/_ScreenSize .xy;
21.      
22.         for (uint i=0 ; i<sampleCount ; ++i )
23.         {
24.             outline =  max( SampleCustomColor( posInput.positionNDC + uvOffsetPerPixel * _OutlineWidth * offsets[i] ), outline );
25.         }
26.  
27.         float4 o = float4(0,0,0,0);
28.      
29.         float alphaFactor = (db > d) ? _BehindFactor : 1;
30.      
31.         float4 innerColor = SAMPLE_TEXTURE2D(_Texture, s_trilinear_repeat_sampler, posInput.positionSS / _TextureSize) * _InnerColor;
32.         innerColor.a *= alphaFactor;
33.      
34.         float4 outerColor = _OuterColor * float4(outline.rgb, 1);
35.         outerColor.a *= alphaFactor;
36.         outline.a *= alphaFactor;
37.      
38.         o = lerp(o, outerColor, outline.a);
39.         o = lerp(o, innerColor * float4(c.rgb, 1), obj);
40.      
41.         return o;
42.     }

 

This is my whole setup:


Draw renderers pass:

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.Experimental.Rendering;
3. using UnityEngine.Rendering;
4. using UnityEngine.Rendering.HighDefinition;
5.  
6. namespace FX.Highlight {
7.    public class HighlightPassDrawRenderers : CustomPass {
8.       //Filter settings
9.       public LayerMask LayerMask = 1; // Layer mask Default enabled
10.  
11.       [SerializeField]
12.       private float _maxDistance = 15f;
13.  
14.       [SerializeField]
15.       private float _lerpMaxDistance = 15f;
16.  
17.       [SerializeField]
18.       private Shader _shader;
19.  
20.       #region [Fields]
21.  
22.       private static ShaderTagId[] _forwardShaderTags;
23.  
24.       private static readonly int MaxDist = Shader.PropertyToID("_MaxDistance");
25.       private static readonly int LerpMaxDist = Shader.PropertyToID("_LerpMaxDistance");
26.  
27.       // Cache the shaderTagIds so we don't allocate a new array each frame
28.       private ShaderTagId[] _cachedShaderTagIDs;
29.  
30.       private Material _material;
31.       private int _pass;
32.  
33.       #endregion
34.  
35.       protected override void Setup(ScriptableRenderContext renderContext, CommandBuffer cmd) {
36.          _shader = Shader.Find("HDRP/PostFX/HighlightPassDrawRenderers");
37.          _material = CoreUtils.CreateEngineMaterial(_shader);
38.  
39.          _pass = _material.FindPass("FirstPass");
40.      
41.          _forwardShaderTags = new[]
42.                               {
43.                                  new ShaderTagId("Forward"), // HD Lit shader
44.                                  new ShaderTagId("ForwardOnly"), // HD Unlit shader
45.                                  new ShaderTagId("SRPDefaultUnlit"), // Cross SRP Unlit shader
46.                                  new ShaderTagId(""), // Add an empty slot for the override material
47.                               };
48.       }
49.  
50.       protected override void AggregateCullingParameters(ref ScriptableCullingParameters cullingParameters,
51.                                                          HDCamera hdCamera) {
52.          cullingParameters.cullingMask |= (uint) (int) LayerMask;
53.       }
54.  
55.       /// <summary>
56.       /// Execute the DrawRenderers with parameters setup from the editor
57.       /// </summary>
58.       protected override void Execute(ScriptableRenderContext renderContext,
59.                                       CommandBuffer cmd,
60.                                       HDCamera hdCamera,
61.                                       CullingResults cullingResult) {
62.          SetRenderTargetAuto(cmd);
63.      
64.          var stateBlock = new RenderStateBlock();
65.  
66.          PerObjectData renderConfig = hdCamera.frameSettings.IsEnabled(FrameSettingsField.Shadowmask)
67.                                          ? HDUtils.k_RendererConfigurationBakedLightingWithShadowMask
68.                                          : HDUtils.k_RendererConfigurationBakedLighting;
69.  
70.          var result = new RendererListDesc(_forwardShaderTags, cullingResult, hdCamera.camera)
71.                       {
72.                          rendererConfiguration = renderConfig,
73.                          renderQueueRange = RenderQueueRange.all,
74.                          sortingCriteria = SortingCriteria.BackToFront,
75.                          excludeObjectMotionVectors = false,
76.                          overrideMaterial = _material,
77.                          overrideMaterialPassIndex = _pass,
78.                          stateBlock = stateBlock,
79.                          layerMask = LayerMask,
80.                       };
81.  
82.          _material.SetFloat(MaxDist, _maxDistance);
83.          _material.SetFloat(LerpMaxDist, _lerpMaxDistance);
84.      
85.          HDUtils.DrawRendererList(renderContext, cmd, RendererList.Create(result));
86.       }
87.  
88.       protected override void Cleanup() {
89.          base.Cleanup();
90.      
91.          CoreUtils.Destroy(_material);
92.       }
93.    }
94. }

Draw renderers shader:

Code (CSharp):

1. Shader "HDRP/PostFX/HighlightPassDrawRenderers"
2. {
3.     Properties
4.     {
5.         _MaxDistance("Max Distance", float) = 15
6.         _LerpMaxDistance("Lerp Max Distance", float) = 20
7.     }
8.  
9.     HLSLINCLUDE
10.  
11.     #pragma target 4.5
12.     #pragma only_renderers d3d11 ps4 xboxone vulkan metal switch
13.  
14.     // #pragma enable_d3d11_debug_symbols
15.  
16.     //enable GPU instancing support
17.     #pragma multi_compile_instancing
18.     #pragma instancing_options renderinglayer
19.  
20.     ENDHLSL
21.  
22.     SubShader
23.     {
24.         Pass
25.         {
26.             Name "FirstPass"
27.             Tags { "LightMode" = "FirstPass" }
28.  
29.             Blend Off
30.             ZWrite On
31.             ZTest LEqual
32.  
33.             Cull Back
34.  
35.             HLSLPROGRAM
36.  
37.             // Toggle the alpha test
38.             #define _ALPHATEST_ON
39.  
40.             // Toggle transparency
41.             // #define _SURFACE_TYPE_TRANSPARENT
42.  
43.             // Toggle fog on transparent
44.             #define _ENABLE_FOG_ON_TRANSPARENT
45.        
46.             // List all the attributes needed in your shader (will be passed to the vertex shader)
47.             // you can see the complete list of these attributes in VaryingMesh.hlsl
48.             #define ATTRIBUTES_NEED_TEXCOORD0
49.             #define ATTRIBUTES_NEED_NORMAL
50.             #define ATTRIBUTES_NEED_TANGENT
51.  
52.             // List all the varyings needed in your fragment shader
53.             #define VARYINGS_NEED_TEXCOORD0
54.             #define VARYINGS_NEED_TANGENT_TO_WORLD
55.             #define VARYINGS_NEED_POSITION_WS
56.        
57.             #include "Packages/com.unity.render-pipelines.high-definition/Runtime/RenderPipeline/RenderPass/CustomPass/CustomPassRenderers.hlsl"
58.        
59.             float _MaxDistance;
60.             float _LerpMaxDistance;
61.        
62.             float invLerp(float A, float B, float T)
63.             {
64.                 return (T - A) / (B - A);
65.             }
66.  
67.             void GetSurfaceAndBuiltinData(FragInputs fragInputs, float3 viewDirection, inout PositionInputs posInput, out SurfaceData surfaceData, out BuiltinData builtinData)
68.             {
69.                 float dist = length(fragInputs.positionRWS);
70.        
71.                 if (dist > _LerpMaxDistance) {
72.                     discard;
73.                 }
74.            
75.                 float percentage = invLerp(_MaxDistance, _LerpMaxDistance, dist);
76.                 percentage = clamp (percentage, 0, 1);
77.            
78.                 // Write back the data to the output structures
79.                 ZERO_INITIALIZE(BuiltinData, builtinData); // No call to InitBuiltinData as we don't have any lighting
80.                 builtinData.opacity = lerp(1, 0, percentage);
81.                 builtinData.emissiveColor = float3(0, 0, 0);
82.                 surfaceData.color = float3(1,1,1);
83.             }
84.  
85.             #include "Packages/com.unity.render-pipelines.high-definition/Runtime/RenderPipeline/ShaderPass/ShaderPassForwardUnlit.hlsl"
86.  
87.             #pragma vertex Vert
88.             #pragma fragment Frag
89.  
90.             ENDHLSL
91.         }
92.     }
93. }

Fullscreen pass:

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.Rendering;
3. using UnityEngine.Rendering.HighDefinition;
4.  
5. namespace FX.Highlight {
6.    /// <summary>
7.    /// FullScreen Custom Pass
8.    /// </summary>
9.    [System.Serializable]
10.    public class HighlightPassFullscreen : CustomPass {
11.       [SerializeField, Range(1, 3)]
12.       private float _samplePrecision = 1;
13.  
14.       [SerializeField]
15.       private float _outlineWidth = 5;
16.  
17.       [SerializeField]
18.       private Color _innerColor = new Color(1, 1, 0, 0.5f);
19.  
20.       [SerializeField]
21.       private Color _outerColor = new Color(1, 1, 0, 0.5f);
22.  
23.       [SerializeField]
24.       private Texture _texture = default;
25.  
26.       [SerializeField]
27.       private Vector2 _texturePixelSize = new Vector2(64,64);
28.  
29.       [SerializeField, Range(0, 1)]
30.       private float _behindFactor = 0.2f;
31.  
32.       [SerializeField]
33.       private Shader _shader;
34.  
35.       #region [Fields]
36.  
37.       private int _fadeValueId;
38.       private Material _material;
39.  
40.       private int _pass;
41.  
42.       private static readonly int SamplePrecision = Shader.PropertyToID("_SamplePrecision");
43.       private static readonly int OutlineWidth = Shader.PropertyToID("_OutlineWidth");
44.       private static readonly int InnerColor = Shader.PropertyToID("_InnerColor");
45.       private static readonly int OuterColor = Shader.PropertyToID("_OuterColor");
46.       private static readonly int Texture = Shader.PropertyToID("_Texture");
47.       private static readonly int TextureSize = Shader.PropertyToID("_TextureSize");
48.       private static readonly int BehindFactor = Shader.PropertyToID("_BehindFactor");
49.  
50.       #endregion
51.  
52.       protected override void Setup(ScriptableRenderContext renderContext, CommandBuffer cmd) {
53.          _shader = Shader.Find("HDRP/PostFX/HighlightPassFullscreen");
54.          _material = CoreUtils.CreateEngineMaterial(_shader);
55.  
56.          _fadeValueId = Shader.PropertyToID("_FadeValue");
57.  
58.          _pass = _material.FindPass("FirstPass");
59.       }
60.  
61.       /// <summary>
62.       /// Execute the pass with the fullscreen setup
63.       /// </summary>
64.       protected override void Execute(ScriptableRenderContext renderContext,
65.                                       CommandBuffer cmd,
66.                                       HDCamera hdCamera,
67.                                       CullingResults cullingResult) {
68.          ResolveMSAAColorBuffer(cmd, hdCamera);
69.          SetRenderTargetAuto(cmd);
70.  
71.          _material.SetFloat(_fadeValueId, fadeValue);
72.          _material.SetFloat(SamplePrecision, _samplePrecision);
73.          _material.SetFloat(OutlineWidth, _outlineWidth);
74.          _material.SetColor(InnerColor, _innerColor);
75.          _material.SetColor(OuterColor, _outerColor);
76.          _material.SetTexture(Texture, _texture);
77.          _material.SetVector(TextureSize, _texturePixelSize);
78.          _material.SetFloat(BehindFactor, _behindFactor);
79.      
80.          CoreUtils.DrawFullScreen(cmd, _material, _pass);
81.       }
82.    }
83. }

Fullscreen shader:

Code (CSharp):

1. Shader "HDRP/PostFX/HighlightPassFullscreen"
2. {
3.     properties
4.     {
5.         _SamplePrecision ("Sampling Precision", Range(1,3) ) = 1
6.         _OutlineWidth ("Outline Width", Float ) = 5
7.    
8.         _InnerColor ("Inner Color", Color) = (1, 1, 0, 0.5)
9.         _OuterColor( "Outer Color", Color ) = (1, 1, 0, 1)
10.         _Texture ("Texture", 2D ) = "black" {}
11.         _TextureSize("Texture Pixels Size", Vector) = (64,64,0,0)
12.    
13.         _BehindFactor("Behind Factor", Range(0,1)) = 0.2
14.     }
15.  
16.     HLSLINCLUDE
17.  
18.     #pragma vertex Vert
19.  
20.     #pragma target 4.5
21.     #pragma only_renderers d3d11 ps4 xboxone vulkan metal switch
22.  
23.     #include "Packages/com.unity.render-pipelines.high-definition/Runtime/RenderPipeline/RenderPass/CustomPass/CustomPassCommon.hlsl"
24.  
25.     // The PositionInputs struct allow you to retrieve a lot of useful information for your fullScreenShader:
26.     // struct PositionInputs
27.     // {
28.     //     float3 positionWS;  // World space position (could be camera-relative)
29.     //     float2 positionNDC; // Normalized screen coordinates within the viewport    : [0, 1) (with the half-pixel offset)
30.     //     uint2  positionSS;  // Screen space pixel coordinates                       : [0, NumPixels)
31.     //     uint2  tileCoord;   // Screen tile coordinates                              : [0, NumTiles)
32.     //     float  deviceDepth; // Depth from the depth buffer                          : [0, 1] (typically reversed)
33.     //     float  linearDepth; // View space Z coordinate                              : [Near, Far]
34.     // };
35.  
36.     // To sample custom buffers, you have access to these functions:
37.     // But be careful, on most platforms you can't sample to the bound color buffer. It means that you
38.     // can't use the SampleCustomColor when the pass color buffer is set to custom (and same for camera the buffer).
39.     // float3 SampleCustomColor(float2 uv);
40.     // float3 LoadCustomColor(uint2 pixelCoords);
41.     // float LoadCustomDepth(uint2 pixelCoords);
42.     // float SampleCustomDepth(float2 uv);
43.  
44.     // There are also a lot of utility function you can use inside Common.hlsl and Color.hlsl,
45.     // you can check them out in the source code of the core SRP package.
46.  
47.     #define v2 1.41421
48.     #define c45 0.707107
49.     #define c225 0.9238795
50.     #define s225 0.3826834
51.  
52.     #define MAXSAMPLES 16
53.     static float2 offsets[MAXSAMPLES] = {
54.         float2( 1, 0 ),
55.         float2( -1, 0 ),
56.         float2( 0, 1 ),
57.         float2( 0, -1 ),
58.    
59.         float2( c45, c45 ),
60.         float2( c45, -c45 ),
61.         float2( -c45, c45 ),
62.         float2( -c45, -c45 ),
63.    
64.         float2( c225, s225 ),
65.         float2( c225, -s225 ),
66.         float2( -c225, s225 ),
67.         float2( -c225, -s225 ),
68.         float2( s225, c225 ),
69.         float2( s225, -c225 ),
70.         float2( -s225, c225 ),
71.         float2( -s225, -c225 )
72.     };
73.  
74.     int _SamplePrecision;
75.     float _OutlineWidth;
76.  
77.     float4 _InnerColor;
78.     float4 _OuterColor;
79.  
80.     Texture2D _Texture;
81.     float2 _TextureSize;
82.  
83.     float _BehindFactor;
84.  
85.     float4 FullScreenPass(Varyings varyings) : SV_Target
86.     {
87.         float depth = LoadCameraDepth(varyings.positionCS.xy);
88.         PositionInputs posInput = GetPositionInput(varyings.positionCS.xy, _ScreenSize.zw, depth, UNITY_MATRIX_I_VP, UNITY_MATRIX_V);
89.         float3 viewDirection = GetWorldSpaceNormalizeViewDir(posInput.positionWS);
90.    
91.         float d = LoadCustomDepth(posInput.positionSS);
92.         float db = LoadCameraDepth(posInput.positionSS);
93.    
94.         float4 c = LoadCustomColor(posInput.positionSS);
95.  
96.         float obj = c.a;
97.    
98.         uint offset = 5;
99.    
100.         int sampleCount = min( 2 * pow(2, _SamplePrecision ), MAXSAMPLES ) ;
101.    
102.         float4 outline = float4(0,0,0,0);
103.    
104.         float2 uvOffsetPerPixel = 1.0/_ScreenSize .xy;
105.    
106.         for (uint i=0 ; i<sampleCount ; ++i )
107.         {
108.             outline =  max( SampleCustomColor( posInput.positionNDC + uvOffsetPerPixel * _OutlineWidth * offsets[i] ), outline );
109.         }
110.  
111.         float4 o = float4(0,0,0,0);
112.    
113.         float alphaFactor = (db > d) ? _BehindFactor : 1;
114.    
115.         float4 innerColor = SAMPLE_TEXTURE2D(_Texture, s_trilinear_repeat_sampler, posInput.positionSS / _TextureSize) * _InnerColor;
116.         innerColor.a *= alphaFactor;
117.    
118.         float4 outerColor = _OuterColor * float4(outline.rgb, 1);
119.         outerColor.a *= alphaFactor;
120.         outline.a *= alphaFactor;
121.    
122.         o = lerp(o, outerColor, outline.a);
123.         o = lerp(o, innerColor * float4(c.rgb, 1), obj);
124.    
125.         return o;
126.     }
127.  
128.     ENDHLSL
129.  
130.     SubShader
131.     {
132.         Pass
133.         {
134.             Name "FirstPass"
135.  
136.             ZWrite Off
137.             ZTest Always
138.             Blend SrcAlpha OneMinusSrcAlpha
139.             Cull Off
140.  
141.             HLSLPROGRAM
142.                 #pragma fragment FullScreenPass
143.             ENDHLSL
144.         }
145.     }
146.     Fallback Off
147. }

Well, at least I've got distance working now.

Depth check (_BehindFactor) doesn't work correctly if objects are behind anything.
No idea why.

 

Hmmm... I've tried these settings. It works weirdly in the scene view (hidden both in front and behind objects). But in game, it still doesn't work (layer object's still visible behind objects).

I've also tried adding depth comparison to the RenderStateBlock like so (with both writeEnabled false and true), but it changes nothing:

Code (CSharp):

1.          RenderStateBlock stateBlock = new RenderStateBlock
2.                                        {
3.                                           depthState = new DepthState
4.                                                        {
5.                                                           compareFunction = CompareFunction.LessEqual,
6.                                                           writeEnabled = true
7.                                                        }
8.                                        };
9.  
10.          PerObjectData renderConfig = hdCamera.frameSettings.IsEnabled(FrameSettingsField.Shadowmask)
11.                                          ? HDUtils.k_RendererConfigurationBakedLightingWithShadowMask
12.                                          : HDUtils.k_RendererConfigurationBakedLighting;
13.  
14.          var result = new RendererListDesc(_forwardShaderTags, cullingResult, hdCamera.camera)
15.                       {
16.                          rendererConfiguration = renderConfig,
17.                          renderQueueRange = RenderQueueRange.all,
18.                          sortingCriteria = SortingCriteria.BackToFront,
19.                          excludeObjectMotionVectors = false,
20.                          overrideMaterial = _material,
21.                          overrideMaterialPassIndex = _pass,
22.                          stateBlock = stateBlock,
23.                          layerMask = LayerMask,
24.                       };

 

Hmmm that somewhat works, but it gives rather inconsistent results when camera is actually moving.

I've set draw renderers target depth buffer to the custom and applied change to the shader. And clear flags for the DrawRenderers is set to Color;


Edit: Alternatively, if you got it working correctly, please upload unitypackage or src somewhere.
I've probably messed up my setup somehow. But at this point idk where.

 

I've tried this one, brought it as is to the scene.
And it still renders outline on top of everything, even with behind factor 0.



Interesting thing, that it seems to work correctly with your scene.
If I bring those outlined objects into it, they aren't rendered behind other objects.

 

OMFG. I got it. Its the double camera setup!

I have a separate world space UI camera, which renders on top.
Which probably caused depth to be used from the UI camera instead. Which always empty.

I can't believe I've missed that.

Man, I wish there somewhere target camera being displayed for the pass.
That would've saved me a couple of days

The question is now, how to make sure custom pass uses specific camera?
Edit: Okay, found it.
There's an override in the Custom Frame Settings in the Camera settings that allows to disable Custom Passes.

Thank you for your support, its really awesome