Best Way to Blur a Depth Map

Hello,

I'm using a render texture to generate a depth map that displaces geometry. While the current result is passable, I think I can do better by blurring the depth map. The issue I'm having is that I just don't know how to go about blurring it. I've looked into Gaussian and Poisson Blurs, but the examples I've found for them are hard to understand and difficult to integrate into my existing surface shader.

Does anyone have any thoughts, tips, or ideas?

Thanks.

 

You want to do the blur to the render texture as blits, not as part of the surface texture. I mean you could do a basic poisson disc blur in the surface shader, but it’ll be far faster and much higher quality to do a separable Gaussian in two passes, or maybe a Kawase blur.

 

Ah okay. So I'll want to do the blur as a C# script then.

Do you have any good resources on how to implement a Gaussian blur?

Also, my interest in doing a Poisson disc blur was from this GDC presentation which tackles the same visual effect I'm working on.

 

That's from the Arkham Origins talk, yes? For a small blur, a poisson blur is fine. It's all about trying to reduce the number of taps (texture samples) being done. A separable gaussian can do a much wider blur for a sqrt() number of taps vs poisson for the same quality, but requires two passes. Poisson can be done in one pass, and if you're only doing 4 taps there's not a lot of reason to go with multiple passes for that.

Note, the above slide is talking about doing the blur in the equivalent of a script side blit() as well.

Also note, "poisson" really just means random points with a minimum separation distance. 4 points in a square are technically a poisson distribution.

 

Yup!

Ah, okay, I'll keep all that in mind. Now to just figure out how to use blit()...

 

Okay so update on my progress: I managed to find a Gaussian blur shader online and am currently trying to use that code to work. In the C# script pipeline I'm using, I have blit() being called in Update(). I feed in my RenderTexture which is then sent to a temporary render texture. The temporary render texture goes through the Gaussian Blue shader and then finally blit is called again to rewrite the original RenderTexture with the blurred version of itself.

Unfortunately, this is not working and I don't know why. I've verified that the shader works by testing it out on the MainCamera and the game view is definitely blurred.

 

Further Update:

Using the frame debugger I was able to determine that the render texture is being blurred (yay!) but is then being overridden. I don't exactly know why. What I do know is that the blur is the first effect to be processed, and then is overridden by a blit call that I don't exactly know where it's coming from as I'm fairly certain I've kept track of all my blits.

Assistance or troubleshooting ideas would be greatly appreciated.

EDIT: Hmm... It appears that the render texture is somehow being swapped from ARGB32 to Depth.

 

If you're doing the Blit() calls in Update(), you're likely modifying the previous frame's render texture, which then gets overridden when the current frame actually updates the depth texture. Basically all rendering starts after most of the C# code runs:
https://docs.unity3d.com/Manual/ExecutionOrder.html

Unless you're manually generating the depth texture using a cam.Render() call or similar, it'll absolutely be overridden.

 

Oh wow. I had no idea those functions existed! Much appreciated. I've switched over to OnRenderObject and tried out the other Render ones, but the same problem persists.

Here's the problem I've identified:

The RenderTexture that gets blurred is in an ARGB32 format. When the final blit is called to send it back to the original RenderTexture it's sent back in a Depth format, which for some reason erases the blurring. When I try setting the temporary RenderTextures to the Depth format they don't seem to hold anything. And then the final blit somehow sends in the depth map.

I'm still messing around, but no luck so far.

Step 1: Downsize Resolution by half


Step 2: Horizontal Blur


Step 3: Vertical Blur


Step 4: Send back to original RenderTexture

 

OnRenderObject somewhat unintuitively runs after everything else has already rendered, so it's too late to be used by other objects in the scene.

OnRenderImage runs after a camera has rendered, but only if the script is attached to a camera, and not if you're using camera.Render(), only if it renders normally (ie: it's enabled).

I guess the real question is why you're copying it back to the original texture to begin with? Why not just blit to a new texture and use that in your material? Also, how are you creating your temporary render textures? I suspect you're not setting a format, or are using RenderTextureFormat.default. For this, you probably want to be using RenderTextureFormat.RHalf or some other single channel float format. I also almost never use the depth format, so I'm not sure if blit copies work with that format ... I would expect it to, but maybe it doesn't, especially if you're trying to copy from an RGBA32?

I'd try making them all use the Depth format and see if that works.

Really, without seeing your code I'm mostly just making wild guesses.

 

Can you share the code for generating the depth map? perhaps it will reveal a further issue.

 

I seriously appreciate the help bgolus, it means a lot.

Okay, so here's the code I'm working with:

Code (CSharp):

1. using UnityEngine;
2.  
3. [ExecuteInEditMode]
4. public class GaussianBlurFilter : MonoBehaviour
5. {
6.     enum DownSampleMode { Off, Half, Quarter }
7.  
8.     [SerializeField]
9.     Shader _shader;
10.  
11.     [SerializeField]
12.     RenderTexture source;
13.  
14.     [SerializeField]
15.     DownSampleMode _downSampleMode = DownSampleMode.Quarter;
16.  
17.     [SerializeField, Range(0, 8)]
18.     int _iteration = 4;
19.  
20.     Material _material;
21.  
22.     void OnRenderObject()
23.     {
24.         if (_material == null)
25.         {
26.             _material = new Material(_shader);
27.             _material.hideFlags = HideFlags.HideAndDontSave;
28.         }
29.  
30.         RenderTexture rt1, rt2;
31.  
32.         if (_downSampleMode == DownSampleMode.Half)
33.         {
34.             rt1 = RenderTexture.GetTemporary(source.width / 2, source.height / 2);
35.             rt2 = RenderTexture.GetTemporary(source.width / 2, source.height / 2);
36.             Graphics.Blit(source, rt1);
37.         }
38.         else if (_downSampleMode == DownSampleMode.Quarter)
39.         {
40.             rt1 = RenderTexture.GetTemporary(source.width / 4, source.height / 4);
41.             rt2 = RenderTexture.GetTemporary(source.width / 4, source.height / 4);
42.             Graphics.Blit(source, rt1, _material, 0);
43.         }
44.         else
45.         {
46.             rt1 = RenderTexture.GetTemporary(source.width, source.height);
47.             rt2 = RenderTexture.GetTemporary(source.width, source.height);
48.             Graphics.Blit(source, rt1);
49.         }
50.  
51.         for (var i = 0; i < _iteration; i++)
52.         {
53.             Graphics.Blit(rt1, rt2, _material, 1);
54.             Graphics.Blit(rt2, rt1, _material, 2);
55.         }
56.  
57.         Graphics.Blit(rt1, source);
58.  
59.         RenderTexture.ReleaseTemporary(rt1);
60.         RenderTexture.ReleaseTemporary(rt2);
61.     }
62.  
63.     private void OnGUI()
64.     {
65.         GUI.DrawTexture(new Rect(0, 0, 256, 256), source);
66.     }
67. }

I haven't touched anything since your last response, so obviously a few things are going to not be optimal.

 

Where is the source render texture coming from? How is that generated?

 

Currently it's just sitting in my texture folder. I drag and drop it into the inspector in the slot for "RenderTexture source" on line 12. However, I'm working on making it generated via a modified version of the script I posted above.

It's updated via a camera's Target Texture slot.

 

Well, first thing, try making all of those GetTemporary use the source texture's format.

RenderTexture.GetTemporary(source.width / 2, source.height / 2, 0, source.format);

Best cast that fixes everything. Worst case nothing works at all anymore.
I'm mostly curious how the original render texture is being filled.

 

Here's my updated code:

Code (CSharp):

1. using UnityEngine;
2.  
3. public class SnowRenderTexturePipeline : MonoBehaviour {
4.  
5.     enum DownSampleMode { Off, Half, Quarter }
6.  
7.     [SerializeField]
8.     Material snowMat;
9.  
10.     [SerializeField]
11.     Shader blurShader;
12.  
13.     [SerializeField]
14.     DownSampleMode downSampleMode = DownSampleMode.Quarter;
15.  
16.     [SerializeField, Range(0, 8)]
17.     int iteration = 4;
18.  
19.     Material blurMat;
20.     RenderTexture depthTexture;
21.     Camera cam;
22.  
23.     private void Start()
24.     {
25.         depthTexture = new RenderTexture(512, 512, 24, RenderTextureFormat.Depth);
26.         cam = GetComponent<Camera>();
27.         cam.targetTexture = depthTexture;
28.     }
29.  
30.     void OnRenderImage(RenderTexture source, RenderTexture destination)
31.     {
32.         if (blurMat == null)
33.         {
34.             blurMat = new Material(blurShader);
35.             blurMat.hideFlags = HideFlags.HideAndDontSave;
36.         }
37.  
38.         RenderTexture rt1, rt2;
39.  
40.         if (downSampleMode == DownSampleMode.Half)
41.         {
42.             rt1 = RenderTexture.GetTemporary(depthTexture.width / 2, depthTexture.height / 2, 24, RenderTextureFormat.Depth);
43.             rt2 = RenderTexture.GetTemporary(depthTexture.width / 2, depthTexture.height / 2, 24, RenderTextureFormat.Depth);
44.             Graphics.Blit(depthTexture, rt1);
45.         }
46.         else if (downSampleMode == DownSampleMode.Quarter)
47.         {
48.             rt1 = RenderTexture.GetTemporary(depthTexture.width / 4, depthTexture.height / 4, 24, RenderTextureFormat.Depth);
49.             rt2 = RenderTexture.GetTemporary(depthTexture.width / 4, depthTexture.height / 4, 24, RenderTextureFormat.Depth);
50.             Graphics.Blit(depthTexture, rt1, blurMat, 0);
51.         }
52.         else
53.         {
54.             rt1 = RenderTexture.GetTemporary(depthTexture.width, depthTexture.height, 24, RenderTextureFormat.Depth);
55.             rt2 = RenderTexture.GetTemporary(depthTexture.width, depthTexture.height, 24, RenderTextureFormat.Depth);
56.             Graphics.Blit(depthTexture, rt1);
57.         }
58.  
59.         for (var i = 0; i < iteration; i++)
60.         {
61.             Graphics.Blit(rt1, rt2, blurMat, 1);
62.             Graphics.Blit(rt2, rt1, blurMat, 2);
63.         }
64.  
65.         RenderTexture blurredDepthMap = new RenderTexture(depthTexture.width, depthTexture.height, 24, RenderTextureFormat.Depth);
66.         Graphics.Blit(rt1, blurredDepthMap);
67.         snowMat.SetTexture("_DispTex", blurredDepthMap);
68.  
69.         RenderTexture.ReleaseTemporary(rt1);
70.         RenderTexture.ReleaseTemporary(rt2);
71.     }
72.  
73.     private void OnGUI()
74.     {
75.         GUI.DrawTexture(new Rect(0, 0, 256, 256), depthTexture);
76.     }
77. }

Hmm.... So it appears nothing works anymore. The only thing that still works is the GUI.DrawTexture() call. The depth Texture shows up, but there's no displacement happening anymore.



Here's the inspector for the camera in question in case you wanted to see that:

 

Okay, yeah, that confirms my expectations there. Use RenderTextureFormat.RHalf instead. Also set your snow camera's depth to -1 or lower, you want to make sure it's less than your main camera's depth.

Other things, do not create a render texture every OnRenderImage. Create one during start and reuse it. make it the same resolution as the downscaled buffer, there's no point in bliting from a low resolution to a higher resolution.

 

Alright, I'll change those things and then report back my findings.

UPDATE: Alright, I'm still getting the same results.

Hmm... I did find this piece of info online. Not quite sure if it relates to what I'm doing though.

https://support.unity3d.com/hc/en-u...aphics-Blit-does-not-copy-RenderTexture-depth

 

Well, I implemented the DepthCopy shader into my pipeline script in the most basic way possible and was able to replicate my existing functionality for the snow shader, so I think it's definitely related. Going to look into modifying the gaussian blur shader copy the depth in the same way.

UPDATE: Getting somewhere...

Got through the full script. No blur applied however. Hmm...

 

I'm getting closer.

Okay, so I've managed to get the half and quarter size working, but I run into a problem. For some reason the resolution change and rescale is being wiped out by a Draw Dynamic event.



Here's the info for the Draw Dynamic event in question:


It's so bizarre. At this point I'm not using any ordinary blit calls anymore, and I can't identify what's causing it.

UPDATE: Scratch that, it actually is working. Had to find a work around to doing an GUI.DrawTexture. Now to figure out how to apply the blur...

UPDATE 2: No luck on applying the blur. Something seems to be wrong with the shader I think. Inputting custom values for the fragment shader doesn't do anything.

Code (CSharp):

1. Shader "Hidden/Gaussian Blur Filter"
2. {
3.     Properties
4.     {
5.         _MainTex("-", 2D) = "white" {}
6.         _MyDepthTex("-", 2D) = "white" {}
7.     }
8.  
9.         CGINCLUDE
10.  
11. #include "UnityCG.cginc"
12.  
13.     sampler2D _MainTex;
14.     sampler2D_float _MyDepthTex;
15.     float4 _MyDepthTex_TexelSize;
16.  
17.     struct appdata
18.     {
19.         float4 vertex : POSITION;
20.         float2 uv : TEXCOORD0;
21.     };
22.  
23.     struct v2f
24.     {
25.         float2 uv : TEXCOORD0;
26.         float4 vertex : SV_POSITION;
27.     };
28.  
29.     v2f vert(appdata v)
30.     {
31.         v2f o;
32.         o.vertex = UnityObjectToClipPos(v.vertex);
33.         o.uv = v.uv;
34.         return o;
35.     }
36.  
37.     // 9-tap Gaussian filter with linear sampling
38.     // http://rastergrid.com/blog/2010/09/efficient-gaussian-blur-with-linear-sampling/
39.     half4 gaussian_filter(float2 uv, float2 stride)
40.     {
41.         half4 s = tex2Dlod(_MyDepthTex, float4(uv, 0, 0)) * 0.227027027;
42.  
43.         float2 d1 = stride * 1.3846153846;
44.         s += tex2Dlod(_MyDepthTex, float4(uv + d1, 0, 0)) * 0.3162162162;
45.         s += tex2Dlod(_MyDepthTex, float4(uv - d1, 0, 0)) * 0.3162162162;
46.  
47.         float2 d2 = stride * 3.2307692308;
48.         s += tex2Dlod(_MyDepthTex, float4(uv + d2, 0, 0)) * 0.0702702703;
49.         s += tex2Dlod(_MyDepthTex, float4(uv - d2, 0, 0)) * 0.0702702703;
50.  
51.         return s;
52.     }
53.  
54.     // Quarter downsampler
55.     half4 frag_quarter(v2f i, out float outDepth : SV_Depth) : SV_Target
56.     {
57.         float depth = SAMPLE_DEPTH_TEXTURE(_MyDepthTex, i.uv);
58.         outDepth = depth;
59.  
60.         float4 d = _MyDepthTex_TexelSize.xyxy * float4(1, 1, -1, -1);
61.         half4 s;
62.         s = tex2D(_MyDepthTex, i.uv + d.xy);
63.         s += tex2D(_MyDepthTex, i.uv + d.xw);
64.         s += tex2D(_MyDepthTex, i.uv + d.zy);
65.         s += tex2D(_MyDepthTex, i.uv + d.zw);
66.         return s * 0.25;
67.     }
68.  
69.     // Separable Gaussian filters
70.     half4 frag_blur_h(v2f i, out float outDepth : SV_Depth) : SV_Target
71.     {
72.         float depth = SAMPLE_DEPTH_TEXTURE(_MyDepthTex, i.uv);
73.         outDepth = depth;
74.  
75.         return gaussian_filter(i.uv, float2(_MyDepthTex_TexelSize.x, 0));
76.     }
77.  
78.     half4 frag_blur_v(v2f i, out float outDepth : SV_Depth) : SV_Target
79.     {
80.         float depth = SAMPLE_DEPTH_TEXTURE(_MyDepthTex, i.uv);
81.         outDepth = depth;
82.  
83.         return gaussian_filter(i.uv, float2(0, _MyDepthTex_TexelSize.y));
84.     }
85.  
86.         ENDCG
87.  
88.         Subshader
89.     {
90.         Pass
91.         {
92.             ZTest Always Cull Off ZWrite On
93.             CGPROGRAM
94.             #pragma vertex vert
95.             #pragma fragment frag_quarter
96.             ENDCG
97.         }
98.             Pass
99.         {
100.             ZTest Always Cull Off ZWrite On
101.             CGPROGRAM
102.             #pragma vertex vert
103.             #pragma fragment frag_blur_h
104.             #pragma target 3.0
105.             ENDCG
106.         }
107.             Pass
108.         {
109.             ZTest Always Cull Off ZWrite On
110.             CGPROGRAM
111.             #pragma vertex vert
112.             #pragma fragment frag_blur_v
113.             #pragma target 3.0
114.             ENDCG
115.         }
116.     }
117. }

 

SOLVED IT! YES!!

Okay, so in the end it was two key things that led to the solution.

1. blit() does not traditionally copy the depth value over to a new RenderTexture, so blit() needed to be modified to do that and the C# script had to be updated accordingly.

2. The fragment shader only returns values of color - no depth information at all! Instead, the fragment shader had to be modified to specifically blur the depth value, ignoring the ARGB values completely.

Here's the final Gaussian blur shader:

Code (CSharp):

1. Shader "Hidden/Gaussian Blur Filter"
2. {
3.     Properties
4.     {
5.         _MainTex("-", 2D) = "white" {}
6.         _MyDepthTex("-", 2D) = "white" {}
7.     }
8.  
9.         CGINCLUDE
10.  
11. #include "UnityCG.cginc"
12.  
13.     sampler2D _MainTex;
14.     sampler2D_float _MyDepthTex;
15.     float4 _MyDepthTex_TexelSize;
16.  
17.     // 9-tap Gaussian filter with linear sampling
18.     // http://rastergrid.com/blog/2010/09/efficient-gaussian-blur-with-linear-sampling/
19.     half gaussian_filter(float2 uv, float2 stride)
20.     {
21.         half s = tex2D(_MyDepthTex, float4(uv, 0, 0)).r * 0.227027027;
22.  
23.         float2 d1 = stride * 1.3846153846;
24.         s += tex2D(_MyDepthTex, uv + d1).r * 0.3162162162;
25.         s += tex2D(_MyDepthTex, uv - d1).r * 0.3162162162;
26.  
27.         float2 d2 = stride * 3.2307692308;
28.         s += tex2D(_MyDepthTex, uv + d2).r * 0.0702702703;
29.         s += tex2D(_MyDepthTex, uv - d2).r * 0.0702702703;
30.  
31.         return s;
32.     }
33.  
34.     // Quarter downsampler
35.     half4 frag_quarter(v2f_img i, out float outDepth : SV_Depth) : SV_Target
36.     {
37.         float depth = SAMPLE_DEPTH_TEXTURE(_MyDepthTex, i.uv);
38.         outDepth = depth;
39.  
40.         float4 d = _MyDepthTex_TexelSize.xyxy * float4(1, 1, -1, -1);
41.         half4 s;
42.         s = tex2D(_MyDepthTex, i.uv + d.xy);
43.         s += tex2D(_MyDepthTex, i.uv + d.xw);
44.         s += tex2D(_MyDepthTex, i.uv + d.zy);
45.         s += tex2D(_MyDepthTex, i.uv + d.zw);
46.         return s * 0.25;
47.     }
48.  
49.     // Separable Gaussian filters
50.     half4 frag_blur_h(v2f_img i, out float outDepth : SV_Depth) : SV_Target
51.     {
52.         outDepth = gaussian_filter(i.uv, float2(_MyDepthTex_TexelSize.x, 0));
53.  
54.         return 0;
55.     }
56.  
57.     half4 frag_blur_v(v2f_img i, out float outDepth : SV_Depth) : SV_Target
58.     {
59.         outDepth = gaussian_filter(i.uv, float2(0, _MyDepthTex_TexelSize.y));
60.  
61.         return 0;
62.     }
63.  
64.         ENDCG
65.  
66.         Subshader
67.     {
68.         Pass
69.         {
70.             ZTest Always Cull Off ZWrite On
71.             CGPROGRAM
72.             #pragma vertex vert_img
73.             #pragma fragment frag_quarter
74.             ENDCG
75.         }
76.             Pass
77.         {
78.             ZTest Always Cull Off ZWrite On
79.             CGPROGRAM
80.             #pragma vertex vert_img
81.             #pragma fragment frag_blur_h
82.             #pragma target 3.0
83.             ENDCG
84.         }
85.             Pass
86.         {
87.             ZTest Always Cull Off ZWrite On
88.             CGPROGRAM
89.             #pragma vertex vert_img
90.             #pragma fragment frag_blur_v
91.             #pragma target 3.0
92.             ENDCG
93.         }
94.     }
95. }

 

Thanks for being awesome enough to share the journey! Most people don't post when they've resolved it

 

Yeah no problem! The results are just what I wanted too!

No blur applied:


Blur applied:


Also, special shout out to bgolus for being generally awesome and helping me troubleshoot!

 

Yep @bgolus is a gem on these forums. He's basically erased all the guesswork and confusion that was here before.