Specular convolution when calculating mip maps for cubemap Render Texture?

I am rendering my own realtime reflection cubemaps by first rendering each face to an intermediate "face" RT, copying all faces to the cubemap RT and then generating mip maps.
This works fine, except the cubemap looks noticeably "mip-mapped" when sampled as a reflection on a rough surface. I know Unity uses a special method ("specular convolution") for generating mip maps for cubemaps when importing them in the inspector, and so my question is: Is this method available through scripting somewhere? I cannot seem to find any info on it.

 

There's no easy to use function for this, but the shader they use to do the convolution is available. It's called "Hidden/CubeBlur" and you can look at it in the built in shader source files in the file CubeBlur.shader.

 

That does look promising. Any idea of how to use it? I tried this but any mip level above 1 is completely black.

Code (CSharp):

1. var convolutionMaterial = new Material(Shader.Find("Hidden/CubeBlur"));
2.  
3. var rt = RenderTexture.GetTemporary(cubeMap.descriptor);
4. rt.useMipMap = false;
5. Graphics.Blit(cubeMap, rt);
6.  
7. int mipCount = 1 + Mathf.FloorToInt(Mathf.Log10(cubeMap.width) / Mathf.Log10(2f));
8.  
9. for (int mip = 0; mip < mipCount; mip++)
10. {
11.     int mipRes = cubeMap.width / (mip + 1);
12.     convolutionMaterial.SetFloat("_Texel", 1f / (float)mipRes);
13.     convolutionMaterial.SetFloat("_Level", mip);
14.     Graphics.SetRenderTarget(cubeMap, mip);
15.     Graphics.Blit(rt, convolutionMaterial, 0);
16. }

 

Unfortunately, no.

Here's some stuff I can guess though.

Blit() uses a screen aligned ortho quad, so likely isn't going to be useful for rendering a cubemap. You'll likely need to setup geometry manually and use GL commands to build out a cube rather than Blit()
https://docs.unity3d.com/ScriptReference/GL.html.

Alternatively you may need to render to one face & mip at a time. Certainly that seems to be how Unity does it since it's an option for time slicing on reflection probes.

 

Your post put me in the right direction and I'm having some luck making this work, but the result is off compared to the default convolution:



This seems to be due to the way the mip map of the source cubemap is sampled. When convoluting like this, are you maybe supposed to use the result of the previous mip level when calculating the current one? Is that even possible to do in Unity? From my experience it is not possible to read from one mip level while simultaneously writing to another (although I think this is supported by the graphics driver).

 

Yeah, I would assume they build off of each other.

And no, I don't believe you can render to a mip map of a texture to you're reading from as there's no guarantee you're not going to sample from that same mip in the shader.

You should either have two cubemap render textures and ping / pong between them, then use CopyTexture() to combine them into one. Or if you're rendering one face at a time have a 2d render texture you're rendering too and use CopyTexture to copy back to that face's mip.

 

You were right on the money with the guess about needing to construct a cube instead of using Blit.
I went with the ping-pong approach and got a bit further, just gotta figure out what parameters the CubeBlur shader wants.

Code (CSharp):

1. RenderTexture srcCubemap;         // Contains the result of the rendered scene in mip 0
2. RenderTexture dstCubemap;         // Will contain the full result with convolved mips when done
3.  
4. Material convolutionMaterial;    // Uses the shader "Hidden/CubeBlur"
5.  
6. int mipCount = 7;                // Unity seems to always use 7 mip steps in total for reflection probes
7.  
8. GL.PushMatrix();
9. GL.LoadOrtho();
10.  
11. for (int mip = 0; mip < mipCount + 1; mip++)
12. {
13.     // Ping to dstCubemap (copy each face)
14.     Graphics.CopyTexture(srcCubemap, 0, mip, dstCubemap, 0, mip);
15.     Graphics.CopyTexture(srcCubemap, 1, mip, dstCubemap, 1, mip);
16.     Graphics.CopyTexture(srcCubemap, 2, mip, dstCubemap, 2, mip);
17.     Graphics.CopyTexture(srcCubemap, 3, mip, dstCubemap, 3, mip);
18.     Graphics.CopyTexture(srcCubemap, 4, mip, dstCubemap, 4, mip);
19.     Graphics.CopyTexture(srcCubemap, 5, mip, dstCubemap, 5, mip);
20.  
21.     // Destination mip level
22.     int dstMip = mip + 1;
23.  
24.     if (dstMip == mipCount)
25.         // We just copied the last mip level, don't need to convolve any further
26.         break;
27.  
28.     // The resolution of the source mip level
29.     int mipRes = dstCubemap.width / (int)Mathf.Pow(2, mip);
30.  
31.     // Pong to srcCubemap (convolve)
32.     convolutionMaterial.SetTexture("_MainTex", dstCubemap);
33.     convolutionMaterial.SetFloat("_Texel", 1f / mipRes);
34.     convolutionMaterial.SetFloat("_Level", mip);
35.  
36.     // Activate the material for rendering
37.     convolutionMaterial.SetPass(0);
38.  
39.     // The CubeBlur shader uses 3D texture coordinates when sampling the cubemap,
40.     // so we can't use Graphics.Blit here.
41.     // Instead we build a cube (sort of) and render using that.
42.  
43.     // Positive X
44.     Graphics.SetRenderTarget(srcCubemap, dstMip, CubemapFace.PositiveX);
45.     GL.Begin(GL.QUADS);
46.     GL.TexCoord3( 1, 1, 1);
47.     GL.Vertex3(0, 0, 1);
48.     GL.TexCoord3( 1,-1, 1);
49.     GL.Vertex3(0, 1, 1);
50.     GL.TexCoord3( 1,-1,-1);
51.     GL.Vertex3(1, 1, 1);
52.     GL.TexCoord3( 1, 1,-1);
53.     GL.Vertex3(1, 0, 1);
54.     GL.End();
55.  
56.     // Negative X
57.     Graphics.SetRenderTarget(srcCubemap, dstMip, CubemapFace.NegativeX);
58.     GL.Begin(GL.QUADS);
59.     GL.TexCoord3(-1, 1,-1);
60.     GL.Vertex3(0, 0, 1);
61.     GL.TexCoord3(-1,-1,-1);
62.     GL.Vertex3(0, 1, 1);
63.     GL.TexCoord3(-1,-1, 1);
64.     GL.Vertex3(1, 1, 1);
65.     GL.TexCoord3(-1, 1, 1);
66.     GL.Vertex3(1, 0, 1);
67.     GL.End();
68.  
69.     // Positive Y
70.     Graphics.SetRenderTarget(srcCubemap, dstMip, CubemapFace.PositiveY);
71.     GL.Begin(GL.QUADS);
72.     GL.TexCoord3(-1, 1,-1);
73.     GL.Vertex3(0, 0, 1);
74.     GL.TexCoord3(-1, 1, 1);
75.     GL.Vertex3(0, 1, 1);
76.     GL.TexCoord3( 1, 1, 1);
77.     GL.Vertex3(1, 1, 1);
78.     GL.TexCoord3( 1, 1,-1);
79.     GL.Vertex3(1, 0, 1);
80.     GL.End();
81.  
82.     // Negative Y
83.     Graphics.SetRenderTarget(srcCubemap, dstMip, CubemapFace.NegativeY);
84.     GL.Begin(GL.QUADS);
85.     GL.TexCoord3(-1,-1, 1);
86.     GL.Vertex3(0, 0, 1);
87.     GL.TexCoord3(-1,-1,-1);
88.     GL.Vertex3(0, 1, 1);
89.     GL.TexCoord3( 1,-1,-1);
90.     GL.Vertex3(1, 1, 1);
91.     GL.TexCoord3( 1,-1, 1);
92.     GL.Vertex3(1, 0, 1);
93.     GL.End();
94.  
95.     // Positive Z
96.     Graphics.SetRenderTarget(srcCubemap, dstMip, CubemapFace.PositiveZ);
97.     GL.Begin(GL.QUADS);
98.     GL.TexCoord3(-1, 1, 1);
99.     GL.Vertex3(0, 0, 1);
100.     GL.TexCoord3(-1,-1, 1);
101.     GL.Vertex3(0, 1, 1);
102.     GL.TexCoord3( 1,-1, 1);
103.     GL.Vertex3(1, 1, 1);
104.     GL.TexCoord3( 1, 1, 1);
105.     GL.Vertex3(1, 0, 1);
106.     GL.End();
107.  
108.     // Negative Z
109.     Graphics.SetRenderTarget(srcCubemap, dstMip, CubemapFace.NegativeZ);
110.     GL.Begin(GL.QUADS);
111.     GL.TexCoord3( 1, 1,-1);
112.     GL.Vertex3(0, 0, 1);
113.     GL.TexCoord3( 1,-1,-1);
114.     GL.Vertex3(0, 1, 1);
115.     GL.TexCoord3(-1,-1,-1);
116.     GL.Vertex3(1, 1, 1);
117.     GL.TexCoord3(-1, 1,-1);
118.     GL.Vertex3(1, 0, 1);
119.     GL.End();
120. }
121.  
122. GL.PopMatrix();

The result currently looks like this:

 

You may be able to reverse engineer the values more easily using something like RenderDoc. Place a real-time / every frame reflection probe in a simple scene and then capture while in play mode and you should be able to track down the draw calls doing the blur to extract the values Unity is using.

 

Hey @Fewes any luck in figuring it out? I'm interested in doing this as well and figured it wouldn't hurt to ask before I attempt to reinvent the wheel myself.

 

Unfortunately not, the code above is as far as I got. I gave reverse-engineering the shader a go but it was a bit above my head. A (stupid) trick you can do to get around it is render your custom cubemap, then render an inverted sphere with a cubemap shader around a native Unity reflection probe and let it handle the convolution for you. Ideally there would be a RenderTexture.GenerateMips(ConvolutionType.Specular) or something but I'm guessing that's not high on the priority list for the engine.

 

Dang that is unfortunate. Clever workaround, but given that it involves rendering twice I have a feeling that is going to be too heavy for me. Might give it a go anyways though. Thanks.

 

Just to also chime in on this, @Fewes, looks like you're comparing exported probes, which seem to have some CPU post processing applied to them on import.

If you compare your cube to a realtime reflection probe, they're pretty much identical.

 

I believe the idea is (or at least my idea is) to do it in realtime but match the quality regardless of whether or not it's drawn by the probe set to "realtime" or by rendering to a cubemap manually.

Having dynamic reflections ideally wouldn't cause everything to appear slightly less rough then it should be (due to the lack of blurring).

 

So I had a thought... perhaps it is expecting the mip downsample to occur prior to running the CubeBlur shader. They do have a CubeCopy, possibly for this reason.

I was playing around with some values and noticed that the ones that logically made sense looked like samples were spaced out appropriately, however they were too sharp. If I could sample the mip below and blur that, that would make more sense to me (than blurring the current mip directly) as well.

I'm going to keep messing with it.

 

Unfortunately that alone did not seem to yield correct results. Furthermore, the overhead of repeatedly syncing between CPU/GPU for the mips is too high for my needs, so I'm investigating the possibility of doing the convolution in a compute shader. This task seems like a bad fit for the CPU to have any part in anyways, so I wouldn't mind if it could be avoided completely.

However I have no experience with physically based specular convolution so who knows if I'll manage to produce something usable.

 

So I actually ended up solving this. Sort of. GuitarBro was correct in guessing that the "normal" cubemap mips was needed as the input. It works, but looks slightly different from Unity's reflection probes. Someone who is more intimate with this kind of stuff can probably point out what is incorrect (I'm guessing something to do with texel size or mip map selection when convolving). In any case, I found it good enough to use. It is really fast and can be amortized easily for even better performance. Here is the code:

Code (CSharp):

1. /// <summary>
2. /// Helper function to perform a blit for each cubemap face.
3. /// The provided material should use a shader where the TEXCOORD0 unit xyz coordinates is the world space view direction.
4. /// See the built-in CubeBlurOdd shader for an example.
5. /// </summary>
6. void CubemapBlit (RenderTexture dstCubemap, Material material, int shaderPass = 0, int dstMip = 0)
7. {
8.     GL.PushMatrix();
9.     GL.LoadOrtho();
10.  
11.     // The CubeBlur shader uses 3D texture coordinates when sampling the cubemap,
12.     // so we can't use Graphics.Blit here.
13.     // Instead we build a cube (sort of) and render using that.
14.  
15.     // Positive X
16.     Graphics.SetRenderTarget(dstCubemap, dstMip, CubemapFace.PositiveX);
17.     material.SetPass(shaderPass);
18.     GL.Begin(GL.QUADS);
19.     GL.MultiTexCoord3(0, 1, 1, 1);
20.     GL.Vertex3(0, 0, 1);
21.     GL.MultiTexCoord3(0, 1,-1, 1);
22.     GL.Vertex3(0, 1, 1);
23.     GL.MultiTexCoord3(0, 1,-1,-1);
24.     GL.Vertex3(1, 1, 1);
25.     GL.MultiTexCoord3(0, 1, 1,-1);
26.     GL.Vertex3(1, 0, 1);
27.     GL.End();
28.  
29.     // Negative X
30.     Graphics.SetRenderTarget(dstCubemap, dstMip, CubemapFace.NegativeX);
31.     material.SetPass(shaderPass);
32.     GL.Begin(GL.QUADS);
33.     GL.MultiTexCoord3(0,-1, 1,-1);
34.     GL.Vertex3(0, 0, 1);
35.     GL.MultiTexCoord3(0,-1,-1,-1);
36.     GL.Vertex3(0, 1, 1);
37.     GL.MultiTexCoord3(0,-1,-1, 1);
38.     GL.Vertex3(1, 1, 1);
39.     GL.MultiTexCoord3(0,-1, 1, 1);
40.     GL.Vertex3(1, 0, 1);
41.     GL.End();
42.  
43.     // Positive Y
44.     Graphics.SetRenderTarget(dstCubemap, dstMip, CubemapFace.PositiveY);
45.     material.SetPass(shaderPass);
46.     GL.Begin(GL.QUADS);
47.     GL.MultiTexCoord3(0,-1, 1,-1);
48.     GL.Vertex3(0, 0, 1);
49.     GL.MultiTexCoord3(0,-1, 1, 1);
50.     GL.Vertex3(0, 1, 1);
51.     GL.MultiTexCoord3(0, 1, 1, 1);
52.     GL.Vertex3(1, 1, 1);
53.     GL.MultiTexCoord3(0, 1, 1,-1);
54.     GL.Vertex3(1, 0, 1);
55.     GL.End();
56.  
57.     // Negative Y
58.     Graphics.SetRenderTarget(dstCubemap, dstMip, CubemapFace.NegativeY);
59.     material.SetPass(shaderPass);
60.     GL.Begin(GL.QUADS);
61.     GL.MultiTexCoord3(0,-1,-1, 1);
62.     GL.Vertex3(0, 0, 1);
63.     GL.MultiTexCoord3(0,-1,-1,-1);
64.     GL.Vertex3(0, 1, 1);
65.     GL.MultiTexCoord3(0, 1,-1,-1);
66.     GL.Vertex3(1, 1, 1);
67.     GL.MultiTexCoord3(0, 1,-1, 1);
68.     GL.Vertex3(1, 0, 1);
69.     GL.End();
70.  
71.     // Positive Z
72.     Graphics.SetRenderTarget(dstCubemap, dstMip, CubemapFace.PositiveZ);
73.     material.SetPass(shaderPass);
74.     GL.Begin(GL.QUADS);
75.     GL.MultiTexCoord3(0,-1, 1, 1);
76.     GL.Vertex3(0, 0, 1);
77.     GL.MultiTexCoord3(0,-1,-1, 1);
78.     GL.Vertex3(0, 1, 1);
79.     GL.MultiTexCoord3(0, 1,-1, 1);
80.     GL.Vertex3(1, 1, 1);
81.     GL.MultiTexCoord3(0, 1, 1, 1);
82.     GL.Vertex3(1, 0, 1);
83.     GL.End();
84.  
85.     // Negative Z
86.     Graphics.SetRenderTarget(dstCubemap, dstMip, CubemapFace.NegativeZ);
87.     material.SetPass(shaderPass);
88.     GL.Begin(GL.QUADS);
89.     GL.MultiTexCoord3(0, 1, 1,-1);
90.     GL.Vertex3(0, 0, 1);
91.     GL.MultiTexCoord3(0, 1,-1,-1);
92.     GL.Vertex3(0, 1, 1);
93.     GL.MultiTexCoord3(0,-1,-1,-1);
94.     GL.Vertex3(1, 1, 1);
95.     GL.MultiTexCoord3(0,-1, 1,-1);
96.     GL.Vertex3(1, 0, 1);
97.     GL.End();
98.  
99.     GL.PopMatrix();
100. }
101.  
102. // Will contain the convolved texture
103. RenderTexture outputCubemap;
104.  
105. const int mipCount = 7; // Unity seems to always use 7 mip steps in total for reflection probe cubemaps
106.  
107. public void SpecularConvolve (RenderTexture inputCubemap)
108. {
109.     // I had to copy this shader from BuiltinShaders to my project
110.     Material convolutionMaterial = new Material(Shader.Find("Hidden/CubeBlurOdd"));
111.  
112.     var desc = new RenderTextureDescriptor(inputCubemap.width, inputCubemap.height, RenderTextureFormat.DefaultHDR, 0, mipCount);
113.     desc.dimension = TextureDimension.Cube;
114.     desc.useMipMap = true;
115.     desc.autoGenerateMips = false;
116.  
117.     outputCubemap = new RenderTexture(desc);
118.     outputCubemap.Create();
119.     outputCubemap.filterMode = FilterMode.Trilinear;
120.  
121.     // Generate mip maps using the standard box filter (these will appear blocky if the cubemap is used for glossy reflections)
122.     // We will use these as the input to the convolution blur shader
123.     // Your cubemap RT must have been created with useMipMap set to true and it should use the maximum number of mipmaps for its resolution
124.     // If you already generating mips you can skip this
125.     inputCubemap.GenerateMips();
126.  
127.     // If you want to amortize, turn the function into a coroutine and yield for a frame here
128.  
129.     // Transfer mip 0 (this is done separately from the loop below as we do not want to blur it)
130.     for (int face = 0; face < 6; face++)
131.         Graphics.CopyTexture(inputCubemap, face, 0, outputCubemap, face, 0);
132.  
133.     // This will track the texel size for the current mip level
134.     float texelSize = 1f / (resolution / 2); // Divided by 2 because we are going to start with mip 1, which is half-res
135.     // Process mips 1-7
136.     for (int mip = 1; mip <= mipCount; mip++)
137.     {
138.         // If you want to amortize, turn the function into a coroutine and yield for a frame here
139.  
140.         convolutionMaterial.SetTexture("_MainTex", inputCubemap);
141.         convolutionMaterial.SetFloat("_Texel", texelSize);
142.         // Output mip range -> normalized range -> input mip range
143.         float level = (float)(mip+1) / outputCubemap.mipmapCount * inputCubemap.mipmapCount;
144.         convolutionMaterial.SetFloat("_Level", level);
145.  
146.         CubemapBlit(outputCubemap, convolutionMaterial, 0, mip);
147.  
148.         texelSize *= 2;
149.     }
150. }

I stripped this code from my own project without testing it, so if you run into problems let me know and I can probably help you get it working.

 

Hey that's great! I never quite figured it out so I'm glad to hear you did. Might try implementing it later in which case I'll let you know how it goes.