Search Unity

Specular convolution when calculating mip maps for cubemap Render Texture?

Discussion in 'General Graphics' started by Fewes, Jan 23, 2019.

  1. Fewes

    Fewes

    Joined:
    Jul 1, 2014
    Posts:
    174
    I am rendering my own realtime reflection cubemaps by first rendering each face to an intermediate "face" RT, copying all faces to the cubemap RT and then generating mip maps.
    This works fine, except the cubemap looks noticeably "mip-mapped" when sampled as a reflection on a rough surface. I know Unity uses a special method ("specular convolution") for generating mip maps for cubemaps when importing them in the inspector, and so my question is: Is this method available through scripting somewhere? I cannot seem to find any info on it.
     
  2. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    8,157
    There's no easy to use function for this, but the shader they use to do the convolution is available. It's called "Hidden/CubeBlur" and you can look at it in the built in shader source files in the file CubeBlur.shader.
     
  3. Fewes

    Fewes

    Joined:
    Jul 1, 2014
    Posts:
    174
    That does look promising. Any idea of how to use it? I tried this but any mip level above 1 is completely black.

    Code (CSharp):
    1. var convolutionMaterial = new Material(Shader.Find("Hidden/CubeBlur"));
    2.  
    3. var rt = RenderTexture.GetTemporary(cubeMap.descriptor);
    4. rt.useMipMap = false;
    5. Graphics.Blit(cubeMap, rt);
    6.  
    7. int mipCount = 1 + Mathf.FloorToInt(Mathf.Log10(cubeMap.width) / Mathf.Log10(2f));
    8.  
    9. for (int mip = 0; mip < mipCount; mip++)
    10. {
    11.     int mipRes = cubeMap.width / (mip + 1);
    12.     convolutionMaterial.SetFloat("_Texel", 1f / (float)mipRes);
    13.     convolutionMaterial.SetFloat("_Level", mip);
    14.     Graphics.SetRenderTarget(cubeMap, mip);
    15.     Graphics.Blit(rt, convolutionMaterial, 0);
    16. }
     
  4. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    8,157
    Unfortunately, no.

    Here's some stuff I can guess though.

    Blit() uses a screen aligned ortho quad, so likely isn't going to be useful for rendering a cubemap. You'll likely need to setup geometry manually and use GL commands to build out a cube rather than Blit()
    https://docs.unity3d.com/ScriptReference/GL.html.

    Alternatively you may need to render to one face & mip at a time. Certainly that seems to be how Unity does it since it's an option for time slicing on reflection probes.
     
  5. Fewes

    Fewes

    Joined:
    Jul 1, 2014
    Posts:
    174
    Your post put me in the right direction and I'm having some luck making this work, but the result is off compared to the default convolution:



    This seems to be due to the way the mip map of the source cubemap is sampled. When convoluting like this, are you maybe supposed to use the result of the previous mip level when calculating the current one? Is that even possible to do in Unity? From my experience it is not possible to read from one mip level while simultaneously writing to another (although I think this is supported by the graphics driver).
     
    Last edited: Jan 24, 2019
  6. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    8,157
    Yeah, I would assume they build off of each other.

    And no, I don't believe you can render to a mip map of a texture to you're reading from as there's no guarantee you're not going to sample from that same mip in the shader.

    You should either have two cubemap render textures and ping / pong between them, then use CopyTexture() to combine them into one. Or if you're rendering one face at a time have a 2d render texture you're rendering too and use CopyTexture to copy back to that face's mip.
     
  7. Fewes

    Fewes

    Joined:
    Jul 1, 2014
    Posts:
    174
    You were right on the money with the guess about needing to construct a cube instead of using Blit.
    I went with the ping-pong approach and got a bit further, just gotta figure out what parameters the CubeBlur shader wants.
    Code (CSharp):
    1. RenderTexture srcCubemap;         // Contains the result of the rendered scene in mip 0
    2. RenderTexture dstCubemap;         // Will contain the full result with convolved mips when done
    3.  
    4. Material convolutionMaterial;    // Uses the shader "Hidden/CubeBlur"
    5.  
    6. int mipCount = 7;                // Unity seems to always use 7 mip steps in total for reflection probes
    7.  
    8. GL.PushMatrix();
    9. GL.LoadOrtho();
    10.  
    11. for (int mip = 0; mip < mipCount + 1; mip++)
    12. {
    13.     // Ping to dstCubemap (copy each face)
    14.     Graphics.CopyTexture(srcCubemap, 0, mip, dstCubemap, 0, mip);
    15.     Graphics.CopyTexture(srcCubemap, 1, mip, dstCubemap, 1, mip);
    16.     Graphics.CopyTexture(srcCubemap, 2, mip, dstCubemap, 2, mip);
    17.     Graphics.CopyTexture(srcCubemap, 3, mip, dstCubemap, 3, mip);
    18.     Graphics.CopyTexture(srcCubemap, 4, mip, dstCubemap, 4, mip);
    19.     Graphics.CopyTexture(srcCubemap, 5, mip, dstCubemap, 5, mip);
    20.  
    21.     // Destination mip level
    22.     int dstMip = mip + 1;
    23.  
    24.     if (dstMip == mipCount)
    25.         // We just copied the last mip level, don't need to convolve any further
    26.         break;
    27.  
    28.     // The resolution of the source mip level
    29.     int mipRes = dstCubemap.width / (int)Mathf.Pow(2, mip);
    30.  
    31.     // Pong to srcCubemap (convolve)
    32.     convolutionMaterial.SetTexture("_MainTex", dstCubemap);
    33.     convolutionMaterial.SetFloat("_Texel", 1f / mipRes);
    34.     convolutionMaterial.SetFloat("_Level", mip);
    35.  
    36.     // Activate the material for rendering
    37.     convolutionMaterial.SetPass(0);
    38.  
    39.     // The CubeBlur shader uses 3D texture coordinates when sampling the cubemap,
    40.     // so we can't use Graphics.Blit here.
    41.     // Instead we build a cube (sort of) and render using that.
    42.  
    43.     // Positive X
    44.     Graphics.SetRenderTarget(srcCubemap, dstMip, CubemapFace.PositiveX);
    45.     GL.Begin(GL.QUADS);
    46.     GL.TexCoord3( 1, 1, 1);
    47.     GL.Vertex3(0, 0, 1);
    48.     GL.TexCoord3( 1,-1, 1);
    49.     GL.Vertex3(0, 1, 1);
    50.     GL.TexCoord3( 1,-1,-1);
    51.     GL.Vertex3(1, 1, 1);
    52.     GL.TexCoord3( 1, 1,-1);
    53.     GL.Vertex3(1, 0, 1);
    54.     GL.End();
    55.  
    56.     // Negative X
    57.     Graphics.SetRenderTarget(srcCubemap, dstMip, CubemapFace.NegativeX);
    58.     GL.Begin(GL.QUADS);
    59.     GL.TexCoord3(-1, 1,-1);
    60.     GL.Vertex3(0, 0, 1);
    61.     GL.TexCoord3(-1,-1,-1);
    62.     GL.Vertex3(0, 1, 1);
    63.     GL.TexCoord3(-1,-1, 1);
    64.     GL.Vertex3(1, 1, 1);
    65.     GL.TexCoord3(-1, 1, 1);
    66.     GL.Vertex3(1, 0, 1);
    67.     GL.End();
    68.  
    69.     // Positive Y
    70.     Graphics.SetRenderTarget(srcCubemap, dstMip, CubemapFace.PositiveY);
    71.     GL.Begin(GL.QUADS);
    72.     GL.TexCoord3(-1, 1,-1);
    73.     GL.Vertex3(0, 0, 1);
    74.     GL.TexCoord3(-1, 1, 1);
    75.     GL.Vertex3(0, 1, 1);
    76.     GL.TexCoord3( 1, 1, 1);
    77.     GL.Vertex3(1, 1, 1);
    78.     GL.TexCoord3( 1, 1,-1);
    79.     GL.Vertex3(1, 0, 1);
    80.     GL.End();
    81.  
    82.     // Negative Y
    83.     Graphics.SetRenderTarget(srcCubemap, dstMip, CubemapFace.NegativeY);
    84.     GL.Begin(GL.QUADS);
    85.     GL.TexCoord3(-1,-1, 1);
    86.     GL.Vertex3(0, 0, 1);
    87.     GL.TexCoord3(-1,-1,-1);
    88.     GL.Vertex3(0, 1, 1);
    89.     GL.TexCoord3( 1,-1,-1);
    90.     GL.Vertex3(1, 1, 1);
    91.     GL.TexCoord3( 1,-1, 1);
    92.     GL.Vertex3(1, 0, 1);
    93.     GL.End();
    94.  
    95.     // Positive Z
    96.     Graphics.SetRenderTarget(srcCubemap, dstMip, CubemapFace.PositiveZ);
    97.     GL.Begin(GL.QUADS);
    98.     GL.TexCoord3(-1, 1, 1);
    99.     GL.Vertex3(0, 0, 1);
    100.     GL.TexCoord3(-1,-1, 1);
    101.     GL.Vertex3(0, 1, 1);
    102.     GL.TexCoord3( 1,-1, 1);
    103.     GL.Vertex3(1, 1, 1);
    104.     GL.TexCoord3( 1, 1, 1);
    105.     GL.Vertex3(1, 0, 1);
    106.     GL.End();
    107.  
    108.     // Negative Z
    109.     Graphics.SetRenderTarget(srcCubemap, dstMip, CubemapFace.NegativeZ);
    110.     GL.Begin(GL.QUADS);
    111.     GL.TexCoord3( 1, 1,-1);
    112.     GL.Vertex3(0, 0, 1);
    113.     GL.TexCoord3( 1,-1,-1);
    114.     GL.Vertex3(0, 1, 1);
    115.     GL.TexCoord3(-1,-1,-1);
    116.     GL.Vertex3(1, 1, 1);
    117.     GL.TexCoord3(-1, 1,-1);
    118.     GL.Vertex3(1, 0, 1);
    119.     GL.End();
    120. }
    121.  
    122. GL.PopMatrix();
    The result currently looks like this:
     
    Last edited: Jan 27, 2019
    RockSPb and superexplosive like this.
  8. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    8,157
    You may be able to reverse engineer the values more easily using something like RenderDoc. Place a real-time / every frame reflection probe in a simple scene and then capture while in play mode and you should be able to track down the draw calls doing the blur to extract the values Unity is using.
     
  9. GuitarBro

    GuitarBro

    Joined:
    Oct 9, 2014
    Posts:
    80
    Hey @Fewes any luck in figuring it out? I'm interested in doing this as well and figured it wouldn't hurt to ask before I attempt to reinvent the wheel myself. :p
     
  10. Fewes

    Fewes

    Joined:
    Jul 1, 2014
    Posts:
    174
    Unfortunately not, the code above is as far as I got. I gave reverse-engineering the shader a go but it was a bit above my head. A (stupid) trick you can do to get around it is render your custom cubemap, then render an inverted sphere with a cubemap shader around a native Unity reflection probe and let it handle the convolution for you. Ideally there would be a RenderTexture.GenerateMips(ConvolutionType.Specular) or something but I'm guessing that's not high on the priority list for the engine.
     
  11. GuitarBro

    GuitarBro

    Joined:
    Oct 9, 2014
    Posts:
    80
    Dang that is unfortunate. Clever workaround, but given that it involves rendering twice I have a feeling that is going to be too heavy for me. Might give it a go anyways though. Thanks.
     
  12. wwweh

    wwweh

    Joined:
    Oct 15, 2012
    Posts:
    15
    Just to also chime in on this, @Fewes, looks like you're comparing exported probes, which seem to have some CPU post processing applied to them on import.

    If you compare your cube to a realtime reflection probe, they're pretty much identical.
     
    Last edited: Nov 6, 2019
  13. GuitarBro

    GuitarBro

    Joined:
    Oct 9, 2014
    Posts:
    80
    I believe the idea is (or at least my idea is) to do it in realtime but match the quality regardless of whether or not it's drawn by the probe set to "realtime" or by rendering to a cubemap manually.

    Having dynamic reflections ideally wouldn't cause everything to appear slightly less rough then it should be (due to the lack of blurring).
     
  14. GuitarBro

    GuitarBro

    Joined:
    Oct 9, 2014
    Posts:
    80
    So I had a thought... perhaps it is expecting the mip downsample to occur prior to running the CubeBlur shader. They do have a CubeCopy, possibly for this reason.

    I was playing around with some values and noticed that the ones that logically made sense looked like samples were spaced out appropriately, however they were too sharp. If I could sample the mip below and blur that, that would make more sense to me (than blurring the current mip directly) as well.

    I'm going to keep messing with it.
     
  15. GuitarBro

    GuitarBro

    Joined:
    Oct 9, 2014
    Posts:
    80
    Unfortunately that alone did not seem to yield correct results. Furthermore, the overhead of repeatedly syncing between CPU/GPU for the mips is too high for my needs, so I'm investigating the possibility of doing the convolution in a compute shader. This task seems like a bad fit for the CPU to have any part in anyways, so I wouldn't mind if it could be avoided completely.

    However I have no experience with physically based specular convolution so who knows if I'll manage to produce something usable. :p
     
unityunity