Search Unity

  1. Are you interested in providing feedback directly to Unity teams? Sign up to become a member of Unity Pulse, our new product feedback and research community.
    Dismiss Notice

RenderTexture with clear flags set to depth has weird outline (Anti-Aliasing issue)

Discussion in 'General Graphics' started by JoePatrick, Apr 22, 2019.

  1. JoePatrick

    JoePatrick

    Joined:
    Nov 3, 2013
    Posts:
    91
    Hi, so I'm rendering out a 3D mesh to a Texture2D and drawing that texture in an editor window but when I turn on depth clear flags so that I can have a transparent background, there is a weird outline around the mesh - is there anyway to fix this?

    Below are two screenshots - left is with clearflags set to solidcolor (no weird outline) and right is set to depth (has weird outline)

    Also below that is the relevant section of code

    Thanks :)



    Code (CSharp):
    1. RenderTexture rt = new RenderTexture(renderSize.x, renderSize.y, 24);
    2. rt.Create();
    3.  
    4. previewRenderer.camera.targetTexture = rt;
    5. previewRenderer.camera.Render();
    6.        
    7. var currentRT = RenderTexture.active;      
    8. RenderTexture.active = rt;
    9. Texture2D render = new Texture2D(rt.width, rt.height);
    10. render.ReadPixels(new Rect(0, 0, rt.width, rt.height), 0, 0);
    11. render.Apply();
    12. RenderTexture.active = currentRT;
    13.  
    14. //render is then drawn in the UI
     
  2. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    10,902
    They both have outlines really. When you set the clear to depth only, really you’re saying “use the default clear color of (0,0,0,0)”. This means the background is “clear” because the alpha defaults to fully transparent, but you’re rendering using MSAA, so the anti-aliased edges are a blend of the black default clear color and your rendered object. The short version is the colors in the final image have been premultiplied by the alpha.

    The easiest solution is to render the image using a premultiplied alpha shader... of which Unity has none built in that actually work properly for the simple case of using an input texture that has premultiplied alpha. Like this:
    Code (CSharp):
    1. Shader "Custom/Premultiplied Alpha Blend"
    2. {
    3.     Properties
    4.     {
    5.         _MainTex("Tex", 2D) = "white" {}
    6.     }
    7.     SubShader
    8.     {
    9.         Tags { "Queue"="Transparent" "PreviewType"="Plane" }
    10.  
    11.         LOD 100
    12.  
    13.         ZWrite Off
    14.         Blend One OneMinusSrcAlpha
    15.  
    16.         Pass
    17.         {
    18.             CGPROGRAM
    19.             #pragma vertex vert
    20.             #pragma fragment frag
    21.            
    22.             #include "UnityCG.cginc"
    23.  
    24.             struct appdata
    25.             {
    26.                 float4 vertex : POSITION;
    27.                 float2 uv : TEXCOORD0;
    28.             };
    29.  
    30.             struct v2f
    31.             {
    32.                 float4 pos : SV_POSITION;
    33.                 float2 uv : TEXCOORD0;
    34.             };
    35.  
    36.             sampler2D _MainTex;
    37.            
    38.             v2f vert (appdata v)
    39.             {
    40.                 v2f o;
    41.                 o.pos = UnityObjectToClipPos(v.vertex);
    42.                 o.uv = v.uv;
    43.                 return o;
    44.             }
    45.            
    46.             fixed4 frag (v2f i) : SV_Target
    47.             {
    48.                 return tex2D(_MainTex, i.uv);
    49.             }
    50.             ENDCG
    51.         }
    52.     }
    53. }
    You can use that along with something like EditorGUI.DrawPreviewTexture to draw a texture with a custom material rather than just the default alpha blending.

    The other options would be to disable MSAA and filtering on your texture, or to use a post process on your images to divide the color values by the alpha to undo the premultiplication, and then dilate the color into the fully transparent areas to ensure filtering doesn't still produce a dark edge.
     
  3. JoePatrick

    JoePatrick

    Joined:
    Nov 3, 2013
    Posts:
    91

    Thanks for that, I'll give it a go - hopefully should do the trick :D
     
  4. JoePatrick

    JoePatrick

    Joined:
    Nov 3, 2013
    Posts:
    91
    Ok so I tried using the custom shader but didn't seem to change anything so maybe I did it wrong?
    Here's a screenshot (above is new method, below is old)


    And here's the code for drawing it (rendersList is a list of Texture2Ds - converted from RenderTextures as in my original post)
    Code (CSharp):
    1. if (rendersList.Count > 0)
    2. {
    3.      Rect r = GUILayoutUtility.GetRect(128, 128);
    4.      r.width = 128;
    5.      EditorGUI.DrawPreviewTexture(r, rendersList[0], new Material(Shader.Find("IconGenerator/ImgShader")));
    6. }
     
  5. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    10,902
    bleh ... obviously something isn't working as there's almost no difference between those two images.

    One random option. Try setting the camera to clear to a solid color, and set that solid color to 0,0,0,0. Otherwise I wonder if custom blend modes are broken for DrawPreviewTexture. :/
     
  6. JoePatrick

    JoePatrick

    Joined:
    Nov 3, 2013
    Posts:
    91
    Still no luck :(

    I also tried setting it to 1,0,0,0 and this is what I got so there is some difference?


    also, if I turn off AA for the camera it gets rid of the outline as expected, from here is there then a way to do some form of AA/smoothing that wouldn't make the outline?

     
    Last edited: Apr 23, 2019
  7. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    10,902
    So, Unity has a long running bug that it displays textures in the UI using the wrong gamma. A small change to the shader should make the original setup work.
    Code (CSharp):
    1. Shader "Editor/Premultiplied Alpha Texture"
    2. {
    3.     Properties
    4.     {
    5.         _MainTex("Tex", 2D) = "white" {}
    6.     }
    7.     SubShader
    8.     {
    9.         Tags { "Queue"="Transparent" "PreviewType"="Plane" }
    10.  
    11.         LOD 100
    12.  
    13.         ZWrite Off
    14.         Blend One OneMinusSrcAlpha
    15.  
    16.         Pass
    17.         {
    18.             CGPROGRAM
    19.             #pragma vertex vert
    20.             #pragma fragment frag
    21.          
    22.             #include "UnityCG.cginc"
    23.  
    24.             struct appdata
    25.             {
    26.                 float4 vertex : POSITION;
    27.                 float2 uv : TEXCOORD0;
    28.             };
    29.  
    30.             struct v2f
    31.             {
    32.                 float4 pos : SV_POSITION;
    33.                 float2 uv : TEXCOORD0;
    34.             };
    35.  
    36.             sampler2D _MainTex;
    37.          
    38.             v2f vert (appdata v)
    39.             {
    40.                 v2f o;
    41.                 o.pos = UnityObjectToClipPos(v.vertex);
    42.                 o.uv = v.uv;
    43.                 return o;
    44.             }
    45.          
    46.             half4 frag (v2f i) : SV_Target
    47.             {
    48.                 half4 col = tex2D(_MainTex, i.uv);
    49.                 // gamma correction for use in the editor UI
    50.                 col.rgb = LinearToGammaSpace(col.rgb);
    51.                 return col;
    52.             }
    53.             ENDCG
    54.         }
    55.     }
    56. }
    edit: nope, it's still wrong. Closer, but now it has a bright edge. :/
    Drawing textures in the editor properly is a huge pain.
     
    Last edited: Apr 23, 2019
  8. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    10,902
    Left: Before, Right: with correction
    upload_2019-4-23_12-59-14.png
     
  9. JoePatrick

    JoePatrick

    Joined:
    Nov 3, 2013
    Posts:
    91
    Could I see your code for the editor script because mines still not quite right :/



    thanks for all the help btw
     
  10. JoePatrick

    JoePatrick

    Joined:
    Nov 3, 2013
    Posts:
    91
    The one thing that did kinda work was rendering the texture at twice the resolution with MXAA turned off, then using EditorGUI.DrawPreviewTexture and drawing at the original size (essentially downscaling it and doing the AA for me)

    only issue is I need to be able to save as png but obviously all I have available is the double scale texture2d - not the nicely scaled down one that shows in the UI
     
  11. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    10,902
    Okay, solved it a different way. Go back to the previous shader without correction. The trick is to set the proper sRGB settings for the render texture (sRGB) and the Texture2D (linear).
    Code (CSharp):
    1. using UnityEngine;
    2. using UnityEditor;
    3.  
    4. [RequireComponent(typeof(Camera))]
    5. public class CameraIcon : MonoBehaviour
    6. {
    7.     public enum IconResolutions
    8.     {
    9.         x64 = 64,
    10.         x128 = 128,
    11.         x256 = 256,
    12.         x512 = 512
    13.     }
    14.  
    15.     public IconResolutions iconResolution;
    16.     public Texture2D icon;
    17.  
    18. #if UNITY_EDITOR
    19.     public void RenderIcon()
    20.     {
    21.         var cam = GetComponent<Camera>();
    22.         int res = (int)iconResolution;
    23.  
    24.         var rtd = new RenderTextureDescriptor(res, res) { depthBufferBits = 24, msaaSamples = 8, useMipMap = false, sRGB = true };
    25.         var rt = new RenderTexture(rtd);
    26.         rt.Create();
    27.  
    28.         cam.targetTexture = rt;
    29.         cam.Render();
    30.         cam.targetTexture = null;
    31.  
    32.         if (icon == null)
    33.         {
    34.             icon = new Texture2D(res, res, TextureFormat.RGBA32, false, true);
    35.         }
    36.         else if (icon.width != res)
    37.         {
    38.             icon.Resize(res, res);
    39.         }
    40.  
    41.         var oldActive = RenderTexture.active;
    42.         RenderTexture.active = rt;
    43.         icon.ReadPixels(new Rect(0, 0, res, res), 0, 0);
    44.         icon.Apply();
    45.         RenderTexture.active = oldActive;
    46.  
    47.         DestroyImmediate(rt);
    48.     }
    49. #endif
    50. }
    51.  
    52. #if UNITY_EDITOR
    53. [CustomEditor(typeof(CameraIcon))]
    54. public class CameraIconEditor : Editor
    55. {
    56.     private Material iconMat;
    57.  
    58.     public override void OnInspectorGUI()
    59.     {
    60.         base.OnInspectorGUI();
    61.  
    62.         var ci = (target as CameraIcon);
    63.  
    64.         if (GUILayout.Button("UpdateIcon"))
    65.             ci.RenderIcon();
    66.         if (ci.icon != null)
    67.         {
    68.             if (iconMat == null)
    69.                 iconMat = new Material(Shader.Find("Editor/Premultiplied Alpha Blend"));
    70.             Rect rect = EditorGUILayout.GetControlRect(false, (int)ci.iconResolution);
    71.             rect.width = rect.height;
    72.             EditorGUI.DrawPreviewTexture(rect, ci.icon, iconMat);
    73.         }
    74.     }
    75. }
    76. #endif
     
  12. JoePatrick

    JoePatrick

    Joined:
    Nov 3, 2013
    Posts:
    91
    Yep that seems to work, now I just need to get it to work with the PreviewRenderUtility - hopefully shouldn't need to change much :p
     
  13. JoePatrick

    JoePatrick

    Joined:
    Nov 3, 2013
    Posts:
    91
    Ehh, still has the border for mine though :/

    It's not a scene camera, its PreviewRenderUtility but I thought this would work
    Code (CSharp):
    1. Test t = previewRenderer.camera.gameObject.AddComponent<Test>();
    2. t.iconResolution = Test.IconResolutions.x512;
    3. Texture2D render = t.RenderIcon(); //I made the function return the texture
    Then
    Code (CSharp):
    1. Material iconMat = new Material(Shader.Find("IconGenerator/ImgShader"));
    2. Rect rect = EditorGUILayout.GetControlRect(false, (int)previewSize);
    3. rect.width = rect.height;
    4. EditorGUI.DrawPreviewTexture(rect, rendersList[0], iconMat); //rendersList[0] is the same as the render Texture2D above


    Any thoughts?
    Might just rewrite my thing to avoid using the PreviewRenderUtility altogether - could just instantiate a camera in scene view, render the image, then delete it straight away
     
    Last edited: Apr 23, 2019
  14. JoePatrick

    JoePatrick

    Joined:
    Nov 3, 2013
    Posts:
    91
    Ok so I did what I said at the bottom of my last comment and ditched PreviewRenderUtility and it works if I use EditorGUI.DrawPreviewTexture with the custom shader however I need to be able to save the render as a png (and really just need it as a Texture2D as I am drawing a SelectionGrid which takes Texture2Ds) - is there anyway to save the render with the shader applied because otherwise the saved image just looks like the bottom pic rather than the top

     
    Last edited: Apr 23, 2019
  15. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    10,902
    The shader isn't modifying the data, it's modifying how the data is displayed. If you save the texture2D to a png, then load that png and display it with the same shader it should work exactly the same. Unity's PNG save and load shouldn't be modifying the data at all.

    The next "step" I would take if you want to try to avoid some of this pain is the "undo the premultiplied alpha" step I mentioned above. This could be done with a blit, or by modifying the data on the CPU before saving it to disk.

    Code (csharp):
    1. Color[] pixels = icon.GetPixels();
    2. for (int i=0; i < pixels.Length; i++)
    3. {
    4.     if (pixels[i].a < 1f && pixels[i].a > 0f)
    5.     {
    6.         pixels[i].r /= pixels[i].a;
    7.         pixels[i].g /= pixels[i].a;
    8.         pixels[i].b /= pixels[i].a;
    9.     }
    10. }
    11. icon.SetPixels(pixels);
    12. var bytes = icon.EncodeToPNG();
    13. // etc
     
    Last edited: Apr 24, 2019
  16. JoePatrick

    JoePatrick

    Joined:
    Nov 3, 2013
    Posts:
    91
    Issue is that I need to be able to view the image without using any special shaders, like in other programs or windows photo viewer for example. Is there anyway to sort of bake the shader into the data if that makes sense?
     
  17. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    10,902
    ... o_O

    See my previous post which you quoted?
     
  18. JoePatrick

    JoePatrick

    Joined:
    Nov 3, 2013
    Posts:
    91
    Sorry I somehow missed that whole bottom part, thanks I'll take a look at that when I'm home tomorrow :)
     
  19. JoePatrick

    JoePatrick

    Joined:
    Nov 3, 2013
    Posts:
    91
    Okay I got a chance to try it out and IT WORKED!
    Seriously dude, thanks for all your help, I'm really grateful for it :D
     
  20. MechaWolf99

    MechaWolf99

    Joined:
    Aug 22, 2017
    Posts:
    105
    Sorry for necro posting a bit, but I was wondering if you could elaborate on this?
    Using this code it improves the result for sure, but doesn't quite remove it. It goes from this:

    To this


    This is using
    GUI.DrawTexture
    , and white using
    EditorGUI.DrawPreviewTexture
    with the material works, it also will no longer respect GUI masking (like in scroll views).
    Of course using point filtering will solve this, but that makes it quite pixel (duh).
    Any further recommendations would be appreciated. :)
     
  21. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    10,902
    Unity uses a hidden shader that's embedded somewhere in the editor itself to render GUI.DrawTexture(), so I have no idea what it looks like. But if we make the assumption that it looks like their runtime UI shaders, you might be able to modify the UI-Default.shader to work for your needs.

    Try using
    EditorGUI.DrawPreviewMaterial()
    using a material with "UI/Default" as its shader and see if it respects the GUI masking in scroll views. If it does, take that shader and modify the Blend from:
    Blend SrcAlpha OneMinusSrcAlpha

    to:
    Blend One OneMinusSrcAlpha


    And change the end of the shader to:
    Code (csharp):
    1. color.rgb = LinearToGammaSpace(color.rgb);
    2. return color;
     
  22. MechaWolf99

    MechaWolf99

    Joined:
    Aug 22, 2017
    Posts:
    105
    So after looking at the source code and a bit of reflection I found that it uses this shader, however when I switch to using:
    Blend One OneMinusSrcAlpha
    , it stops respecting scroll view and makes the parts outside solid white. I tried changing a couple of different things, but this is beyond capabilities when it comes to shaders.
    Could the texture be directly edited to fix this so we wouldn't have to deal with the shaders? Or is as simple as making a different change to the shader?
    I really appreciate your help with this!
     
  23. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    10,902
    Blend One OneMinusSrcAlpha
    is for when you don't do the
    pixels[i].r /= pixels[i].a;
    I posted above. If you're doing that, keep the original
    Blend SrcAlpha OneMinusSrcAlpha
    .
     
  24. MechaWolf99

    MechaWolf99

    Joined:
    Aug 22, 2017
    Posts:
    105
    If I use the code then there is still an outline (like the second image), but if I don't use the code and instead use the
    Blend One OneMinusSrcAlpha
    , it will clip out of GUI masks/areas, but no border.

    Adding
    color.rgb = LinearToGammaSpace(color.rgb);
    only seemed to make it lighter, nothing else really.
     
  25. MechaWolf99

    MechaWolf99

    Joined:
    Aug 22, 2017
    Posts:
    105
    The more I think about it I guess there just isn't a way to have both
    One
    and
    SrcAlpha
    which seems to be what I need in order to have both transparent pixels blending and have it respect GUI scopes.
    Is there a way to recreate what the
    Blend One OneMinusSrcAlpha
    does on the C# side? I know there is that code, but it doesn't work as well as the shader.
     
  26. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    10,902
    Coming back to this, some thoughts:

    Your specific issue is because the c# script fixes the problem, but only for textures that are displayed with point sampling or pixel perfect. It took me until today to work out that the problem is bilinear filtering on the texture you're using means you're still sometimes blending to the black pixels outside of that. There are ways to fix that with a dilation pass, which Unity's "Alpha is Transparency" option for imported textures handles, or can be very slowly implemented in c#.

    But...
    No, you can't have both. You also don't need both, because you can convert "SrcAlpha" to "One" by multiplying the color by the alpha in the shader.

    You can skip all of the c# related stuff by changing the shader to
    Blend One OneMinusSrcAlpha
    and modifying the end to be like this:
    Code (csharp):
    1. fixed4 col = colTex * i.color;
    2. col.rgb *= i.color.a;
    3. col.rgba *= tex2D(_GUIClipTexture, i.clipUV).a;
    Also I only just notice that shader is already doing the gamma conversion I mentioned before. So they finally added that themselves.
     
  27. MechaWolf99

    MechaWolf99

    Joined:
    Aug 22, 2017
    Posts:
    105
    Ah, I did notice that it worked with point sampling, that makes sense.
    That works great! Thank you so much for your help!! :D
     
  28. MechaWolf99

    MechaWolf99

    Joined:
    Aug 22, 2017
    Posts:
    105
    Hey, sorry to come back to this. I went to use the tool that I made in a project using URP and of course it broke (thanks Unity), as it would show a semi transparent background instead of fully transparent.
    Reverting the shader to using
    Blend SrcAlpha OneMinusSrcAlpha
    works, but then it is back to having the outline around it. I was wondering if you had any thoughts on this?

    I totally get if you don't, but thought it was worth asking at least. :)
     
unityunity