How to keep sprites sharp and crisp, even while rotating [Solved]

How can I maintain crisp, sharp edges on sprites, while avoiding the aliasing caused by rotation?

None of the things I've tried produce a pixel-perfect non-rotated block, while allowing rotation without aliasing. (Row 5 is my best effort so far)

Here's a sample project: https://github.com/trenthm/Maintaining-Sharpness-Without-Pixelation

Here's my StackExchange question (has some more details): https://gamedev.stackexchange.com/questions/169423/unity-how-to-keep-sprites-sharp-and-crisp-even-while-rotating

Ideally, this is what it would look like

From Photoshop, not Unity.

- Flawless edges, smooth corners, and very faint aliasing when rotated.​

Screenshot from Github project:

 

The next things I was thinking about trying were:

1. Custom mip map level management (both creation of mip maps and which ones are displayed)
2. Tinkering around shader solutions

 

Shouldn’t need mip maps, assuming your sprite is showing at the same resolution on screen as the original sprite texture. Otherwise, yes, mip maps are the appropriate solution. But these will add some blurring. Trilinear filtering can help, as can setting a mip bias (has to be done via a custom script or shader).

If your sprite touches the edge of the quad geometry, then when rotated you’ll be limited to the aliasing of basic geometry. Enabling MSAA can help, but padding is much cheaper.

My usual solution is to add a few pixels of padding, and then do super sampling in the shader on a mip mapped texture using a biased sample. That’ll produce results basically identical to Photoshop. The other option is to use a SDF shape.

 

I really appreciate the answer, @bgolus – And sorry for all the questions! I'm in a bind here with this problem, and you seem to really know how to fix it.

"My usual solution is to add a few pixels of padding, and then do super sampling in the shader on a mip mapped texture using a biased sample."

• This sounds awesome. It's a bit over my head, though. (I noticed you recommended the same thing here: https://forum.unity.com/threads/dow...s-image-editing-software.553807/#post-3669721)

Since you said "usual solution," I'm guessing you've done this a time or two before. So, maybe you have a code snippet lying around on how to achieve SS "on a mip mapped texture using a biased sample"? I feel pretty lost here.

Does "biased sample" just mean a texture has it's mip map bias set in the Unity editor? Meaning all the shader does is perform SSAA, and Unity provides the "biased sample" automatically?

[Edit]

After a bit more digging I found http://inverseblue.com/?p=287 – So, I'm thinking that what you're suggesting is that I use tex2Dbias and run a super sampling algorithm on the higer-res texture (via a negative bias in tex2Dbias).

Is that right? And, if so, should the bias be as high as possible to sample to original highest-res texture (i.e. -10 bias), or just the one higher mip map level (i.e. -1 bias).

(I'm hoping you have a shader snippet you could share lol – not that I'm trying to be a cheater or anything )

 

Been doing VR related stuff, which is super sensitive to both aliasing and texture blurring of images with sharp details, like gameplay icons or text. So yeah, I've experimented a lot with various techniques for improving that.

I think I've posted a version of the code I use somewhere, but I don't remember when or where and my googlefu is falling me. So I'll post it again here later once I get a chance.

But yes, tex2Dbias is part of it. I think I use -0.75 these days, as a full -1 is still a little aliased for my use.

 

Here's an example shader that does super sampling on a texture using 3 different techniques, as well as just biasing the mip:

Spoiler: SuperSampledTexture.shader

Code (CSharp):

1. Shader "Custom/SuperSampled Texture"
2. {
3.     Properties {
4.         [NoScaleOffset] _MainTex ("Texture", 2D) = "white" {}
5.         _Bias ("Mip Bias", Range(-4, 4)) = -0.75
6.         [KeywordEnum(Off, 2x2 RGSS, 8x Halton, 16x16 OGSS)] _SuperSampling ("Super Sampling Technique", Float) = 1
7.         _AAScale ("AA Pixel Width", Range(0.75, 10.0)) = 1.25
8.     }
9.  
10.     SubShader {
11.         Tags { "Queue"="Transparent" "RenderType"="Transparent" }
12.  
13.         Pass {
14.             Blend SrcAlpha OneMinusSrcAlpha
15.             ZWrite Off
16.  
17.             CGPROGRAM
18.             #pragma vertex vert
19.             #pragma fragment frag
20.             #include "UnityCG.cginc"
21.  
22.             #pragma shader_feature _ _SUPERSAMPLING_2X2_RGSS _SUPERSAMPLING_8X_HALTON _SUPERSAMPLING_16X16_OGSS
23.  
24.             half4 tex2DSS(sampler2D tex, float2 uv, float bias, float aascale)
25.             {
26.                 // get uv derivatives, optionally scaled to reduce aliasing at the cost of clarity
27.                 // used by all 3 super sampling options
28.                 float2 dx = ddx(uv * aascale);
29.                 float2 dy = ddy(uv * aascale);
30.              
31.                 half4 col = 0;
32.  
33.             #if defined(_SUPERSAMPLING_2X2_RGSS)
34.                 // MSAA style "four rooks" rotated grid super sampling
35.                 // samples the texture 4 times
36.  
37.                 float2 uvOffsets = float2(0.125, 0.375);
38.  
39.                 col += tex2Dbias(tex, float4(uv + uvOffsets.x * dx + uvOffsets.y * dy, 0, bias));
40.                 col += tex2Dbias(tex, float4(uv - uvOffsets.x * dx - uvOffsets.y * dy, 0, bias));
41.                 col += tex2Dbias(tex, float4(uv + uvOffsets.y * dx - uvOffsets.x * dy, 0, bias));
42.                 col += tex2Dbias(tex, float4(uv - uvOffsets.y * dx + uvOffsets.x * dy, 0, bias));
43.  
44.                 col *= 0.25;
45.             #elif defined(_SUPERSAMPLING_8X_HALTON)
46.                 // 8 points from a 2, 3 Halton sequence
47.                 // similar to what TAA uses, though they usually use more points
48.                 // samples the texture 8 times
49.                 // better quality for really fine details
50.  
51.                 float2 halton[8] = {
52.                     float2(1,-3) / 16.0,
53.                     float2(-1,3) / 16.0,
54.                     float2(5,1) / 16.0,
55.                     float2(-3,-5) / 16.0,
56.                     float2(-5,5) / 16.0,
57.                     float2(-7,-1) / 16.0,
58.                     float2(3,7) / 16.0,
59.                     float2(7,-7) / 16.0
60.                 };
61.  
62.                 for (int i=0; i<8; i++)
63.                     col += tex2Dbias(tex, float4(uv + halton[i].x * dx + halton[i].y * dy, 0, bias));
64.  
65.                 col *= 0.125;
66.             #elif defined(_SUPERSAMPLING_16X16_OGSS)
67.                 // brute force ground truth 16x16 ordered grid super sampling
68.                 // samples the texture 256 times! you should not use this!
69.                 // does not use tex2Dbias, but instead always samples the top mip
70.  
71.                 float gridDim = 16;
72.                 float halfGridDim = gridDim / 2;
73.  
74.                 for (float u=0; u<gridDim; u++)
75.                 {
76.                     float uOffset = (u - halfGridDim + 0.5) / gridDim;
77.                     for (float v=0; v<gridDim; v++)
78.                     {
79.                         float vOffset = (v - halfGridDim + 0.5) / gridDim;
80.                         col += tex2Dlod(tex, float4(uv + uOffset * dx + vOffset * dy, 0, 0));
81.                     }
82.                 }
83.  
84.                 col /= (gridDim * gridDim);
85.             #else
86.                 // no super sampling, just bias
87.  
88.                 col = tex2Dbias(tex, float4(uv, 0, bias));
89.             #endif
90.                 return col;
91.             }
92.  
93.             sampler2D _MainTex;
94.             float _Bias;
95.             float _AAScale;
96.  
97.             struct v2f {
98.                 float4 pos : SV_Position;
99.                 float2 uv : TEXCOORD0;
100.             };
101.  
102.             void vert(appdata_base v, out v2f o)
103.             {
104.                 o.pos = UnityObjectToClipPos(v.vertex);
105.                 o.uv = v.texcoord;
106.             }
107.  
108.             half4 frag(v2f i) : SV_Target
109.             {
110.                 return tex2DSS(_MainTex, i.uv, _Bias, _AAScale);
111.             }
112.  
113.             ENDCG
114.         }
115.     }
116. }

And here's a texture I made to test it with:

Spoiler: TextTexture.png

And here's what that shader looks like in use:

​

 

Wow. I can't wait to try it out! Huge thank you for the effort! Will report back...

 

Things are working satisfactorily! All thanks to you, @bgolus !!

You have been invaluable, man. And based on all the times your name comes up when I comb the forums and see all the help you've given people, I'm sure I'm not the only one that owes you a debt of gratitude.

Some things I found:

• The non-rotated edges are not perfect, but they're reaall close!
• Of the variations you provided, 2X2 RGSS looks great and works fine on mobile.
  
• Things look best with no mip maps at all. Even your test texture with the text on it looks much sharper with SSAA and no mip maps.
  • This doesn't make a ton of sense to me, but the edges remain crisper, and the anti aliasing that SSAA produces alone is great. (Mip maps make things blurrier with SSAA)
• I modified the UI shader and TextMesh Pro sprite shader to use SSAA and even with 512x512 textures are scaled down very small is looks very sharp. It's like magic.

A couple questions:

1. If SSAA is averaging pixel colors, why does it produce such sharp edges? (Where a 100% transparent pixel is adjacent 100% colored pixel.) I would think the edges would have at least 1 pixel of half-ish transparency.
2. Why use mip maps with this at all, based on my findings?

TextureSSAA.cginc:

Code (CSharp):

1. /*
2.  
3.     Original SSAA code supplied by bgolus, via Unity Forums, March 2019
4.  
5. */
6.  
7. #ifndef TEXTURE_SSAA_INCLUDED
8. #define TEXTURE_SSAA_INCLUDED
9.  
10. half4 col;
11. // static const float DEFAULT_BIAS = -0.75;
12. static const float2 UV_OFFSETS = float2(0.125, 0.375);
13.  
14. // fixed4 Tex2DSS(sampler2D tex, float2 uv, float bias)
15. fixed4 Tex2DSS(sampler2D tex, float2 uv)
16. {
17.     // get uv derivatives
18.     float2 dx = ddx(uv);
19.     float2 dy = ddy(uv);
20.  
21.     col = 0;
22.  
23.     // MSAA style "four rooks" rotated grid super sampling
24.     // samples the texture 4 times
25.     // col += tex2Dbias(tex, float4(uv + UV_OFFSETS.x * dx + UV_OFFSETS.y * dy, 0, bias));
26.     // col += tex2Dbias(tex, float4(uv - UV_OFFSETS.x * dx - UV_OFFSETS.y * dy, 0, bias));
27.     // col += tex2Dbias(tex, float4(uv + UV_OFFSETS.y * dx - UV_OFFSETS.x * dy, 0, bias));
28.     // col += tex2Dbias(tex, float4(uv - UV_OFFSETS.y * dx + UV_OFFSETS.x * dy, 0, bias));
29.  
30.     // No longer using mip maps, so no bias needed
31.     col += tex2D(tex, float2(uv + UV_OFFSETS.x * dx + UV_OFFSETS.y * dy));
32.     col += tex2D(tex, float2(uv - UV_OFFSETS.x * dx - UV_OFFSETS.y * dy));
33.     col += tex2D(tex, float2(uv + UV_OFFSETS.y * dx - UV_OFFSETS.x * dy));
34.     col += tex2D(tex, float2(uv - UV_OFFSETS.y * dx + UV_OFFSETS.x * dy));
35.  
36.     col *= 0.25;
37.  
38.     return col;
39. }
40.  
41. #endif // TEXTURE_SSAA_INCLUDED

SSAA_Sprite.shader:

Code (CSharp):

1. Shader "Custom/SSAA Sprite"
2. {
3.     Properties {
4.         [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {} // Used to have [NoScaleOffset] set
5.         [HideInInspector] _Flip ("Flip", Vector) = (1,1,1,1)
6.     }
7.     SubShader {
8.         Tags
9.         {
10.             "Queue"="Transparent"
11.             "IgnoreProjector"="True"
12.             "RenderType"="Transparent"
13.             "PreviewType"="Plane"
14.         }
15.  
16.         Blend SrcAlpha OneMinusSrcAlpha // Alpha blending
17.         Cull Off
18.         Lighting Off
19.         ZWrite Off
20.         Pass {
21.             CGPROGRAM
22.             #pragma vertex vert
23.             #pragma fragment frag
24.             #include "UnityCG.cginc"
25.             #include "TextureSSAA.cginc"
26.  
27.             fixed2 _Flip;
28.             sampler2D _MainTex;
29.             inline float4 UnityFlipSprite(in float3 pos, in fixed2 flip) // From UnitySprites.cginc
30.             {
31.                 return float4(pos.xy * flip, pos.z, 1.0);
32.             }
33.  
34.             struct v2f {
35.                 float4 pos : SV_Position;
36.                 float2 uv : TEXCOORD0;
37.                 fixed4 color : COLOR;
38.             };
39.  
40.             struct appdata_t
41.             {
42.                 float4 vertex   : POSITION;
43.                 fixed4 color    : COLOR;
44.                 float2 texcoord : TEXCOORD0;
45.             };
46.             void vert(appdata_t v, out v2f o)
47.             {
48.                 o.pos = UnityFlipSprite(v.vertex, _Flip);
49.                 o.pos = UnityObjectToClipPos(o.pos);
50.                 o.uv = v.texcoord;
51.                 o.color = v.color;
52.             }
53.             half4 frag(v2f i) : SV_Target
54.             {
55.                 return Tex2DSS(_MainTex, i.uv) * i.color;
56.             }
57.             ENDCG
58.         }
59.     }
60.  
61.     Fallback "Sprites/Default"
62. }

UI_Default_SSAA.shader:

Code (CSharp):

1. Shader "Custom/UI/UI-Default-SSAA"
2. {
3.     Properties
4.     {
5.         [PerRendererData] _MainTex ("Sprite Texture", 2D) = "white" {}
6.         _Color ("Tint", Color) = (1,1,1,1)
7.  
8.         _StencilComp ("Stencil Comparison", Float) = 8
9.         _Stencil ("Stencil ID", Float) = 0
10.         _StencilOp ("Stencil Operation", Float) = 0
11.         _StencilWriteMask ("Stencil Write Mask", Float) = 255
12.         _StencilReadMask ("Stencil Read Mask", Float) = 255
13.  
14.         _ColorMask ("Color Mask", Float) = 15
15.  
16.         [Toggle(UNITY_UI_ALPHACLIP)] _UseUIAlphaClip ("Use Alpha Clip", Float) = 0
17.     }
18.  
19.     SubShader
20.     {
21.         Tags
22.         {
23.             "Queue"="Transparent"
24.             "IgnoreProjector"="True"
25.             "RenderType"="Transparent"
26.             "PreviewType"="Plane"
27.             "CanUseSpriteAtlas"="True"
28.         }
29.  
30.         Stencil
31.         {
32.             Ref [_Stencil]
33.             Comp [_StencilComp]
34.             Pass [_StencilOp]
35.             ReadMask [_StencilReadMask]
36.             WriteMask [_StencilWriteMask]
37.         }
38.  
39.         Cull Off
40.         Lighting Off
41.         ZWrite Off
42.         ZTest [unity_GUIZTestMode]
43.         Blend SrcAlpha OneMinusSrcAlpha
44.         ColorMask [_ColorMask]
45.  
46.         Pass
47.         {
48.             Name "Default"
49.         CGPROGRAM
50.             #pragma vertex vert
51.             #pragma fragment frag
52.             #pragma target 2.0
53.  
54.             #include "UnityCG.cginc"
55.             #include "UnityUI.cginc"
56.             #include "TextureSSAA.cginc"
57.  
58.             #pragma multi_compile __ UNITY_UI_CLIP_RECT
59.             #pragma multi_compile __ UNITY_UI_ALPHACLIP
60.  
61.             struct appdata_t
62.             {
63.                 float4 vertex   : POSITION;
64.                 float4 color    : COLOR;
65.                 float2 texcoord : TEXCOORD0;
66.                 UNITY_VERTEX_INPUT_INSTANCE_ID
67.             };
68.  
69.             struct v2f
70.             {
71.                 float4 vertex   : SV_POSITION;
72.                 fixed4 color    : COLOR;
73.                 float2 texcoord  : TEXCOORD0;
74.                 float4 worldPosition : TEXCOORD1;
75.                 UNITY_VERTEX_OUTPUT_STEREO
76.             };
77.  
78.             sampler2D _MainTex;
79.             fixed4 _Color;
80.             fixed4 _TextureSampleAdd;
81.             float4 _ClipRect;
82.             float4 _MainTex_ST;
83.  
84.             v2f vert(appdata_t v)
85.             {
86.                 v2f OUT;
87.                 UNITY_SETUP_INSTANCE_ID(v);
88.                 UNITY_INITIALIZE_VERTEX_OUTPUT_STEREO(OUT);
89.                 OUT.worldPosition = v.vertex;
90.                 OUT.vertex = UnityObjectToClipPos(OUT.worldPosition);
91.  
92.                 OUT.texcoord = TRANSFORM_TEX(v.texcoord, _MainTex);
93.  
94.                 OUT.color = v.color * _Color;
95.                 return OUT;
96.             }
97.  
98.             fixed4 frag(v2f IN) : SV_Target
99.             {
100.                 // half4 color = (tex2D(_MainTex, IN.texcoord) + _TextureSampleAdd) * IN.color;
101.                 half4 color = (Tex2DSS(_MainTex, IN.texcoord) + _TextureSampleAdd) * IN.color;
102.  
103.                 #ifdef UNITY_UI_CLIP_RECT
104.                 color.a *= UnityGet2DClipping(IN.worldPosition.xy, _ClipRect);
105.                 #endif
106.  
107.                 #ifdef UNITY_UI_ALPHACLIP
108.                 clip (color.a - 0.001);
109.                 #endif
110.  
111.                 return color;
112.             }
113.         ENDCG
114.         }
115.     }
116. }
117.  

 

If your sprite isn't being scaled to less than half its original pixel size, the SSAA should produce near perfect results. Below that and mip maps w/ a bias of -1 or higher can still be helpful in avoiding aliasing on small details, but for something like your example solid red block it won't really matter either way. The sampling pattern I'm using is based off of 4x MSAA, and that's being used to apply anti-aliasing on geometry edges which is always a binary "on" or "off" in terms of the the individual samples, so bilinear filtering on the texture is just a bonus. You could probably set the texture to point sampled and still get good results with that super sampling.

 

Point does produce great results, with bilinear maybe seeming a bit smoother (without blurring)

I am using SSAA for virtually ALL of my graphics now (which are originally vector-based), not just blocks. And the clarity is amazing. (The blocks just show off the aliasing the best, and blocks are a big part of the game)

I wish vector graphics worked without relying on MSAA, since I could reduce my game size by ~90% by switching from textures to svg.

Just for clarity, with mip maps enabled and SSAA, the graphics got muddy, even when graphics were far scaled down very little, and the bias was very negative. (Since my graphics are vector-style, muddiness is extremely easy to see, esp. on detailed graphics)

• I'm really mainly saying this because I didn't expect it, since I thought the large negative bias would force it to use the original texture.

(For future reference, I just asked about SSAA on the vector package thread today – https://forum.unity.com/threads/vector-graphics-preview-package.529845/page-11#post-4397302)

Huge thanks, again, @bgolus!