How to do anti-alias a palette swapping shader?

I have a palette swapping shader that maps the red channel of an input texture to positions in a palette (e.g. red 123 means "use color of pixel 123 in the palette").

Code (CSharp):

1. half4 base = tex2D(_MainTex, uv);
2. half2 paletteUV = half2(base.r + 1 / 512, paletteY);
3. half3 rgb = tex2D(_Palette, paletteUV);
4. return half4(rgb, base.a);

It works fine, except for filtering: When I enable any filtering on the input texture, the shader will blend the palette indices together, so when I have an area that uses palette color 80 right next to one with palette color 250, the border between them will end up as palette color 165, which is most likely not what I want.

So I need to use point sampling, but that creates heavy aliasing. What I really want is to basically create a temporary texture in the native resolution of the base texture where I apply the palette, and then use filtering on that to blend the final colors together (instead of blending the palette indices). Is there a way to do this directly in the shader?

If not, what is a good way to implement my own texture sampling in the shader? So I could fetch, let's say, 5 samples from the base texture, look up each of them in the palette, and then blend the colors together.

Note that this is a sprite shader that needs to be batchable across instances that use different input textures and palettes (both are sprites). So I don't think I can use a RenderTexture.

 

Hi,
Can you show some image what kind of artifacts you're having, as I can't exactly understand what looks wrong etc. I'm not sure how replacing colors in a limited palette / pixel art image would cause bad aliasing, unless colors are radically different in hue/value etc. But just a small piece of image would help a lot. Sampling your palette texture and sampling your actual image don't have to use the same sampling method in lockstep... You could use point sampling for your palette and linear sampling for the actual image. But maybe this is something you already did.

"Own" texture sampling - nothing prevents you from doing 5 lookups, like -2,-1,0,1,2 and then take an average of those. But I don't see much point in that and it would waste extra texture reads. And that would most likely result into just the same look as having linear sampling used when you sample the palette texture.

But I'm not sure if I undestood correctly what you're trying to achieve...

 

The problem is that you cannot bilinear filter the palette indices: you have to read the four indices from your texture without, fetch the four colors from your palette texture, then bilinear filter the result manually in your shader. Use the TextureObject.Gather() function (https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-to-gather) to fetch the four red values that would be used in the bilinear operation so you don't need to read the _MainTex four times (you'll still have to read from your palette texture four times, there's no way around that).

There's an example here:
https://www.gamedev.net/forums/topi...lation-in-hlsl-using-gather-functions/5403870

 

Thank you, KokkuHub, that's exactly what I've been looking for!

@Olmi: That's what I've ended up doing now, and I didn't like it for exactly those reasons, but I think KokkuHub's solution should be cheaper.

 

Ahhh, I celebrated too early. Gather requires shader model 4.1, this is a mobile project and I need to support OpenGL ES 2. So I guess I need to go back to this: https://developer.nvidia.com/gpugem...ng/chapter-27-advanced-high-quality-filtering

 

Yeah, without gather support, you'll have to sample the texture 4 times (not 5: you only need the current texel and the ones at the right, bottom, and bottom-right), unless you think of a different way to color your sprites which doesn't involve using a texture to index into another.

 

The 5 sample version is for doing additional filtering over and above bilinear filtering. For replicating bilinear filtering you only want 4 samples. The easiest way to do this is by setting the texture to use point filtering and offset the sample uv position by +/- half a texel. That’ll get you the 4 samples you need to blend between. You also need to calculate the current uv’s fractional value to know how much to blend between each to get the proper bilinear result.

Code (csharp):

1. // sample 4 positions offset by half a texel to get the 4 texel values
2. float2 halfTexel = _MainTex_TexelSize.xy * 0.5;
3. fixed ll_base = tex2D(_MainTex, i.uv + float2(-halfTexel.x, -halfTexel.y).r;
4. fixed lr_base = tex2D(_MainTex, i.uv + float2( halfTexel.x, -halfTexel.y).r;
5. fixed ul_base = tex2D(_MainTex, i.uv + float2(-halfTexel.x,  halfTexel.y).r;
6. fixed ur_base = tex2D(_MainTex, i.uv + float2( halfTexel.x,  halfTexel.y).r;
7.  
8. // get color from palette for each base pixel
9. fixed4 ll_rgb = tex2D(_Palette, float2(ll_base + (1/512), 0.5);
10. fixed4 lr_rgb = // etc
11.  
12. // calculate the bilinear blended color
13. float2 uv_blend = frac(i.uv * _MainTex_TexelSize.zw + 0.5);
14. fixed4 rgb = lerp(
15.     lerp(ll_rgb, lr_rgb, uv_blend.x),
16.     lerp(ul_rgb, ur_rgb, uv_blend.x),
17.     uv_blend.y);