Writing a shader to replace/mask colors on a map image with textures?

Hey everyone!

So I'm currently working on a character customization system for my game, where the player can swap clothing models and material presets to get what they want. However, I've run into a bit of an issue with my textures--to do what I want to do, I'd have to create thousands and thousands of images to be swapped out.

This isn't ideal, so the best idea I've been able to come up with involves a custom shader. For each of my clothing models, I have them fully UV-mapped along with a color map (such as what would be used in modeling program) where each region that I want to set a different texture to has a separate color. The most detailed piece of clothing I have has 8 color regions, so I need to write a shader that supports up to 8 textures to be masked.

​

​

Splitting up the meshes into separate regions was another idea I thought of, but it won't be efficient because I have too many separate objects and each are already configured and weight painted. On top of that, I wanted to keep the options that come with the Standard shader (normal/metallic/ambient maps, smoothness, and main color) and simply add this functionality. I used a copy of the Standard shader to add the options I want along with these variables, so I can simply pick the region to replace based on its color.

​

My only issue is that I know almost nothing about writing shaders; so far, I've only had to use the shaders that ship with Unity until now. What would be the best way of implementing this into the existing shader I have? Or, better yet, is there already a shader that exists that can accomplish this?

Thank you everyone! Sorry for the long post

 

There's no shader that I know of that does exactly what you're asking for. It's very similar conceptually to how a terrain shader works, and there are plenty of color replacement shader examples.

So start with, you can use a Surface Shader to easily interface with the Standard lighting model which will let you focus on the albedo color generation. Unity will create one for you if you right click in the project view and create a new Surface Shader, though you'll need to add additional properties beyond the _MainTex. That's the easy part.

The second part is layering the different textures. If you have a single float value mask (I'll get to that in a moment) you can lerp() from one texture's color to another.

Code (csharp):

1. fixed4 col1 = tex2D(_tex1, IN.uv_MainTex);
2. fixed4 col2 = tex2D(_tex2, IN.uv_MainTex);
3. fixed4 col3 = tex2D(_tex3, IN.uv_MainTex);
4. fixed4 col4 = tex2D(_tex4, IN.uv_MainTex);
5.  
6. fixed4 col = lerp(col1, col2, mask2);
7. col = lerp(col, col3, mask3);
8. col = lerp(col, col4, mask4);

The problem is going to be the masks themselves. Yes, you can extract a mask from your texture by seeing if a pixel's color matches the mask color defined, but you're going to run into issues on the edges between the masks if you do this due to texture filtering. By default, Unity sets textures to use bilinear filtering, which interpolates between texel colors rather than changing abruptly.

This has a lot of benefits in the general case, but for masks like the one you have it's going to be a problem. Because you're testing to see if a sampled pixel is a specific color, those pixels that are blending between colors won't match any one color. If you test the pixel color against a distance from the mask color (like chroma keying), you'll get some false overlap. You could use point filtering on your mask texture, but that'll be kind of ugly.

The best answer is to break up your masks into individual color channels in multiple textures. Mask 1 is anywhere that's red in Mask Texture 1, Mask 2 is anywhere that's green in Mask Texture 1, etc. For 8 separate masks, you'd need two RGBA textures, though Terrain systems usually use a hack of anywhere that's black is considered a base texture, aka "Mask 1", so for 8 layers you'd only need an RGBA (DXT5) texture and an RGB (DXT1) texture.

This still won't be perfect though. You'll need to make sure your per channel masks slightly overlap along their edges otherwise you'll potentially get odd seams.
https://medium.com/@bgolus/the-team-color-problem-b70ec69d109f


There's one more option. Don't use a shader to do this. Or perhaps use your shader to render to a render texture and use that.

The problem with the seams and needing to use multiple single component mask textures is down to interpolation. If you could do everything accurate to the original texture's pixels it wouldn't have that problem. The idea would be to render out your "composite" shader to a render texture with the same resolution as your input textures, then use that render texture on the character. This would remove the issue of interpolation as you're not sampling the mask texture "between" the texel centers!

If you just have a single hero character that you're doing this with, then you can just use the resulting render texture without much worry. If you're going to be doing this on multiple characters in a scene, you may need to read the render texture back into a Texture2D and then do real time compression and use that texture.

 

Hey! I'm sorry about the super late reply, I’ve been a bit behind on some things.

Thanks a lot for taking the time to write such a detailed post! This is really helpful for trying to make sure they don’t overlap, but if you don’t mind me asking, I’m still not entirely sure how I can pass all my different textures into the one image. I was able to use the custom shader that Unity creates automatically to get started, but I couldn’t seem to get it to work with my metallic and ambient maps, which is why I tried working off of the downloadable Standard shader here on the forums. This worked great, acting just like the normal Standard shader, so I went ahead and defined my properties for each of the textures and the colors they would apply to.

Code (JavaScript):

1. _MainTex("Original Texture", 2D) = "white" {}            // Most likely will be ignored, as other textures will layer over this and fully cover it.
2.  
3. _MaskTex("Mask Texture", 2D) = "white" {}            // Stores the main texture that is used as a reference for what region each texture masks to depending on the color.
4.  
5. _MaskColor1("Mask Color 1", Color) = (1,1,1,1)            // Stores the color of what to mask. Eyedropper in Inspector will set each of these colors.
6. _MaskReplace1("Replacement Texture", 2D) = "white" {}    // Stores the texture that will replace the mask color.
7.  
8. _MaskColor2("Mask Color 2", Color) = (1,1,1,1)
9. _MaskReplace2("Replacement Texture", 2D) = "white" {}
10.  
11. _MaskColor3("Mask Color 3", Color) = (1,1,1,1)
12. _MaskReplace3("Replacement Texture", 2D) = "white" {}
13.  
14. _MaskColor4("Mask Color 4", Color) = (1,1,1,1)
15. _MaskReplace4("Replacement Texture", 2D) = "white" {}
16.  
17. _MaskColor5("Mask Color 5", Color) = (1,1,1,1)
18. _MaskReplace5("Replacement Texture", 2D) = "white" {}
19.  
20. _MaskColor6("Mask Color 6", Color) = (1,1,1,1)
21. _MaskReplace6("Replacement Texture", 2D) = "white" {}
22.  
23. _MaskColor7("Mask Color 7", Color) = (1,1,1,1)
24. _MaskReplace7("Replacement Texture", 2D) = "white" {}
25.  
26. _MaskColor8("Mask Color 8", Color) = (1,1,1,1)
27. _MaskReplace8("Replacement Texture", 2D) = "white" {}

Filtering may very well be a significant issue, but my main problem is that I’m not sure how I would go about telling Unity to only display each texture over its correct corresponding color—I’m not really experienced in shader coding . I’ve seen this accomplished with one texture before, but not with more than one on different parts of a model. I assume that I would have to pass these textures into my _MaskTex through only the albedo property, but I’m not entirely sure where or how to do this part. I’m working in a deferred setup, so I know it has to fit somewhere in here within the custom shader (although it happens to reference the UnityStandardCore shader as well, so that may require editing):

Code (JavaScript):

1. // ------------------------------------------------------------------
2.         //  Deferred pass
3.         Pass
4.         {
5.             Name "DEFERRED"
6.             Tags { "LightMode" = "Deferred" }
7.  
8.             CGPROGRAM
9.             #pragma target 3.0
10.             #pragma exclude_renderers nomrt
11.  
12.  
13.             // -------------------------------------
14.  
15.             #pragma shader_feature _NORMALMAP
16.             #pragma shader_feature _ _ALPHATEST_ON _ALPHABLEND_ON _ALPHAPREMULTIPLY_ON
17.             #pragma shader_feature _EMISSION
18.             #pragma shader_feature _METALLICGLOSSMAP
19.             #pragma shader_feature _ _SMOOTHNESS_TEXTURE_ALBEDO_CHANNEL_A
20.             #pragma shader_feature _ _SPECULARHIGHLIGHTS_OFF
21.             #pragma shader_feature ___ _DETAIL_MULX2
22.             #pragma shader_feature _PARALLAXMAP
23.  
24.             #pragma multi_compile_prepassfinal
25.             #pragma multi_compile_instancing
26.             // Uncomment the following line to enable dithering LOD crossfade. Note: there are more in the file to uncomment for other passes.
27.             //#pragma multi_compile _ LOD_FADE_CROSSFADE
28.  
29.             #pragma vertex vertDeferred
30.             #pragma fragment fragDeferred
31.  
32.             #include "UnityStandardCore.cginc"
33.  
34.             ENDCG
35.         }
36. // ------------------------------------------------------------------

I’m not sure if there’s a certain way to accomplish this, but I think this is giving me the most difficulty right now. Unfortunately, I think a shader is my only solution in this case, as I have six different character models and each has about 80 different pieces of clothing that can be swapped. As for the filtering, I could very well separate each of my color masks into layers, but right now I don’t believe any of my masking textures have overlap between their colors—this may or may not be helpful with Unity’s filtering, but the pixels are only solid colors for each different region.

​

(That picture actually blurs it a little, as it's not blurry at all in the original file) I wish there was really an easier way of doing something like this. I have seen this done in modeling programs before, but only when the textures were not meant to be interchanged often. Still, thanks a lot for getting back so quickly earlier! Would you have an idea of how I could implement this?

 

Time to start learnings the basics of shaders!
https://www.alanzucconi.com/2015/06/10/a-gentle-introduction-to-shaders-in-unity3d/

Modifying the built in shader is a massive undertaking I would not recommend attacking as your first foray into shader writing. Doing something like adding a single extra texture requires modifying several files (at least 4, pontentially more). The Surface Shader that Unity generates when you make a new shader exists explicitly to allow you to add new features while still getting the benefits of the Standard shading model and Unity’s lighting system.

See the my entire previous post.

 

Hey!

You were absolutely right on that topic, haha. I figured that out pretty quickly, I had thought that starting with the base shader was smarter when really that was probably the dumbest decision I could have made. After going through a lot of the basic tutorials and trying to get an idea of how to setup everything up, I was able to figure it out after a lot of tinkering! I struggled at first with using too many samplers, but I also found out how to reuse one for each set of maps so that I could have even more than eight textures if I wanted to. Thanks so much for all of your help! I know I've got a lot to learn when it comes to shaders in the future. I've simply never worked with them before, so I'm sure my script isn't optimized at all, but I'm mainly going for functionality at this point. What's interesting is I never ran into a problem with filtering—the textures ended up staying in their respective regions, oddly enough. Not that I'm complaining, it's exactly what I wanted to do!

​

Still, thanks for taking the time to break things down for me! It took me a while, but I actually understand how a lot of the basics work now

 

Having the same problem now. I want to replace at least 5 colors. Could you tell us how you managed to do so, please? Is it necessary to split the texture into multiple parts or did you managed to pass all textures/colors through the albedo property?

 

To selectively change a color with another color;
* you first get the luminance value of the original color and define a tolerance value Range(0,1 or much smaller actually)
* do a check if the value original color channels are in between this tolerance (+- range)
* if it is, you change the luminance value to the color you want to change and return the value
this way you can change as many colors as you want instead of the regular rgb

If need to see code, you can check Sincity shader inside my TOZ Image Effects pack.