Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Stripping scriptable shader variants using ShaderVariantCollections?

Discussion in 'Shaders' started by jashan, Feb 13, 2019.

  1. jashan


    Mar 9, 2007
    I'm currently trying to implement shader variant stripping into our project - and the most obvious approach seemed to be using ShaderVariantCollections for that purpose. This would be a very simply workflow:
    1. Create all the ShaderVariantCollections for the project by playing the game in the editor, using Settings / Graphics / Save to Asset (after clearing / tracking the shaders and shader variants). Relevant documentation.
    2. Write an implementation of IPreprocessShaders.OnPreprocessShader() that uses those ShaderVariantCollections to strip out any shader keywords that are not used.
    This could probably also be really useful to completely remove shaders from the project that are not used in any of the ShaderVariantCollections.

    Unfortunately, it turns out that the Unity APIs that are available to Unity users don't cover that use case. ShaderVariantCollections has shaderCount and variantCount, methods to add, remove or clear all variants, or test if a given variant is already in there - but no way to access the actual shaders or shader variants.

    The ShaderVariantCollectionInspector accesses this information but I don't see a way to access this from our own scripts, e.g. a custom implementation of IPreprocessShaders.OnPreprocessShader().
  2. jashan


    Mar 9, 2007
    @hippocoder and/or @aliceingameland could you ping Christophe about this? I have a rather hacky solution working now by taking ShaderVariantCollectionInspector and adding a button into the custom inspector that basically creates my own custom ShaderVariantCollection that has all the info available that I need. But it would be nice to have a more solid approach (I could post about it on the alpha-list - but it's not really alpha-related, so I felt it's more appropriate to post it here). It really seems to me that something built-in that's based on ShaderVariantCollections would probably be the most elegant solution most of the time but I might be completely missing something.

    Also, it turns out that stripping scriptable shader variants doesn't help with the maximum number of shader keywords (256). So, one thing that I feel is also missing with this approach is a way to tell Unity to globally drop specific shader keywords right at the start of shader processing (and ideally even in the editor runtime). Something like "ignore shader keywords" in platform-specific project graphics settings might do the trick. It is often not feasible to change the shader files.
  3. jashan


    Mar 9, 2007
    It gets worse: I removed my IPreprocessShaders implementation to have a new "reference build" where everything is included. But the build-time is still 35 minutes (before, a build was almost two hours), and it seems that some of the shaders variants are still missing. I have deleted ShaderCache and ShaderCache.db, as well as the ScriptAssemblies just to be sure. Also, I don't get the log statements from my IPreprocessShaders implementation - but it very much looks as if I'm no longer getting those variants anymore.

    Are the results from IPreprocessShaders cached somewhere?
    Last edited: Jul 13, 2019
  4. KovaPx


    Jan 27, 2018
    Have you ever solved this problem? I have a similar idea and am clearing the variants collection for some shaders but they end up in the build anyway, wondering if anything is cached or not.
  5. havchr


    Jun 18, 2009
    Our implementation for this is :

    Code (CSharp):
    2. using System;
    3. using System.Collections.Generic;
    4. using System.IO;
    5. using Agens;
    6. using UnityEditor;
    7. using UnityEditor.Build;
    8. using UnityEditor.Callbacks;
    9. using UnityEditor.Rendering;
    10. using UnityEngine;
    11. using UnityEngine.Rendering;
    13. class ShaderVariantsStripperProject : IPreprocessShaders
    14. {
    16.     private ShaderStrippingSetting shaderStrippingSetting;
    17.     private static StreamWriter shaderStripLogFile;
    18.     public ShaderVariantsStripperProject()
    19.     {
    20.         var shaderStripSettings = Resources.LoadAll<ShaderStrippingSetting>("");
    21.         if (shaderStripSettings.Length == 0)
    22.         {
    23.             Debug.LogWarning("Did not find shader stripping settings asset , please create one. Compiling all shaders");
    24.         }
    25.         else
    26.         {
    27.             if (shaderStripSettings.Length > 1)
    28.             {
    29.                 Debug.LogWarning("Warning More than one shader strip setting asset -- only using first found");
    30.             }
    31.             shaderStrippingSetting = shaderStripSettings[0];
    32.         }
    33.     }
    34.     public int callbackOrder { get { return (int)ShaderVariantsStripperOrder.Project; } }
    35.     public bool KeepVariant(Shader shader, ShaderSnippetData snippet, ShaderCompilerData shaderVariant)
    36.     {
    37.         bool resultKeepVariant = true;
    38.         if (shaderStripLogFile == null)
    39.         {
    40.             string timeStamp = String.Format("{0}d_{1}m__{2}h_{3}m",
    41.                 DateTime.Now.Day, DateTime.Now.Month,
    42.                 DateTime.Now.Hour, DateTime.Now.Minute);
    43.             shaderStripLogFile = new StreamWriter("Stripped_ShaderLog_" + timeStamp + ".txt");
    44.         }
    46.         var shaderKeywords = shaderVariant.shaderKeywordSet.GetShaderKeywords();
    47.         string[] keywords = new string[shaderKeywords.Length];
    48.         int i = 0;
    49.         foreach (var shaderKeyword in shaderVariant.shaderKeywordSet.GetShaderKeywords())
    50.         {
    51.             keywords[i]=shaderKeyword.GetKeywordName();
    52.             i++;
    53.         }
    54.         ShaderVariantCollection.ShaderVariant variant = new ShaderVariantCollection.ShaderVariant(shader,snippet.passType,keywords);
    55.         if (shaderStrippingSetting != null)
    56.         {
    57. #if UNITY_IOS || UNITY_OSX
    58.         resultKeepVariant = shaderStrippingSetting.collectionOfShadersToKeepMetal.Contains(variant);
    60.         resultKeepVariant = shaderStrippingSetting.collectionOfShadersToKeepPC.Contains(variant);
    61. #endif
    62.         }
    63.         if (!resultKeepVariant)
    64.         {
    65.             string prefix = "not keepeing VARIANT: " + + " (";
    66.             if (snippet.passName.Length > 0)
    67.                 prefix += snippet.passName + ", ";
    68.             prefix += snippet.shaderType.ToString() + ") ";
    69.             string log = prefix;
    70.             for (int labelIndex = 0; labelIndex < keywords.Length; ++labelIndex)
    71.                 log += keywords[labelIndex] + " ";
    72.             shaderStripLogFile.Write(log + "\n");
    73.         }
    75.         return resultKeepVariant;
    76.     }
    77.     public void OnProcessShader(
    78.         Shader shader, ShaderSnippetData snippet, IList<ShaderCompilerData> shaderVariants)
    79.     {
    81.         int inputShaderVariantCount = shaderVariants.Count;
    82.         for (int i = 0; i < shaderVariants.Count; ++i)
    83.         {
    84.             bool keepVariant = KeepVariant(shader, snippet, shaderVariants[i]);
    85.             if (!keepVariant)
    86.             {
    87.                 shaderVariants.RemoveAt(i);
    88.                 --i;
    89.             }
    90.         }
    91.         if (shaderStripLogFile != null)
    92.         {
    93.             float percentage = (float)shaderVariants.Count / (float)inputShaderVariantCount * 100f;
    94.             shaderStripLogFile.Write("STRIPPING(" + snippet.shaderType.ToString() + ") = Kept / Total = " + shaderVariants.Count + " / " + inputShaderVariantCount + " = " + percentage + "% of the generated shader variants remain in the player data\n");
    95.         }
    96.     }
    97.      [PostProcessBuild(1)]
    98.     public static void OnPostprocessBuild(BuildTarget target, string pathToBuiltProject) {
    99.         if (shaderStripLogFile != null)
    100.         {
    101.             shaderStripLogFile.Close();
    102.             shaderStripLogFile = null;
    103.         }
    105.     }
    106. }
    107. #endif
    It seems to be working, we went from a memory footprint of 200 MB of shaderlab to 50 MB , which seems much more sane. However, the process of stripping the shaders seems to be very slow. 30 minutes added build time or so for metal.

    The next thing I am going to try is to use the log file of what we are stripping out - because maybe there's a configuration of the project that can be made, to not have Unity bundle that many variants to begin with.

    Also, we are going to try to use the ShaderVariant collection to generate a more effecient strip file.
    The hunch: it might be faster to use the shaderVariants file to generate a list of shaders&variants that probably should be stripped, and then directly strip based on that list - instead of calling shaderVariantCollection.Contains - which probably is the slow part of this mix.

    I'll report once I know more.

    Update : We might be seeing similar caching based behaviour. Don't really know what's going on.

    Update : also - we were returning Package in the shader stripping order, so that's why it was so slow, I've updated the code to change it into Project
    Last edited: Apr 16, 2019
    jashan likes this.
  6. havchr


    Jun 18, 2009
    Something broke in our project big time after having the stripper run. The iOS build just crashes on startup - it seems to go to a shader - that is looking for a fallback and then into lza-decompress-code which finally crashes.

    I'm deleting the Library folder now and rebuilding without stripping code - hope to have a build that runs again. It would be really nice to get the shader stripping features working. The variant explosion is so annoying (I understand why shaders are built like this though), especially when it's not needed for the final product.
  7. jashan


    Mar 9, 2007
    It would really be great if someone from UT could chime in. I still think this is a great and useful feature - but not if it breaks things the way it currently seems to break things.
  8. fherbst


    Jun 24, 2012
    I would also be very interested in this. It's just not feasible to have ~100MB of shader code in an Android build when I could define exactly which variants I actually want to use.
  9. havchr


    Jun 18, 2009
    We got it quite stable now and it has reduced our memory footprint by maybe 250 MB - our build times are good as well. I will write a blog post and share our code soon. I think maybe our project was configured in a very unfortunate way in regards to shader variants, because our team is small and no one had complete view over these things until we had to figure out a way to reduce them.

    Also the weird issues we experienced at the beginning are no longer an issue - we have since upgraded unity as well as done some changes to our code : we are using the 2018.4 LTS version.
  10. tspk91


    Nov 19, 2014
    Any news on this? ShaderVariantCollections still don't expose their variant lists.
    BradZoob and jashan like this.
  11. DonCornholio


    Feb 27, 2017
    Hi there ! Any Update on this? We're a small team as well and would love to see your solution to this.
  12. DonCornholio


    Feb 27, 2017
    Hey there ! I just found this amazing asset on github :
    This will basically give you all the control and flexibility you need and is very easy to use.
    I don't know if this is the solution havchr promised but, and i still would love to read that blogpost, but this solves my problems so far :)
  13. havchr


    Jun 18, 2009
    I was about to finally post my code but I would rather suggest using SixWays UnityShaderStripper instead - it is much better.