Search Unity

Stripping scriptable shader variants using ShaderVariantCollections?

Discussion in 'Shaders' started by jashan, Feb 13, 2019.

  1. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    3,085
    I'm currently trying to implement shader variant stripping into our project - and the most obvious approach seemed to be using ShaderVariantCollections for that purpose. This would be a very simply workflow:
    1. Create all the ShaderVariantCollections for the project by playing the game in the editor, using Settings / Graphics / Save to Asset (after clearing / tracking the shaders and shader variants). Relevant documentation.
    2. Write an implementation of IPreprocessShaders.OnPreprocessShader() that uses those ShaderVariantCollections to strip out any shader keywords that are not used.
    This could probably also be really useful to completely remove shaders from the project that are not used in any of the ShaderVariantCollections.

    Unfortunately, it turns out that the Unity APIs that are available to Unity users don't cover that use case. ShaderVariantCollections has shaderCount and variantCount, methods to add, remove or clear all variants, or test if a given variant is already in there - but no way to access the actual shaders or shader variants.

    The ShaderVariantCollectionInspector accesses this information but I don't see a way to access this from our own scripts, e.g. a custom implementation of IPreprocessShaders.OnPreprocessShader().
     
  2. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    3,085
    @hippocoder and/or @aliceingameland could you ping Christophe about this? I have a rather hacky solution working now by taking ShaderVariantCollectionInspector and adding a button into the custom inspector that basically creates my own custom ShaderVariantCollection that has all the info available that I need. But it would be nice to have a more solid approach (I could post about it on the alpha-list - but it's not really alpha-related, so I felt it's more appropriate to post it here). It really seems to me that something built-in that's based on ShaderVariantCollections would probably be the most elegant solution most of the time but I might be completely missing something.

    Also, it turns out that stripping scriptable shader variants doesn't help with the maximum number of shader keywords (256). So, one thing that I feel is also missing with this approach is a way to tell Unity to globally drop specific shader keywords right at the start of shader processing (and ideally even in the editor runtime). Something like "ignore shader keywords" in platform-specific project graphics settings might do the trick. It is often not feasible to change the shader files.
     
  3. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    3,085
    It gets worse: I removed my IPreprocessShaders implementation to have a new "reference build" where everything is included. But the build-time is still 35 minutes (before, a build was almost two hours), and it seems that some of the shaders variants are still missing. I have deleted ShaderCache and ShaderCache.db, as well as the ScriptAssemblies just to be sure. Also, I don't get the log statements from my IPreprocessShaders implementation - but it very much looks as if I'm no longer getting those variants anymore.

    Are the results from IPreprocessShaders cached somewhere?
     
    Last edited: Jul 13, 2019
  4. KovaPx

    KovaPx

    Joined:
    Jan 27, 2018
    Posts:
    5
    Have you ever solved this problem? I have a similar idea and am clearing the variants collection for some shaders but they end up in the build anyway, wondering if anything is cached or not.
     
  5. havchr

    havchr

    Joined:
    Jun 18, 2009
    Posts:
    51
    Our implementation for this is :


    Code (CSharp):
    1.  
    2. using System;
    3. using System.Collections.Generic;
    4. using System.IO;
    5. using Agens;
    6. using UnityEditor;
    7. using UnityEditor.Build;
    8. using UnityEditor.Callbacks;
    9. using UnityEditor.Rendering;
    10. using UnityEngine;
    11. using UnityEngine.Rendering;
    12. #if !AGENS_NO_SHADER_STRIPPING
    13. class ShaderVariantsStripperProject : IPreprocessShaders
    14. {
    15.      
    16.     private ShaderStrippingSetting shaderStrippingSetting;
    17.     private static StreamWriter shaderStripLogFile;
    18.     public ShaderVariantsStripperProject()
    19.     {
    20.         var shaderStripSettings = Resources.LoadAll<ShaderStrippingSetting>("");
    21.         if (shaderStripSettings.Length == 0)
    22.         {
    23.             Debug.LogWarning("Did not find shader stripping settings asset , please create one. Compiling all shaders");
    24.         }
    25.         else
    26.         {
    27.             if (shaderStripSettings.Length > 1)
    28.             {
    29.                 Debug.LogWarning("Warning More than one shader strip setting asset -- only using first found");
    30.             }
    31.             shaderStrippingSetting = shaderStripSettings[0];
    32.         }
    33.     }
    34.     public int callbackOrder { get { return (int)ShaderVariantsStripperOrder.Project; } }
    35.     public bool KeepVariant(Shader shader, ShaderSnippetData snippet, ShaderCompilerData shaderVariant)
    36.     {
    37.         bool resultKeepVariant = true;
    38.         if (shaderStripLogFile == null)
    39.         {
    40.             string timeStamp = String.Format("{0}d_{1}m__{2}h_{3}m",
    41.                 DateTime.Now.Day, DateTime.Now.Month,
    42.                 DateTime.Now.Hour, DateTime.Now.Minute);
    43.             shaderStripLogFile = new StreamWriter("Stripped_ShaderLog_" + timeStamp + ".txt");
    44.         }
    45.      
    46.         var shaderKeywords = shaderVariant.shaderKeywordSet.GetShaderKeywords();
    47.         string[] keywords = new string[shaderKeywords.Length];
    48.         int i = 0;
    49.         foreach (var shaderKeyword in shaderVariant.shaderKeywordSet.GetShaderKeywords())
    50.         {
    51.             keywords[i]=shaderKeyword.GetKeywordName();
    52.             i++;
    53.         }
    54.         ShaderVariantCollection.ShaderVariant variant = new ShaderVariantCollection.ShaderVariant(shader,snippet.passType,keywords);
    55.         if (shaderStrippingSetting != null)
    56.         {
    57. #if UNITY_IOS || UNITY_OSX
    58.         resultKeepVariant = shaderStrippingSetting.collectionOfShadersToKeepMetal.Contains(variant);
    59. #elif UNITY_STANDALONE_WIN
    60.         resultKeepVariant = shaderStrippingSetting.collectionOfShadersToKeepPC.Contains(variant);
    61. #endif
    62.         }
    63.         if (!resultKeepVariant)
    64.         {
    65.             string prefix = "not keepeing VARIANT: " + shader.name + " (";
    66.             if (snippet.passName.Length > 0)
    67.                 prefix += snippet.passName + ", ";
    68.             prefix += snippet.shaderType.ToString() + ") ";
    69.             string log = prefix;
    70.             for (int labelIndex = 0; labelIndex < keywords.Length; ++labelIndex)
    71.                 log += keywords[labelIndex] + " ";
    72.             shaderStripLogFile.Write(log + "\n");
    73.         }
    74.      
    75.         return resultKeepVariant;
    76.     }
    77.     public void OnProcessShader(
    78.         Shader shader, ShaderSnippetData snippet, IList<ShaderCompilerData> shaderVariants)
    79.     {
    80.      
    81.         int inputShaderVariantCount = shaderVariants.Count;
    82.         for (int i = 0; i < shaderVariants.Count; ++i)
    83.         {
    84.             bool keepVariant = KeepVariant(shader, snippet, shaderVariants[i]);
    85.             if (!keepVariant)
    86.             {
    87.                 shaderVariants.RemoveAt(i);
    88.                 --i;
    89.             }
    90.         }
    91.         if (shaderStripLogFile != null)
    92.         {
    93.             float percentage = (float)shaderVariants.Count / (float)inputShaderVariantCount * 100f;
    94.             shaderStripLogFile.Write("STRIPPING(" + snippet.shaderType.ToString() + ") = Kept / Total = " + shaderVariants.Count + " / " + inputShaderVariantCount + " = " + percentage + "% of the generated shader variants remain in the player data\n");
    95.         }
    96.     }
    97.      [PostProcessBuild(1)]
    98.     public static void OnPostprocessBuild(BuildTarget target, string pathToBuiltProject) {
    99.         if (shaderStripLogFile != null)
    100.         {
    101.             shaderStripLogFile.Close();
    102.             shaderStripLogFile = null;
    103.         }
    104.      
    105.     }
    106. }
    107. #endif
    108.  
    109.  
    It seems to be working, we went from a memory footprint of 200 MB of shaderlab to 50 MB , which seems much more sane. However, the process of stripping the shaders seems to be very slow. 30 minutes added build time or so for metal.

    The next thing I am going to try is to use the log file of what we are stripping out - because maybe there's a configuration of the project that can be made, to not have Unity bundle that many variants to begin with.

    Also, we are going to try to use the ShaderVariant collection to generate a more effecient strip file.
    The hunch: it might be faster to use the shaderVariants file to generate a list of shaders&variants that probably should be stripped, and then directly strip based on that list - instead of calling shaderVariantCollection.Contains - which probably is the slow part of this mix.

    I'll report once I know more.

    Update : We might be seeing similar caching based behaviour. Don't really know what's going on.

    Update : also - we were returning Package in the shader stripping order, so that's why it was so slow, I've updated the code to change it into Project
     
    Last edited: Apr 16, 2019
    jashan likes this.
  6. havchr

    havchr

    Joined:
    Jun 18, 2009
    Posts:
    51
    Something broke in our project big time after having the stripper run. The iOS build just crashes on startup - it seems to go to a shader - that is looking for a fallback and then into lza-decompress-code which finally crashes.

    I'm deleting the Library folder now and rebuilding without stripping code - hope to have a build that runs again. It would be really nice to get the shader stripping features working. The variant explosion is so annoying (I understand why shaders are built like this though), especially when it's not needed for the final product.
     
  7. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    3,085
    It would really be great if someone from UT could chime in. I still think this is a great and useful feature - but not if it breaks things the way it currently seems to break things.
     
  8. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    209
    I would also be very interested in this. It's just not feasible to have ~100MB of shader code in an Android build when I could define exactly which variants I actually want to use.
     
  9. havchr

    havchr

    Joined:
    Jun 18, 2009
    Posts:
    51
    We got it quite stable now and it has reduced our memory footprint by maybe 250 MB - our build times are good as well. I will write a blog post and share our code soon. I think maybe our project was configured in a very unfortunate way in regards to shader variants, because our team is small and no one had complete view over these things until we had to figure out a way to reduce them.

    Also the weird issues we experienced at the beginning are no longer an issue - we have since upgraded unity as well as done some changes to our code : we are using the 2018.4 LTS version.
     
    SentryGames and jashan like this.