Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Feature Request Question regarding shader stripping

Discussion in 'Shaders' started by TJHeuvel-net, Sep 15, 2022.

  1. TJHeuvel-net

    TJHeuvel-net

    Joined:
    Jul 31, 2012
    Posts:
    838
    We are using Unity 2021.3.0, builtin render pipeline. Especially on platforms that share memory between GPU and CPU (consoles) we have had memory issues with shaders. The regular shader stripping was insufficient for us, so we have made an extra layer on top of this.

    Its mostly the same as described here, this is our workflow:

    1. Build the game without any (custom) shader stripping, ensure LogShaderComplication is on
    2. Run the build, make sure to render objects with these shader variants at least once(*)
    3. Merge in the player log file into our ShaderWhiteList, this is a long list of shaders and used stage/variant etc.
    4. Build the game with shader stripping, remove any shader that is not in our whitelist.

    This works. Ish. The results were spectacular, (standard)shaders that were 300mb now are mere kilobytes in the memory profiler. We could finally start our game on console!

    However workflow wise this is really, really terrible. These unstripped builds take nearly an hour, which has to be done every time some random visual glitch happens. We just released our game, two release issues were related to this.

    What makes it even harder to do this flow are two things; we dont have any idea *when* a variant cannot be found, and we dont know what logic the fallback-to-best-variant is.
    A log for "Hey i couldnt find variant A, B so i retorted to A" would be tremendously helpful for us in order to find issues. What fallback is chosen also is a mystery, and one that makes it very hard to find potential issues.
    From my guestimate; it will fall back to an already loaded variant. That means that depending on my previous ingame actions, the chosen shader/variants will be different. We've had these wrong variants being chosen only after a player does something particular in our game, which could be explained by this behaviour.

    So my questions:

    1. Is this a completely stupid approach, am i doing it wrong? I am seeing a lot of unity users struggling with shader stripping.
    2. Is there a way to be informed when a variant cant be found?
    3. Is there a way to suggest which variant should be used, when one cant be found?

    Thanks in advance!
    * Vulkan is a special case where the option to log shaders will always log `All` for the stage, instead of Vertex/Fragment etc. This greatly reduces how much we can strip because we cant be sure if a shader is only using vertex/fragment, or also tesselation (etc).
     
  2. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,751
    Did you try using the similar feature that collects shader variants into a collection, but inside the editor instead? It's in the graphic settings, and tells you how many variants the editor has seen and has a button that exports them into a ShaderVariantCollection.

    At the very least you can skip making a build and collect variants in editor instead.
     
  3. TJHeuvel-net

    TJHeuvel-net

    Joined:
    Jul 31, 2012
    Posts:
    838
    Thanks, but this doesnt work for us.

    These are variants used in the editor, and this often wildly differs from what is in the player. Plus; this only gathers shaders and variants, in our method we also get the shader stage and strip based on that.
     
  4. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    Hi!
    I think you'll be better off with Dynamic Shader Loading - it will ensure you're only loading variants that are actually used. I'm also working on reducing memory usage in shaders with many variants, so you'll benefit from that as well when it's available.

    See https://forum.unity.com/threads/improvements-to-shader-build-time-and-runtime-memory-usage.1305615/ for more details.

    2022.1 has a checkbox that does something similar.

    Your guesstimate is incorrect. It will fall back to a variant that has a closest matching keyword set, which is defined by a metric on enabled and disabled keywords.

    No, there isn't.
     
    TJHeuvel-net likes this.
  5. TJHeuvel-net

    TJHeuvel-net

    Joined:
    Jul 31, 2012
    Posts:
    838
    Thank you for the in depth response, its much appreciated.

    Once we update to 2022.1 i'll definitely turn on Strict Shader Variant matching, that will help us solve many of our issues.

    For dynamic shader loading, as i understand it, i hope we wont have to use this much. Although this will definitely reduce memory usage for players, i'm concerned about our build. Shaders do seem to compress incredibly well, a non-stripped build isnt much larger on disk, but our buildtimes increase immensly when it has to process all shader variants. It will cut us some slack with variants that are still erronously included, but it doesnt mean we can remove our custom solution entirely. Or perhaps the variant prefiltering solution described is already enough.

    Anyway; i'll push to update to 2022.1, and its very good to know its in such active development.
     
  6. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,983
    They do compress well, however when we load the data we decompress it (all of it). That's where dynamic shader loading helps - it only keeps around up to X decompressed chunks.

    Build times for shaders are normally dominated by shader compilation time. You could use Accelerator to speed this up.
     
    TJHeuvel-net likes this.