Search Unity

Official Shader Stripping Improvements in URP

Discussion in 'Universal Render Pipeline' started by LukasCh, Nov 5, 2021.

  1. LukasCh

    LukasCh

    Unity Technologies

    Joined:
    Mar 9, 2015
    Posts:
    102
    Shader Stripping Improvements in URP
    In recent years, URP has grown and introduced a lot of amazing features. This resulted in a huge amount of shader variants to support those features.

    To ensure that these variants don't add to build times, file size, and runtime memory usage, we have expanded URP's shader stripping capabilities.

    In 2021.2 we are introducing a few new features that will be covered in detail.

    Stripping unused feature variants
    Many features in URP require "enabled" and "disabled" variants of prebuilt shaders, depending on whether the feature is enabled or disabled. URP already performs some automatic stripping of these variants, but we have improved this from version 2021.2 onwards.

    In all versions of URP, if a feature is disabled in all URP Assets included in your build, URP strips the "enabled" variants and compiles only the "disabled" variants.

    In versions prior to 2021.2, if a feature is enabled in any URP Asset included in your build, URP compiles both "enabled" and "disabled" shader variants.

    From 2021.2 onwards, you can enable Strip Unused Variants in the URP Global Settings. When this setting is enabled, if a feature is enabled in all URP Assets included in your build, URP strips the "disabled" variants and compiles only the "enabled" variants. If this setting is disabled, URP behaves as before.

    Here is the list of features that this applies to:
    • Light Layers
    • Render Pass
    • Reflection Probe Blending
    • Reflection Probe Box Projection
    • Forward+
    • SSAO
    • Decals
    • Main Light Shadows *
    • Additional Light Shadows *
    • Additional Lights *
    Keyword changes
    In versions of URP prior to 2021.2, URP enables and disables the shader keywords for Main Light Shadow, Additional Light Shadow and Additional Light based on the configuration of the current scene. This means that URP can switch between the "enabled" and the "disabled" variants for different scenes.

    From 2021.2, URP enables and disables these keywords based on settings in the URP Asset. This means that URP uses either the "enabled" or "disabled" shader variant as long as the current URP Asset is in use, rather than switching per-scene.

    This change allows URP to strip unneeded variants when Strip Unused Variants is enabled.

    We felt this was a less common case, and the overall performance change is quite small. Keep in mind that you can disable Strip Unused Variants to retain the previous functionality.

    Stripping unused Quality Settings variants
    You can assign different URP Assets to different Quality Settings levels. At build time, URP examines these URP Assets to determine which URP features are enabled or disabled in your build, and therefore which variants to compile or strip.

    In versions of URP prior to 2021.2, URP examines every URP Asset that is assigned to a Quality Settings level, regardless of whether that Quality Settings level is enabled for the current build target. To avoid including unneeded variants in these versions of URP, you must ensure that unneeded URP Assets are not assigned to a Quality Settings level at build time.

    From 2021.2, URP only examines URP Assets that are assigned to Quality Settings levels that are enabled for the current build target.

    Stripping unused post-processing variants
    In URP 2021.1, we added an option to strip all post-processing shader variants from your build. In URP 2021.2, we introduce the option to strip post-processing shader variants that are not used by any Volume Profiles in your project. For example, if you only use Bloom in your project, all other unneeded post processing variants and shaders will be stripped.

    To enable this feature, go to URP Global Settings and enable Strip Post Processing. Note that this feature works by checking all Volume Profiles in the project - for now it does not check if it is actually used in a scene or not.

    XR/VR Stripping
    If you do not plan to use XR/VR in your project, you can disable XR and VR modules (How to disable a Module). This allows URP to strip XR and VR related shader variants. This functionality has not changed in 2021.2, but it is a useful tip.

    "Small" Project
    Testing small project with enabled features:
    • SSAO
    • Decals
    • Reflection Probe Blending
    • Reflection Probe Box Projection
    • Bloom


    Results from Windows Player D3D11 build:

    --------------------- Before ------- After
    Build Time ------ 2min 27s ---- 28s
    Variant Count -- 4407 --------- 944
    Build Size ------- 70.2MB ----- 69.4MB



    Template Project
    Testing changes with URP Template project.



    Results from Windows Player D3D11 build:

    --------------------- Before ------- After
    Build Time ------ 20min 14s --- 7min 12s
    Variant Count -- 7964 ---------- 2277
    Build Size ------- 82.4MB ------ 78.6MB



    Boat Attack Project
    Testing changes with URP Boat Attack Demo.



    Results from Windows Player D3D11 build:

    --------------------- Before ------- After
    Build Time ------ 58min 57s --- 39min 19s
    Variant Count -- 22890 -------- 11895
    Build Size ------- 495MB ------ 492MB
     
    Last edited: Nov 5, 2021
  2. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,361
    Any chance of giving some "always include" override options for the PP stripping? For example right now if you want to use URP PP stripping and custom AO solution that still feeds into URP's Lit shader, only ways to not have the required parts stripped away is to either disable PP shader stripping, modify the URP package itself for this or enable built-in SSAO but set it's intensity to 0 and make it run before your real AO solution (this is quite hacky IMHO). Ultimately you'd want this system to be flexible enough to not forcing users to modify the URP package.

    As another topic, I've mentioned this few times in other related threads: it would be nice if URP and HDRP shader strippers could coexist on same build. I don't know what exactly changed on 2021 cycle for this because this used to work in past Unity versions (or at least build times used to be very reasonable). Now if I try to make this kind of setup work it either results on one of the SRPs shaders getting fully stripped out or alternatively having one SRP including all it's shader variants - which obviously can't work as it would take forever to build anything like this.
     
    goncalo-vasconcelos likes this.
  3. LukasCh

    LukasCh

    Unity Technologies

    Joined:
    Mar 9, 2015
    Posts:
    102
    Thanks for feedback. Yes this is something we will look into in the future. Overall If I understood correctly you want SSAO variants in lit shader not to be stripped as your custom SSAO would supply texture there. I am curious if exposing this as option from etc renderer "require SSAO variant" would suffice in your scenario?
    As for now I think the other workaround would be just have another renderer in urp asset which uses our builtin ssao. This way you will still able to have all benefits of stripping and also not pay the price of bultin ssao.

    Yes, we have initiative to address in general URP and HDRP compatability in the same project. Which should also address this issue, sadly I do not have anything concrete about it yet.
     
    rz_0lento likes this.
  4. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,361
    Oh right, didn't see this alternative at all. I like this workaround more than just having the SSAO on your main data asset.

    I don't have a good idea on what would be the ultimate solution for making this type usage cleaner, but I can imagine as URP grows more there will be other things beyond SSAO that users may want to bypass with some custom solution as well.

    I don't expect it to happen over night but it's nice to hear that it's at least on some backlog. :)

    As another note on the shader stripping: I'd love to have some kind of visual indicator on the unity inspector per shader if it was actually included or stripped away on previous build. Like just small note when you look at the shader telling it wasn't included on previous build etc. This would mainly be useful for custom shaders that may get stripped accidentally so wouldn't need to add them to that always included shaders -list "just in case they might get axed".
     
  5. Thermos

    Thermos

    Joined:
    Feb 23, 2015
    Posts:
    148
    Alongside automatic stripping, IPreprocessShader interface is a powerful tool to strip unwanted shader variations.

    Would love to see a built in tool (or user interface) that could show all shader variations in previous build and let users to decided which variations are unwanted so the get stripped away in the after builds.
     
    zackblack and rz_0lento like this.
  6. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    802
    Just want to chime in and note that "stripping is based on project content" is kinda dangerous, since often Packages or AssetStore content ships with scenes, post processing volumes, ...

    Would be great if there was a visual post-build overview of "we included this and that variant because we think you'll need it based on this (pingable) asset".
    Additionally, would be great if an alternative to this would be to specify an explicit list of assets to be used (e.g. just because I'm using post processing in a non-build test scene doesn't mean I want it to be in my build).
     
  7. LukasCh

    LukasCh

    Unity Technologies

    Joined:
    Mar 9, 2015
    Posts:
    102
    Yes, this is first step. We plan to have in future to do it based on which profiles are referenced in scene.
     
  8. burningmime

    burningmime

    Joined:
    Jan 25, 2014
    Posts:
    845
    This is probably a similar complain t to @rz_0lento 's but I have a custom shadow solution (stencil shadows), which I am injecting into the screen space shadow map and then enabling in a render pass by calling
    cmd.EnableShaderKeyword("_MAIN_LIGHT_SHADOWS_SCREEN";);
    . I struggled for a bit to get this working in a build because the shader was being stripped. The solution was to create a separate URP Renderer called "DO NOT USE" with the normal ScreenSpaceShadowmapFeature, include it in the build under Quality Settings, and then never use it. But that seems very hacky.

    I'd love for there to be some way to force shader variants with a certain keyword to be included even if the shader stripping thing does not detect that it will be used.
     
  9. tatoforever

    tatoforever

    Joined:
    Apr 16, 2009
    Posts:
    4,369
    Unity 2021.2.2f1 and URP12.1 Bug case: 1381759

    Build fails after hours trying to compile a custom URP Lit shader (of 900 000 variants).
    Worth mention the memory leak is umbelivably high (~25GB of ram) while compiling variants. Btw nothing gets cached, if I start a new build the whole variant compilation process start from zero.
     
    ImpossibleRobert and lilacsky824 like this.
  10. tatoforever

    tatoforever

    Joined:
    Apr 16, 2009
    Posts:
    4,369
    Still happens in Unity 2021.2.4f1:



    Still can't build build always fail, editor crashes on a 64GB ram system. Doesn't crash on 128GB ram system but give a bunch of IL2CPP errors and all shaders contain compile errors too.
    Worth mention that in the 128GB ram system the shader compilation is started again from zero (nothing gets cached) despite successfully compiling all shader variants for hours.
     
  11. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    Please check the Editor log, this is the source of truth for what has been cached and what hasn't been. The progress bar always says "Compiling shader variants" and always starts over, but that doesn't mean the compilation results haven't been taken from the cache.
     
  12. tatoforever

    tatoforever

    Joined:
    Apr 16, 2009
    Posts:
    4,369
    In the 128GB system, after few hours compiling my shader variants the build just fails with a bunch of IL2CPP errors. I then tried to restart the build and the whole shder variant compilation, starts from zero once again.
    The 64GB system can't event get pass half of my shader variants compilation build because Unity crashes after reaching 60-61GB of ram trying to compile my shader.
     
  13. momings

    momings

    Joined:
    Sep 11, 2015
    Posts:
    2
    We also met build crash after upgrading to 2021.2.6 from 2020.3. It seems unity ran out of memory.
    OnProcessShader and ShaderVariantCollection is useless because too many variants be prepared.

    Editor logs:

    Compiling shader "Universal Render Pipeline/Lit" pass "ForwardLit" (fp)
    [6.75s] 100M / ~503M prepared
    [13.66s] 200M / ~503M prepared
    [18.39s] 300M / ~503M prepared
    [27.64s] 400M / ~503M prepared
    [32.40s] 500M / ~503M prepared
    94371840 variants, prepared in 32.56 seconds, starting stripping...
    Obtained 0 stack frames.
     
  14. burningmime

    burningmime

    Joined:
    Jan 25, 2014
    Posts:
    845
    It should at least warn you if you're going over even like 1000 (maybe 10k) variants of one shader. There's no call for that.
     
  15. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
  16. Kevin-VFX

    Kevin-VFX

    Joined:
    Apr 17, 2016
    Posts:
    54
    Did I miss something, when did Forward+ get added?
     
  17. burningmime

    burningmime

    Joined:
    Jan 25, 2014
    Posts:
    845
    You didn't; it's still in progress.
     
  18. Anthelmed

    Anthelmed

    Joined:
    Mar 2, 2017
    Posts:
    15
    Hello,
    I just did a quick test and I build an empty URP project and it seems that the post processing Film Grain textures are included by default even if I don't use the post processing pass.

    Unity v 2021.2.8f1, URP v 12.1.3

     
    Last edited: Feb 5, 2022
    apkdev, Deleted User and daneobyrd like this.
  19. jimmikaelkael

    jimmikaelkael

    Joined:
    Apr 27, 2015
    Posts:
    796
    Any news about this?
    Now that URP does deferred the workaround only works at the condition that rendering path is the same on both renderers.
    Also not having such option makes it hard to distribute custom SSAO solution on URP because many customers requiring support for no AO in builds...
     
  20. victorhs_studio

    victorhs_studio

    Joined:
    Feb 16, 2019
    Posts:
    2
    What is the point in compiling shader variantes that will be striped later?

    if Variante.goingToBeStriped
    Variante = DontEvenCompile_NextPlease();
     
  21. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    727
    Is there no option to entirely disable URP's terrible shader stripping and just let us do it ourselves? I've disabled the options available to me in 2022.3.10f1 and it still manages to strip variants I need in asset bundle builds.
     
  22. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,025
    @funkyCoty
    There are several layers of variant stripping.
    What are you trying to get in ABs that gets stripped?
     
  23. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    727
    In this specific case: All the variants of main light + main light shadow casting on/off + additional light + additional light shadow casting on/off + shadow cascades + shadow filtering quality. The game is a PC game where users can change their quality settings - so of course all of my shaders need all variants. What URP apparently expects me to do is to create a new "Render Pipeline Asset" with every permutation of every shader I want..?! Then the only way to include it is to clutter up the Quality Settings??? Just let me turn it off and do it myself.
     
    BOXOPHOBIC likes this.