Resolved Extremely Slow Shader Compiling for Builds

Hi Josh, thank you for your question. This is on top of our minds.

We are aware the shader variant explosion is a problem for productions, and it’s a hard problem to solve. This was also a big issue highlighted by Gigaya and we know it’s impacting several other projects as well. This is something that multiple teams are working on to improve. I also acknowledge we can always do better. So let me share some thoughts on why this is a hard problem to solve and what we are doing.

Why was this not a problem in Built-in RP?
Built-in RP would also have reached the scalability issues on shader explosion if it had continue developing graphics features in the same pace we are adding for SRPs. The scalability limitations and some of the problems around uber shaders are hinted here in this blog post by Aras.

Why does build time keeps increasing for URP?
URP is a pipeline rapidly in development, we are on a way to overcome built-in RP in every way. We are landing several features in each release. In 2023 we are close in feature parity with built-in RP feature while also offering a lot more. In some platforms, whenever we add a feature we face a choice: Do we regress runtime performance or do we add more variants? It’s a hard balance that we try to best reason with every feature for all the platforms we support. In the past we tried to optimize and turn some of the variants into branches but identified runtime regressions in some platforms.

Why do users get a regression in build time even if they don’t use new features in their project?

This could happen for a combination of reasons.
1) The shader stripping system is not ready to handle scalability at this amount of variants. There’s a cost in the shader stripper as it processes every variant, and it could be that there are multiple C# shader strippers in a single project. There's a massive improvement on this system by pre-filtering variants, in our test case it's a 100x reduction in variant processing for the C# shader stripping for URP Lit shader. Developers will discuss the backport of this improvement to older versions of Unity.

2) The current shader stripping and Unity project cooking is too manual and relies a lot on users to understand the settings and optimize it themselves. We don’t prevent settings to be changed at runtime. We have to be careful If we strip features by default or have some implicit static analysis heuristic then this could cause situations that users have features being stripped unwillingly from the build. The impact of that conservative stripping we have is that we end up adding many variants that are not really needed in the build in most cases.

3) We also lack API for users to fine tune the balance of runtime vs build time performance, by f.ex selecting which variants they would like to turn into a branch. We also lack API to explicitly strip some of the features.

A few things that we are doing to improve this:

• We have been landing improvements to shader systems in Unity. The shader pre-filtering work is one of them, we also recently landed some improvement to reduce shader memory footprint by allowing users to control how shaders are evicted from memory. In URP every release we do a pass to look for potential optimizations that we might have missed, f.ex in 2021.2 we did this, and in 22.2 we did already some other work for PRs. There's an opportunity here to improve and be better at catching up these optimization opportunities at PR level instead.
  
• We can communicate better to users the known issues / expectations when they update to a newer version and define on a release acceptance criteria when it comes to build time regression from one LTS to the other. We are defining these release checklist and guidelines for our upcoming 2023 release, which is the one we are working now. However I just want to make it clear that we support LTS so any improvements that we find in this area we are considering to backport to older versions.
• This is a challenge for release and testing. We support 25 platforms and several graphics APIs. Quite frequently an approach that works well in one platform might not in the other when it comes to decide between shader keyword or branching. We have already some performance test automation for URP, but we are expanding that to run on more platforms and capture more metrics, such as build times.
• We have added more documentation on shader stripping [1][2][3][4]. We currently have documentation in progress to document shader keyword in URP.
  
  
  The above list is not the end of it. We will need to keep getting feedback and iteratively improving it.
  

 

Strict Shader Variant Matching was introduced in 2022.1.0a7, and can be used to fallback on a pink debug shader when a requested variant is missing, logging an error in the console detailing the variant and its keywords.

We shared some useful information on recent and upcoming improvements to Shader Variant Management in the following Shaders forum thread, and will keep updating as we move forward with additional improvements!
https://forum.unity.com/threads/improvements-to-shader-build-time-and-runtime-memory-usage.1305615/

 

Since processing a large amount of variants for potential stripping and compilation can still take a long time, and so we are introducing the Variant Prefiltering improvement that @phil_lira mentioned above.

Prefiltering allows to perform early exclusion of shader keywords based on prefiltering rules (driven by RP settings), resulting in less variants to process thus significantly reducing shader processing time.

With prefiltering, the stripping would also look a bit different, and so we are now updating the documentation to reflect and will soon include new meaningful examples.

 

dynamic_branch keywords were introduced in 2022.1.0a15, to abstract over dynamic branching in shaders.
See more detail in: https://docs.unity3d.com/2022.1/Documentation/Manual/SL-MultipleProgramVariants.html

Keep in mind that before 2022.1, you can also implement dynamic branching traditionally via if statements:
https://docs.unity3d.com/2021.3/Documentation/Manual/shader-branching.html

One improvement we are investigating, is allowing to define runtime usage flags for certain RP features, while specifying when its ok to use dynamic branching for certain features. Features controlled at runtime should generally use 'multi_compile' keywords, but it may be acceptable to use 'dynamic_branch' keywords on certain platforms (thus generating less variants). Features not controlled at runtime generally should use 'shader_feature' keywords, which are stripped out based on material usage, thus producing less variants.

 

As part of the larger improvements to Shader Variant Management, we plan to provide a meaningful Editor tool that will help to easily understand the project's shader variant usage. This will allow you to view which shaders and keywords generate the most variants included the build, and potentially also the runtime usage of variants (which variants are requested, missing or unused at runtime).

This would help with more effective authoring of conditional shader features, as well as help perform more effective variant stripping. We will share more about our plans and provide additional detail as soon as we can!

 

We are still exploring how to introduce such an improvement, and make sure users are aware of the tradeoffs - but as you highlighted dynamic branching in shaders should be handled with care, and with platform considerations taken into account. Sensible defaults and an opt-in approach could definitely help.

As a note, as of 2021.3 we officially support conditional declaration of shader keywords in platform conditionals (to declare 'dynamic_branch' / 'multi_compile' / 'shader_feature' based on the target platform)

 

This is something we are investigating, to be able to provide API to allow a multi compile to be either turned into a dynamic branch of be compiled as shader variant.

 

'shader_feature' keywords address this, as we will not include their resulting shader variants in the build if no material in the project enable these. However, many render pipeline settings can be controlled at runtime and thus we need to include their possible variants, as they may be requested at runtime. As @phil_lira mentioned above, we could allow you to flag whether certain features can utilize dynamic branching instead, to reduce the amount of compiled variants.

With improvements to our build time shader processing, we will pre-filter unnecessary multi_compile keywords, so this should significantly reduce variant processing time though!

 

The solution called Variant Keyword Prefiltering has shipped in 2021 LTS (f15) and reduces warm build times by 40 to 90%. You can read more here.

 

Yes, Variant Prefiltering landed in 2023.1.0a14 and was backported all the way back to 2021.3.15.