Official Improvements to shader build time and runtime memory usage

Thanks for info

Waiting for editor tool to restrict some shader variant at authoring time and see it in edit, play mode and even in material inspector i.e. (this shader variant is disallowed by project settings)

 

Cool stuff. Glad this area of the engine is getting attention.

Do you have anything planned for the following issue:

We have our own custom uber-shader which can't be included in "Always include". Then for each level we have a scene that gets built as its own AssetBundle. Each of those AssetBundles will include its own copy of commonly used shaders which in our case results in a 300mb extra disk space.

For example for us the solution would be to be able to specify Shader Variant Collections as Always Include.

 

If Unity will make API that helps us treat each shader variant as a separate sub-asset or something like that → than we potentially can grab all shader variants and put them into separate bundle on build time

 

@dnach does the "This optimization is targeted for 23.1" means that 2021/2022 won't receive this and still have the *128 (2^7, since 7 new keywords were added, and both are compiled as enabled/disabled permutation regardless of scene settings) increased shader compilation times for the shaders that are either based on Lit or that are created using SG?

Maybe I misunderstood so don't get me wrong (frankly I don't know who merged those changes to URP 12 without seeing the mayhem they will cause, so it's hard for me to place any kind of blame), but if 2021 nor 2022 won't get any reasonable fix for that, how are people actually expected to use URP in them (with no possibility to downgrade URP packages in newer Unity Editors, obviously) at all?

 

We had to write our own shader variant system that creates lots of sub shaders or else we would end up with GBs of loaded shader programs (with custom shader variant system it is less than 30 MB iirc).

I would like to see very aggressive shader loading at runtime. If only necessary shaders are loaded in, I wouldn't expect more than a MB of loaded shader variants. Looking forward to this!

 

This seems to be available from 2022.1 onwards?

 

@dnach Can you elaborate on how Variant Keyword Prefiltering is different from OnProcessShader() where we can already skip specific keywords?
Is it just a fancy way of saying there's going to be a UI to control keywords instead of going through coding with OnProcessShader()?

 

Thanks sounds good. Waiting for the backport to 2021 LTS
Updated the project from 2019 lts to 2021 1.5 Weeks ago increasing the build time by xx% with shader variants in the millions.

 

Is there any ETA on how long these backports take? Having some incredibly bad issues lately with these exports, it's pretty costly to what our studio is doing.

 

Thanks for sharing this. Is there any news yet on when backports to 2021.3.x will arrive?

 

Checking in on the status of the back ports since it is a critical issue for my team as well.

 

This appears on 2021.3.13. Is this it?



Also, will this setting allows us to use multiple HDRP assets without expecting several gb of memory usage for HDRP Lit?

 

why can't unity come with precompiled shaders for the most common usages?
seems like reinventing the wheel each time anyone does a build

 

Any ideas about a potential ETA?

 

So exciting to hear.

 

Any news @aleksandrk ?

 

Just installed the update (2021.3.15f1) and so far it has been an improvement.

Cold build which was basically the first ever build done on the new version was an expected 6 hours, somewhat of a large game and it's expected for a new version to be doing a large build on first go. If I remember correctly, the version I was on before this one took me a total 8-12 hours to do my first build.

Second build it was also an improvement, from my usual 15+ minutes, it's gone down to 7 minutes. Again, I'm unsure if this was an expected result the engineers wanted, but this is me not really fine tuning everything, it was just an installation and seeing how things went.

I'm unsure if I was really supposed to change much, but to me it's a decent improvement anyway. Thanks again for listening to everyone here, and I appreciate the support.

 

It seems like 2021.3.15f1 is back to the pre-crazy build time levels. Back from 1.5-2hrs to 37m for cold build, and for warm builds, went from 36m to 8m.

 

2021.3.15f1 brings warm builds down to ~14 minutes for us. This is still significantly slower than 2020 LTS (currently at ~4 minutes), however we have no choice but to take it.

StripUnusedMeshComponents option has a larger impact in 2021 than 2020. Disabling it further reduces warm builds down to under 7 minutes.

 

Unity 2020.3.40 + HDRP

• With one referenced HDRP Asset, most features unticked: 150mb runtime memory usage
  
• With one referenced HDRP Asset, most features on: 500mb
• With three referenced HDRP Assets (low, medium, high): 2gb+
• Initial build time took a long night, more than 10 hours
• More details in this thread

Unity 2021.3.15 + HDRP

• With one referenced HDRP Asset, most features unticked: 150mb
  
• With one referenced HDRP Asset, most features on: 150mb
• With three referenced HDRP Assets (low, medium, high): 320mb
• Initial build time took around 3 hours

Not sure when exactly the steps of progression occured, as we did notice runtime memory gains when jumping from Unity 2020 to 2021, but eh! That's some major progress anyway.

Thanks to everyone involved in this at Unity, this is really nice.

 

Just to add: I'm also seeing slightly shorter build times.

 

Btw. since "cold" times are so huge, would it not be worth it to memory-dump all that that's computed and save it on drive for reload?

 

After upgrading from 2021 3.6f1 LTS to 3.15f1 with the latest URP version and deleting the Library folder, the cold build (I assume it means the build into an empty folder without cashed shaders?) now takes more than 2 hours, creating from 1,5k to 3k variants for each custom shader (and the default Lit shaders for URP, which I shouldn't even use in the project, really), in a project with quite a few custom shaders. Is this normal? I'm using Amplify Shader Editor and Addressables, wonder if it could affect the build times. I don't really notice any speed gain in builds, is there something wrong with my project?

The next build without any changes is lighting fast, less than a minute.

Then I slightly edited 2 shaders made in Amplify Shader Editor and the next build took 25 minutes (worked mainly on those 2 shaders, each needed 3k variants)

Then I slightly edited just 1 shader made in ASE, but also unticked some checkboxes for different platforms that I don't use (DX9, Switch, PS4, XBoxes, OpenGL, etc, just left DX11 for now). After that the build took ~12,5 minutes, which is roughly twice as fast as the previous build where I edited 2 shaders. So it looks like ASE is not the culprit, as unchecking the boxes didn't speed up the shader compilation time (I haven't upgraded ASE for the most recent version yet, though, because they disabled the Undo feature for some reason and I'm waiting for it to be resolved).

Only after that I decided to check if there any new settings for variant prefiltering, and I found none.

This is quite a long buildup for each new or edited shader, and it promotes testing the game in the editor instead of in builds and it's quite dangerous in my experience.

Currently the source project is 4,5GB with the Library folder and 1,5GB without it. The build is 253MB.

 

Hey, I already saw a few forum posts about Variant Keyword Prefiltering, but no examples/doc on how we can use it, or maybe I missed it, or don't fully understand it Is it applicable for keywords in the shader directly? Can we use something like this?

Code (CSharp):

1. #pragma shader_feature AWESOME_FEATURES
2.  
3. #if AWESOME_FEATURES
4.    #pragma multi_compile _ RED GREEN BLUE WHITE
5. #else
6.    #pragma shader_feature RED
7. #endif

 

Hi there,

We got some really bad build times here too, over four days to build the entire game, but since 2021.3.15f1 with Variant Keyword Prefiltering, for the second build it's very similar to before on built-in.

For those who are still wondering how the Variant Keywork Prefiltering works, here is the keynote for the Unity 2021.3.15f1 version that explained it for me:

Editor: Enabling shader keyword pre-filtering so that build process does not have to enumerate through full shader variant space. This fixes the issue of URP builds even with warm shader cache taking really long time. This fix does not reduce the time spent on actually compiling shaders.​

I also had a confirmation from Unity that this is automatically done and there is no need to do any setting about it.

I'm still looking how we can reduce the first build time because we need to build for PC, Android and PS4 (soon for PS5) and this is painfull. Also, updating the URP Package or Unity version makes the cache go away and the build starts from scratch again. A minor change on a shader also takes hours to build it again, with about 40k variants on PC.

I'm looking forward to some understandable ways to reduce this variants if already possible.

Thanks,

 

https://therealmjp.github.io/posts/shader-permutations-part1/

We do an approach described here:

1. Build the game without shader stripping, and enable shader logging. This takes long.
2. Go through your project, parse what shaders were used from the logfile.
3. Next builds *know* what is used, so make a shader postprocessor that strips out anything unused.
4. When something new is used, repeat cycle.

 

Upgraded from 2022.1 to 2022.2.1f1
Cold Build times went up from ~1.5 hours to 3hours
Warm Build times went up from ~2minutes to 10minutes.

Most of the time is spent "Compiling Shader variants"

 

Is there any way to extract a report of what multi_compile keywords were stripped by Variant Keyword Prefiltering on a warm build?
So that we can use that information to either rewrite our shaders to remove keywords we never used, or to add them to our OnProcessShader call backs to make sure they get stripped even on cold builds

 

Ah great thanks!

 

Panick led me here.
I just upgraded a project from 2021.3.11f1 and URP 12.1.7 to 2021.3.17f1 and URP 12.1.9, hoping on some shader compilation time improvements as I casually read about that in the Release notes. Since I already spent quite some time optimising the use of keywords in our custom shaders, compilation time and memory usage at runtime were already quite decent.
However, the first android build after upgrade has taken nearly 4 hours, compared to max 2 hours before.
Build size doubled (463 Mo to 898 Mo).
And now when I run the application on Focus 3, it simply crashes at loading the one scene that contains most of the materials (so the shader variants) used in the project. What's happening...?


Edit: My problems seem to be related to the URP package upgrade from 12.1.7 to 12.1.9, not Unity itself. When I reverted to URP 12.1.7 while keeping Unity 2021.3.17 everything went back to normal (build size, time and performances. No major build time improvement though, compared to 2021.3.11).
I checked the Memory Profiler's Tree Map before and after URP upgrade in a scene that doesn't crash, the memory usage of each loaded shader doubled with URP 12.1.9.
As URP 12.1.10 was released today I tested with this version as well but the problem is still there.
There is something going on with keywords and shader variants management in URP but I guess this is no longer within the scope of this thread...

 

Henlo all, I am using Unity 2022.1.10f1+URP, and I can't find "Shader Settings" and "Shader Variant Loading Settings" shown in the screenshot. Do them not exist in 2022.1.10f1+URP?
It would be awesome if anyone would shed some light!

 

Thank you!