Official Improvements to shader build time and runtime memory usage

Just chiming in on this comment. We updated from 2021.3.14 to 2021.3.18 today and are using whatever URP the editor update did (URP went from 12.1.8 to 12.1.10) and our build times have hugely blown out. We're seeing clean build times doubling and warm builds near doubling on one machine and tripling on another.

It's really frustrating to lose so much build performance in the latest editor that we had to update to so that we could fix a showstopper asset pipeline garbage collection editor issue on macOS Ventura (that was fixed in 2021.3.16). Argh, just setting fire to time here.

 

Actually I sent a bug report about this issue a few days ago: IN-31285.

At least on my side it appears to be related to some recently added code in Editor/ShaderPreprocessor.cs in URP 12.1.8 or .9, that intentionally prevents the exclusion of additional lights "off" variants from the build, for performance reasons (see comment in script). Except that it doubles the amount of variants for all lit shaders at least if additional lights are enabled in the URP asset.

@greg-harding I got reasonable build times again after commenting lines 360 and 367 in the script mentionned above, so it can be done as temporary fix I guess, not sure if it's entirely safe though.

Code (CSharp):

1. // features |= ShaderFeatures.AdditionalLightsKeepOffVariants;

Apart from that, I am not seeing significant build time improvements in 2021.3.18f1 compared to 2021.3.11f1.

 

Thanks for your reply. Yep, I filed a general ticket yesterday: IN-31718 mentioning a large build performance drop we experienced moving from 2021.3.14 to 2021.3.18. It's light on details as I haven't analysed the editor logs from builds on both versions yet, and I only stumbled onto this thread talking about intended improvements where some replies were mentioning a big increase in build times which is what we're seeing.

I'm going to analyse the editor logs building from both versions to see what I can add to the bug report.

Update: I've run some comparisons on an M2 MacBook Pro and here are some stats from building the same project with clean and warm builds.

clean:
2021.3.14 - 765s (12m 45s)
2021.3.18 - 1165s (19m 25s)

warm:
2021.3.14 - 213s (3m 33s)
2021.3.18 - 756s (12m 36s)

@aleksandrk Just confirming that I've added all of these builds from the Editor.log along with our current package manifests to the bug ticket.

 

Have you happened to see this bug report (N-31718) come through? I haven't had any response from QA yet and these build times are a killer in 2021.3.18. Thanks for any info or advice you might have.

Update: 2021.3.19 just landed and build times are back to normal again - thanks!

 

We were forced to upgrade from 2021.3.6f1 to 2021.3.16f1 due to an issue with an SDK. Now our build times have increased from ~15 mins to over an hour. Is there some code I can backport to fix this?

 

Perhaps this would help?
https://forum.unity.com/threads/imp...time-memory-usage.1305615/page-2#post-8796451

 

It's never as easy as just upgrade to Unity 2021.3.19 because only you know how much effort it is for you and your team and what other dependencies there are, but moving from 2021.3.18 to 2021.3.19 fixed all our build issues again. If you can't upgrade and want to find some code to potentially change, perhaps install both versions and do a diff on the URP package... we didn't have any luck with that but I didn't try too hard.

 

Wasn't that lucky, although I did some changes in my shader code that may have affected build time negatively when I tried a build in 2021.3.19.

If trying to upgrade to 2021.3.19 is not an option and if you indeed use URP:
I found the following code has been added between URP 12.1.7 and 12.1.9 (maybe 12.1.8) and I believe it's responsible for increasing cold build time a lot since I upgraded to 2021.3.17 and above.

Code (CSharp):

1.  
2.             // XRTODO: We need to figure out what's the proper way to detect HL target platform when building. For now, HL is the only XR platform available on WSA so we assume this case targets HL platform.
3.             var wsaTargetSettings = XRGeneralSettingsPerBuildTarget.XRGeneralSettingsForBuildTarget(BuildTargetGroup.WSA);
4.             if (wsaTargetSettings != null && wsaTargetSettings.AssignedSettings != null && wsaTargetSettings.AssignedSettings.activeLoaders.Count > 0)
5.             {
6.                 // Due to the performance consideration, keep addtional light off variant to avoid extra ALU cost related to dummy additional light handling.
7.                 features |= ShaderFeatures.AdditionalLightsKeepOffVariants;
8.             }
9.  
10.             var questTargetSettings = XRGeneralSettingsPerBuildTarget.XRGeneralSettingsForBuildTarget(BuildTargetGroup.Android);
11.             if (questTargetSettings != null && questTargetSettings.AssignedSettings != null && questTargetSettings.AssignedSettings.activeLoaders.Count > 0)
12.             {
13.                 // Due to the performance consideration, keep addtional light off variant to avoid extra ALU cost related to dummy additional light handling.
14.                 features |= ShaderFeatures.AdditionalLightsKeepOffVariants;
15.             }
16.  

It's in Editor/ShaderPreprocessor.cs script. You can try to comment out line 360 and 367. Build time went back to normal after I did that, however it may cancel any runtime performance improvement this code was added for in the first place.
Since I cannot give up on these runtime improvements and knowing URP features/keywords won't decrease over time anyway (on the contrary), I went for improving my shader variants management and stripping scripts to compensate. Still hoping the URP team adresses this build time issue though, it would be a relief.

 

Thanks, but we are using HDRP. Maybe the issue is similar though.

 

Also seeing way longer build times where it's just compiling shaders for 30-60 minutes where our builds would normally take around 20 minutes. Do we know if anything came from the bug ticket submissions mentioned above?

 

I've extracted shader variant stripping data from the log file output of builds of our project from 2021.3.6f1 (~30m) and 2021.3.16f1 (~1h 20m). They seem to be compiling and stripping roughly the same amount of shaders.

2021.3.6f1 Remaining variants = 31,156 / 818,930
2021.3.16f1 Remaining variants = 31,290 / 821,670

It doesn't seem likely that 1% additional variants is going to make that much difference to the build time.

 

We've been forced to update again due to a different issue with a platform SDK. Now we're on 2021.3.21f1 and the build time is back to normal (~12 minutes)

 

Cant Unity just give us a simple built-in function:
1. collect all shader variants I used in one Editor's playmode session.
2. ask me, would I like to add this collection to strip whitelist

And when build, unity native strips all shader variants not included in the whitelist.

I think it would be a life saver function.
The KPIs of the QoL team or optimization team for the whole year can be completed in this version.

----
Well, there is still one more function point to close the loop.
3. Automatically log missing shader variants when running development build versions.
4. And automatically collect these missing variants in Player.log, and goto step 2.

 

Sadly, we've had to revert to 2021.3.16f1 due to other bugs. Did anyone ever figure out any work arounds for the slow build times?

 

Is this normal? Anyone else getting compile times like this?

I'm currently on Unity 2021.3.24f1 and not sure if I'm using the editor settings correctly.

I read through this and have to admit I only understand a small portion of what was written.
https://blog.unity.com/engine-platform/2021-lts-improvements-to-shader-build-times-and-memory-usage

 

Unfortunately, yes, that is what we have decided is "normal".

It should only be this bad "the first time" you build, though. After that, much of it should be cached and it should build faster. But, watch out, because "first time" means any time:

* You are building on a build agent as part of CI and don't aggressively cache the library (which you cannot always do)
* You have switched platforms
* You are trying out different rendering API's
* You are on a different workstation than you were on yesterday (who woulda thunk this ever happens?)
* You are trying a new version of Unity

It's a little bit crazy that we've actually decided this is an OK state to be in. But believe it or not, you can still be quite productive while working around the list above if you're careful.

 

I guess it's better than nothing. With the recent layoffs at Unity.. probably a skeleton crew left to work on this stuff.

 

Funny, I came to this thread because Unity guys pointed me here on this other thread Why is a URP Simple Lit shader so large in memory? - Unity Forum

I see that as usual... many words but little action. Things just keep getting worse.

 

The times have been improved 2022 onwards!

 

So many parameters to tune Unity is starting to feel like a neural network

 

2022 and newer is even a bit worse, plus sync shader loading takes 10x time and hangs the editor while it happens.

 

What helped in my case was running the URP material converter ( only on materials) it ran for 27 hours and did not finish i canceled it and the next build was back to normal build time. (40 min instead of 13 hours)

The issue appeared when upgrading from LTS 21 to LTS 22

 

I just wanted to say quickly, this helped me greatly, thank you. With this change, I was able to see the variants generated by shader quite clearly in the Editor.log file. I wrote a tool to parse that into something more excel friendly - and discovered the Unity Sample pack for Spline was generating a whopping 90,000+ shader variants from a single "Shader Graph / Lit" shader that it came with. Wow. And confirmed more of my worst offenders were third-party imported Shader Graph shaders.

I knew something was up as our builds went overnight from ~10 minutes to ~6 hours. Removing this, the build times dropped back to 10 minutes again. Thank you for sharing this. Had I not fixed this before contributing it back to the repo my team would have killed me .