Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Official Improvements to shader build time and runtime memory usage

Discussion in 'Shaders' started by dnach, Jul 7, 2022.

  1. greg-harding

    greg-harding

    Joined:
    Apr 11, 2013
    Posts:
    525
    Just chiming in on this comment. We updated from 2021.3.14 to 2021.3.18 today and are using whatever URP the editor update did (URP went from 12.1.8 to 12.1.10) and our build times have hugely blown out. We're seeing clean build times doubling and warm builds near doubling on one machine and tripling on another.

    It's really frustrating to lose so much build performance in the latest editor that we had to update to so that we could fix a showstopper asset pipeline garbage collection editor issue on macOS Ventura (that was fixed in 2021.3.16). Argh, just setting fire to time here.
     
    futurlab_peterh likes this.
  2. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,047
    @greg-harding we'd like to get a bug report, if possible. It should have decreased the build times.
     
  3. Beauque

    Beauque

    Joined:
    Mar 7, 2017
    Posts:
    61
    Actually I sent a bug report about this issue a few days ago: IN-31285.

    At least on my side it appears to be related to some recently added code in Editor/ShaderPreprocessor.cs in URP 12.1.8 or .9, that intentionally prevents the exclusion of additional lights "off" variants from the build, for performance reasons (see comment in script). Except that it doubles the amount of variants for all lit shaders at least if additional lights are enabled in the URP asset.

    @greg-harding I got reasonable build times again after commenting lines 360 and 367 in the script mentionned above, so it can be done as temporary fix I guess, not sure if it's entirely safe though.
    Code (CSharp):
    1. // features |= ShaderFeatures.AdditionalLightsKeepOffVariants;
    Apart from that, I am not seeing significant build time improvements in 2021.3.18f1 compared to 2021.3.11f1.
     
    Last edited: Feb 19, 2023
    futurlab_peterh and greg-harding like this.
  4. greg-harding

    greg-harding

    Joined:
    Apr 11, 2013
    Posts:
    525
    Thanks for your reply. Yep, I filed a general ticket yesterday: IN-31718 mentioning a large build performance drop we experienced moving from 2021.3.14 to 2021.3.18. It's light on details as I haven't analysed the editor logs from builds on both versions yet, and I only stumbled onto this thread talking about intended improvements where some replies were mentioning a big increase in build times which is what we're seeing.

    I'm going to analyse the editor logs building from both versions to see what I can add to the bug report.

    Update: I've run some comparisons on an M2 MacBook Pro and here are some stats from building the same project with clean and warm builds.

    clean:
    2021.3.14 - 765s (12m 45s)
    2021.3.18 - 1165s (19m 25s)

    warm:
    2021.3.14 - 213s (3m 33s)
    2021.3.18 - 756s (12m 36s)

    @aleksandrk Just confirming that I've added all of these builds from the Editor.log along with our current package manifests to the bug ticket.
     
    Last edited: Feb 11, 2023
    aleksandrk and DevDunk like this.
  5. greg-harding

    greg-harding

    Joined:
    Apr 11, 2013
    Posts:
    525
    Have you happened to see this bug report (N-31718) come through? I haven't had any response from QA yet and these build times are a killer in 2021.3.18. Thanks for any info or advice you might have.

    Update: 2021.3.19 just landed and build times are back to normal again - thanks!
     
    Last edited: Feb 17, 2023
    latrodektus and Beauque like this.
  6. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    We were forced to upgrade from 2021.3.6f1 to 2021.3.16f1 due to an issue with an SDK. Now our build times have increased from ~15 mins to over an hour. Is there some code I can backport to fix this?
     
  7. Invertex

    Invertex

    Joined:
    Nov 7, 2013
    Posts:
    1,557
    paulatwarp likes this.
  8. greg-harding

    greg-harding

    Joined:
    Apr 11, 2013
    Posts:
    525
    It's never as easy as just upgrade to Unity 2021.3.19 because only you know how much effort it is for you and your team and what other dependencies there are, but moving from 2021.3.18 to 2021.3.19 fixed all our build issues again. If you can't upgrade and want to find some code to potentially change, perhaps install both versions and do a diff on the URP package... we didn't have any luck with that but I didn't try too hard.
     
    paulatwarp likes this.
  9. Beauque

    Beauque

    Joined:
    Mar 7, 2017
    Posts:
    61
    Wasn't that lucky, although I did some changes in my shader code that may have affected build time negatively when I tried a build in 2021.3.19.

    If trying to upgrade to 2021.3.19 is not an option and if you indeed use URP:
    I found the following code has been added between URP 12.1.7 and 12.1.9 (maybe 12.1.8) and I believe it's responsible for increasing cold build time a lot since I upgraded to 2021.3.17 and above.
    Code (CSharp):
    1.  
    2.             // XRTODO: We need to figure out what's the proper way to detect HL target platform when building. For now, HL is the only XR platform available on WSA so we assume this case targets HL platform.
    3.             var wsaTargetSettings = XRGeneralSettingsPerBuildTarget.XRGeneralSettingsForBuildTarget(BuildTargetGroup.WSA);
    4.             if (wsaTargetSettings != null && wsaTargetSettings.AssignedSettings != null && wsaTargetSettings.AssignedSettings.activeLoaders.Count > 0)
    5.             {
    6.                 // Due to the performance consideration, keep addtional light off variant to avoid extra ALU cost related to dummy additional light handling.
    7.                 features |= ShaderFeatures.AdditionalLightsKeepOffVariants;
    8.             }
    9.  
    10.             var questTargetSettings = XRGeneralSettingsPerBuildTarget.XRGeneralSettingsForBuildTarget(BuildTargetGroup.Android);
    11.             if (questTargetSettings != null && questTargetSettings.AssignedSettings != null && questTargetSettings.AssignedSettings.activeLoaders.Count > 0)
    12.             {
    13.                 // Due to the performance consideration, keep addtional light off variant to avoid extra ALU cost related to dummy additional light handling.
    14.                 features |= ShaderFeatures.AdditionalLightsKeepOffVariants;
    15.             }
    16.  
    It's in Editor/ShaderPreprocessor.cs script. You can try to comment out line 360 and 367. Build time went back to normal after I did that, however it may cancel any runtime performance improvement this code was added for in the first place.
    Since I cannot give up on these runtime improvements and knowing URP features/keywords won't decrease over time anyway (on the contrary), I went for improving my shader variants management and stripping scripts to compensate. Still hoping the URP team adresses this build time issue though, it would be a relief.
     
    Last edited: Feb 24, 2023
    futurlab_peterh and paulatwarp like this.
  10. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    Thanks, but we are using HDRP. Maybe the issue is similar though.
     
  11. Halfspacer

    Halfspacer

    Joined:
    Sep 13, 2014
    Posts:
    23
    Also seeing way longer build times where it's just compiling shaders for 30-60 minutes where our builds would normally take around 20 minutes. Do we know if anything came from the bug ticket submissions mentioned above?
     
    paulatwarp likes this.
  12. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    I've extracted shader variant stripping data from the log file output of builds of our project from 2021.3.6f1 (~30m) and 2021.3.16f1 (~1h 20m). They seem to be compiling and stripping roughly the same amount of shaders.

    2021.3.6f1 Remaining variants = 31,156 / 818,930
    2021.3.16f1 Remaining variants = 31,290 / 821,670

    It doesn't seem likely that 1% additional variants is going to make that much difference to the build time.
     
    Last edited: Mar 24, 2023
  13. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    We've been forced to update again due to a different issue with a platform SDK. Now we're on 2021.3.21f1 and the build time is back to normal (~12 minutes)
     
    sabojako, DragonCoder and aleksandrk like this.
  14. dairectx

    dairectx

    Joined:
    Jan 31, 2015
    Posts:
    5
    Cant Unity just give us a simple built-in function:
    1. collect all shader variants I used in one Editor's playmode session.
    2. ask me, would I like to add this collection to strip whitelist

    And when build, unity native strips all shader variants not included in the whitelist.

    I think it would be a life saver function.
    The KPIs of the QoL team or optimization team for the whole year can be completed in this version.

    ----
    Well, there is still one more function point to close the loop.
    3. Automatically log missing shader variants when running development build versions.
    4. And automatically collect these missing variants in Player.log, and goto step 2.
     
    Last edited: Apr 21, 2023
    EvOne likes this.
  15. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,047
    FWIW 1 and 3 are there already.
     
  16. paulatwarp

    paulatwarp

    Joined:
    May 17, 2018
    Posts:
    135
    Sadly, we've had to revert to 2021.3.16f1 due to other bugs. Did anyone ever figure out any work arounds for the slow build times?
     
  17. KarlKarl2000

    KarlKarl2000

    Joined:
    Jan 25, 2016
    Posts:
    608

    Attached Files:

    Last edited: May 29, 2023
    blackbird likes this.
  18. Claytonious

    Claytonious

    Joined:
    Feb 16, 2009
    Posts:
    904
    Unfortunately, yes, that is what we have decided is "normal".

    It should only be this bad "the first time" you build, though. After that, much of it should be cached and it should build faster. But, watch out, because "first time" means any time:

    * You are building on a build agent as part of CI and don't aggressively cache the library (which you cannot always do)
    * You have switched platforms
    * You are trying out different rendering API's
    * You are on a different workstation than you were on yesterday (who woulda thunk this ever happens?)
    * You are trying a new version of Unity

    It's a little bit crazy that we've actually decided this is an OK state to be in. But believe it or not, you can still be quite productive while working around the list above if you're careful.
     
    kmowers, fullerfusion and andreiagmu like this.
  19. KarlKarl2000

    KarlKarl2000

    Joined:
    Jan 25, 2016
    Posts:
    608
    I guess it's better than nothing. With the recent layoffs at Unity.. probably a skeleton crew left to work on this stuff. :(
     
  20. creat327

    creat327

    Joined:
    Mar 19, 2009
    Posts:
    1,756
  21. DragonCoder

    DragonCoder

    Joined:
    Jul 3, 2015
    Posts:
    1,745
    The times have been improved 2022 onwards!
     
  22. yoloandoyolo

    yoloandoyolo

    Joined:
    Jul 1, 2022
    Posts:
    8
    So many parameters to tune Unity is starting to feel like a neural network
     
    timmehhhhhhh likes this.
  23. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,984
    2022 and newer is even a bit worse, plus sync shader loading takes 10x time and hangs the editor while it happens.
     
  24. DavidZobrist

    DavidZobrist

    Joined:
    Sep 3, 2017
    Posts:
    234
    What helped in my case was running the URP material converter ( only on materials) it ran for 27 hours and did not finish i canceled it and the next build was back to normal build time. (40 min instead of 13 hours)

    The issue appeared when upgrading from LTS 21 to LTS 22
     
  25. lloydsummers

    lloydsummers

    Joined:
    May 17, 2013
    Posts:
    359
    I just wanted to say quickly, this helped me greatly, thank you. With this change, I was able to see the variants generated by shader quite clearly in the Editor.log file. I wrote a tool to parse that into something more excel friendly - and discovered the Unity Sample pack for Spline was generating a whopping 90,000+ shader variants from a single "Shader Graph / Lit" shader that it came with. Wow. And confirmed more of my worst offenders were third-party imported Shader Graph shaders.

    I knew something was up as our builds went overnight from ~10 minutes to ~6 hours. Removing this, the build times dropped back to 10 minutes again. Thank you for sharing this. Had I not fixed this before contributing it back to the repo my team would have killed me :D.
     
    dnach likes this.
  26. fendercodes

    fendercodes

    Joined:
    Feb 4, 2019
    Posts:
    197
    When I compile my custom Shader Graph shader within a built-in project on 2021.3.36f1 I am seeing massive memory usage when profiling a standalone build (500mb).
    upload_2024-5-20_11-57-14.png

    Here are the logs showing number of variants. Is this a lot? I'm not really familiar with what a reasonable number is.
    upload_2024-5-20_11-56-33.png

    I also used the Shader Control asset from the store to identify the keywords for the shader:


    There are only 3 keywords being added from my own parameters in the Shader (an enum keyword with AXIS_X, Y, Z as shown above) so I am a bit confused as to why this is using so much memory.
     
  27. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,047
    Yes, it's a lot. Hundreds (or thousands) would be normal, but not tens of thousands.

    I don't really know what the Shader Control asset shows you exactly. Maybe it's individual keywords it encountered during the build process.
     
  28. fendercodes

    fendercodes

    Joined:
    Feb 4, 2019
    Posts:
    197
    Apologies, the screenshot didn't attach correctly. Essentially, the Shader Control asset just implements IPreprocessShaders as described in the official Unity docs and provides a GUI to view keywords and setup stripping.

    Screenshot 2024-05-22 084811.png

    One thing that I am noticing is that Shader Graph shaders that are targetting Built-In are including keywords which are for URP (all the ones starting with an _ underscore in the screenshot above).

    By manually excluding all of those URP keywords as well as lightmap-specific ones as those aren't needed for this shader, I can reduce the variants down to about 50k which still seems excessive.

    Any other tips?

    Note: I updated from 2021.3 to 2022.3 and that didn't improve the situation.
     
  29. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,047
    What's the order of the callback? It may influence the data you get. It should run after all other stripping scripts.
     
  30. fendercodes

    fendercodes

    Joined:
    Feb 4, 2019
    Posts:
    197
    I set it to 99999:

    public int callbackOrder { get { return 99999; } }
     
  31. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,047
    OK :)

    I'm really surprised URP keywords are there in a BiRP shader graph. It doesn't look right that you have 50K variants after getting rid of those.
    Would it be possible to file a bug report so that we could take a look?
     
  32. fendercodes

    fendercodes

    Joined:
    Feb 4, 2019
    Posts:
    197
    Can you confirm if those keywords starting with _ are actually URP keywords? I did some searching online and it seemed that way, but if I strip the _ADDITIONAL_LIGHTS ones, I get some pink in builds on BiRP so now I'm not so sure.

    Regardless, I created a fresh project to test this more and get a bug report going. BiRP 2021.3.36f1, empty Lit Shader Graph on a single cube in a fresh scene.

    Here were my findings for total variants after stripping when building:

    - Initial: 886 :D
    - With Lightmap baked: 40k :eek:
    - With Lightmap baked and Linear Fog enabled: 100k :eek::eek:
    - With no lightmap and Linear Fog enabled: 2k

    So having a lightmap baked is causing the majority of the issues I am seeing. I have logged a bug for just that part for now. Bug case ID: IN-76881
     
  33. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,047
    These are not a built-in BiRP keywords for sure.
    Thank you for the bug report!