Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. We’re making changes to the Unity Runtime Fee pricing policy that we announced on September 12th. Access our latest thread for more information!
    Dismiss Notice
  3. Dismiss Notice

Bug Build time taking almost a day - URP

Discussion in '2021.2 Beta' started by RenOli, Jun 29, 2021.

  1. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,819
    This is the number of variants before any stripping. `M` stands for `million`.
    This number depends on the number of materials that end up in the build for a particular scene and reference the given shader.
     
  2. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    I find it impossible to believe there's hundreds of millions of these. I'm really not writing a complicated game. On top of that, if Unity actually told me WHICH shader had 415M possible variants, I might be able to do something about it.
     
  3. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,819
    It's the number of variants before stripping. This means it's before any render pipeline settings are considered, just what's in the shader itself.
    I'll make a note that it would be more useful if it showed the shader name there.
     
    Lars-Steenhoff likes this.
  4. kogi_rc

    kogi_rc

    Joined:
    Apr 23, 2019
    Posts:
    39
    Editor.log outputs quite a lot of information that could be useful, for example:

    STRIPPING: Shader Graphs/ObjectHighlightShader (Universal Forward pass) (Vertex) - Remaining shader variants = 12/96 = 12.5% - Total = 1422/108174 = 1.314549% TimeMs=5.5725

    I'm not sure if some toggle needs to be switched on to enable logging of shader stripper. Maybe someone else in this topic can clarify that.
     
  5. dnach

    dnach

    Unity Technologies

    Joined:
    Mar 9, 2022
    Posts:
    84
    Since 2020.2, the editor.log was extended to include this useful information on shader variants processing (stripped, processing time, compilation time, cache hits, amount compiled...)

    URP/HDRP Settings also include a setting for 'Shader Variant Log Level', which logs similar information in console, and more recently (22.2) a setting to export the log as a JSON ('Export Shader Variants' ).
     
    Last edited: Aug 9, 2022
    kogi_rc likes this.
  6. Neozman

    Neozman

    Joined:
    Sep 29, 2016
    Posts:
    56
    Any progress for this problem? We are still waiting for upgrade from 2020 to 2021)))
     
    TJNBG likes this.
  7. dnach

    dnach

    Unity Technologies

    Joined:
    Mar 9, 2022
    Posts:
    84
    We are aiming to land the optimizations for shader variant processing ("variant keyword prefiltering") in 2023.1, and hope to backport to 2022 and 2021.
     
    Catsploration, spamove and funkyCoty like this.
  8. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    ...you *hope* to backport this fix???
     
  9. dnach

    dnach

    Unity Technologies

    Joined:
    Mar 9, 2022
    Posts:
    84
    Landing the shader variant processing improvements is a high priority, and we are planning to backport this optimization as far back as 21.3.
     
    valarus likes this.
  10. RealHandy

    RealHandy

    Joined:
    Jul 11, 2020
    Posts:
    21
    @aleksandrk It looks like this bug is still not being mentioned in the known issues as of 2021.3.8f1
     
  11. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,819
  12. RealHandy

    RealHandy

    Joined:
    Jul 11, 2020
    Posts:
    21
  13. Bwacky

    Bwacky

    Joined:
    Nov 2, 2016
    Posts:
    153
    Please consider plugging this in for 2021.3.9 or 10.
    Build times are one thing, but the same issue affect building addressable groups. With a project using UMA and building addressablse that include only 2 shaders (a custom one I've used since 2019 with regular updates till this date + the default unity lit shader) compile times are hitting up to 7 hours. It's making work incredibly frustrating and it's not like you can do anything else in a project while it's frozen in compilation.
     
    timmehhhhhhh likes this.
  14. CPULL

    CPULL

    Joined:
    Aug 1, 2019
    Posts:
    5
    upload_2022-8-24_18-10-47.png
    URP project, simple scene, all default shaders except one quite simple done in shadergraph.
    Powerful PC (i9 16 cores/32 threads, 64gb memory, 3080ti graphics, only ssd drives)
     
    tlamiczka, IS_Twyker and timmehhhhhhh like this.
  15. lclemens

    lclemens

    Joined:
    Feb 15, 2020
    Posts:
    702
    5 shaders (all URP Simple-Lit based). 30 minutes and the bar hasn't moved.

    upload_2022-8-29_14-26-12.png

    100% cpu utilization
    upload_2022-8-29_14-27-19.png

    2021.3.8f1

    Good times.
     
    IS_Twyker, Bwacky and TJNBG like this.
  16. timmehhhhhhh

    timmehhhhhhh

    Joined:
    Sep 10, 2013
    Posts:
    154
    @aleksandrk @dnach Sorry for the ping, but are there additional updates to this? We're > 1 year from the OP and many months now in 'LTS' and it's still very unclear both when a fix might land and what might be expected in terms of improvements that real projects will see (ie, not synthetic tests).

    Given that nothing relating to this seems to have been included in the recent 2022 release, at this point I've lost confidence in the 2021 cycle entirely and, barring advice from you that says otherwise, will be recommending our projects stick to 2020 LTS and / or the built-in render pipeline wherever possible, and generally avoiding Unity packages (ie, Shader Graph and VFX Graph) going forward.
     
  17. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,819
    @timmehhhhhhh We have a PR to 2023.1 - it's being reviewed. Once this PR lands, we can start backporting it to earlier versions, including 2021.3.

    As to improvements, total clean build times of our graphics tests (including shader variant compilation) went down by 30-75% depending on the target platform. The variants needing processing for URP Lit in this test suite, for example, went down from ~2 000M to ~18M, so it's done in seconds instead of tens of minutes.
    I can try to get you some numbers using Gigaya - probably by mid next week.
     
  18. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    233
    Down by 75%? So from 15 hours to 3.75 hours?

    To quote the *first post* :

    Why do I feel this isn't being taken seriously? :/ You seem *happy* that the build time has gone from between 10 seconds and 5 minutes (assuming OP wasn't exaggerating) to 3.75 hours.
     
  19. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    674
    Are there plans to improve this further? We're using entirely custom shaders in our project (quest 2) and we are pretty careful about not adding too many variants, but to stay compatible with URP shader source we still end up having like 100 M variants, and this stripping phase seems to take longer than actually compiling.
     
  20. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,819
    You're comparing different things.
    The original post was about a tiny project; graphics tests are definitely not tiny.

    That's exactly what the fix addresses.
     
    djarcas likes this.
  21. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    233
    I really really hope you're right; but we're still talking about percentages going up by hundreds or thousands, and then a drop of 75%.

    Given as I'm doing a multiplayer game and quite frankly having to wait before doing a network test is driving me mad, the sooner this improvement arrives the better.
     
  22. timmehhhhhhh

    timmehhhhhhh

    Joined:
    Sep 10, 2013
    Posts:
    154
    Cheers, I genuinely appreciate this and would very much look forward to that.

    Would it be prudent to guesstimate a timeline of months then for LTS? Appreciate the effort the team is making in communication and resolution now, but it feels like the kind of issue that should not have been part of an LTS release to begin with; given that it is I'd expect more of an all-hands-on-deck response, where even if it doesn't make technical sense to do so, the work would first be done for branches being promoted as 'recommended for production', with PRs accelerated however possible. It's being backported, so it seems like effort is being made here already, but nowhere near enough.

    The 'driving me mad' sentiment is something I've also felt as I've tried to navigate this via the recommended steps and come up with a scalable strategy around it. The root of the issue for me is not that you've introduced a ton of cool new features that have had some unfortunate negative side effects, it's that it's trivially easy now in Unity to kill creativity for hours, days, or longer at a time. And for what? We're years into the scriptable renderer experiment and we've only very recently reached something like feature parity with BiRP (which is still very much a viable option), at the cost of nuking productivity for some and fragmenting a community and asset ecosystem.

    Given that Unity no longer has an internal productions team battle testing and championing these things (and while they did exist, it seems the concerns they raised WRT this particular problem were not taken seriously enough), how will the eventual fix for this issue be vetted for production use, and how can we have confidence in SRP and LTS releases in the future? Much of the roadmap has dependencies on this...
     
    Last edited: Sep 2, 2022
  23. MedicManDan

    MedicManDan

    Joined:
    Oct 30, 2018
    Posts:
    6
    I've gotta say... Building a project out of 2021.3.0 has been an absolute nightmare.

    If I take my project... and make a new scene, and I add a couple basic components to that scene to test VR... I'm looking at 540 MILLION VARIANTS... it takes 6-8 hours for that singular... basic scene... to build.

    I have read dozens and dozens of threads on the issues. Tried everything they've suggested... but I can't get the build times down to anything reasonable. Listen, if I have to have PHD level shader and graphical engineering knowledge to understand whats going on, Unity Engine is useless to me.

    I am trying to build my project to the quest 2. I'm having some trouble getting that working. I can't test a damn thing because I have to build my project to test it on the quest 2, and builds are an ALL DAY process.

    How is this a thing? How is this acceptable?

    Sorry... Rant Over... but I'm just lost. I've spent the last 4 days cooped up in an office troubleshooting a problem that requires me to wait all day between testing fixes. I have had enough.
     
  24. lclemens

    lclemens

    Joined:
    Feb 15, 2020
    Posts:
    702
    Any chance you could downgrade to an older version? 2019 and other older versions were much faster.
     
  25. HellsPlumber

    HellsPlumber

    Joined:
    Aug 17, 2014
    Posts:
    42
    I'm in a similar boat.
    I too am I using URP and developing for VR and the problem is exacerbated by using a combination of the two.
    Thankfully I'm not targeting Quest so don't need to build EVERY time I make a change.

    When I DO have to build the project it takes HOURS and my PC is unusable the entire time.
    I'm running a Ryzen 9 5950x 16 core CPU and Unity uses 100% of it, making my PC completely unusable for the hours it spends building.

    The project I'm working on relies on user-generated content built using Unity.
    It's unreasonable for me to expect any user to essentially lose access to their PC for hours to build content for the game.

    I've had this post bookmarked and keep checking back month after month to see if it's improved but I've yet to see any progress on this. Right now it's halting the release of my game and it's crazy to me that this was seen as acceptable as by Unity themselves.
     
  26. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    https://forum.unity.com/threads/improvements-to-shader-build-time-and-runtime-memory-usage.1305615/
     
  27. LightYarn

    LightYarn

    Joined:
    Nov 18, 2012
    Posts:
    75
    @djarcas maybe a bit off-topic but I'm also developing a multiplayer game right now. "ParrelSync" helped me a LOT with this because it basically removes the need to build you project every time if you just want to test a small thing.
    Maybe it helps you too: https://github.com/VeriorPies/ParrelSync
     
  28. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,819
    OK, so here are the build times for Gigaya. Build target DX11, 8-core laptop CPU. Note that Gigaya build time is completely dominated by shader compilation time. Clean build means empty shader cache; warm build means the shader cache is there from a previous build.

    Without the fix: clean build took 14h 15m, warm build 46m.
    With the fix: clean build 13h 49m, warm build 4m 14s.

    We'll aim for October, although I can't promise we'll make it (there's too many factors influencing that). After the fix lands in 2023.1 we can prioritise 2021.3 LTS.
     
    Last edited: Sep 7, 2022
  29. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    What do you think a clean build should take for a regular game?

    Basically is 13-14 hours is considered a reality these days for a clean build Unity project?

    Also does changing any option at all, or shader incur a new clean build penalty?
     
  30. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,819
    I think it really depends on the content and the HW you build it on.

    Let's say I'm not happy with these numbers.

    A clean build is "empty shader cache". If you delete "Library/ShaderCache", basically, before doing a build.
    If you change a single option somewhere, it's a warm build; however, if this option has effect on shader stripping it may need to compile additional variants.
    If you change a shader, all variants of this shader will be rebuilt. We keep around the data for several most recently used modifications, so if you change something in the shader, build, then undo the cache is still there.
    If you change an include file, all shader variants of all shaders that depend on it will get recompiled.
     
    djarcas likes this.
  31. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Hmm, perhaps a possibility is to be able to have a more static approach where shaders are considered inflexible for the build - so you won't be able to change any material features at runtime, and have to manage that yourself. Is that a viable path? I see a lot of asset store authors preferring to just generate a shader without compile time variants to combat this.

    Does seem a little high, hope there's a path - even if manual - for developers to work with at some point.
     
  32. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,170
    Deleting the Shader Cache doesn't make any difference in build times for me. If I make an IL2CPP build of Teslagrad 2, any subsequent build without any changes takes 10 seconds, no matter if I delete the shader cache or not.

    Usually building on my computer takes 2-3 minutes, I assume because I have changed code and scenes so less things are in cache. 12GB Assets folder, URP with some minor edits.


    So like, yeah, if you want to have fast builds, do 2D games, I guess? We do use quite a few custom shaders for things, so it's not just that we don't have that. I wonder what on earth is going on with Gigaya, and 3D games in general.
     
    hippocoder likes this.
  33. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    674
    13 hours for a build is crazy! What is going on? Does that project just have 200 m shader variants or what? Why?
     
  34. Andy-Touch

    Andy-Touch

    A Moon Shaped Bool Unity Legend

    Joined:
    May 5, 2014
    Posts:
    1,439
    Im quite confused how build times are taking THAT long now. What changed since the end of June?

    When I was working on Gigaya, a clean local build would take roughly 6 hours. This was with 2021.3.5f1. About 95% of this being millions of URP Shader Variants and 1 hour of this being URP Lit. We were talking to URP Team on methods of shader stripping but also in general WHY using vanilla Unity can result in that much length of build time. Even though I wasn't on the Art team and I made probably only 1 shader for Gigaya, the art team weren't doing anything too crazy or exotic with shader graph.

    Because of the 6 hour clean build length we shifted to automated nightly builds so the editor wasnt locked for nearly a whole day every day. :)

    In general, Unity should ultra prioritise reducing build times exploding in length to an unusable state. Or atleast offer some better build in tooling to diagnose and improve the situation.
     
    Last edited: Sep 7, 2022
    djarcas and LightYarn like this.
  35. timmehhhhhhh

    timmehhhhhhh

    Joined:
    Sep 10, 2013
    Posts:
    154
    Cheers, and again genuinely appreciate your team's presence and candidness in this thread.

    Even with the fix, without major realignment from Unity here I'm finding it difficult to justify relying on URP (and SRPs generally) going forward. Not to say that there have been no improvements, but we're two years out from this official statement (which already was painful) and SRPs are in many dimensions still a nightmare to use, with the added cost of this gigantic productivity leap backward.

    I don't expect features and performance overnight and for free, but what is going on here? I won't speak for anyone else, but I'm at a point where I'd prefer Unity take an 'un-easy' and 'controversial' route out of this, rather than continuing to believe these solutions will gradually improve over time.
     
    djarcas, pm007 and Andy-Touch like this.
  36. HellsPlumber

    HellsPlumber

    Joined:
    Aug 17, 2014
    Posts:
    42
    hippocoder likes this.
  37. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Sounds like Plumbing Hell to be honest. Good luck with it though!
     
  38. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,819
    Variant usage from materials is a fixed set of variants. It's the
    multi_compile
    s that hurt the build time.

    No, just ~50K variants of URP/Lit :) They comprise the majority of build time.

    Keep in mind that these numbers come from a laptop, not a desktop PC.

    FWIW if a project is using Accelerator to cache compiled shader variants, the problem becomes less prominent.
     
  39. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,746
    I'm sorry, but over 10 hours for a clean uncached build for something like Gigaya, which was quite early in terms of content development, is insane. If this is what it takes to make URP match the built-in feature set then something went really wrong along the way.
     
  40. Andy-Touch

    Andy-Touch

    A Moon Shaped Bool Unity Legend

    Joined:
    May 5, 2014
    Posts:
    1,439
    Quick Q; What is Accelerator?
     
  41. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,746
    Question: how large does the shader cache becomes with hundreds of millions of shader variants in it? Specially if you enable the PS4/XBOX/Switch shader debug information generation (which can already balloon into tens of GBs for projects using built-in).
     
  42. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,170
    The replacement for/evolution of the cache server.

    The idea is that you put all the results of importing assets that's not in version control (ie. everything that goes into the Library/ folder) on a server on the local network, and every time Unity wants to import an asset it checks if it's in the cache server instead.

    It's a bit too hard to set up, not very well documented/advertised, and it's not doing a good job of visualizing to you as the user that "hey you saved time now because we fetched something from the server!". But it's a great product.
    Of course it's completely useless as a solo dev unless you happen to switch platforms very often.
     
  43. Andy-Touch

    Andy-Touch

    A Moon Shaped Bool Unity Legend

    Joined:
    May 5, 2014
    Posts:
    1,439
    Aha, makes sense! Thank you. :D
     
  44. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,819
    I'm not sure about Accelerator, but we trim the local variant cache. We keep several revisions of cache (it's updated each time the shader changes) and we look in previous revisions for the variant we need to compile right now. If we find it, we move it to the latest revision.

    As to hundreds of millions variants, if it takes many hours to compile a hundred thousand variants, you're unlikely to ever get to that point :)

    The ShaderCache folder after doing the Gigaya build is 2.32 GiB, a bit over 120K files.
     
  45. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    How long are we realistically looking at before the solution reaches a public Unity alpha / beta build?

    And what kind of estimate for time savings do you think we should expect?

    If it's possible: what version of Unity were the 14 hour cold build times measured with, and what laptop hardware? Just so we can figure out perhaps an LTS or better hardware to fix this thing.

    I'm kind of leaning toward 45 mins being the clean build. 5 mins for a warm build is tolerable, but 14 hours for something that can still reset at times, is just too much work lost. That's actually a lot of money lost for developers using URP.

    Some things can't skip builds. It means that the pain point would push people to other engines or maybe a different pipeline (if it's any better on HDRP / BiRP).


    Ultimately for me: 45 mins is acceptable for a cold build, and 5 mins is acceptable for a warm build. If it is not solvable to that extent, please at least let developers know.
     
    tlamiczka likes this.
  46. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,819
    Alpha - probably in several weeks. As I mentioned above, we'll try to prioritise the backport to LTS.

    Depends on the shaders you use and the number of variants you end up with after stripping. And the hardware. Ultimately, very hard to predict :) Warm builds will be on the order of several minutes instead of tens of minutes.

    I measured this on 2023.1.0a11; Lenovo ThinkPad X1 Extreme Gen 3 (Core i9-10885H @2.4 GHz, 64 GB RAM).
    The laptop was at 100% charge and plugged in all the time.

    The warm build part is solvable (this fix we're talking about does exactly that). The cold build part is mostly in the hands on the URP team.
     
    hippocoder likes this.
  47. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Thank you for your candour, I guess the URP team has a little bit of work to do!
     
    djarcas likes this.
  48. timmehhhhhhh

    timmehhhhhhh

    Joined:
    Sep 10, 2013
    Posts:
    154
    For anyone interested, there's a great bit of history on this here.

    @aleksandrk Am I right in understanding that the work on variant keyword pre-filtering is essentially including URP variants in the process 'Now: can I have hundred billion variants? or don't search that other data'?

    Also, how might I better understand why a clean URP build would require so much longer than the HDRP builds aras-p describes? Or were those numbers with potentially billions of variants cached builds?
     
    lclemens likes this.
  49. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,746
    Modern feature-rich single-pass rendering is a recipe for shader permutations explosion. Early URP/LWRP avoided it by being too simple, built-in avoided it by relying on multipass additive rendering, HDRP avoids it by dividing workloads due to clustered/deferred rendering and screen-space shadowing.

    (There's also the question of whether URP is (or even if it can) avoiding the generation of nonsense/excludent keyword combinations.)
     
  50. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,819
    Right now the full variant space for URP shaders is too big. Scriptable stripping that reduces it to the set that needs to be compiled comes in very late in the process, so the build process has to feed it with hundreds of millions of variants (those 'M's during the "Preparing variants for compilation" step).
    Pre-filtering moves part of this reduction to the front of the pipeline that figures out variants to compile. Example: if URP says "no shadows", we fully exclude all variants that have shadows; with pre-filtering this is done before the build process figures out, which variants to generate.
    Quote from the post you linked: "Looping over a hundred billion simple things is a long process! So they were like “oh we wanted to do a build to check performance on a console, but gave up waiting”."
    They ended up working around that. IIRC they generate an include file with fixed defines based on settings and don't allow to modify those at runtime.
     
    timmehhhhhhh likes this.