Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Bug Build time taking almost a day - URP

Discussion in '2021.2 Beta' started by RenOli, Jun 29, 2021.

  1. neoangelique

    neoangelique

    Joined:
    Nov 19, 2014
    Posts:
    31
    I was experiencing this same issue in 2021 LTS and 2022 Beta. I currently have 2023 alpha 15 installed for about 4 days now and it seems to have solved the issue, my build times are approx 3 mins now instead of 40mins++(Might be longer since I cancelled the build). Did not have to do anything in the settings.
     
    LightYarn likes this.
  2. Bwacky

    Bwacky

    Joined:
    Nov 2, 2016
    Posts:
    214
    Personally, I tried jumping ship to 2023.1 but my projects crash on opening and the crash logs show absolutely nothing of value to go off on why the crash occured, so I imagine that's still not an option for most people.
    Great it works for you though!
     
  3. colinsauce00

    colinsauce00

    Joined:
    Jul 13, 2021
    Posts:
    16
    Hi, you can read my above post where I had similar issues. My fix? It just started... working again. There was no change in anything I did. I went from 40 mins etc. build times back to 4 minutes, except for linux which took 8 minutes. It is definitely an issue with shaders though yes as it compiles all of them for 30 minutes.
     
  4. Neozman

    Neozman

    Joined:
    Sep 29, 2016
    Posts:
    67
    Hello, any news for 2021 lts?
     
    PieterAlbers likes this.
  5. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    A pull request is ready, waiting to be merged.
    I'll post here when we know in which version the fix will appear.
     
    TJNBG and Bwacky like this.
  6. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    Known Issues in 2021.3.12f1

    • Universal RP: Shader compilation time is higher in 2021.3 than in 2020.3 (UUM-17405)


      Well, that's understated. Do I take it that we're waiting ANOTHER month for this?
     
  7. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    @djarcas that's another issue that we uncovered recently. Individual variants of URP/Lit and some other URP shaders are compiling slower on 21.3 than on 20.3.
    We already found the culprit and URP will fix this.
     
    djarcas likes this.
  8. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    Just so I'm not misunderstanding; 2021.3.12 neither has the long-build-time-fix and doesn't list it on the known issues either?
     
  9. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    Yes, that's right. I don't know why it's not showing it, it says it should on the issue internally.
     
    djarcas likes this.
  10. LightYarn

    LightYarn

    Joined:
    Nov 18, 2012
    Posts:
    77
    Will the fix also be implemented in 2022 versions?
     
  11. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    Of course.
     
    LightYarn likes this.
  12. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    IS_Twyker likes this.
  13. gatisp09

    gatisp09

    Joined:
    Nov 24, 2021
    Posts:
    40
    Like srsly UNITY get a grip.. My system runs on Latest Intel CPU I have RTX 3080TI, SSD with read/write speeds of 6000mb/s, and in order to build my simple medium to low side size game i need to w8 more than 2 hours. like what is this? How are u guys calling yourself a reliable and fast platform?
     
  14. gatisp09

    gatisp09

    Joined:
    Nov 24, 2021
    Posts:
    40
    and in my build progress bar, I see that it's doing something with forward shaders. i don't even use forward mode in my game its all deferred..
     
  15. gatisp09

    gatisp09

    Joined:
    Nov 24, 2021
    Posts:
    40
    like it clearly shows used variants, but no when i make build it takes a mythical amount of 25k+ ....
     

    Attached Files:

  16. gatisp09

    gatisp09

    Joined:
    Nov 24, 2021
    Posts:
    40
    and this is what it shows during the build process.... 364 million variants? where from
     

    Attached Files:

    • wtf.PNG
      wtf.PNG
      File size:
      3.2 KB
      Views:
      155
  17. Arnold_2013

    Arnold_2013

    Joined:
    Nov 24, 2013
    Posts:
    289
    I think this is how it works. Every variant used/unused is compiled and put into the cache. Now on a second build everything is much faster. In the ideal case when someone wants to do a build with different keywords (low vs high quality or something), it would still build fast the second time because all variants are already in the cache.

    There is no magic numbers, its just not a very efficient for most developers. They know the problem, they know the solution. Re-ranting won't make it go any faster.
     
  18. gatisp09

    gatisp09

    Joined:
    Nov 24, 2021
    Posts:
    40
    but how can there be milion variants like what ? just make a option to not load all the variants and the ones that u are using thats it simple easy peasy
     
  19. Arnold_2013

    Arnold_2013

    Joined:
    Nov 24, 2013
    Posts:
    289
    I think in few words this is what they are doing :). But it is a very different way of filtering so they will have to do more work to make sure they don't break something. Broken < Slow. They might also have decided to put some extra time into fixing it in a good way versus hacking in another "if(shaderUsed)" somewhere and adding to the long term technical dept.

    aleksandrk in this thread mentioned some reasons why the URP build time exploded. I see it as every feature/keyword added doubles the amount of shader variants. So you go from a reasonable buildtime of 15 minutes to 8 hours with only 5 extra features.
     
  20. Destruxion

    Destruxion

    Joined:
    Jun 5, 2019
    Posts:
    37
    Actually we could automate this whole process and fix things on our side if the tools provided would actually work.

    Strikt Shader Variant Matching: Great idea, but tests showed many missing keyword combinations are not logged.
    Editor Shader Variant Collections: Same problem, only a subset of shaders and keywords is tracked.

    Obviously there must be a place, where shaders and keywords are requested by Unity. We need a consistent way to track those exact combinations and store them in a deserializable format, not just a log that must be parsed or a Shader Variant Collection blackbox. Either in the player (no matter if the very first build is just purple, by simply not compiling ANY shader at all, just gathering data) or even better live in the editor or at least when running play mode (something that Shader Variant Collections should theoretically do).

    During the build process Unity can still throw those millions of keyword combinations at us, but we can then filter the EXACT combination of keywords required. Meaning 12 detected URP Lit shader keywords that are actually in use will not result in 4096 shader variants, but maybe in only 3, because no other combination is actually used.

    I built that pretty extensive tool shown on a previous page in this thread. We are able to activate shaders and a global keyword list manually in an UI (everything Unity refuses to track) and then rely on simple asset and memory scans get everything else. That combined with our own builder tool currently fully automates the process and we are back to normal build times. If Unity would solve the shader/keyword tracking issue, this solultion would fully automate the process.
     
    Arnold_2013 likes this.
  21. gatisp09

    gatisp09

    Joined:
    Nov 24, 2021
    Posts:
    40
    question is if they want to ..
     
  22. neoangelique

    neoangelique

    Joined:
    Nov 19, 2014
    Posts:
    31
    I thought this too. Until I installed Unity 2023 alpha, and build times dropped to 3 mins, even if its the first build it took no more than 5 mins.
     
  23. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    As mentioned here, we're very well aware of the larger problem with the whole shader variant workflow. We want to address it as soon as we have capacity to do this :)
     
    Destruxion likes this.
  24. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    2021.3.14f1
    Released: 18 November 2022 - STILL no mention of the fix?
     
  25. Bwacky

    Bwacky

    Joined:
    Nov 2, 2016
    Posts:
    214
    It'll be out before you know it, go for a long walk, catch some air, check back around April next year and I'm sure it'll be fixed by then. Patience is a virtue.
     
    djarcas likes this.
  26. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    The fix for long build times with warm cache will be available in 2021.3.15f1 and 2022.2.0b15.
     
    Destruxion, LightYarn and Bwacky like this.
  27. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    To quote the goblins of gadgetzan; time is money, friend!
     
    Destruxion likes this.
  28. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    Can we also talk about the time taken for cold builds?

    upload_2022-11-24_13-50-47.png

    This is so far from acceptable that it's unreal, and I'm genuinely at the point of considering abandoning Unity after over a decade of using it for the other engine with a U in it.
     
    LightYarn likes this.
  29. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    Sure.
    Absolutely. And we have stripping for this :)
    Do you have it configured correctly? Did you turn off all the stuff you know you won't ever need in the player?
    The number of variants comes from two things: everything that says "shader_feature" in the shader is controlled by materials that end up in the build; everything "multi_compile" is stripped based on the render pipeline settings assets assigned for different quality levels.
    So, if you have two quality levels and, say, one says "Enable SSAO" and the other one says "Disable SSAO", you get 2x variants.
     
  30. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,352
    @aleksandrk What Unity could do with that to make things better is;
    - Not spam new projects with 5 quality levels by default. We always delete all of those, they're a really bad way to control quality. So for a lot of users, things are slow and bad because you made it slow and bad by default.

    - Rejig things such that shader variants are opt-in rather than opt-out. You have this absurd situation where games that should take minutes to build takes hours because users have to juggle confusing settings in order for you to not do a bunch of work that they didn't want you to do in the first place. A good place to start would simply to find-and-replace
    multi-compile
    with
    shader_feature
    in the monster that is the standard shader, and then deal with the fallout of that on the other end instead of playing whack-a-mole with stripping.
     
  31. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    So why am I seeing 40+ hours cold builds? >_<

    ...how would I possibly know the answer these questions? Where am I told? Am I supposed write a parser for the log file? Does UNITY have a parser for the log file that'll break down what's going on?

    I stopped writing .shader files the moment shadergraph came along; are these settings or tickboxes within shadergraph that should be labelled with a big 'WARNING THIS SHADER HAS 84 MILLION VARIANTS' ? Should ShaderGraph list how many variants there are?

    Your example of SSAO confuses the hell out of me tho - isn't SSAO a post-processing full-screen effect? Why would that make a shader compile twice? You must understand that I am working from a position of a *ton* of Unity experience, but never at any point have I needed to start worrying about shader_feature or multi_compile, and, as I said, we are talking about URP here - everything is shadergraph and there is *zero* information or warning about something I do that's stupid. Also, this is a *one man project* and it takes FORTY EIGHT hours to compile on a high-end Intel processor.

    My HDRP projects and BIRP projects also build in a tiny percentage of the same time.
     
  32. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    A quick check through the 36+ hour build log shows this terrifying line:

    Cuz, uh, I didn't write that shader. Y'all did. And that's 78 minutes.

    And that's 12.2 hours to compile that many variants, for whatever reason unbeknownst to me for Unity deciding to compile that many. Quite what the other 24 hours is, I don't know, as Unity really doesn't have any helpful information spat out at the end. Maybe a nice pie chart? ;)
    35 hours. This was after a previous attempt on the cold build, which crashed after 12-15 hours.

    ...!
     

    Attached Files:

  33. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    The URP team is working on improving the cold build time.
    Just replacing
    multi_compile
    with
    shader_feature
    will simply not work. These two modes for dealing with keywords are controlled by different things.
    shader_feature
    is used for something that's controlled by materials.
    multi_compile
    is for features that can be switched at runtime - either by changing settings or by toggling individual keywords from scripts.
    Bad defaults? This is a tough balance between "a feature work by default" and "a user needs to go and change something in order to make a feature work". Maybe URP errs on the incorrect side of this. But one can't simply go and change how things work for everyone - if people upgrade to a new version something will stop working. So any change in this area can only happen in new projects.

    We have a good idea about how to deal with "too many variants" in general, but we need time to get it done properly.

    I mentioned a post with details on how to reduce build times two months ago.
    It's implemented a bit differently in URP. But we could take a different keyword as an example and the logic would still be the same.
    Take soft shadows. There's a keyword that controls that, and you can enable and disable soft shadows in different quality levels. If there are two quality levels in the build that have a different value for it (one says enable, another one says disable), you get 2x variants. Only if all quality levels agree on one state (enabled or disabled), the feature is considered as not switchable at runtime and the variants get stripped.
    The high number of possible variants comes from the number of features implemented in URP that can be toggled at runtime by default.
    This part in particular is getting improved with the fix that helps warm builds. Cold builds will also have a build time improvement, it just won't be as large because variant compilation is taking most of the time.
     
  34. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,352
    Yeah, I know. But with the combination of:
    - The URP standard shader having something like 40 multi_compile flags
    - Multi-compile being a thing you have to work around to not create a combinatoric explosion

    You will never have a good build pipeline for URP. Prefiltering is helping, but it is to some degree or another S*** polishing.

    You might have some analytics numbers on this, but I assume that the vast majority of projects either doesn't set any multi_compile keywords on materials/globally at all, or only set them as a side-effect of something Unity does internally. If we got to have a checklist for each shader with "these are the multi_compile keywords I intend to set", and errors if we tried to set any that we didn't check, that'd massively reduce the number of shaders compiled by default.

    This is an area where both you and Unreal are just massively bad (google Unreal Engine Shader Compile Memes), so if you don't have the resources to actually do the good thing here available, your bosses are bad at prioritizing.
     
  35. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    Last time I checked (a couple of months ago) it had around 25.
    As I said above, we have a good idea on how to deal with it, but it will take time to implement. And it will be better than a checklist for each shader with "these are the multi_compile keywords I intend to set".
     
  36. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    https://forum.unity.com/threads/improvements-to-shader-build-time-and-runtime-memory-usage.1305615/

    That one? It just seems to be detailing ways in which Unity is improving things. If there's information that's useful for me trying to work out what combination of tickboxes makes shaders compile slowly, it's lost on me.

    I've spent a little bit of time writing a little parse to actually give useful information, as opposed to the multi-megabyte logs Unity gives. I spotted this:

    "Shader [ "Shader Graphs/Hologram" pass "Universal Forward" (fp)] took 4.0m to strip and 5.0hours to compile. 15360 variants were compiled."

    Is that a lot of variants?

    But why? What did I do wrong? How could I make the shader not take so much time? I feel this particular deep dive should lead to a Best Practises page, especially as one of the four PILLARS of URP is "Faster iteration to get to quality"



    But nothing about this shader, as far as I know, jumps out at me as requiring *more than half a work day* to compile. Hell, it doesn't even cast shadows. I would like to learn and be enlightened as to what I've done so wrong, and further more to have Unity display the total variant space + stripped count *when* editing the shader or (as I've learned from your post) I link against a different URP Pipeline Asset)
     
  37. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    @djarcas there's a section in this post called "Build-time shader variant stripping". It has links to URP and HDRP specific pages.

    15360 is definitely a lot of variants. Most of them come from URP itself, not from your shader. Please check the links in the post I linked to and configure URP :)
     
  38. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    Again, I'm trying to push very hard here for a place within Unity that explains that my shader has 83 base variants, and it's x2 because of SSAO and x2 because of reflection probe blending, and (etc etc etc). I've ticked both of the tick boxes that should be ticked related to stripping, and I've *un*ticked 'optimize mesh data', and now I'm back to "I have zero idea why this shader takes 5 hours to compile'

    If no-where within Unity tells me how many variants there are *and why*, how can it ever be possible to fix this?
     
  39. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    There is no information about what causes the number of variants to grow within the Editor currently. We know it's missing, and we'll fix this, as soon as we get time for this.

    The closest thing you can do right now is open the generated shader, check what multi_compile directives it has and then check which options in the render pipeline settings they correspond to. It's definitely not straightforward, but this is a way to get the info you're interested in.
     
    djarcas likes this.
  40. Blepius

    Blepius

    Joined:
    Mar 9, 2021
    Posts:
    68
    There seem to be a lot of people heavily invested in this issue. If there are any aspects of this process that can be made public, the community may be able to help.

    I personally would likely be of little help, but I would love to read along and learn about more about how Unity works.
     
  41. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    That's surprisingly useful, and not something I've ever looked at previously. I notice that it supports 3 types of fog, for example (which I believe is 2x, 4x, 8x the variant count?) So that alone, if reduced to the single type of fog I *actually* use, would quarter this shader to a more reasonable (but still outrageous) 1.25 hours? (Is it even possible to disable the other 2 types of fog? Quartering the render time of *all* my shaders is no small gain)

    A lot of these are internal, and ofc there's nothing I expect to be able to do about those.

    upload_2022-11-30_23-15-47.png
     
  42. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    No, it's a single directive, so it's 4x variants (no fog, linear fog, exponential fog, exponential squared fog).
    I also think it's taken care of already - we perform stripping of fog modes internally based on the Graphics settings.

    But you can! :)
    There's information on this in the post I linked above.
    E.g. to strip any variant that uses the DEBUG_DISPLAY keyword, you need to check `Strip Debug Variants` in the URP global asset. Most of the light and shadows keywords are controlled by the Lighting and Shadows settings in the URP quality settings.
    If you make sure all the quality settings agree with each other (e.g. all enable soft shadows or all disable soft shadows), you can reduce the number of variants.
     
    Bwacky likes this.
  43. Bwacky

    Bwacky

    Joined:
    Nov 2, 2016
    Posts:
    214
    Happy to say, if someone hasn't tested yet, build times are normal again even in cold builds. Went from 30+ hours of waiting and canceling due to losing my patience to building under an hour with 2021.3.15f1. Similarly, addressables now build in a matter of minutes cold rather than over an hour for my case, and rebuilding afterwards with slight changes takes a few seconds. Didn't have to play around with stripping variants either.

    Thanks a lot for explaining the issue and helping us deal with the wait for a fix @aleksandrk - I think we all really appreciate it. Still, I wish issues like this would be considered priority fixes for whoever decides on priorities for LTS fixes when they cost everyone here a lot of time and money.
     
  44. haim96

    haim96

    Joined:
    May 24, 2013
    Posts:
    107
    confirmed also in 2022.2.0f ... yes, just released! i just cancel over 4 hour compilation...
     
  45. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    So, some numbers on the 2021.3.15 build. Bear in mind they will be massively skewed since my previous 48 hour cold build. Rather than expect Unity to get their act together and develop background asynchronous builds or actually optimise stuff, I went out and bought the fastest i9 that money can buy. That took my cold build time from 48 hours to 16.3 hours. My warm build time is now 96 minutes, which is far better, but obviously is still S***. That's 96 minutes for Unity to do *nothing*, to check everything and go 'no work needed to be done'. If I was running a warm build on my previous computer, I'd guesstimate that a warm build would be 4.8 hours.

    It's good that we have improvements, but quite frankly they're a day late and a dollar short.
     
  46. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    @djarcas can you share the Editor log for the warm build in DM? I'd like to take a look at what takes so long.
     
  47. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    Super happy to! This build log is a cold build, then a cancelled build - not sure why, then the warm build immediately afterwards. All I did was shove an extra whitespace into a single script to compare cold and build times.
     

    Attached Files:

  48. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    It looks like several shaders (mostly created using Shader Graph) are not affected by the change, this is why you have long warm build times.
    Could you please send me the "Shader Graphs/Hologram" one, for example?
     
  49. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    248
    Oh, the plot thickens!
     

    Attached Files:

    aleksandrk likes this.
  50. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,055
    @djarcas looks like prefiltering is picking it up properly. I took a deeper look into the log, and it seems that all shaders are left as is. This means that your render pipeline settings assets check all possible variants of settings.