Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Bug Build time taking almost a day - URP

Discussion in '2021.2 Beta' started by RenOli, Jun 29, 2021.

  1. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    226
    This was broken 15 months ago. I hope the URP decal fixes don't take this long. >_<
     
  2. kogi_rc

    kogi_rc

    Joined:
    Apr 23, 2019
    Posts:
    32
    BTW is this normal that using "Patch and run" option to alter only scripts in a Mono build also causes the shader variant processing to be triggered?

    Isn't it supposed to be faster way to alter scripts only without touching resources? Or am I doing something wrong?

    I was under impression that there is some caching going one, yet changing one line in an entirely graphics-unrelated code requires gigantic shader variant processing operation.

    EDIT: And then after the build doesn't work because addressable resources are not found after patching the installed .apk.
     
    Last edited: Sep 22, 2022
  3. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,749
    @kogi_rc no idea, sounds like a separate topic to me.
     
  4. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    226
    upload_2022-9-24_10-51-9.png

    I am eagerly awaiting the backport...
     
  5. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    226
    upload_2022-9-24_17-46-54.png

    Someone hold me :|
     
    tlamiczka and valarus like this.
  6. Andy-Touch

    Andy-Touch

    A Moon Shaped Bool Unity Legend

    Joined:
    May 5, 2014
    Posts:
    1,404
    valarus likes this.
  7. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    Ah haha. It's still building mate.

    upload_2022-9-25_0-52-47.png
     
  8. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,629
    Wait, did the progress number go down?
     
  9. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    I've stopped looking at it tbh, and have retreated into insanity.

    upload_2022-9-25_10-48-53.png
     
    IS_Twyker likes this.
  10. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    You are only being perverse at the moment... haha

    Also for science!
     
  11. Arnold_2013

    Arnold_2013

    Joined:
    Nov 24, 2013
    Posts:
    202
    Maybe you should start a new thread, because this one is labeled "build time taking almost a day urp". :p

    Hopefully it finished before the bug is fixed ;).

    Seriously once it stops being possible to run a build over night (~8 hours), it just becomes useless to do a build at all.

    Good luck.
     
    hippocoder likes this.
  12. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Agreed, for most use-cases a build should not take more than an hour, realistically. At least the competition takes less time than that for anything I ever tried. So past an hour, it's reasonable to look into how to manually control the process, generate less variants and so on.

    If you can't, I'd go as far as to say that I recommend using HDRP or Built-in instead since utterly crippling build times is extremely expensive in time/money.
     
  13. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,629
    The numbers going up and down show that something is probably going very wrong under the hood that goes beyond the raw number of shader variants. Compiling 300k shaders shouldn't take anywhere near that long, I worked on <other engine> projects which had at least that many shaders and build in a fraction of that time.

    We got our first request for porting a game made using 2021.1 for consoles (everyone else seemed to be on 2019 or 2020) and I'm very wary of this bug showing up and making the porting process unfeasible because the project needs to be upgraded to 2021.3 because 2021.1 doesn't support the console SDK versions required for publishing.

    Just for reference, our 2020.3 projects build in under 15 minutes when building addressables and in under 3 when building only code and we do several builds a day to test the many things that cannot be tested on editor/PC like achievements, platform-specific multiplayer integration, process lifetime management, platform-specific save logic, and (specially) performance optimization.
     
    Last edited: Sep 25, 2022
    hippocoder likes this.
  14. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    I hate science.

    upload_2022-9-25_15-42-10.png

    (for reference, I went to the latest 2021 LTS as it has a fix for decal rendering (note - it does not have a fix) - after TEN HOURS of reimporting my scene, I then did a build - 40 hours later, it's still going. GTX 3080, 64 gigs of fast ram, NVME drives all around and overclocked i9-9900K - I can't even throw any more hardware at this)

    I guess this is what's been described as a 'cold' build? Feels more like Captain America being frozen at this point.
     
    hippocoder likes this.
  15. Andy-Touch

    Andy-Touch

    A Moon Shaped Bool Unity Legend

    Joined:
    May 5, 2014
    Posts:
    1,404
    The unfortunate thing is this is not an isolated project issue; it is affecting a lot of projects with URP (including Gigaya). Hopefully Unity leadership takes note of this and puts it at a very high priority to tackle sooner rather than later. :)
     
    projectorgames_unity likes this.
  16. Catsploration

    Catsploration

    Joined:
    Mar 9, 2021
    Posts:
    55
    It's hard to not feel as if this is part of why Gigaya was canceled. However, this is exactly the reason why we all felt that Gigaya was so important.

    It would have been amazing if the Gigaya team was able to trailblaze these new Unity versions instead of releasing them onto the entire development community. As it is, each individual bug is discovered hundreds of times.

    (Although, it does make me really happy to see that some Unity developers are still internally using Gigaya as a test bed.)
     
    Last edited: Sep 25, 2022
    projectorgames_unity likes this.
  17. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    And completed a hair short of 47 hours. I just want to stress that I'm a one-man dev team, not some sort of AAA project with a hundred artists and thousands of shaders :/ No idea if the log file is of use to anyone, but here it is. Obviously if there was some sort of solution or something I'm doing wrong, I'm *all for it*, otherwise, well; I started to update to the new Unity version at about 10am Friday morning, and it's now midnight on Sunday. Please, Unity lads; fix this as soon as possible.
     

    Attached Files:

    chriseborn likes this.
  18. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Almost a day, twice.
     
  19. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,629
    I took a look at the log. Picked one example of one pass for a single shader:

    Code (CSharp):
    1. Compiling shader "Shader Graphs/Detail TriPlanar Glow Shader" pass "Universal Forward" (fp)
    2. STRIPPING: Shader Graphs/Detail TriPlanar Glow Shader (Universal Forward pass) (Fragment) - Remaining shader variants = 2688/1000000 = 0.2688% - Total = 866861/293832849 = 0.2950184% TimeMs=39022.0982
    3. STRIPPING: Shader Graphs/Detail TriPlanar Glow Shader (Universal Forward pass) (Fragment) - Remaining shader variants = 1152/572864 = 0.2010949% - Total = 868013/294405713 = 0.2948356% TimeMs=21898.2799
    4.     3840 / 1572864 variants left after stripping, processed in 61.57 seconds
    5.     starting compilation...
    6.     [ 62s] 282 / 3840 variants ready
    7.     [128s] 527 / 3840 variants ready
    8.     [190s] 703 / 3840 variants ready
    9.     [255s] 1108 / 3840 variants ready
    10.     [318s] 1531 / 3840 variants ready
    11.     [385s] 1771 / 3840 variants ready
    12.     [448s] 2186 / 3840 variants ready
    13.     [512s] 2354 / 3840 variants ready
    14.     [575s] 2777 / 3840 variants ready
    15.     [636s] 3031 / 3840 variants ready
    16.     [699s] 3409 / 3840 variants ready
    17.     [761s] 3648 / 3840 variants ready
    18.     finished in 785.98 seconds. Local cache hits 1 (0.00s CPU time), remote cache hits 0 (0.00s CPU time), compiled 3839 variants (12574.25s CPU time), skipped 0 variants
    If I'm reading it correctly:
    1) Stripping was run twice for some reason, resulting in 3840 variants stripped from of over 1.5 million and took 61.57 seconds total.
    2) Compiling the 3.8 thousand variants took 13 minutes (785.98 seconds) using 16 threads.
    3) Out of 3839 variants only one was found in cache.

    Individually it doesn't look too bad, but I added all the "TimeMs" values from the lines starting with "STRIPPING" and they show stripping alone took 3.25 hours.

    Then I did the same with the times from the "finished in" lines and the result is 42.5 hours compiling shaders. Ouch! Even if stripping took 0 seconds it would reduce the build time by only 7% in this case.

    The "STRIPPING" logs seem to show the running totals, so I guess this to be the total compiled variants versus the total possible variants, is that right?
    Code (CSharp):
    1. Total = 900665/301893621
    So, compiled 900 thousand shaders out from 300 million possible variants.

    Skimming the log for outliers, seems the ForwardLit pass generates large amount of variants. But the real problem is that it's doing it hundreds of times:
    Code (CSharp):
    1. STRIPPING: Universal Render Pipeline/Lit (ForwardLit pass) (Vertex) - Remaining shader variants = 3840/15360 = 25% - Total = 153173/12476297 = 1.227712% TimeMs=996.2921
    2. STRIPPING: Universal Render Pipeline/Lit (ForwardLit pass) (Fragment) - Remaining shader variants = 0/1000000 = 0% - Total = 153173/13476297 = 1.136611% TimeMs=37701.4033
    3. STRIPPING: Universal Render Pipeline/Lit (ForwardLit pass) (Fragment) - Remaining shader variants = 0/1000000 = 0% - Total = 153173/14476297 = 1.058095% TimeMs=37561.3079
    4. STRIPPING: Universal Render Pipeline/Lit (ForwardLit pass) (Fragment) - Remaining shader variants = 0/1000000 = 0% - Total = 153173/15476297 = 0.9897264% TimeMs=37967.461
    5. STRIPPING: Universal Render Pipeline/Lit (ForwardLit pass) (Fragment) - Remaining shader variants = 0/1000000 = 0% - Total = 153173/16476297 = 0.9296567% TimeMs=37721.2604
    6. STRIPPING: Universal Render Pipeline/Lit (ForwardLit pass) (Fragment) - Remaining shader variants = 0/1000000 = 0% - Total = 153173/17476297 = 0.8764614% TimeMs=37743.7309
    7. STRIPPING: Universal Render Pipeline/Lit (ForwardLit pass) (Fragment) - Remaining shader variants = 0/1000000 = 0% - Total = 153173/18476297 = 0.8290244% TimeMs=37538.2389
    8. ...
    Why is it processing the same shader fragment pass over and over again, stripping a different amount of variants (many times zero) at each pass?

    Hey @projectorgames_unity a question: do you have this setting enabled?
     
    Last edited: Sep 26, 2022
  20. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,749
    Stripping is being run in batches to keep the number of variants being processed in C# sane. A single batch is up to 1 million variants. If the build process has to go over 300 million variants for a single shader, it will call stripping 300 times.

    We're in the final stages of testing a PR that reduces that time. I already wrote about it before.

    You should look at "3840 / 1572864 variants left after stripping" - this is the final number of variants, these need to be compiled or fetched from the cache.

    This post contains a link to a guide on how to reduce build times.
     
    projectorgames_unity likes this.
  21. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    I do indeed have that ticked. I also came to the same conclusion you did, that everyone has been talking about the stripping part, yet my log showed 42-something hours of actual shader compile time.

    @aleksandrk Is there any sort of ETA? I'm writing a multiplayer game, so as you can imagine, I have to make constant builds.

     
  22. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,629
    Try disabling it. There are other threads about build times going completely out of control with it enabled with the same shaders being compiled over and over again like in your case.
     
  23. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    I'm going to be quite honest here.

    https://forum.unity.com/threads/improvements-to-shader-build-time-and-runtime-memory-usage.1305615/

    It doesn't. It's a post about memory usage of shaders. I also have done *nothing* fancy in this project, nothing at all. There's maybe 50 or 60 shaders, the vast majority of which are unused, and all the ones I've written are simple shadergraph things. Furthermore, please, telling people to go away and see if they can improve their build times due to a Unity change between 2021.2.19a and 2021.2.0b1 is the most horrible sort of victim blaming. I had 6-ish minute long warm builds, and a cold build was a painful hour or two. Now I've literally watched my computer churn for 60 hours, simply moving to a new minor Unity version bump and then making a build to test nothing's broken. Unity is supposed to be the other AAA competitor to Unreal. If I was still working in the AAA industry, and the build times rocketed 60-fold, that engine would be put in the bin, either immediately, or for the next project. It's been broken at LEAST since :
    - at LEAST 15 months. I genuinely cannot think of anything higher priority than getting this fixed, and the fact that Unity's regression testing doesn't even contain 'build a reasonably-sized URP game' is utterly terrifying. Please - when will this be available?
     
    psyconius and pm007 like this.
  24. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    I'm going to go out on a limb here and say that because I've just watched my PC build 800 trillion shaders in the last 48 hours, I've probably weathered the storm and they're all cached now... right? I'll find out this evening when I attempt my nightly build. (I had plans to setup a second build PC, but that won't have finished the initial build until at least...er...Wednesday) If the next 'warm' build is still overly painful, I'll disable it, ofc.
     
  25. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,629
    Fair enough.
     
  26. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    Please tell me this isn't a tacit agreement that, after the fix, the 3.7 hours of stripping is improved, but the 42 hours of shader compilation time is the same.

    Btw, I realise this has been a very negative thread, especially from me, but I do want to thank you for actually responding to us, attempting to help, and letting us know that things are changing. Quite often, Unity is a vast wall of silence and fixes are a surprise. <3
     
    Last edited: Sep 26, 2022
    gnp89 likes this.
  27. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,629
    It's strange how the total number of pre-stripping variants got into 300 million when the URP forward lit pass has a max of 1 million variants. That implies close to 300 shaders with a forward lit pass.
     
  28. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    upload_2022-9-26_11-31-5.png

    Unless I can't count (possible, it's a Monday morning), there's 141 shaders in the entire project. Loads of those aren't used in the build however, and there's no simple way to check that (right click -> find references in scene hilariously takes over a minute to complete with 'legacy search', so I ain't doing that by hand ;))
     
  29. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,629
    You really should try unticking the "optimize mesh data" when possible. I had bad experiences with that all the way back in Unity 2019, and seems the flag has become completely broken as of late.

    That flag is supposed to check which data the materials used by a mesh actually read (like uv channels, vertex colors, tangents, etc) and remove unused data from the mesh itself at build time to reduce memory and file size. Exactly how it does it, and what does it do when it finds a mesh reference that doesn't have a material attached, I don't know, but I suspect the logic behind it triggers redundant shader compilations ballooning the build times.

    I know it's frustrating to have to scrap for workarounds but it's the only thing we have any control over when it comes to Unity, while groveling to get our pleas heard and waiting months/years for a fix.
     
  30. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,749
    I just checked, it's 12 million based on multi_compile directives.
    The rest should come from material usage (e.g. if you have 25 materials using different keywords for the shader_feature directives, you end up with 300 million variants before stripping).

    Yes, here: #178
     
    projectorgames_unity and djarcas like this.
  31. colinsauce00

    colinsauce00

    Joined:
    Jul 13, 2021
    Posts:
    6
    Hi guys, using latest version of unity LTS. Full finished game, builds were sub 5 minutes consistently. I imported an asset from the store that was only meant for SRP (using URP) and it made all materials pink, im guessing it changed my project to SRP when I clicked "fix now" at 1am. I instantly changed my project back to URP. Issue is, I now get 15 minute build times with the issue being forward lit shaders. I have tried multiple things and am now reimporting the project to try and fix. The build itself is the same in every way it seems the shaders are just no longer cached I guess. Thought I would tell you since this seems to be a year long issue.
     
  32. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,104
    URP is a SRP.

    SRP => Scriptable Render Pipeline. There are two built in SRP's; URP and HDRP. You can also in theory make your own.

    Maybe the asset was based on the built in rendering pipeline (aka. BIRP)?
     
  33. Bwacky

    Bwacky

    Joined:
    Nov 2, 2016
    Posts:
    90
    Did the fix ship for 2023.1 yet by any chance? Still hoping for 2021.3 sometimes soon.
     
    PieterAlbers likes this.
  34. PieterAlbers

    PieterAlbers

    Joined:
    Dec 2, 2014
    Posts:
    222
    Yeah, me too!
     
  35. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,749
    Yes, it will be available in 2023.1.0a14.
    Backports have been started, but they will take some time.
     
    funkyCoty likes this.
  36. Bwacky

    Bwacky

    Joined:
    Nov 2, 2016
    Posts:
    90
    :oops::oops::oops:
    Some time being a few days, a few weeks or something closer to a year?
    Genuinely question, I'm considering moving up to 2023.1 just so we can keep builds running even if it breaks all our shaders and frameworks. It's been a massive detriment to our release cycle.
     
  37. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,749
    It's going to be in "several weeks" range. Can't say for sure - the backport is complex and release management will likely want to wait a bit and see that it doesn't break user projects on latest alpha.
     
    funkyCoty and Bwacky like this.
  38. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    226
    Yup, ditto.
     
  39. psyconius

    psyconius

    Joined:
    May 16, 2018
    Posts:
    5
    Yeah.. I've checked all these threads and tried all the tricks. I have no idea why I am always here. My last project started doing it and randomly decided to not do it for me and had a build in 30 mins in VR.

    This is a rather large demo scene (Tower by PolyArt) that I am testing my project with. I use the URP normal shaders, a few shaders included with that, and then a single custom shader from the store, FlatKit. This is the 4th time I've restarted. I let it go for 3 hours and then tried tips and tricks again and I just dunno. I'd love to get a quick build for some friends to test. Otherwise, like everyone it runs fine in editor. I haven't even seen anyone with a number like mine.. 154618M
    upload_2022-10-12_7-12-50.png

    Still no clue what this number means then
    upload_2022-10-12_7-12-54.png
     
  40. LightYarn

    LightYarn

    Joined:
    Nov 18, 2012
    Posts:
    69
    2023.1.0a14 is available since yesterday with:
    Editor: Enabling shader keyword pre-filtering so that build process does not have to enumerate through full shader variant space. This fixes the issue of URP builds even with warm shader cache taking really long time. This fix does not reduce the time spent on actually compiling shaders


    Did anybody test it? Any actual improvements?
     
    DavidZobrist likes this.
  41. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    226
    I assume this is also coming, given that I think I spent 42 hours compiling the shaders in my project? ;)
     
  42. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,629
    I doubt it.

    BTW, did you have the same long times with a warm build? Did you get to try disabling the "optimize mesh data" check box?
     
  43. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    226
    I unticked it, but the chances of me losing 2 days of work by forcing a cold build are about zero. But guess who's installing 2023.1.0a14 right now!
     
  44. fortunacio

    fortunacio

    Joined:
    Aug 30, 2014
    Posts:
    5
    Any update?
     
  45. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    1,629
    From older threads and based on the logs you shared there's a good chance this is the problem in your case. Having "optimize mesh data" enabled has been known to drastically increase the number of compiled shaders all the way back to before URP was a thing.

    My suspicion is that the feature has been broken for years and causes Unity to re-compile the same shader variants for each mesh in the project to find out what mesh vertex data can be safely stripped away because no shader uses it. When combined with the high number of variants generates by URP in 2021 and later the problem blows up into ridiculous proportions, and Unity never notices it because their internal test projects have only a fraction of the amount of assets of a real world project.

    You could always backup your library folder and make a test. If things go awry, just bring back your backed up library folder and your already compiled shaders are safe. Or test it on a full copy of the project.
     
  46. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    226
    Yes. The import process is [expletive] horrible these days. I just imported a 20 gig + project into 2023 BIRP and it took under 15 minutes.

    Importing a project of about the same size into URP...?

    upload_2022-10-17_13-26-8.png

    See you in... 8 hours? Ridiculous. Why does stuff like this not happen on demand? 90% of these models and textures aren't used in any given scene. Import stuff in the background and on-demand ffs.
     
    Last edited: Oct 17, 2022
  47. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,104
    Because the project to fix Unity's main-thread asset importer and replace it with an async one never got finished and we haven't heard about it for a while.

    Got nothing to do with build times, though!
     
  48. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    226
    Well, kinda/sorta it does. I'm upgrading Unity versions to get around a crippling bug that takes up literal days of my time, only to be hit with... a cripplingly poor design that's taking up a day of my time. ;)

    upload_2022-10-17_16-58-22.png

    Again, just to be clear, if I was still in the AAA space, this would be the last time I ever used Unity in any project. Imagine a team of 200 people, all losing a day of work as you had to bump versions.
     
  49. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    226
    upload_2022-10-17_21-3-51.png

    *squint* Someone did test this before releasing it right.
     
  50. djarcas

    djarcas

    Joined:
    Nov 15, 2012
    Posts:
    226
    I've abandoned trying. So much else is broken; lightmapping, resolution scaling, ScreenPositionNode, a weird error that says "No script asset for AssetVersion. Check that the definition is in a file of the same name and that it compiles properly." but with no callstack or any other clue where it's come from; I think that error leads to a >half hour import process of the SAME meshes on each restart, along with it telling me all my URP materials are out of date. So now comes an overnight rollback and reimport, and I'll wait for Unity to get their act together ;)