Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only. On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live. Read our full announcement for more information and let us know if you have any questions.

Bug Build time taking almost a day - URP

Discussion in '2021.2 Beta' started by RenOli, Jun 29, 2021.

  1. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    3,143
     
  2. iceb_

    iceb_

    Joined:
    Nov 10, 2015
    Posts:
    98
  3. Kronnect

    Kronnect

    Joined:
    Nov 16, 2014
    Posts:
    2,915
    We’re told by Unity support that this specific case of lengthy compilation time only happened in Unity 2021 due to a FXC compiler bug and that they couldn’t do anything (the issue didn’t replicate in 2019/2020).
    We already fixed it by changing the shader so it ‘s no longer an issue with the latest version of the asset (5.0) in Unity 2021.
     
  4. iceb_

    iceb_

    Joined:
    Nov 10, 2015
    Posts:
    98
    The issue for me and probably others is that it also happens in other situations too, when not using that particular volumetric light asset from Kronnect.
     
  5. TJNBG

    TJNBG

    Joined:
    Apr 7, 2021
    Posts:
    26
    Correct.
    It reproduces on Unity's standard URP sample project.
     
    spamove and iceb_ like this.
  6. iceb_

    iceb_

    Joined:
    Nov 10, 2015
    Posts:
    98
    I haven't seen an official issue tracker for it yet, aside from Kronnect's separate issue, do you know of any?
     
  7. TJNBG

    TJNBG

    Joined:
    Apr 7, 2021
    Posts:
    26
    No, my bug report did not convert to an issue on the public tracker.
     
  8. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    Yes, we're looking into improving build times in general. This has nothing to do with the issue reported by Kronnect.
     
    valarnur and iceb_ like this.
  9. iceb_

    iceb_

    Joined:
    Nov 10, 2015
    Posts:
    98
    By the way, it builds faster if it's only targeting DX11. But once you add on others aside from DX11, it adds up to very long build times.
     
  10. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    Others as in Vulkan or GLCore?
     
  11. iceb_

    iceb_

    Joined:
    Nov 10, 2015
    Posts:
    98
    yes, vulvan, GLCore, DX12. Adding some or all of them will add more and more non-linearly to the build time by a lot. And it compounds the problem with more complex shaders too.
     
  12. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    DX12 shouldn't add anything.

    Others are a known issue, and a fix is in the queue :)
     
    iceb_ likes this.
  13. iceb_

    iceb_

    Joined:
    Nov 10, 2015
    Posts:
    98
    that's awesome to hear! Is there an issue tracker for this or expected patch where it'll come in on?
     
  14. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    Unfortunately, there isn't. I'll post here when I know the patch numbers.
     
  15. PPPoe252110

    PPPoe252110

    Joined:
    Apr 11, 2019
    Posts:
    1
  16. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    Here a Screesnhot of my results in 2021.3.x....
    Check the Folder generation amount ( ~3000).
    This should be introduced around 2021.2.13+.
    Before there were only around 16 Folders.
    These LTS release are still not usable with this building behaviour.

    1.JPG
     

    Attached Files:

    • 1.jpg
      1.jpg
      File size:
      62.1 KB
      Views:
      222
  17. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    @keeponshading
    The folder count changed after we introduced shader cache trimming. We reorganised the way variants are cached. It's completely unrelated :)
    I think I mentioned this at some point. The URP build times went up because there were many new features added to URP in 2021.2.

    @PPPoe252110 It's totally possible to make a shader with too many variants. This is a custom shader it's preparing for compilation - you can check the Editor log and reduce the number of variants there to a sane number.
     
  18. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937

    The build time from Adressables and normal build is still 5 to 10h in 2021.3.2f1.
    This makes the LTS releases completly unusable.

    Before (~2021.2.15) around 20 to 40min.
    Tested on +10 mid size (50GB) to large (300GB) projects on 4 PC´s.

    If you enable Development build ist goes up to +15h. I am stoped the builds around 15h.
    Without Develompment build like i said around 10h.

    The question is how can i get back to 20 to 40min build time i had ahave the last 1.5 years across all 2020 and 2021 releases before 2021.2.15, allowing to use the LTS releases?
     
  19. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    @keeponshading can you share (in a DM) the Editor log from one of the long builds you did?
     
  20. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    Yes. I have to check the information inside and send it to you when this one is finished.
    It´s running since around 2h in 2021.3.2f1 on

    3.JPG


    4.JPG

    1.JPG



    I thought these fixes in 2021.3.2 change something.
    5.JPG
    But they doesn´t.
     
    Last edited: May 9, 2022
  21. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    This behaviour scales about all projects i tested. (around 20)
    You could simply take the SpaceShipDemo and Unity Raytracing examples from Github
    and compare the build time from 2021.2.14 and 2021.3.x.

    In a second step simply add some Assets to simulate real world project size and build again and comapre
    Build time from 2021.2.14 vs 2021.3.x..

    In a third step simply mark some Adressables in these projects and build these to simulate further real world projects and comare build time from 2021.2.14 and 2021.3.x.

    In all three cases you will get same behaviour and exploding build times due to shader compiler factor 10 to 40..

    I am still wondering why this hat not been done before releasing an LTS.
     
    Last edited: May 9, 2022
    projectorgames_unity likes this.
  22. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    The problem is that all Workstations become unusable for hours.

    Over 200 ShaderCompiler Tasks blocking 40GB of RAM...
    Still ongoing....

    6t.JPG

    and this is the fast PC we have (specs above).
    It further scales by using slower workstations.
     
  23. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,725
    Can you check how many variants are being compiled? Also are you doing any shader stripping yourself?
     
  24. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    No. I dont do custom shader stripping by myself.

    Here some data.
    200 000 to 500 000 files in shader cache.
    16 folders in older releases. Now 3000 Folders plus in LTS releases.
    Around 5 to 10 GB Shadercache madness.

    https://forum.unity.com/threads/build-time-taking-almost-a-day-urp.1133605/page-2#post-8112830

    Development build disabled because it s not possible to wait 20h +.
     
  25. kogi_rc

    kogi_rc

    Joined:
    Apr 23, 2019
    Posts:
    39
    My build times also jumped from 4 minutes to 30 minutes (Vulkan mobile development build). This is not a case of some features being added and reasonably increasing build time. Something is broken beyond that. I'm eagerly waiting for a fix mentioned in #62.
     
  26. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    This would be factor 8 and matches with my 10x to 20x according to project size.
    Actual running one

    7.JPG

    Alone the Lit Tesselation Forward vp is running now for 25min+ on over 200 Threads...

    9.JPG
     
    Last edited: May 9, 2022
  27. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    live update - still running

    10.JPG
     
  28. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    @keeponshading are you aware this thread is about URP build time?
    Shaders in HDRP take a while to compile, so if you have over 2500 variants, it's expected that the build takes long.
     
  29. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    yes
    yes. and i am aware that the hdrp mess started by changes made in urp shader compiler around 2021.2.15. I am also aware that there is a general denial that any problem exists in this thread.

    So we could extend the description to urp/hdrp or i could convert all my shaders in projects to urp to show that the shader compiler is massively broken and not tested at all before releasing this LTS
     
    Last edited: May 9, 2022
  30. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    There's no such thing as "URP shader compiler".
    If you read the thread carefully, you'll find that this statement is incorrect.
     
    hippocoder likes this.
  31. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    Ok. So what i have to do get in a workable state with my projects in LTS?
    Downloading common Unity Github Demos, optionally integrate Adressables, then building these in 2021.2.14 and 2021.3.x to show the build increase factor around 10+?
     
  32. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    @keeponshading it would be great if you could send me the Editor log from one of those lengthy builds as I asked in #69
     
  33. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    Hi. Which part do you would need.
    It ´s 190MB in size?
     
  34. swantonb

    swantonb

    Joined:
    Apr 10, 2018
    Posts:
    177
    So all this is normal? Ive read that it was addressed but this happens in 2021.3 lts and even in 2022.2, even more time and more variants in 2022.2. 3500M variants for URP/Lit.

    Build takes 14 hours.

    At least sometimes it goes fast through the variants (i mean the "fp" things, like 43000/43000fp) but that happens rarely and randomly, not even sure whats the criteria, its not the first build. It used to be the first build 14 hours, stay for the fps for huge amount of times but when building again over that project it would go fast through the fps. In 2022.2 that doesnt apply, it seems every time it will take 14 hours, basically make around 15 fps every 1 second and as it advances it makes 15 fps every ~5 seconds, even more. Why is it this slow? And why in earlier versions of unity (2021.2 or .3) it sometimes moved very fast through the fps even if it wasnt the first build? Could you tell me (@aleksandrk) whats the criteria that makes it move fast or slow through those fps? If it was the fast moving it would be acceptable, im ok even with 2 hours builds but 14 is just too much. I dont have realtime protection enabled from windows defender as Ive read a while ago in some thread that it would be the cause of the moving slow through the fps, but its not true, like i said, sometimes it moves fast and sometimes slow, could you tell me what can i do for it to at least move fast and not 15 fps every 1 second and as it advances 15 fps every ~5 seconds? I guess no need to mention that this is done twice for the lit shader, so 43000 fps x2.

    video 1 showing around 15fps every 1 second: https://gyazo.com/ccd26c45e9395ff6ff28b543b35ba6ef
    video 2 showing around 15 fps every ~5 seconds: https://gyazo.com/83d11da3750c73753448cb7d8741df64


    Additionally, I even commented light cookies, dots instancing and some more keywords that i dont use (actually i did use cookies but there is the nasty bug where the sampler limit is exceded that was even postponed, i dont even understand why you guys keep the light cookie keyword if it prevents the player from building. I mean this: https://issuetracker.unity3d.com/issues/sampler-limit-16-exceeded-in-urp-lit-shader ). Anyway, point is even with some keywords commented i have those 43000 variants, without those comments it would probably be much higher than that.


    edit: it's even more than 5 seconds. Now it's at 35000/43000 and it adds around 15fps every 7 seconds. Probably will be even more as it gets to the end
     
    Last edited: May 16, 2022
  35. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    This is the number of variants before stripping. We're working on addressing this.

    There's a couple possibilities here. One is that "fast" goes through the variants that have already been compiled earlier and cached, and "slow" compiles each variant anew. Another is that with that many variants to compile, your machine runs out of resources (e.g. RAM) and slows down at some point. It also could be something else - hard to say.

    Either way, 43000 variants is way too much. I would suggest to try and reduce the number of variants you use.
     
  36. swantonb

    swantonb

    Joined:
    Apr 10, 2018
    Posts:
    177
    How would I do that though? It's not like I use that many, I mean I have a few materials that use detail normals, detail albedo and emmisions, a few transparent ones but overall the rest of the materials are only using albedo map and normal map. But probably not even close to 43000 variants. Or maybe, I guess, since this is the number.
     
  37. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    I would go and check the URP settings - they drive the variant stripping.
     
  38. TJNBG

    TJNBG

    Joined:
    Apr 7, 2021
    Posts:
    26
    Thanks for keeping everyone updated @aleksandrk .
    Do you know why this is not listed in the known issues of 2021.3? Meaning bugs 1137857 and 1383867?
     
  39. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    One of these is closed as it never got reproduced, another one got reopened recently and is in QA.
    I don't really know how the list of known issues is generated, sorry.
     
    TJNBG likes this.
  40. swantonb

    swantonb

    Joined:
    Apr 10, 2018
    Posts:
    177
  41. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    No, the ones that have things like enabling shadows, shadow quality, etc.
     
  42. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    The fix for Vulkan and GL landed, should be available in 2022.2.0a14, 2022.1.2f1 and 2021.3.4f1 when they're published.
     
  43. kogi_rc

    kogi_rc

    Joined:
    Apr 23, 2019
    Posts:
    39
    Since the fix is coming in 2021.3.4f1 and the currently released LTS editor is 2 versions behind I suspect we're still a couple of weeks away from this fix being public. I've been also trying to debug this problem for our team but it is extremely time consuming debugging process and it's hard to test more than few options per day because of the build time.

    Is there any more information that could let people with the sudden increease in build times verify that we are in fact affected by this specific problem that has just been fixed in unreleased versions? It would save us some time if we can assume that the situtation will become better next month and we don't need to troubleshoot on our own.

    Is there anything specific to look in our project setup or log that triggers the problem?
     
  44. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    @kogi_rc you can check the Editor log - if you're compiling not only for Vulkan and/or GL, you'll see a drastic difference in the number of variants between them and the other graphics API(s).
    The difference in Gigaya: 26K variants of URP/Lit on DX11, 400K variants on Vulkan (before the fix, obviously).
     
  45. kogi_rc

    kogi_rc

    Joined:
    Apr 23, 2019
    Posts:
    39
    Thanks for the information.

    I tested both builds, here are the logs for lines including and following "Compiling shader "Universal Render Pipeline/Lit". Time measurements aren't comparable because they were built on different machines. Besides, the time measurements don't really reflect actual time spent between these lines appearing because these lines were filtered out of the 30k line log.

    Android - Auto Graphics API - Vulkan and OpenGLES3 when the option is clicked off:
    https://pastebin.com/L7uJAN53

    PC - D3D11:
    https://pastebin.com/LJPPiY8u

    PC one has one line with very high amount of variants:

    Compiling shader "Universal Render Pipeline/Lit" pass "ForwardLit" (fp)
    800 / 1474560 variants left after stripping, processed in 134.54 seconds

    but generally everything else seems much more reasonable than Android one.

    Command to filter log files is included in pastebin logs, if anyone else wants to check it.
     
  46. TJNBG

    TJNBG

    Joined:
    Apr 7, 2021
    Posts:
    26
    Unity is actually recompiling a few variants every time, making incremental builds slower for no reason. That was the initial cause for the bug report.

    The fact that 2021.2 introduced several new URP features should only have caused the initial build to be longer.
     
  47. swantonb

    swantonb

    Joined:
    Apr 10, 2018
    Posts:
    177
    I see, so you mean the URP asset. Then if I have 4 URP assets for quality presets which are changeable from the settings, then there may be more variants?
     
  48. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    I fixed something recently that looks like this issue - should be available in 2021.3.4f1.
    https://issuetracker.unity3d.com/is...-building-the-player-after-each-editor-launch

    Yes. If one of them uses, for example, hard shadows and another one uses soft shadows, both hard and soft shadow variants will end up in the build.
     
    valarnur, hippocoder and swantonb like this.
  49. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    748
    Wanted to chime in. We shipped a game on PC/Switch using Unity 2019.x, when the game was completed we had roughly 100,000 variants in one of our shaders haha. But still, the build maybe took a few hours. It seems that shader compile times themselves have drastically increased somewhere between 2019 and 2022.

    We were of course using the built in render pipeline back then, and now we're using URP for future projects. We have a project with 10,000 or so variants and it takes hours the first time we compile. It just seems like each shader compile is taking drastically longer than a single shader compile used to. In both cases we are writing custom vert/frag shaders (both back in built in and even now with urp).

    I guess what I want to know is, is there something we can do to lower individual shader compile times?
     
    kogi_rc likes this.
  50. TJNBG

    TJNBG

    Joined:
    Apr 7, 2021
    Posts:
    26
    aleksandrk likes this.