Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice
  2. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  3. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Bug Build time taking almost a day - URP

Discussion in '2021.2 Beta' started by RenOli, Jun 29, 2021.

  1. RenOli

    RenOli

    Joined:
    Jan 23, 2013
    Posts:
    102
    Hi, I am experiencing some weird, extreme long, build time since I got the latest Unity 2021.2.0b1

    On the 2021.2.19a I was able to build it in around 10 seconds, and first builds around 3 - 5 minutes.

    Since I got the latest, it has been around 15 hours and it is still going.

    It seems to be stuck in Compiling Shaders Variants (which, for whatever reason, also compiles the Deferred even though I am just using Forward)

    I am making a build for Windows. I am working with Multiplayer, I just wanted to do a fast test on the progress I've done.

    I've worked on massive projects before, thousands of shaders, and the build was around 1 - 3 hours. (There were some PCs that was able to build that in 8 minutes, for unknowing reasons.)

    My current project is still in early stages, very small, I am working alone and I deleted almost all shaders in the project.

    Not sure if it is a bug or not, but it started at the exact time I updated to the latest version.

    Thank you.
     
    spamove likes this.
  2. PutridEx

    PutridEx

    Joined:
    Feb 3, 2021
    Posts:
    1,136
    If you still have it, editor logs of the build process would be helpful
     
  3. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    3,136
    Hi @RenOli,

    Could you please submit a bug report with a reproduction project attached?
     
    lhy_ps2 likes this.
  4. jaguero

    jaguero

    Joined:
    Dec 17, 2018
    Posts:
    8
    Hi, any advance in this topic? I have a similar problem in Unity 2021.2.0b13:
    The "Compiling shaders" phase is taking a very long time in build, i think it should be a bug because it tries to compile high number of variants as seen in the next image:

    upload_2021-10-18_10-54-21.png

    This is not the first try on making a build, but every time I tried in "mono" ended with an "OutOfMemory" error after several hours stuck in "Compiling shaders".

    This last build i'm making now is made in IL2CPP, but the only thing i see is that the build is a bit faster, but still slow for a non massive project, asset folder is only 5.9Gb and there are less than 10 custom shaderGraph-made shaders. Is this any known issue solved in a next patch? Or is it an isolated case, if i need to provide info please tell me what do you need.

    Project is now in Unity 2021.2.0b13 using the Universal Render Pipeline.

    Thanks in advance
     
  5. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    3,136
    Hi @jaguero,

    Can you please try updating to b16 to see whether it helps with this problem? A couple of fixes and improvements to shader variant stripping and the incremental build pipeline have landed in the last updates.
     
  6. jaguero

    jaguero

    Joined:
    Dec 17, 2018
    Posts:
    8
    Hi @LeonhardP , thanks for the advice, but the build is still taking too long so far;

    upload_2021-10-19_8-31-30.png

    I let the computer on while it was away, but i don't know if the hdd tuned off, so i will wait a bit longer to see if it can complete it or if it crashes, if you have any ideas on what can be causing this long times, i will be glad to hear and try any solution
     
  7. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    6,342
    Check your task manager - is Windows Defender running? You might try to add Unity install directory and the project directory from Defender's ignored directories, as it might be continually scanning all the temp files created by the shader compiler.
     
  8. jaguero

    jaguero

    Joined:
    Dec 17, 2018
    Posts:
    8
    Hi @Baste , Windows defender is not running, but i have another antivirus, i don't know if it is the problem as in task manager all the CPU usage was used by "unity shader compiler". Anyway, it has ended shading compiling right now, so i will try another build after the build had finished without the antivirus to see if anything changes.
     
  9. rz_0lento

    rz_0lento

    Joined:
    Oct 8, 2013
    Posts:
    2,361
    @jaguero Do you have URP asset assigned to your main graphics RP asset? If not, I think the shader preprocessor takes early out and doesn't strip ANY of your URP shaders (means it will take forever to get them compiled then).
     
    Last edited: Oct 19, 2021
  10. jaguero

    jaguero

    Joined:
    Dec 17, 2018
    Posts:
    8
    Hi @rz_0lento . Yes, i do have it
    upload_2021-10-20_9-43-43.png

    I tried "strip unused" and "strip all" and i didn't notice changes in time.

    Actually I'm trying to change the project to an SSD to see if the problem is the HDD.
     
  11. jaguero

    jaguero

    Joined:
    Dec 17, 2018
    Posts:
    8
    Well, it crashed again, dont know if having a very small SSD could be the problem

    upload_2021-10-20_10-29-17.png
     
  12. jaguero

    jaguero

    Joined:
    Dec 17, 2018
    Posts:
    8
     

    Attached Files:

  13. jaguero

    jaguero

    Joined:
    Dec 17, 2018
    Posts:
    8
    Well, apparently changing to the SSD solved the problem, time now went from 19 hours to less than 30 minutes, don't know if it could be better, but at least is a reasonable time, besides not having the problem in previous versions, maybe the antivirus overcharged the HDD as @Baste suggested.

    Thank you all for the help!
     
  14. hippocoder

    hippocoder

    Digital Ape

    Joined:
    Apr 11, 2010
    Posts:
    29,723
    Builds are thousands of tiny files so seek time on a traditional hard drive is the bottleneck rather than data itself. SSDs are good at this. But 30 mins for a build is too much even for a final game. In these scenarios one is best off separating assets that won't change into asset bundles and then just compiling code becomes the main thing for each build you do - and should take a few minutes at most.

    The exception being shaders - these can be a hot mess for compile times but can also be cached if I'm not mistaken, between builds.
     
  15. anatolyV

    anatolyV

    Joined:
    Nov 29, 2014
    Posts:
    63
    Can attest to the last point. On my project the first build after a reimport for Android/iOS takes about an hour (all about those shaders), all subsequent ones are within 5-10 minutes.
     
  16. alloystorm

    alloystorm

    Joined:
    Jul 25, 2019
    Posts:
    88
    This is also happening to me.

    My project now takes 2 hour + to build if the shader is changed. Previously this takes around 30 minutes tops.
    And every time I switch build target, it's like all the build cache is cleared that every variant needs to be re-compiled again. HDRP seems normal, only URP is experiencing this issue.
     
  17. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    3,136
    If you're affected by regressions like this, please submit a bug report with a reproducible attached to it so we can investigate.
     
  18. Kevin-VFX

    Kevin-VFX

    Joined:
    Apr 17, 2016
    Posts:
    54
    Happening to me too. I have tried all the above fixes. Just building the simple construction sample scene takes hours. You guys added A LOT of variant keywords.
     
  19. TJNBG

    TJNBG

    Joined:
    Apr 7, 2021
    Posts:
    26
    Hey,

    This is reproducible for us as well. Incremental game build went up from a few minutes to closer to 1 hour.
    Empty URP example project incremental build: 40 -> 102 seconds

    Bug report 1383867
     
  20. iceb_

    iceb_

    Joined:
    Nov 10, 2015
    Posts:
    96
    Been testing since 2021.2, and all the way up to 2021.2.6f1, super long build times, mostly due to shader stripping I believe. In 2021.2.6f1 particularly, unverified but it seems to have made the build size larger as well.
     
  21. mrCharli3

    mrCharli3

    Joined:
    Mar 22, 2017
    Posts:
    981
    Since upgrading to 2021.2.8 my builds take 2-3 hours instead of 10 minutes. 95% of the time is spent on our shaders. Shouldn’t these be cached?

    Never had these issue before upgrading.

    Each shader does:

    upload_2022-1-26_20-14-42.png

    upload_2022-1-26_20-14-51.png

    upload_2022-1-26_20-14-58.png

    then again:
    upload_2022-1-26_20-15-8.png

    Om on an SSD.
     
    Last edited: Jan 27, 2022
    iceb_ likes this.
  22. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    3,136
    The team recently published this thread as a guide to a range of new shader stripping features in 2021.2. Please have a look if you're experiencing issues due to increased shader variant compilation times.
     
  23. mrCharli3

    mrCharli3

    Joined:
    Mar 22, 2017
    Posts:
    981
    I am already stripping everything:

    upload_2022-2-1_16-30-10.png

    Builds still take around 3 hours.
     
    iceb_ likes this.
  24. iceb_

    iceb_

    Joined:
    Nov 10, 2015
    Posts:
    96
    Have tried all the various shader stripping toggles, still takes a very very long time for each Windows build compared to Unity 2020.3
     
  25. TJNBG

    TJNBG

    Joined:
    Apr 7, 2021
    Posts:
    26
    The problem reproduces in the URP Template project and that's what has been reported and confirmed by QA.
     
    iceb_ likes this.
  26. allencook200

    allencook200

    Joined:
    Oct 2, 2020
    Posts:
    178
    Same issue. This is beyond annoying. Waited around 30 minutes before quitting the build. There's hardly anything in my scene. It seems to have triggered it by regenerating lightmaps.

    In 2021.2.10f1
     
    unnanego likes this.
  27. fendercodes

    fendercodes

    Joined:
    Feb 4, 2019
    Posts:
    195
    Same. Build was taking up to 45mins so I just cancelled it. Cmon URP..
     
    unnanego likes this.
  28. TJNBG

    TJNBG

    Joined:
    Apr 7, 2021
    Posts:
    26
    2021.2.14f1 contains a fix which reduces the magnitude of the problem.

    Now our main project builds only 4x slower than on 2021.1, versus 10-12x slower.
     
    unnanego likes this.
  29. mrCharli3

    mrCharli3

    Joined:
    Mar 22, 2017
    Posts:
    981
    I dont dare upgrade any more until the LTS version comes out. But thats something at least
     
    unnanego likes this.
  30. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    Last edited: Mar 21, 2022
  31. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,045
    It's compiling over 10K variants. It can take a lot of time depending on your HW and the shader complexity.
    The hint about the compiler being broken is misleading. The problem here lies in variant stripping.
     
  32. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937

    I work on this project since 3+ years.
    It takes 5 to 8 hours on 90% CPU.
    All other Unity versions before 2021.2.x did not have the problem and did the build in max 30min even after reimport.
    This problem also happens on HighestEnd workstations.
    Intel(R) Core(TM) i9-9940X CPU @ 3.30GHz 3.31 GHz

    Total Cores 14
    Total Threads 28
    Max Turbo Frequency 4.40 GHz
    Intel® Turbo Boost Max Technology 3.0 Frequency ‡ 4.50 GHz
    Processor Base Frequency 3.30 GHz


    I often saw this BetterLit Forward 0 to 10368 progress bar.
    It took some/one minutes to finalize. Not 3h+

    e.g. The progress on the screenhsot on 2497 of 10368 stands there for 10to 15min before it iterates to ~2550.of 10368.

    Did the Shader complexity of the BetterLit forward changes as much in last HDRP releases?
     
    Last edited: Mar 22, 2022
  33. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,045
    I have no idea about the complexity of BetterLit.
    From what you're describing, it sounds like it was taking everything from the cache previously, and now it's actually compiling variants. You can check the variant stats for each pass in the Editor.log.
     
  34. mrCharli3

    mrCharli3

    Joined:
    Mar 22, 2017
    Posts:
    981
    The fact still remains that all of us here had "normal" build times of 10-20 minutes before upgrading Unity to 2021.2.x, and now it takes us several hours. If you actually read through the thread youd see that all of us has tried all possible settings youve suggested and nothing helps.If it was because it "used the cache before", then why doesnt it anymore?

    I really hope you guys don't close your eyes to this problem going forward because right now it's just not good enough.
     
    ilia_infable, tlamiczka and iceb_ like this.
  35. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,045
    One big part of the build time regression was fixed in 2021.2.14f1.

    URP added a lot of features in 2021.2, and the logic that figures out the variants became a bottleneck (it has to process a lot of variants). We'll address it, but it will require more time.
     
  36. Kevin-VFX

    Kevin-VFX

    Joined:
    Apr 17, 2016
    Posts:
    54
    I guess we'll have to suffer though our 20 hour build times until it's addressed. :(
     
  37. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,045
    @Kevin-VFX it shouldn't be that bad after 2021.2.14f1 - what shader authoring tools are you using? .14f1 addressed variant stripping issues with ShaderGraph.
     
  38. mrCharli3

    mrCharli3

    Joined:
    Mar 22, 2017
    Posts:
    981
    Nice to hear that it is being adressed at least! :)
     
  39. TJNBG

    TJNBG

    Joined:
    Apr 7, 2021
    Posts:
    26
    Is there a workaround we could do in the mean time?
     
  40. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,045
    The only workaround is to use shaders with less possible variants, unfortunately.
     
  41. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    In HDRP it started with 2021.2.15+.
    TransparentBackface and HDRP Lit also takes hours to build.
     
  42. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,045
    I think it's a different issue then. Can you please report a bug?
    Thank you!
     
  43. fendercodes

    fendercodes

    Joined:
    Feb 4, 2019
    Posts:
    195
    Could you give more details on what regressions are being fixed and how? One of our production titles on Unity 2020 (BRP) with hundreds of shaders compiles in less than 10 minutes. A newer project on 2021.2 (URP) with barely 10 shaders takes hours...
     
  44. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,045
    Sure. There was an issue with the variant stripper that got updated in URP in a way very specific to the keyword setup in URP shaders. This resulted in shaders created using Shader Graph having around 100x vertex variants compared to URP/Lit.
    This is what got fixed - Shader Graph now declares the keywords in the same way as URP/Lit.
     
    spamove and fendercodes like this.
  45. Kronnect

    Kronnect

    Joined:
    Nov 16, 2014
    Posts:
    2,906
    Hi,

    We're also facing some issues in shader compilation times with URP (with no Shader Graph use):

    Same project in 2020.3.30 loads in Editor in a few seconds, builds in less than 1 minute.

    In 2021.2.17 however, the same scene takes 10 minutes to load in Editor and build takes like hours? (all due to shader variant compilation).

    It's same project so I tag this as a perf regression. Cases #1416284 & #1416268.
     
  46. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,045
    Thanks for making a bug report! We'll investigate :)
     
    Kronnect likes this.
  47. Kronnect

    Kronnect

    Joined:
    Nov 16, 2014
    Posts:
    2,906
    Some info on the bug report:

    It turns out the shader is forced to be unrolled in 2021.2.17 (due to a texture sampling macro that's resolved to a sampling with no explicit LOD in a loop). The URP base code in 2021.2.17 could be more extensive so that could be causing too many instructions in the resulting shader. We're seeing this error "shader can't be unrolled..." in some machines that could be related. We fixed that on our side by forcing LOD sampling, so it's ok now.

    Since the same shader compiled fine in 2020.3 LTS, I guess some change along the chain of the sampling macro or URP increased base code caused the instructions overflow, so the bug repro can still be useful to investigate potential optimizations during the shader compilation.

    Cheers
     
  48. mrCharli3

    mrCharli3

    Joined:
    Mar 22, 2017
    Posts:
    981
    Just made my first build with 2021 LTS, this is still an issue. Its a bit silly you call it a "production ready environment" with this issue still not fixed. Are we just gonn have to deal with 5-hour builds from now on if we want to use Unity? If thats the case I def wont use the engine for any future games.

    upload_2022-4-17_13-51-0.png
     
    Last edited: Apr 17, 2022
    ilia_infable likes this.
  49. LeonhardP

    LeonhardP

    Unity Technologies

    Joined:
    Jul 4, 2016
    Posts:
    3,136
    Thanks again for submitting these reports. Here's the public Issue Tracker link: https://issuetracker.unity3d.com/is...e-to-compile-when-using-the-volumetric-lights
     
  50. TJNBG

    TJNBG

    Joined:
    Apr 7, 2021
    Posts:
    26