Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only. On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live. Read our full announcement for more information and let us know if you have any questions.
    Dismiss Notice
  2. Dismiss Notice

Bug Build time taking almost a day - URP

Discussion in '2021.2 Beta' started by RenOli, Jun 29, 2021.

  1. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    I suppose this could go in a separate thread :)
    But two things that immediately come to my mind are: large arrays and unrolling loops with resource access. The compiler is quite slow with these.
     
  2. Destruxion

    Destruxion

    Joined:
    Jun 5, 2019
    Posts:
    37
    Unity 2021.3.4 did not change anything here. We were able to reduce URP Windows build times before using the Shader Control asset from about 6 hours to 10 minutes.

    Unity 2021.3.4 still does that x / 138M preparing variants pass and then starts recompiling all variants though I'd expect them to be cached already.

    As soon as we start adding any 3D stuff using URP Lit shaders to a project and do not supress individual shader keywords using shader control, build times jump from 2 minutes to several hours. Some shader graph shaders multiply this shader build madness.

    This issue needs way more attention. We are at a point where we are questioning if Unity is the right tool to use for productions with tight deadlines.
     
    ilia_infable, gnp89, Novack and 5 others like this.
  3. RealHandy

    RealHandy

    Joined:
    Jul 11, 2020
    Posts:
    22
    Sorry to be doing little more than piling on, but after seeing how slow our URP builds are due to shader variants, I found the Nov 2021 article which brags about cutting the build time of the URP Template project (the construction tools and wall) from 20 minutes to 7. Seven minutes to build the Template project seems more like it should be a warning about the performance, not an indication that things are in good shape. It just took us 14:30 to build one test scene that's just a terrain base, three characters, and one small tree. Yikes.
     
    spamove and funkyCoty like this.
  4. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
  5. Destruxion

    Destruxion

    Joined:
    Jun 5, 2019
    Posts:
    37
    Looks like it is finding a cached version, but while it says 0.0 seconds, the process of stepping through those variants takes very long. In case of the URP Lit shader with 10k versions displayed in Unitys build progress bar window it takes almost an hour to step through all those files (for the Lit shader alone). Does not look like it is "really" taking the cached version.

    In case of ForwardLit I found this:

    Compiling shader "Universal Render Pipeline/Lit" pass "ForwardLit" (fp)
    [188.88s] 100M / ~138M prepared
    288 / 15138816 variants left after stripping, processed in 249.23 seconds
    starting compilation...
    finished in 17.93 seconds. Local cache hits 0 (0.00s CPU time), remote cache hits 0 (0.00s CPU time), compiled 288 variants (563.38s CPU time), skipped 0 variants

    Seems this never uses the local cache. We stripped this version down by disabling keywords via Shader Control.

    I can PM you the complete log file in case that helps.
     
    Last edited: Jun 2, 2022
  6. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    Yes please, this would be great.
     
  7. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    After nearly two months
    the 5h+ build time bug was solved in Unitry 2021.3.3f1.

    Buildtime was back on normal 20min to 40min.

    Starting with Unity 2021.2.4 the behaviour is back. Back on 5h +.


    Saw this changes in
    https://unity3d.com/unity/whats-new/2021.3.4
    • Shaders: Fixed a crash when warming up a shader variant collection that had a shader with UsePass. (1407692)

    • Shaders: Fixed include dependencies being reported incorrectly when guarded by defines that depend on keywords. (1401580)

    • Shaders: Fixed redundant shader variant recompilation after Editor relaunch. (1425799)

    • Shaders: Fixed ShaderKeywordSet.Disable not disabling keywords. (1417644)
    However they killed the ability to work with Unity again.
    Checked again on 3 independent large projects on 3 different PC´s.
     
  8. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    Are you sure it was solved? As in, do you have a link to a bug report that says that this problem is fixed in 2021.3.3f1?
     
  9. RealHandy

    RealHandy

    Joined:
    Jul 11, 2020
    Posts:
    22
    @keeponshading I just tried a build with 2021.3.3f1 and it wasn't any faster than 2021.3.4f1 for me :(.
     
  10. TJNBG

    TJNBG

    Joined:
    Apr 7, 2021
    Posts:
    26
    Unfortunately 2021.3.4 did not fix the issue.


    [B]2021.3.4:[/B]

    [18:51:44] : [Running Unity editor] Compiling shader "Universal Render Pipeline/MilkshakeLit" pass "Universal Forward" (fp)
    [18:56:36] : [Running Unity editor] 9216 / 14155776 variants left after stripping, processed in 292.50 seconds
    [18:56:36] : [Running Unity editor] starting compilation...
    [18:56:38] : [Running Unity editor] finished in 1.75 seconds. Local cache hits 9216 (27.99s CPU time), remote cache hits 0 (0.00s CPU time), compiled 0 variants (0.00s CPU time), skipped 0 variants


    [B]2021.1.23:[/B]

    [14:43:12] : [Running Unity editor] Compiling shader "Universal Render Pipeline/MilkshakeLit" pass "Universal Forward" (fp)
    [14:43:12] : [Running Unity editor] 82944 variants, prepared in 0.01 seconds, starting stripping...
    [14:43:12] : [Running Unity editor] finished in 0.05 seconds. 18432 variants left
    [14:43:12] : [Running Unity editor] starting compilation...
    [14:43:18] : [Running Unity editor] finished in 6.34 seconds. Local cache hits 18432 (101.38s CPU time), remote cache hits 0 (0.00s CPU time), compiled 0 variants (0.00s CPU time)


    Whatever tool or code is performing shader stripping has exponential complexity now, so it seems.

    It also scales with the amount of scenes you're building.
     
    Last edited: Jun 3, 2022
  11. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    The fix I mentioned was for "Unity sometimes recompiles variants that it compiled already".

    The fix for stripping taking that long is still in the works.
     
  12. TJNBG

    TJNBG

    Joined:
    Apr 7, 2021
    Posts:
    26
    Thanks for the clarification @aleksandrk
    The fix for "Unity sometimes recompiles variants that it compiled already" is definitely verified as fix on our end :)
     
    aleksandrk likes this.
  13. Blepius

    Blepius

    Joined:
    Mar 9, 2021
    Posts:
    68
    Hi all,

    Any update?

    I've been working this month without building to avoid this, but I'm really kind of at a loss here.

    Kronnect's Shader Control helps-- It took my project's build times from days to hours. However, my project used to build in about ~10-15 minutes. Unfortunately, a lot of shaders cannot be stripped by Shader Control and rely on Unity to keep variants in check.
     
    Last edited: Jun 16, 2022
    timmehhhhhhh and IS_Twyker like this.
  14. Zeronev

    Zeronev

    Joined:
    Jun 26, 2015
    Posts:
    6
    An update is needed on this issue.
    Multiplayer games are impossible to debug and test because of this problem.
    Rotating a simple gameobject, saving the scene and then making a build, makes a check on every single shader variant (400M+, 25-40minutes), even modifying a prefab.
    Porting my project into another engine might even be faster.

    Is there a way to simulate play of 2 different players inside the editor?
     
  15. iMer

    iMer

    Joined:
    May 21, 2013
    Posts:
    29
    Could look at https://github.com/VeriorPies/ParrelSync or similar

    Havent used it myself, but it does pop up every now and then
     
    Zeronev likes this.
  16. IS_Twyker

    IS_Twyker

    Joined:
    Sep 6, 2021
    Posts:
    35
    +1

    Building Addressables / compiling shaders takes forever currently. :(
     
  17. iMer

    iMer

    Joined:
    May 21, 2013
    Posts:
    29
    Any chance the caching is too aggressive now?
    Had a bunch of shaders complain about errors during a build that didnt exist anymore and the only fix was a manual reimport of the shaders ("could not open include file xyz" since I deleted & reimported URP for double checking a bug)
     
    Blepius likes this.
  18. Temka_193

    Temka_193

    Joined:
    Oct 24, 2015
    Posts:
    47
    Unfortunately 2021.3.5 did not fix the issue.
     
  19. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    upload_2022-7-1_20-26-24.png

    This is getting silly now.
     
  20. YayapipiStudio

    YayapipiStudio

    Joined:
    Feb 11, 2017
    Posts:
    9
  21. Julien_at_work

    Julien_at_work

    Joined:
    Aug 9, 2018
    Posts:
    35
    First-time builds in Built-in pipeline were like 10min for our visualization projects. For a current project we tried switching to URP, but build times really are unusual as i discovered way too late. For now i've always had to cancel my testbuilds to continue work and hope the next patch would possibly bring a fix for this, but it seems it still hasn't happend (2021.3.5f1). I hope unity is aware of this issue and prioritizes it accordingly. So far it seems unusable compared to built-in.
    Maybe i should have tested building earlier; i just didn't expect this to be an issue but now becomes a problem under time constraints.
     
    Blepius likes this.
  22. kogi_rc

    kogi_rc

    Joined:
    Apr 23, 2019
    Posts:
    39
    Could any of the Unity team members assigned to this bug at least give us some info - is this problem being worked on and is there a rough estimate when the fix is going to be available? Almost 3 months after the LTS release, 47 days after the issue has been added to issue tracker we are still completely in the dark.

    This problem slows down test runs massively and our team would like to know if it's going to be fixed in coming weeks because we are considering rolling back from LTS to last stable Tech Stream 2021.1.x or 2021.2.x.
     
    timmehhhhhhh, Novack and JamesArndt like this.
  23. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    Hi all!

    This problem is being worked on. It has been given high priority a while ago, so it's not being left in the limbo :)
    The fix involves a partial re-architecture of the shader stripping pipeline. The main work is finished already, we're testing things right now.

    We'll post here when there is an update on the issue.
     
  24. kogi_rc

    kogi_rc

    Joined:
    Apr 23, 2019
    Posts:
    39
    Thank you for an update. That's great to hear.
     
  25. Destruxion

    Destruxion

    Joined:
    Jun 5, 2019
    Posts:
    37
    @aleksandrk, great to hear this issue has high priority now.

    I have been digging into this whole shader compilation thing over the past weeks, since our projects usually have a very tight deadline and lengthy compile times are a serious issue.

    Shader Control from the asset store has been a help, but only resolved the issue to some degree.

    I would like to show what we have done so far and show some quirks where Unity could help out, but still fails because some data are missing.

    We use a custom build interface, so we can easily enable things like "Log Shader Compilation" or "Strict Shader Variant Compilation" temporarily, without the need to search project settings.

    upload_2022-7-10_17-28-25.png

    Based on this I use a custom Shader Manager (scriptable object) that completely replaces Shader Control. We can "Disable All Shader" to run a very fast non functional build, that only scans for all shader and keywords requested during the compilation process.

    upload_2022-7-10_17-46-3.png

    The remaining four tabs provide four methods to include shader variants in the build.

    Player Log
    With the "Strict Shader Variant Matching" option enabled scan the player log for missing variants. Actually, this is a great way to detect missing shaders, but unfortunately it does not catch all variants that are missing.

    upload_2022-7-10_17-49-52.png

    Shader Variant Collection
    Shader variant collections are another great feature to detect variants by simply starting play mode in the editor. Again, this method does unfortunately not detect all variants being used.

    The shader variant collection API definitely needs some love. There is no way to query the variant array, which is the most basic method I would expect to exist. This forces us to evaluate the collection during the build process using the "contains" method against all variants the callback throws at us. Yikes!

    upload_2022-7-10_17-53-26.png

    Global Keywords

    Here we can enable variants manually via a selection of all keywords found across all shader. The shader itself must be enabled for this to work.


    Shader & Keywords
    Enable individual shader and corresponding keywords. The keyword methods are inferior to direct variant detection, because enabling 8 keywords results in 256 variants being compiled, where actually only very few variants are finally used.

    upload_2022-7-10_18-1-29.png
     
  26. dnach

    dnach

    Unity Technologies

    Joined:
    Mar 9, 2022
    Posts:
    98
    It is recommended to disable any unneeded rendering features in the URP assets used in the build's quality settings, which will strip the unnecessary shader variants and exclude them from compilation. That being said, simply processing a huge amount of variants can still take a long time.

    This is definitely a huge pain at the moment, and we are working on an urgent optimization to greatly reduce shader build times, via early exclusion of unnecessary variant keywords. Please check the following post for more detail on this and other upcoming improvements, as well as some useful information on shader variant logging and stripping: https://forum.unity.com/threads/improvements-to-shader-build-time-and-runtime-memory-usage.1305615/
     
    TJNBG, timmehhhhhhh and IS_Twyker like this.
  27. IS_Twyker

    IS_Twyker

    Joined:
    Sep 6, 2021
    Posts:
    35
    We also went from 10-30min up to 7h when building addressables, if we leave in all shaders/keywords.
    We created an automated step where all keywords are being disabled and all unity shaders are copied, keywords disabled and then reassigned - with this we went down to about 1h.

    All of this escalated after going from 2020 LTS to 2021 LTS.

    Looking forward to the fix.
     
    timmehhhhhhh and keeponshading like this.
  28. pavelsvs2012

    pavelsvs2012

    Joined:
    Oct 6, 2020
    Posts:
    8


    Hello everyone, the problem is relevant in version 2022.1.5f1. The first build was done for 14 hours, but there was little space on the hard disk and apparently hung up. Restarted again freeing up 40 GB of space. The build is being done for almost 5 hours. Using URP
     

    Attached Files:

  29. Firnu

    Firnu

    Joined:
    May 28, 2019
    Posts:
    2
    Would it be possible to get a "fix" for this earlier, by providing an option in Unity preferences to simply revert to the old build method? We'll gladly turn off this "feature" to be able to return to normal operations. This is not something that we can wait months for to get a "proper" fix.

    We even considered returning to the previous LTS version but downgrading the project is undoable at this point.

    This was supposed to be an upgrade but instead we received a complete halt in the studio development iterations... We completely lost trust in LTS releases due to this.
     
    Novack and projectorgames_unity like this.
  30. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    No. There's no "old build method" or "new build method". This is simply caused by the number of possible variants in shaders. We already optimized this code as a quick fix, it went down by roughly 20%. Further improvements require more time.
     
  31. Firnu

    Firnu

    Joined:
    May 28, 2019
    Posts:
    2
    All people in this topic are getting hundreds of % increase in build times which was not the case in previous LTS and it completely obstructs the way they conduct business.

    Then you come here on a white horse and announce that Unity team barely did anything and we should be thankful for 20% speed improvement. It's not an improvement, it's not a downgrade - it's a complete obstruction of gamedev studio operations.

    You guys charge a substantial amount of money for the engine. DO NOT screw with people's money or even the biggest marketing agency merger won't help the company. HELP us instead of providing a stupid PR response.
     
  32. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    No, I merely gave an expanded answer to your question above. I thinks it's preferable to "we're working on it".

    @pavelsvs2012 it's compiling over 100K variants. Please check the link that @dnach provided in post #126, it should help you reduce the number of variants - and the build time.
     
  33. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    Bear in mind that, for me, 20% improvement means I'm saving half an hour. What boggles my mind is that if it takes 3 hours to compile the build, and I IMMEDIATELY re-compile the build, it takes 3 hours. It's like nothing at all is cached. Is this also being addressed?

    I beg.

    Edit : I'm still begging. This is madness. It was about a 6 minute build only a few months ago, and I've really not done much other than half a dozen shadergraphs...



    Build completed with a result of 'Succeeded' in 34940 seconds (34940468 ms) (~9.7 hours) So a 20% improvement would save me 2 hours. Again, this is NOT a complicated project - and Unity has no UI at all to help me track down if I've managed to tick the wrong box somewhere.
     
    Last edited: Jul 22, 2022
    IS_Twyker likes this.
  34. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    Uh. Just to check. You think this is... correct?

    upload_2022-7-23_23-42-24.png
     
  35. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    Edit : Interestingly, deleting all of these files caused my next build to be 10 hours. However, most of them weren't recreated, and my next build? "Build completed with a result of 'Succeeded' in 3457 seconds (3456729 ms)"

    Under an hour.
     
  36. RealHandy

    RealHandy

    Joined:
    Jul 11, 2020
    Posts:
    22
    Got a question about releases and bug tracking. 2021.3.3f7 just came out. It doesn't have a fix for this build perf issue. But I also notice that the build perf issue is not listed as a known issue in the release. Now, both 3f7 and 3f6 do list ongoing known issues (for example, AI Navigation Core: NavMesh::Raycast freezes the whole editor in an infinite loop on Application.UpdateScene (UUM-2496) is in both 3f6 and 3f7 known issues).

    So my question is this: does the fact that the known issues don't mention this thread's performance issue mean that Unity doesn't consider it a confirmed known issue? If so, is that because Unity is unable to reproduce the circumstances that cause the reported order of magnitude build time increase?
     
  37. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    What Unity version are you on? As I mentioned above, we changed the cache structure and introduced trimming so it should be less files. But I'd like to double-check that what you're seeing is indeed this and not something else.
    Also, is your project on a SSD?
    Btw, I think that 50K variants is a bit too much...

    No. I'll take a look later this week, why it's not listed there.
     
    spamove and projectorgames_unity like this.
  38. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    I don't think anyone is disagreeing there. People have been showing variants in the millions. As I've said in this thread; I'm perfectly happy to accept that I have ticked something or done something to cause the variants to exponentially explode. But I don't know what it is. And there's nowhere that Unity tells me how many variants there are (except during build time) and literally nowhere where it tells me WHY.

    2021.3.6f1, running off an NVME drive.

    Edit : The number of variants also appears to be completely random.

    https://cdn.discordapp.com/attachments/371347509467152384/1004074271758364682/unknown.png
     
    Last edited: Aug 2, 2022
  39. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    You're right that it's quite limited right now. We definitely need better tooling for shader variant handling :)

    I double-checked the code and it should clean up all the old data. 1.4K folders and 573K files sounds a lot. How many do you have after killing the shader cache?
     
  40. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    After blowing it away and desperately deleting shaders I probably don't want, we're back to...

    As I said, if there's something I could tick or untick that might solve it, I'm more than happy to.
     
  41. funkyCoty

    funkyCoty

    Joined:
    May 22, 2018
    Posts:
    748
    upload_2022-8-2_15-7-17.png

    thought mine would be more
     
  42. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    You can check the URP settings and make sure that the ones that you don't need are turned off.
    The link that @dnach provided above has a brief description of where to look (see "Build-time shader variant stripping").
     
  43. IS_Twyker

    IS_Twyker

    Joined:
    Sep 6, 2021
    Posts:
    35
    @aleksandrk Thanks for being so active in this thread. Any new info maybe when the planned fix might hit 2021 LTS?
    The build times are a crippling issue for the whole company.
     
  44. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    As soon as we know the versions the fix will appear in, we'll share it here.
     
    IS_Twyker and Lars-Steenhoff like this.
  45. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    3,568
    Can I ask one thing about build times, why are textures compressed during build time that are not going to be part of the build?

    This takes time and disk space while not needed.

    I wish there was a way to exclude them from the compression stage.
     
  46. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,074
    Sounds like a bug to me.

    But this question probably deserves its own thread.
     
    Lars-Steenhoff likes this.
  47. Lars-Steenhoff

    Lars-Steenhoff

    Joined:
    Aug 7, 2007
    Posts:
    3,568
  48. keeponshading

    keeponshading

    Joined:
    Sep 6, 2018
    Posts:
    937
    Same here still since several months now. We are now in 2021.3.7 LTS.
    Are there any news?
     
  49. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    You mean in the Forward Renderer? I've got SSAO, Decals and ScreenSpace Shadows. There's not really anything to remove.
     
  50. projectorgames_unity

    projectorgames_unity

    Joined:
    Oct 15, 2018
    Posts:
    107
    upload_2022-8-5_21-50-40.png

    One of the things that really does boggle my mind is that the number that appears here seems to be completely random. This time it's 415M. I assume the 'm' means 'million'
     
    IS_Twyker likes this.