Bug Build time taking almost a day - URP

I was experiencing this same issue in 2021 LTS and 2022 Beta. I currently have 2023 alpha 15 installed for about 4 days now and it seems to have solved the issue, my build times are approx 3 mins now instead of 40mins++(Might be longer since I cancelled the build). Did not have to do anything in the settings.

 

Personally, I tried jumping ship to 2023.1 but my projects crash on opening and the crash logs show absolutely nothing of value to go off on why the crash occured, so I imagine that's still not an option for most people.
Great it works for you though!

 

Hi, you can read my above post where I had similar issues. My fix? It just started... working again. There was no change in anything I did. I went from 40 mins etc. build times back to 4 minutes, except for linux which took 8 minutes. It is definitely an issue with shaders though yes as it compiles all of them for 30 minutes.

 

Hello, any news for 2021 lts?

 

Known Issues in 2021.3.12f1

• Universal RP: Shader compilation time is higher in 2021.3 than in 2020.3 (UUM-17405)
  
  
  Well, that's understated. Do I take it that we're waiting ANOTHER month for this?

 

Just so I'm not misunderstanding; 2021.3.12 neither has the long-build-time-fix and doesn't list it on the known issues either?

 

Will the fix also be implemented in 2022 versions?

 

https://unity3d.com/unity/whats-new/2021.3.13 is out - fix still not present?

 

Like srsly UNITY get a grip.. My system runs on Latest Intel CPU I have RTX 3080TI, SSD with read/write speeds of 6000mb/s, and in order to build my simple medium to low side size game i need to w8 more than 2 hours. like what is this? How are u guys calling yourself a reliable and fast platform?

 

and in my build progress bar, I see that it's doing something with forward shaders. i don't even use forward mode in my game its all deferred..

 

like it clearly shows used variants, but no when i make build it takes a mythical amount of 25k+ ....

 

and this is what it shows during the build process.... 364 million variants? where from

 

I think this is how it works. Every variant used/unused is compiled and put into the cache. Now on a second build everything is much faster. In the ideal case when someone wants to do a build with different keywords (low vs high quality or something), it would still build fast the second time because all variants are already in the cache.

There is no magic numbers, its just not a very efficient for most developers. They know the problem, they know the solution. Re-ranting won't make it go any faster.

 

but how can there be milion variants like what ? just make a option to not load all the variants and the ones that u are using thats it simple easy peasy

 

I think in few words this is what they are doing . But it is a very different way of filtering so they will have to do more work to make sure they don't break something. Broken < Slow. They might also have decided to put some extra time into fixing it in a good way versus hacking in another "if(shaderUsed)" somewhere and adding to the long term technical dept.

aleksandrk in this thread mentioned some reasons why the URP build time exploded. I see it as every feature/keyword added doubles the amount of shader variants. So you go from a reasonable buildtime of 15 minutes to 8 hours with only 5 extra features.

 

Actually we could automate this whole process and fix things on our side if the tools provided would actually work.

Strikt Shader Variant Matching: Great idea, but tests showed many missing keyword combinations are not logged.
Editor Shader Variant Collections: Same problem, only a subset of shaders and keywords is tracked.

Obviously there must be a place, where shaders and keywords are requested by Unity. We need a consistent way to track those exact combinations and store them in a deserializable format, not just a log that must be parsed or a Shader Variant Collection blackbox. Either in the player (no matter if the very first build is just purple, by simply not compiling ANY shader at all, just gathering data) or even better live in the editor or at least when running play mode (something that Shader Variant Collections should theoretically do).

During the build process Unity can still throw those millions of keyword combinations at us, but we can then filter the EXACT combination of keywords required. Meaning 12 detected URP Lit shader keywords that are actually in use will not result in 4096 shader variants, but maybe in only 3, because no other combination is actually used.

I built that pretty extensive tool shown on a previous page in this thread. We are able to activate shaders and a global keyword list manually in an UI (everything Unity refuses to track) and then rely on simple asset and memory scans get everything else. That combined with our own builder tool currently fully automates the process and we are back to normal build times. If Unity would solve the shader/keyword tracking issue, this solultion would fully automate the process.

 

question is if they want to ..

 

I thought this too. Until I installed Unity 2023 alpha, and build times dropped to 3 mins, even if its the first build it took no more than 5 mins.

 

2021.3.14f1
Released: 18 November 2022 - STILL no mention of the fix?

 

It'll be out before you know it, go for a long walk, catch some air, check back around April next year and I'm sure it'll be fixed by then. Patience is a virtue.

 

To quote the goblins of gadgetzan; time is money, friend!

 

Can we also talk about the time taken for cold builds?



This is so far from acceptable that it's unreal, and I'm genuinely at the point of considering abandoning Unity after over a decade of using it for the other engine with a U in it.

 

@aleksandrk What Unity could do with that to make things better is;
- Not spam new projects with 5 quality levels by default. We always delete all of those, they're a really bad way to control quality. So for a lot of users, things are slow and bad because you made it slow and bad by default.

- Rejig things such that shader variants are opt-in rather than opt-out. You have this absurd situation where games that should take minutes to build takes hours because users have to juggle confusing settings in order for you to not do a bunch of work that they didn't want you to do in the first place. A good place to start would simply to find-and-replace

multi-compile

with

shader_feature

in the monster that is the standard shader, and then deal with the fallout of that on the other end instead of playing whack-a-mole with stripping.

 

So why am I seeing 40+ hours cold builds? >_<

...how would I possibly know the answer these questions? Where am I told? Am I supposed write a parser for the log file? Does UNITY have a parser for the log file that'll break down what's going on?

I stopped writing .shader files the moment shadergraph came along; are these settings or tickboxes within shadergraph that should be labelled with a big 'WARNING THIS SHADER HAS 84 MILLION VARIANTS' ? Should ShaderGraph list how many variants there are?

Your example of SSAO confuses the hell out of me tho - isn't SSAO a post-processing full-screen effect? Why would that make a shader compile twice? You must understand that I am working from a position of a *ton* of Unity experience, but never at any point have I needed to start worrying about shader_feature or multi_compile, and, as I said, we are talking about URP here - everything is shadergraph and there is *zero* information or warning about something I do that's stupid. Also, this is a *one man project* and it takes FORTY EIGHT hours to compile on a high-end Intel processor.

My HDRP projects and BIRP projects also build in a tiny percentage of the same time.

 

A quick check through the 36+ hour build log shows this terrifying line:

Cuz, uh, I didn't write that shader. Y'all did. And that's 78 minutes.

And that's 12.2 hours to compile that many variants, for whatever reason unbeknownst to me for Unity deciding to compile that many. Quite what the other 24 hours is, I don't know, as Unity really doesn't have any helpful information spat out at the end. Maybe a nice pie chart?

35 hours. This was after a previous attempt on the cold build, which crashed after 12-15 hours.

...!

 

Yeah, I know. But with the combination of:
- The URP standard shader having something like 40 multi_compile flags
- Multi-compile being a thing you have to work around to not create a combinatoric explosion

You will never have a good build pipeline for URP. Prefiltering is helping, but it is to some degree or another S*** polishing.

You might have some analytics numbers on this, but I assume that the vast majority of projects either doesn't set any multi_compile keywords on materials/globally at all, or only set them as a side-effect of something Unity does internally. If we got to have a checklist for each shader with "these are the multi_compile keywords I intend to set", and errors if we tried to set any that we didn't check, that'd massively reduce the number of shaders compiled by default.

This is an area where both you and Unreal are just massively bad (google Unreal Engine Shader Compile Memes), so if you don't have the resources to actually do the good thing here available, your bosses are bad at prioritizing.

 

https://forum.unity.com/threads/improvements-to-shader-build-time-and-runtime-memory-usage.1305615/

That one? It just seems to be detailing ways in which Unity is improving things. If there's information that's useful for me trying to work out what combination of tickboxes makes shaders compile slowly, it's lost on me.

I've spent a little bit of time writing a little parse to actually give useful information, as opposed to the multi-megabyte logs Unity gives. I spotted this:

"Shader [ "Shader Graphs/Hologram" pass "Universal Forward" (fp)] took 4.0m to strip and 5.0hours to compile. 15360 variants were compiled."

Is that a lot of variants?

But why? What did I do wrong? How could I make the shader not take so much time? I feel this particular deep dive should lead to a Best Practises page, especially as one of the four PILLARS of URP is "Faster iteration to get to quality"



But nothing about this shader, as far as I know, jumps out at me as requiring *more than half a work day* to compile. Hell, it doesn't even cast shadows. I would like to learn and be enlightened as to what I've done so wrong, and further more to have Unity display the total variant space + stripped count *when* editing the shader or (as I've learned from your post) I link against a different URP Pipeline Asset)

 

Again, I'm trying to push very hard here for a place within Unity that explains that my shader has 83 base variants, and it's x2 because of SSAO and x2 because of reflection probe blending, and (etc etc etc). I've ticked both of the tick boxes that should be ticked related to stripping, and I've *un*ticked 'optimize mesh data', and now I'm back to "I have zero idea why this shader takes 5 hours to compile'

If no-where within Unity tells me how many variants there are *and why*, how can it ever be possible to fix this?

 

There seem to be a lot of people heavily invested in this issue. If there are any aspects of this process that can be made public, the community may be able to help.

I personally would likely be of little help, but I would love to read along and learn about more about how Unity works.

 

That's surprisingly useful, and not something I've ever looked at previously. I notice that it supports 3 types of fog, for example (which I believe is 2x, 4x, 8x the variant count?) So that alone, if reduced to the single type of fog I *actually* use, would quarter this shader to a more reasonable (but still outrageous) 1.25 hours? (Is it even possible to disable the other 2 types of fog? Quartering the render time of *all* my shaders is no small gain)

A lot of these are internal, and ofc there's nothing I expect to be able to do about those.

 

Happy to say, if someone hasn't tested yet, build times are normal again even in cold builds. Went from 30+ hours of waiting and canceling due to losing my patience to building under an hour with 2021.3.15f1. Similarly, addressables now build in a matter of minutes cold rather than over an hour for my case, and rebuilding afterwards with slight changes takes a few seconds. Didn't have to play around with stripping variants either.

Thanks a lot for explaining the issue and helping us deal with the wait for a fix @aleksandrk - I think we all really appreciate it. Still, I wish issues like this would be considered priority fixes for whoever decides on priorities for LTS fixes when they cost everyone here a lot of time and money.

 

confirmed also in 2022.2.0f ... yes, just released! i just cancel over 4 hour compilation...

 

So, some numbers on the 2021.3.15 build. Bear in mind they will be massively skewed since my previous 48 hour cold build. Rather than expect Unity to get their act together and develop background asynchronous builds or actually optimise stuff, I went out and bought the fastest i9 that money can buy. That took my cold build time from 48 hours to 16.3 hours. My warm build time is now 96 minutes, which is far better, but obviously is still S***. That's 96 minutes for Unity to do *nothing*, to check everything and go 'no work needed to be done'. If I was running a warm build on my previous computer, I'd guesstimate that a warm build would be 4.8 hours.

It's good that we have improvements, but quite frankly they're a day late and a dollar short.

 

Super happy to! This build log is a cold build, then a cancelled build - not sure why, then the warm build immediately afterwards. All I did was shove an extra whitespace into a single script to compare cold and build times.

 

Oh, the plot thickens!