Bug Build time taking almost a day - URP

I find it impossible to believe there's hundreds of millions of these. I'm really not writing a complicated game. On top of that, if Unity actually told me WHICH shader had 415M possible variants, I might be able to do something about it.

 

Editor.log outputs quite a lot of information that could be useful, for example:

STRIPPING: Shader Graphs/ObjectHighlightShader (Universal Forward pass) (Vertex) - Remaining shader variants = 12/96 = 12.5% - Total = 1422/108174 = 1.314549% TimeMs=5.5725

I'm not sure if some toggle needs to be switched on to enable logging of shader stripper. Maybe someone else in this topic can clarify that.

 

Any progress for this problem? We are still waiting for upgrade from 2020 to 2021)))

 

...you *hope* to backport this fix???

 

@aleksandrk It looks like this bug is still not being mentioned in the known issues as of 2021.3.8f1

 

Thanks @aleksandrk ! I know it's just clerical drudgery, but it matters to see it there.

 

Please consider plugging this in for 2021.3.9 or 10.
Build times are one thing, but the same issue affect building addressable groups. With a project using UMA and building addressablse that include only 2 shaders (a custom one I've used since 2019 with regular updates till this date + the default unity lit shader) compile times are hitting up to 7 hours. It's making work incredibly frustrating and it's not like you can do anything else in a project while it's frozen in compilation.

 

URP project, simple scene, all default shaders except one quite simple done in shadergraph.
Powerful PC (i9 16 cores/32 threads, 64gb memory, 3080ti graphics, only ssd drives)

 

5 shaders (all URP Simple-Lit based). 30 minutes and the bar hasn't moved.



100% cpu utilization


2021.3.8f1

Good times.

 

@aleksandrk @dnach Sorry for the ping, but are there additional updates to this? We're > 1 year from the OP and many months now in 'LTS' and it's still very unclear both when a fix might land and what might be expected in terms of improvements that real projects will see (ie, not synthetic tests).

Given that nothing relating to this seems to have been included in the recent 2022 release, at this point I've lost confidence in the 2021 cycle entirely and, barring advice from you that says otherwise, will be recommending our projects stick to 2020 LTS and / or the built-in render pipeline wherever possible, and generally avoiding Unity packages (ie, Shader Graph and VFX Graph) going forward.

 

Down by 75%? So from 15 hours to 3.75 hours?

To quote the *first post* :

Why do I feel this isn't being taken seriously? :/ You seem *happy* that the build time has gone from between 10 seconds and 5 minutes (assuming OP wasn't exaggerating) to 3.75 hours.

 

Are there plans to improve this further? We're using entirely custom shaders in our project (quest 2) and we are pretty careful about not adding too many variants, but to stay compatible with URP shader source we still end up having like 100 M variants, and this stripping phase seems to take longer than actually compiling.

 

I really really hope you're right; but we're still talking about percentages going up by hundreds or thousands, and then a drop of 75%.

Given as I'm doing a multiplayer game and quite frankly having to wait before doing a network test is driving me mad, the sooner this improvement arrives the better.

 

Cheers, I genuinely appreciate this and would very much look forward to that.

Would it be prudent to guesstimate a timeline of months then for LTS? Appreciate the effort the team is making in communication and resolution now, but it feels like the kind of issue that should not have been part of an LTS release to begin with; given that it is I'd expect more of an all-hands-on-deck response, where even if it doesn't make technical sense to do so, the work would first be done for branches being promoted as 'recommended for production', with PRs accelerated however possible. It's being backported, so it seems like effort is being made here already, but nowhere near enough.

The 'driving me mad' sentiment is something I've also felt as I've tried to navigate this via the recommended steps and come up with a scalable strategy around it. The root of the issue for me is not that you've introduced a ton of cool new features that have had some unfortunate negative side effects, it's that it's trivially easy now in Unity to kill creativity for hours, days, or longer at a time. And for what? We're years into the scriptable renderer experiment and we've only very recently reached something like feature parity with BiRP (which is still very much a viable option), at the cost of nuking productivity for some and fragmenting a community and asset ecosystem.

Given that Unity no longer has an internal productions team battle testing and championing these things (and while they did exist, it seems the concerns they raised WRT this particular problem were not taken seriously enough), how will the eventual fix for this issue be vetted for production use, and how can we have confidence in SRP and LTS releases in the future? Much of the roadmap has dependencies on this...

 

I've gotta say... Building a project out of 2021.3.0 has been an absolute nightmare.

If I take my project... and make a new scene, and I add a couple basic components to that scene to test VR... I'm looking at 540 MILLION VARIANTS... it takes 6-8 hours for that singular... basic scene... to build.

I have read dozens and dozens of threads on the issues. Tried everything they've suggested... but I can't get the build times down to anything reasonable. Listen, if I have to have PHD level shader and graphical engineering knowledge to understand whats going on, Unity Engine is useless to me.

I am trying to build my project to the quest 2. I'm having some trouble getting that working. I can't test a damn thing because I have to build my project to test it on the quest 2, and builds are an ALL DAY process.

How is this a thing? How is this acceptable?

Sorry... Rant Over... but I'm just lost. I've spent the last 4 days cooped up in an office troubleshooting a problem that requires me to wait all day between testing fixes. I have had enough.

 

Any chance you could downgrade to an older version? 2019 and other older versions were much faster.

 

I'm in a similar boat.
I too am I using URP and developing for VR and the problem is exacerbated by using a combination of the two.
Thankfully I'm not targeting Quest so don't need to build EVERY time I make a change.

When I DO have to build the project it takes HOURS and my PC is unusable the entire time.
I'm running a Ryzen 9 5950x 16 core CPU and Unity uses 100% of it, making my PC completely unusable for the hours it spends building.

The project I'm working on relies on user-generated content built using Unity.
It's unreasonable for me to expect any user to essentially lose access to their PC for hours to build content for the game.

I've had this post bookmarked and keep checking back month after month to see if it's improved but I've yet to see any progress on this. Right now it's halting the release of my game and it's crazy to me that this was seen as acceptable as by Unity themselves.

 

https://forum.unity.com/threads/improvements-to-shader-build-time-and-runtime-memory-usage.1305615/

 

@djarcas maybe a bit off-topic but I'm also developing a multiplayer game right now. "ParrelSync" helped me a LOT with this because it basically removes the need to build you project every time if you just want to test a small thing.
Maybe it helps you too: https://github.com/VeriorPies/ParrelSync

 

What do you think a clean build should take for a regular game?

Basically is 13-14 hours is considered a reality these days for a clean build Unity project?

Also does changing any option at all, or shader incur a new clean build penalty?

 

Hmm, perhaps a possibility is to be able to have a more static approach where shaders are considered inflexible for the build - so you won't be able to change any material features at runtime, and have to manage that yourself. Is that a viable path? I see a lot of asset store authors preferring to just generate a shader without compile time variants to combat this.

Does seem a little high, hope there's a path - even if manual - for developers to work with at some point.

 

Deleting the Shader Cache doesn't make any difference in build times for me. If I make an IL2CPP build of Teslagrad 2, any subsequent build without any changes takes 10 seconds, no matter if I delete the shader cache or not.

Usually building on my computer takes 2-3 minutes, I assume because I have changed code and scenes so less things are in cache. 12GB Assets folder, URP with some minor edits.


So like, yeah, if you want to have fast builds, do 2D games, I guess? We do use quite a few custom shaders for things, so it's not just that we don't have that. I wonder what on earth is going on with Gigaya, and 3D games in general.

 

13 hours for a build is crazy! What is going on? Does that project just have 200 m shader variants or what? Why?

 

Im quite confused how build times are taking THAT long now. What changed since the end of June?

When I was working on Gigaya, a clean local build would take roughly 6 hours. This was with 2021.3.5f1. About 95% of this being millions of URP Shader Variants and 1 hour of this being URP Lit. We were talking to URP Team on methods of shader stripping but also in general WHY using vanilla Unity can result in that much length of build time. Even though I wasn't on the Art team and I made probably only 1 shader for Gigaya, the art team weren't doing anything too crazy or exotic with shader graph.

Because of the 6 hour clean build length we shifted to automated nightly builds so the editor wasnt locked for nearly a whole day every day.

In general, Unity should ultra prioritise reducing build times exploding in length to an unusable state. Or atleast offer some better build in tooling to diagnose and improve the situation.

 

Cheers, and again genuinely appreciate your team's presence and candidness in this thread.

Even with the fix, without major realignment from Unity here I'm finding it difficult to justify relying on URP (and SRPs generally) going forward. Not to say that there have been no improvements, but we're two years out from this official statement (which already was painful) and SRPs are in many dimensions still a nightmare to use, with the added cost of this gigantic productivity leap backward.

I don't expect features and performance overnight and for free, but what is going on here? I won't speak for anyone else, but I'm at a point where I'd prefer Unity take an 'un-easy' and 'controversial' route out of this, rather than continuing to believe these solutions will gradually improve over time.

 

I have this forum post along with the one you linked in my bookmarks bar and I check them both religiously each day in hopes of this issue being improved upon.
As I'm targeting 2021 LTS it looks like I'll need to wait for a backport of the eventual improvements

 

Sounds like Plumbing Hell to be honest. Good luck with it though!

 

I'm sorry, but over 10 hours for a clean uncached build for something like Gigaya, which was quite early in terms of content development, is insane. If this is what it takes to make URP match the built-in feature set then something went really wrong along the way.

 

Quick Q; What is Accelerator?

 

Question: how large does the shader cache becomes with hundreds of millions of shader variants in it? Specially if you enable the PS4/XBOX/Switch shader debug information generation (which can already balloon into tens of GBs for projects using built-in).

 

The replacement for/evolution of the cache server.

The idea is that you put all the results of importing assets that's not in version control (ie. everything that goes into the Library/ folder) on a server on the local network, and every time Unity wants to import an asset it checks if it's in the cache server instead.

It's a bit too hard to set up, not very well documented/advertised, and it's not doing a good job of visualizing to you as the user that "hey you saved time now because we fetched something from the server!". But it's a great product.
Of course it's completely useless as a solo dev unless you happen to switch platforms very often.

 

Aha, makes sense! Thank you.

 

How long are we realistically looking at before the solution reaches a public Unity alpha / beta build?

And what kind of estimate for time savings do you think we should expect?

If it's possible: what version of Unity were the 14 hour cold build times measured with, and what laptop hardware? Just so we can figure out perhaps an LTS or better hardware to fix this thing.

I'm kind of leaning toward 45 mins being the clean build. 5 mins for a warm build is tolerable, but 14 hours for something that can still reset at times, is just too much work lost. That's actually a lot of money lost for developers using URP.

Some things can't skip builds. It means that the pain point would push people to other engines or maybe a different pipeline (if it's any better on HDRP / BiRP).


Ultimately for me: 45 mins is acceptable for a cold build, and 5 mins is acceptable for a warm build. If it is not solvable to that extent, please at least let developers know.

 

Thank you for your candour, I guess the URP team has a little bit of work to do!

 

For anyone interested, there's a great bit of history on this here.

@aleksandrk Am I right in understanding that the work on variant keyword pre-filtering is essentially including URP variants in the process 'Now: can I have hundred billion variants? or don't search that other data'?

Also, how might I better understand why a clean URP build would require so much longer than the HDRP builds aras-p describes? Or were those numbers with potentially billions of variants cached builds?

 

Modern feature-rich single-pass rendering is a recipe for shader permutations explosion. Early URP/LWRP avoided it by being too simple, built-in avoided it by relying on multipass additive rendering, HDRP avoids it by dividing workloads due to clustered/deferred rendering and screen-space shadowing.

(There's also the question of whether URP is (or even if it can) avoiding the generation of nonsense/excludent keyword combinations.)