Bug Build time taking almost a day - URP

@LeonhardP

I just noticed that the response to this Long build time issue is "Won't Fix" because it's not possible to fix? Will the general long build time unrelated to the volumetric lights be looked into as well?

https://issuetracker.unity3d.com/is...e-to-compile-when-using-the-volumetric-lights

 

We’re told by Unity support that this specific case of lengthy compilation time only happened in Unity 2021 due to a FXC compiler bug and that they couldn’t do anything (the issue didn’t replicate in 2019/2020).
We already fixed it by changing the shader so it ‘s no longer an issue with the latest version of the asset (5.0) in Unity 2021.

 

The issue for me and probably others is that it also happens in other situations too, when not using that particular volumetric light asset from Kronnect.

 

Correct.
It reproduces on Unity's standard URP sample project.

 

I haven't seen an official issue tracker for it yet, aside from Kronnect's separate issue, do you know of any?

 

No, my bug report did not convert to an issue on the public tracker.

 

By the way, it builds faster if it's only targeting DX11. But once you add on others aside from DX11, it adds up to very long build times.

 

yes, vulvan, GLCore, DX12. Adding some or all of them will add more and more non-linearly to the build time by a lot. And it compounds the problem with more complex shaders too.

 

that's awesome to hear! Is there an issue tracker for this or expected patch where it'll come in on?

 

Here a Screesnhot of my results in 2021.3.x....
Check the Folder generation amount ( ~3000).
This should be introduced around 2021.2.13+.
Before there were only around 16 Folders.
These LTS release are still not usable with this building behaviour.

 

The build time from Adressables and normal build is still 5 to 10h in 2021.3.2f1.
This makes the LTS releases completly unusable.

Before (~2021.2.15) around 20 to 40min.
Tested on +10 mid size (50GB) to large (300GB) projects on 4 PC´s.

If you enable Development build ist goes up to +15h. I am stoped the builds around 15h.
Without Develompment build like i said around 10h.

The question is how can i get back to 20 to 40min build time i had ahave the last 1.5 years across all 2020 and 2021 releases before 2021.2.15, allowing to use the LTS releases?

 

Yes. I have to check the information inside and send it to you when this one is finished.
It´s running since around 2h in 2021.3.2f1 on










I thought these fixes in 2021.3.2 change something.

But they doesn´t.

 

This behaviour scales about all projects i tested. (around 20)
You could simply take the SpaceShipDemo and Unity Raytracing examples from Github
and compare the build time from 2021.2.14 and 2021.3.x.

In a second step simply add some Assets to simulate real world project size and build again and comapre
Build time from 2021.2.14 vs 2021.3.x..

In a third step simply mark some Adressables in these projects and build these to simulate further real world projects and comare build time from 2021.2.14 and 2021.3.x.

In all three cases you will get same behaviour and exploding build times due to shader compiler factor 10 to 40..

I am still wondering why this hat not been done before releasing an LTS.

 

The problem is that all Workstations become unusable for hours.

Over 200 ShaderCompiler Tasks blocking 40GB of RAM...
Still ongoing....



and this is the fast PC we have (specs above).
It further scales by using slower workstations.

 

Can you check how many variants are being compiled? Also are you doing any shader stripping yourself?

 

No. I dont do custom shader stripping by myself.

Here some data.
200 000 to 500 000 files in shader cache.
16 folders in older releases. Now 3000 Folders plus in LTS releases.
Around 5 to 10 GB Shadercache madness.

https://forum.unity.com/threads/build-time-taking-almost-a-day-urp.1133605/page-2#post-8112830

Development build disabled because it s not possible to wait 20h +.

 

My build times also jumped from 4 minutes to 30 minutes (Vulkan mobile development build). This is not a case of some features being added and reasonably increasing build time. Something is broken beyond that. I'm eagerly waiting for a fix mentioned in #62.

 

This would be factor 8 and matches with my 10x to 20x according to project size.
Actual running one



Alone the Lit Tesselation Forward vp is running now for 25min+ on over 200 Threads...

 

live update - still running

 

yes

yes. and i am aware that the hdrp mess started by changes made in urp shader compiler around 2021.2.15. I am also aware that there is a general denial that any problem exists in this thread.

So we could extend the description to urp/hdrp or i could convert all my shaders in projects to urp to show that the shader compiler is massively broken and not tested at all before releasing this LTS

 

Ok. So what i have to do get in a workable state with my projects in LTS?
Downloading common Unity Github Demos, optionally integrate Adressables, then building these in 2021.2.14 and 2021.3.x to show the build increase factor around 10+?

 

Hi. Which part do you would need.
It ´s 190MB in size?

 

So all this is normal? Ive read that it was addressed but this happens in 2021.3 lts and even in 2022.2, even more time and more variants in 2022.2. 3500M variants for URP/Lit.

Build takes 14 hours.

At least sometimes it goes fast through the variants (i mean the "fp" things, like 43000/43000fp) but that happens rarely and randomly, not even sure whats the criteria, its not the first build. It used to be the first build 14 hours, stay for the fps for huge amount of times but when building again over that project it would go fast through the fps. In 2022.2 that doesnt apply, it seems every time it will take 14 hours, basically make around 15 fps every 1 second and as it advances it makes 15 fps every ~5 seconds, even more. Why is it this slow? And why in earlier versions of unity (2021.2 or .3) it sometimes moved very fast through the fps even if it wasnt the first build? Could you tell me (@aleksandrk) whats the criteria that makes it move fast or slow through those fps? If it was the fast moving it would be acceptable, im ok even with 2 hours builds but 14 is just too much. I dont have realtime protection enabled from windows defender as Ive read a while ago in some thread that it would be the cause of the moving slow through the fps, but its not true, like i said, sometimes it moves fast and sometimes slow, could you tell me what can i do for it to at least move fast and not 15 fps every 1 second and as it advances 15 fps every ~5 seconds? I guess no need to mention that this is done twice for the lit shader, so 43000 fps x2.

video 1 showing around 15fps every 1 second: https://gyazo.com/ccd26c45e9395ff6ff28b543b35ba6ef
video 2 showing around 15 fps every ~5 seconds: https://gyazo.com/83d11da3750c73753448cb7d8741df64


Additionally, I even commented light cookies, dots instancing and some more keywords that i dont use (actually i did use cookies but there is the nasty bug where the sampler limit is exceded that was even postponed, i dont even understand why you guys keep the light cookie keyword if it prevents the player from building. I mean this: https://issuetracker.unity3d.com/issues/sampler-limit-16-exceeded-in-urp-lit-shader ). Anyway, point is even with some keywords commented i have those 43000 variants, without those comments it would probably be much higher than that.


edit: it's even more than 5 seconds. Now it's at 35000/43000 and it adds around 15fps every 7 seconds. Probably will be even more as it gets to the end

 

How would I do that though? It's not like I use that many, I mean I have a few materials that use detail normals, detail albedo and emmisions, a few transparent ones but overall the rest of the materials are only using albedo map and normal map. But probably not even close to 43000 variants. Or maybe, I guess, since this is the number.

 

Thanks for keeping everyone updated @aleksandrk .
Do you know why this is not listed in the known issues of 2021.3? Meaning bugs 1137857 and 1383867?

 

https://gyazo.com/cdd083512b1bcbedc30634f73b527cbe

do you mean those settings?

 

Since the fix is coming in 2021.3.4f1 and the currently released LTS editor is 2 versions behind I suspect we're still a couple of weeks away from this fix being public. I've been also trying to debug this problem for our team but it is extremely time consuming debugging process and it's hard to test more than few options per day because of the build time.

Is there any more information that could let people with the sudden increease in build times verify that we are in fact affected by this specific problem that has just been fixed in unreleased versions? It would save us some time if we can assume that the situtation will become better next month and we don't need to troubleshoot on our own.

Is there anything specific to look in our project setup or log that triggers the problem?

 

Thanks for the information.

I tested both builds, here are the logs for lines including and following "Compiling shader "Universal Render Pipeline/Lit". Time measurements aren't comparable because they were built on different machines. Besides, the time measurements don't really reflect actual time spent between these lines appearing because these lines were filtered out of the 30k line log.

Android - Auto Graphics API - Vulkan and OpenGLES3 when the option is clicked off:
https://pastebin.com/L7uJAN53

PC - D3D11:
https://pastebin.com/LJPPiY8u

PC one has one line with very high amount of variants:

Compiling shader "Universal Render Pipeline/Lit" pass "ForwardLit" (fp)
800 / 1474560 variants left after stripping, processed in 134.54 seconds

but generally everything else seems much more reasonable than Android one.

Command to filter log files is included in pastebin logs, if anyone else wants to check it.

 

Unity is actually recompiling a few variants every time, making incremental builds slower for no reason. That was the initial cause for the bug report.

The fact that 2021.2 introduced several new URP features should only have caused the initial build to be longer.

 

I see, so you mean the URP asset. Then if I have 4 URP assets for quality presets which are changeable from the settings, then there may be more variants?

 

Wanted to chime in. We shipped a game on PC/Switch using Unity 2019.x, when the game was completed we had roughly 100,000 variants in one of our shaders haha. But still, the build maybe took a few hours. It seems that shader compile times themselves have drastically increased somewhere between 2019 and 2022.

We were of course using the built in render pipeline back then, and now we're using URP for future projects. We have a project with 10,000 or so variants and it takes hours the first time we compile. It just seems like each shader compile is taking drastically longer than a single shader compile used to. In both cases we are writing custom vert/frag shaders (both back in built in and even now with urp).

I guess what I want to know is, is there something we can do to lower individual shader compile times?

 

Epic. We will give that version a go as soon as it is out. Thanks!