Bug Build time taking almost a day - URP

Unity 2021.3.4 did not change anything here. We were able to reduce URP Windows build times before using the Shader Control asset from about 6 hours to 10 minutes.

Unity 2021.3.4 still does that x / 138M preparing variants pass and then starts recompiling all variants though I'd expect them to be cached already.

As soon as we start adding any 3D stuff using URP Lit shaders to a project and do not supress individual shader keywords using shader control, build times jump from 2 minutes to several hours. Some shader graph shaders multiply this shader build madness.

This issue needs way more attention. We are at a point where we are questioning if Unity is the right tool to use for productions with tight deadlines.

 

Sorry to be doing little more than piling on, but after seeing how slow our URP builds are due to shader variants, I found the Nov 2021 article which brags about cutting the build time of the URP Template project (the construction tools and wall) from 20 minutes to 7. Seven minutes to build the Template project seems more like it should be a warning about the performance, not an indication that things are in good shape. It just took us 14:30 to build one test scene that's just a terrain base, three characters, and one small tree. Yikes.

 

Looks like it is finding a cached version, but while it says 0.0 seconds, the process of stepping through those variants takes very long. In case of the URP Lit shader with 10k versions displayed in Unitys build progress bar window it takes almost an hour to step through all those files (for the Lit shader alone). Does not look like it is "really" taking the cached version.

In case of ForwardLit I found this:

Compiling shader "Universal Render Pipeline/Lit" pass "ForwardLit" (fp)
[188.88s] 100M / ~138M prepared
288 / 15138816 variants left after stripping, processed in 249.23 seconds
starting compilation...
finished in 17.93 seconds. Local cache hits 0 (0.00s CPU time), remote cache hits 0 (0.00s CPU time), compiled 288 variants (563.38s CPU time), skipped 0 variants

Seems this never uses the local cache. We stripped this version down by disabling keywords via Shader Control.

I can PM you the complete log file in case that helps.

 

After nearly two months
the 5h+ build time bug was solved in Unitry 2021.3.3f1.

Buildtime was back on normal 20min to 40min.

Starting with Unity 2021.2.4 the behaviour is back. Back on 5h +.


Saw this changes in
https://unity3d.com/unity/whats-new/2021.3.4

• Shaders: Fixed a crash when warming up a shader variant collection that had a shader with UsePass. (1407692)
  
  
• Shaders: Fixed include dependencies being reported incorrectly when guarded by defines that depend on keywords. (1401580)
  
  
• Shaders: Fixed redundant shader variant recompilation after Editor relaunch. (1425799)
  
  
• Shaders: Fixed ShaderKeywordSet.Disable not disabling keywords. (1417644)

However they killed the ability to work with Unity again.
Checked again on 3 independent large projects on 3 different PC´s.

 

@keeponshading I just tried a build with 2021.3.3f1 and it wasn't any faster than 2021.3.4f1 for me .

 

Unfortunately 2021.3.4 did not fix the issue.

[B]2021.3.4:[/B]



    [18:51:44] :             [Running Unity editor] Compiling shader "Universal Render Pipeline/MilkshakeLit" pass "Universal Forward" (fp)

    [18:56:36] :             [Running Unity editor]     9216 / 14155776 variants left after stripping, processed in 292.50 seconds

    [18:56:36] :             [Running Unity editor]     starting compilation...

    [18:56:38] :             [Running Unity editor]     finished in 1.75 seconds. Local cache hits 9216 (27.99s CPU time), remote cache hits 0 (0.00s CPU time), compiled 0 variants (0.00s CPU time), skipped 0 variants





[B]2021.1.23:[/B]



[14:43:12] :            [Running Unity editor] Compiling shader "Universal Render Pipeline/MilkshakeLit" pass "Universal Forward" (fp)

[14:43:12] :            [Running Unity editor]     82944 variants, prepared in 0.01 seconds, starting stripping...

[14:43:12] :            [Running Unity editor]     finished in 0.05 seconds. 18432 variants left

[14:43:12] :            [Running Unity editor]     starting compilation...

[14:43:18] :            [Running Unity editor]     finished in 6.34 seconds. Local cache hits 18432 (101.38s CPU time), remote cache hits 0 (0.00s CPU time), compiled 0 variants (0.00s CPU time)

Whatever tool or code is performing shader stripping has exponential complexity now, so it seems.

It also scales with the amount of scenes you're building.

 

Thanks for the clarification @aleksandrk
The fix for "Unity sometimes recompiles variants that it compiled already" is definitely verified as fix on our end

 

Hi all,

Any update?

I've been working this month without building to avoid this, but I'm really kind of at a loss here.

Kronnect's Shader Control helps-- It took my project's build times from days to hours. However, my project used to build in about ~10-15 minutes. Unfortunately, a lot of shaders cannot be stripped by Shader Control and rely on Unity to keep variants in check.

 

An update is needed on this issue.
Multiplayer games are impossible to debug and test because of this problem.
Rotating a simple gameobject, saving the scene and then making a build, makes a check on every single shader variant (400M+, 25-40minutes), even modifying a prefab.
Porting my project into another engine might even be faster.

Is there a way to simulate play of 2 different players inside the editor?

 

Could look at https://github.com/VeriorPies/ParrelSync or similar

Havent used it myself, but it does pop up every now and then

 

+1

Building Addressables / compiling shaders takes forever currently.

 

Any chance the caching is too aggressive now?
Had a bunch of shaders complain about errors during a build that didnt exist anymore and the only fix was a manual reimport of the shaders ("could not open include file xyz" since I deleted & reimported URP for double checking a bug)

 

Unfortunately 2021.3.5 did not fix the issue.

 

This is getting silly now.

 

terrible

 

First-time builds in Built-in pipeline were like 10min for our visualization projects. For a current project we tried switching to URP, but build times really are unusual as i discovered way too late. For now i've always had to cancel my testbuilds to continue work and hope the next patch would possibly bring a fix for this, but it seems it still hasn't happend (2021.3.5f1). I hope unity is aware of this issue and prioritizes it accordingly. So far it seems unusable compared to built-in.
Maybe i should have tested building earlier; i just didn't expect this to be an issue but now becomes a problem under time constraints.

 

Could any of the Unity team members assigned to this bug at least give us some info - is this problem being worked on and is there a rough estimate when the fix is going to be available? Almost 3 months after the LTS release, 47 days after the issue has been added to issue tracker we are still completely in the dark.

This problem slows down test runs massively and our team would like to know if it's going to be fixed in coming weeks because we are considering rolling back from LTS to last stable Tech Stream 2021.1.x or 2021.2.x.

 

Thank you for an update. That's great to hear.

 

@aleksandrk, great to hear this issue has high priority now.

I have been digging into this whole shader compilation thing over the past weeks, since our projects usually have a very tight deadline and lengthy compile times are a serious issue.

Shader Control from the asset store has been a help, but only resolved the issue to some degree.

I would like to show what we have done so far and show some quirks where Unity could help out, but still fails because some data are missing.

We use a custom build interface, so we can easily enable things like "Log Shader Compilation" or "Strict Shader Variant Compilation" temporarily, without the need to search project settings.



Based on this I use a custom Shader Manager (scriptable object) that completely replaces Shader Control. We can "Disable All Shader" to run a very fast non functional build, that only scans for all shader and keywords requested during the compilation process.



The remaining four tabs provide four methods to include shader variants in the build.

Player Log
With the "Strict Shader Variant Matching" option enabled scan the player log for missing variants. Actually, this is a great way to detect missing shaders, but unfortunately it does not catch all variants that are missing.



Shader Variant Collection
Shader variant collections are another great feature to detect variants by simply starting play mode in the editor. Again, this method does unfortunately not detect all variants being used.

The shader variant collection API definitely needs some love. There is no way to query the variant array, which is the most basic method I would expect to exist. This forces us to evaluate the collection during the build process using the "contains" method against all variants the callback throws at us. Yikes!



Global Keywords
Here we can enable variants manually via a selection of all keywords found across all shader. The shader itself must be enabled for this to work.


Shader & Keywords
Enable individual shader and corresponding keywords. The keyword methods are inferior to direct variant detection, because enabling 8 keywords results in 256 variants being compiled, where actually only very few variants are finally used.

 

We also went from 10-30min up to 7h when building addressables, if we leave in all shaders/keywords.
We created an automated step where all keywords are being disabled and all unity shaders are copied, keywords disabled and then reassigned - with this we went down to about 1h.

All of this escalated after going from 2020 LTS to 2021 LTS.

Looking forward to the fix.

 

Hello everyone, the problem is relevant in version 2022.1.5f1. The first build was done for 14 hours, but there was little space on the hard disk and apparently hung up. Restarted again freeing up 40 GB of space. The build is being done for almost 5 hours. Using URP

 

Would it be possible to get a "fix" for this earlier, by providing an option in Unity preferences to simply revert to the old build method? We'll gladly turn off this "feature" to be able to return to normal operations. This is not something that we can wait months for to get a "proper" fix.

We even considered returning to the previous LTS version but downgrading the project is undoable at this point.

This was supposed to be an upgrade but instead we received a complete halt in the studio development iterations... We completely lost trust in LTS releases due to this.

 

All people in this topic are getting hundreds of % increase in build times which was not the case in previous LTS and it completely obstructs the way they conduct business.

Then you come here on a white horse and announce that Unity team barely did anything and we should be thankful for 20% speed improvement. It's not an improvement, it's not a downgrade - it's a complete obstruction of gamedev studio operations.

You guys charge a substantial amount of money for the engine. DO NOT screw with people's money or even the biggest marketing agency merger won't help the company. HELP us instead of providing a stupid PR response.

 

Bear in mind that, for me, 20% improvement means I'm saving half an hour. What boggles my mind is that if it takes 3 hours to compile the build, and I IMMEDIATELY re-compile the build, it takes 3 hours. It's like nothing at all is cached. Is this also being addressed?

I beg.

Edit : I'm still begging. This is madness. It was about a 6 minute build only a few months ago, and I've really not done much other than half a dozen shadergraphs...



Build completed with a result of 'Succeeded' in 34940 seconds (34940468 ms) (~9.7 hours) So a 20% improvement would save me 2 hours. Again, this is NOT a complicated project - and Unity has no UI at all to help me track down if I've managed to tick the wrong box somewhere.

 

Uh. Just to check. You think this is... correct?

 

Edit : Interestingly, deleting all of these files caused my next build to be 10 hours. However, most of them weren't recreated, and my next build? "Build completed with a result of 'Succeeded' in 3457 seconds (3456729 ms)"

Under an hour.

 

Got a question about releases and bug tracking. 2021.3.3f7 just came out. It doesn't have a fix for this build perf issue. But I also notice that the build perf issue is not listed as a known issue in the release. Now, both 3f7 and 3f6 do list ongoing known issues (for example, AI Navigation Core: NavMesh::Raycast freezes the whole editor in an infinite loop on Application.UpdateScene (UUM-2496) is in both 3f6 and 3f7 known issues).

So my question is this: does the fact that the known issues don't mention this thread's performance issue mean that Unity doesn't consider it a confirmed known issue? If so, is that because Unity is unable to reproduce the circumstances that cause the reported order of magnitude build time increase?

 

I don't think anyone is disagreeing there. People have been showing variants in the millions. As I've said in this thread; I'm perfectly happy to accept that I have ticked something or done something to cause the variants to exponentially explode. But I don't know what it is. And there's nowhere that Unity tells me how many variants there are (except during build time) and literally nowhere where it tells me WHY.

2021.3.6f1, running off an NVME drive.

Edit : The number of variants also appears to be completely random.

https://cdn.discordapp.com/attachments/371347509467152384/1004074271758364682/unknown.png

 

After blowing it away and desperately deleting shaders I probably don't want, we're back to...

As I said, if there's something I could tick or untick that might solve it, I'm more than happy to.

 

thought mine would be more

 

@aleksandrk Thanks for being so active in this thread. Any new info maybe when the planned fix might hit 2021 LTS?
The build times are a crippling issue for the whole company.

 

Can I ask one thing about build times, why are textures compressed during build time that are not going to be part of the build?

This takes time and disk space while not needed.

I wish there was a way to exclude them from the compression stage.

 

https://forum.unity.com/threads/exc...build-from-compression-at-build-time.1317945/

I created a new thread, thanks for the suggestion.

And if its a bug or a feature request I don't know for sure.
Thanks

 

Same here still since several months now. We are now in 2021.3.7 LTS.
Are there any news?

 

You mean in the Forward Renderer? I've got SSAO, Decals and ScreenSpace Shadows. There's not really anything to remove.

 

One of the things that really does boggle my mind is that the number that appears here seems to be completely random. This time it's 415M. I assume the 'm' means 'million'