Compiling Shader Variants Taking Ages

Does anyone know a relevant Unity person we could @ ping here to get their attention?

 

As far as I am aware they are all aware. Although I think "how could any unity user, let alone developer not notice the change" because of how bad it is.

I fail to believe anyone has not already dealt with this internally. Either a fix is imminent, or they have not found the issue yet so cannot resolve it.

 

We're on 2018.1.6 and seeing this more and more now... anyone know if 2018.3 version fixes any of this?

 

Can't remember which version, but my builds used to take around 8 hours each time I upgraded unity due to the shader compiling. This was before the shader compile progress bar even existed, so Unity just appeared to be frozen. Since 2017/2018, the first build still takes an hour+, but subsequent builds around 40 minutes. I can live with that because most often I'm doing scripts only builds which only take a couple minutes. My main shader problem is that now my built game is taking a full 21 seconds on the Shader.Parse step (seen in the profiler). I've asked around from help, on here, and Reddit, but I've had no responses. In total it takes around 45 seconds to launch my (Vive/Rift) game which is way too long.

 

With Unity 2018.2.4f1 and upward If you create any new project and install at least one 3rd party asset, with some new shaders included, you will see this issue. When I also upgraded an older project to these versions, builds that previously took 20 to 25 mins now take 6 to 7 hours, unusable. I had to revert back everything after a few weeks of trying all new unity versions (Project is 120gb) so takes a long time to see if upgrades work. I did test out different settings for the new render pipelines and nothing seemed to help, just destroy the project a little more.

On a side note, if you happened to purchase all of the bonus assets from the August promotion for the asset store and import them into one project you will very quickly hit the 256 keyword limit.

 

Yep broken as. Build times are still atrocious. I had the same thing, projects that took minutes now take stupendous amounts of time, when you have to do three build of one project for a realease it extremely painful, then if you have to quickly iterate well that’s not possible when each build takes 8 hours.

 

Start a project import an asset and click build

 

"Optimizing Shader Load Time

Shaders are small programs that execute on the GPU, and loading them can take some time. Each individual GPU program typically does not take much time to load, but shaders often have a lot of “variants” internally.

For example, the Standard shader
, if fully compiled, ends up being many thousands of slightly different GPU programs. This creates two potential problems:

• Large numbers of these shader variants increase game build
  time, and game data size.
• Loading large numbers of shader variants during game is slow and takes up memory."

-https://docs.unity3d.com/Manual/OptimizingShaderLoadTime.html

Can someone who isn't experiencing this issue explain what their difference is. My project is being run on Unity 2018.2.5f1. I have a project size of about 120gigs and am using several AAA massive scene asset packages and 10 smaller ones. The problem made it impossible for me to develop a multiplayer game for about a month, as to test any multiplayer feature you need to make a quick build to have an alternate player to show the other client. I have managed to fix this and now a build which includes 8 big scenes that ends up with a project build of 3.5gb. It takes about 10-20 mins to build. Each different shader used in the project will make about 2,000 - 10,000 shader varients to compile on build. I've personally found by making sure to stick mainly to the standard shaders (rough and specular) and followed all the instructions above I've been able to knock down the build time from several hours to about 10-20 minutes. This was a development breaking issue, but I've been able to fix it for my project. It still compiles shader varients, but only for the minimal shaders that I use in my project. Other things I've followed:

- Graphics Strip Unused variants is selected.
- Saving a new compiled shaders asset before build.
- Confirm shader instances is not ticked on your material's shader.

That's sorted it for me, but it was a massive massive problem and completely stunted development for a time. There's been a major graphical overhaul in the build process, but it hasn't been explained or documented.

@Tim-C are you experiencing a pop-up on build which says 'compiling shader variants'? What defines how much work this process has to do (using more shaders for example)? What's the way to minimise the length of this job on builds? There has been a new invisible hoop to jump through that hasn't been explained to us.

 

Me too having this stupid problem perhaps after trying out post processing fx pack... Even after removing the package it still compiles some 15k shaders everytime I build......

UPDATE:
Actually for me it turned out I had for one reason or another still PostProcessing folder in our project. I just searched for all shaders in project and found it out this way.

 

We have the same problem.. the tricks posted by @Nigey helped a bit, and compiling into the same folder too, but in 5.6 builds took me 1 minute, now it takes 10 minutes.. why the f*ck, (now on 2018.2 latest patch)
it seems like as before in 5.6 it is not ACTUALLY recompiling any shaders on the 2nd+ build, it only compiles any shaders the first time you build into a new folder, (I see all my 18 cores go 100%) in all other builds its going through all the shader variants for ages (only single threaded) and compiles nothing, cause it probably somehow detects that nothing changed. great. awesome.. but why can't it just skip the whole step at all then?

@Unity, this is a real pain and I mean a REAL PAIN.
We are developing a multiplayer game here, recompiling a server is something I have to do often, and having to wait 10 minutes every time I want to try out some fast test is absolutely painfull, I bought a 10.000 euro PC to make my life faster, but your stupid system is mostly single threaded. jeez.

 

When adding preloaded shader variants the number of shaders compiling increases dramatically for me. Having strip all and no preloaded shaders seems fastest. But either way it takes forever. At least 100x slower than 2017

 

@FastKillteam1 add the pre-loaded shaders and keep strip all. The first build will be slow. Try building again after that first initial build. It 'should' have computed all the shader variants already and the subsequent builds 'should' be faster. Try it and see. This is a bit like taking artillary shots in the dark right now, but give it a go.

 

When i build my project on my laptop it also compiles every shader and takes about 1 hour. But when i build the same project on my PC it does not compile the shaders but finishes the build very fast. I'm using Unity 2018.2.6f1 on both devices.

Edit:
Turns out it did compile the shaders on my PC aswell but extremly fast so i didn't notice it.

 

I have tried that, believe me I have. I have tried absolutely everything. Maybe it doesn’t work like that for me as I have to change the build target after each build?

 

In my case I haven't been changing build targets. That could well be causing some issue I don't know about. We're dealing with unknowns about this issue unfortunately.

 

Does anyone know how exactly the shader cache works?

From my understanding, shaders that have been compiled are saved to Library/ShaderCache?
So when i do the next build, those shaders should be re-used?!

So even if the shader compilation takes a very long time, when they are cached, the next time should be quicker?

The problem is that i just tested identical builds one after another and the shader compilation again took ages (40 minutes).
I checked the Library/ShaderCache folder and it is full of stuff. It has not been cleared between build 1 and build 2.
However, the build seems to compile all shaders again from scratch and 'ignores' the cache?!

 

Any update on this? it makes me wish to go back to 2017.4 -_-

 

wtf ... for me too.

 

I have the same issues, trying to fix this somehow since hours, thanks Nigey for all the hints, but nothing really helped. Desperatly searching and/or waiting for a fix to this. Right now, the only thing that builds with no problems is my rage...

 

So has noone done an actual build at unity ? I just dont understand how not a single person could have noticed this, it really is appallingly obvious the first time you do a build.

I understand the need for a bug filed, but surely you guys actually do a build etc before releasing and must have already caught this? Its not a one off or specific use case bug, it affects ANY project using ANY shader that has ANY variant at all. It will compile ALL variants regardless of what you do. Including a shader variant collection only vaguely increased speed, but that could also be a placebo to be honest, as it really was not thast noticeable.

 

I think I found a solution:

tl;dr: create a new project, import all your files and build; it shouldn't compile thousands of shader variants.

I created a new empty project to test if shared variants would be compiled for a scene with a single cube, and turned out it didn't. So I imported the files from the old project (which compiled like 4k shader variants), and it built, it actually built as fast as with older Unity versions. I was amazed, I felt I was born again.

Now I have to note that the project I'm talking about is very tiny; it consists of some scripts and a couple models (despite that, however, it was compiling 4k shader variants). This worked with Unity 2018.2.11f1.

 

For anyone moving forward on this. It looks like shader variants are an unavoidable part of the Unity workflow now. However, there are ways to optionally either circumvent them, or dig into the depth of shader variants and add only what you need to compile.

Important Reading
Unity Blog Post: https://blogs.unity3d.com/2018/05/14/stripping-scriptable-shader-variants/
Manual: https://docs.unity3d.com/Manual/OptimizingShaderLoadTime.html

tl;dr - For LWRP there is a stripping script that allows you to remove 98% of the shader variants. HDRP/Custom/Regular you will need to dig into understanding shader variants a little more to remove the ones you don't need. Unfortunately you'll just need a bite a pillow and go for it with this one. Read the links above and start understanding shaders in more detail. You can follow my instructions that can 'sometimes' work (Select StripAll and create shadervariants file and assign in the Graphics tab). No one here is sure what the criteria of it working and not working is, but give it a go.

 

@Tim-C any more news on this?

Its getting very hard to explain to rest of team why a process that always took 5 mins now takes 1 hour 22 minutes, in a project that has not increased in size or complexity.

A double increase I could understand, but this ratio makes unity almost unusable. We are foregoing doing builds most of the time which is hurting QA but it cannot be avoided because its getting hard to explain to employer why our PCs lock up for half of the day if we iterate at the rate they require.


Please please can someone respond with a fix, or at least say "a fix is not coming, this is life now" so we know where we stand. We are not using the new pipelines, so having a LWRP script that fixes this as mentioned by @Nigey does not really help us. Nor can we revert to an older version of unity.

We are on 2018.2.11, however all versions of 2018.2 do this. Using beta is not an option for a commercial project of this size.

 

@GameDevCouple_I You tried all the tricks in this topic? it helped a bit at least, also making mono builds instead of c++ helped me..

And make sure to always build into the same folder, if for your project you need to keep old builds, simply copy paste from that export folder to the folder you require.

 

LWRP, unity 2018.3b11 building for WebGL here, I needed to have LWRP Lit, SimpleLit and Unlit shaders to be used at runtime (not having pre-instantiated objects that used this shaders).

As this 'LWRP-built-in' shaders cannot be added to a 'shadervariants' file (the ui just won't find them, as when those shader files are searched for in the project) we ended up having a disabled object per shader (different material each), pre-instantiated in the scene, to avoid stripping those shaders (even if you configure to 'keep all' shaders in the graphics settings, these get stripped if they aren't used anywhere).

This way we avoid the unbearable building time of shader variants compilation for these 'LWRP-built-in' shaders.

Hope this idea helps somebody save some time until this workflow is better.

 

Having the same problem. I have a fairly small and simple project but it's taking Unity 10 minutes to build all the shaders. I am building for the Android platform.

 

UPDATE: After the initial build took 15 minutes, I followed the workaround @Nitey suggested and there is a significant reduction in shader variant compile time.

 

Another thing I've noticed is that this problem doesn't occur when building for PC. It seems that Unity's main focus might not be on mobile platforms.

 

I found that thread after having the same problem with 2018.3.0b12 / LWRP VR.
Simply trying to build the example scene to see how it runs on Oculus Go.

 

I started with LWRP/VR template, then imported Oculus Utilities and Realistic Car Controller. Did Nigey trick. That's pretty much it.

Build time: ~1.5 hours

 

You have to do Nigey's trick after the first build for it to work. At least that's what happened for me.

 

@embeddedt I did Nigey's trick and I don't think it had much impact on the build time. It's still unacceptably long.
Again, this seem to be an issue only with the LWRP VR template. Was your project VR?

 

No, my project was for Android.

 

The "trick" doesn't seem to work with mobile VR.

I could slightly improve the building time by:
- removing Vulkan support
- using only 1 additional light
- using no shadow cascade
- deleting all unused shaders from the project
(just to be clear, I'm not saying these steps are all helping, it's only what I tried)

But the build time is still not acceptable and I'm going to stay from LWRP VR for now.

 

We are having the same issues. Is there any way to work with Unity 2018 for mobile VR? The current build times are just unacceptable. With Unity 5.6 we had 10 minutes build times. Now we are at 3-4 hours. After we tried stripping shaders it even got worse and did not finish building over night.
We can't work with this. Is there any workaround that works for mobile VR?

 

Same issue here. Does anyone know the last version of Unity that is stable?

 

POSSIBLE WORKAROUND
The only way we got around this was to remove all multi_compile statements and move to shader_feature, despite what the latest blog post on shader stripping recommends. I recommend you all do the same.

Ofcourse you will need to manually include every single shader variant in the build yourself as they will get stripped out otherwise, but its better than this mess. Build went from 1.5-2 hours to 6-10 minutes.

 

I don't really have any idea how to start with your suggestion. Any pointers or links would be very appreciated!!

 

Yeah, just like @dariuspowell , I'd love some info on how to do that (with the LWRP VR template for example). Your help is greatly appreciated.

I guess it would be useful for people bumping onto that issue in the future.

 

POSSIBLE WORKAROUND
Hey guys, I am at work currently and will post properly later but for now read this:


https://docs.unity3d.com/Manual/SL-MultipleProgramVariants.html

basically any usage of #pragma multi_compile needs to be replaced with #pragma shader_feature

However, you then need to create a material per variant for your shader and put thgat into resources folder somewhere so that it gets included in the build, otherwise any shader features that are enabled by code will be stripped from the build.

Its a massive pain and very painful when you have a giant uber shader like we do but its better than 2 hour build times!

 

It's unbelievable that Unity released this and does not fix it. We tried most of the things in this thread and currently wait for the next results. The current build is already building since 23 hours.
This totally killed the whole development since more than a week now as we are doing nothing else than waiting for compilation to be done and see if the new try has brought any benefit. If there is no suitable solution soon, we'll probably switch to Unreal for our next project. This is just unacceptable.

 

For me too (2018.2.6f1).

 

try what I posted above @konradkunze as it was literally the only thing that cut down the time, all other suggestions did nothing for us

 

I read the documentation page you shared and that gave a better idea on what's going on in the background. Thanks for that. But I'm still looking forward to know how you create those "materials per variant"

Also, if you manually modify LWRP shaders, does that mean you can't use the Package Manager for LWRP anymore? Or do you create copies of Lit/Unlit shaders and make sure to never use any of them so they don't ever get compiled? I don't have any custom shader (yet), I only want to use Lit/Unlit.

Thanks man.

 

Seems like unity is getting worse with each increased version.
I'm still using 5.6.6. because I absolutely see no reason to pay a monthly fee for a broken tool.

Anyways, I have a similar problem in 5.6.6. in relation to the Standard Shader (always include so assetbundles do not contain duplicate standard shaders --> causes infinite loop on compiling shaders build process). So my hacked together workaround is to have a separate project that I only use to build the AssetBundles and my main project where I avoid the settings that cause the stuck build process.

Maybe this is a regression bug that came back to life in the new version? I would not be surprised, because in my past experiences of upgrading from one unity version to another (Unity 4 and Unity 5 period) there were often bugs reappearing in the same or similar fashion after an update that seemed to be solved in one of the prior updates.

I'm glad I dont have to endure this pain of the update process anymore, I stick with 5.6.6. and if I ever had to use a more modern engine, I rather spend my time on another viable alternative.

 

This thread is not about 5.6, It's about the very recent 2018.2 and 2018.3 BETA, and your message does not help. You can stick to it if you want, who cares? Unity is an amazing tool. We love it even with its bugs. Please go send your troll messages somewhere else.

 

@13Flo I'm not a Unity expert, but I think the problem @AxiomIT has may be related. It seems that even Unity 5.6 has issues with compiling shaders.

 

It looks like unity have listened, as they have added async shader compilation to 2019!

Source: https://forum.unity.com/threads/2019-1-0a11-editor-weird-look-while-building-a-project.594892/

Very happy about this!

 

Okay this is annoying af. Builds used to take about 3 times faster.