Compiling Shader Variants Taking Ages

Just here to report the same unoptimized problem under 2019.3.4f1 with the default scene over URP and building for Android.

I don't know if can call it a bug, but it's very regressive!

All the time, energy and material we could save if this wasn't a thing.
it would be worth finding a solution.

 

For me, checking "Optimize Meshes" did the trick

Edit: For those who checking optimize meshes didn't work for them, I suspect that the very first time you build your game it'll take a while (or when you make changes to a shader it'll recompile that shader) but try rebuilding it a second time (overwriting your first build) and see if that makes it much faster.

 

I can report that deleting library helped me. The project had 120 possible shader variants with only a few shaders, but the build still got stuck compiling them over and over. Must be a Unity bug. Using 2018.4.19f1.

 

I'm currently using Unity 2018.4.19f1 and trying to build fr UWP to deploy on the HoloLens and only using about 8 Shaders (Just added a partical shader btw too standard XR settings) and all the steps mentioned in here didnt fix. Tried deleting Library, Mesh Optimization is unchecked. Unity is building forever with the message: Particles/Standard Surface: x of about 61440.
Any ideas what I could try out? Upgrade to 2019 maybe?

Edit: I deleted the particel shader from the "always included" list and this brought me back to 1 min. build times and just tried it out on my hololens and I can see the gameobject with the material and the shader on it without any problems. Is there any problem not adding my used shader in this list?

 

I am building my little project for the first time. A single scene that is using about 10 materials (URP/Lit/PBR shader).

But what's going on??

 

Unity has announced Caching Shader Preprocessor. See here:
https://blogs.unity3d.com/2020/05/0...new-caching-shader-preprocessor-experimental/

 

I tried checking "Optimize Meshes" and also saving the variant file, still no go. Had to delete Library folder and let Unity rebuild it. First build took about 8 minutes, now subsequent builds are much faster at about 2.5 minutes. I'm on 2019.3.13f1

 

For me using the using the JDK that comes installed with unity fixed the issue.

 

In my case, do this:
ProjectSetting -> Graphics -> Built-in shader settings -> Always include shaders -> Remove Standard Shader

If you want to use AssetBundle with standard shader, download official standard shader, and package into AssetBundle.
https://unity3d.com/get-unity/download/archive

 

Hi, how did you find out which shaders were causing this problem?

 

When build

When building shader, it's name will display under the progressbar.

 

It takes me like 5-10 minutes to make a very simple shader modification, I dont know if its because the project is too big or what, I was running into problems where I was running of kewords (256).

 

it's funny how people complain about 10 min... i left my computer run overnight and it never ended. In the morning, the "cacheshaderlibrary" folder had millions of file. It's completely insade. i had to force close unity. I'm mostly using standard shader, plus I have beautify, NGSS, AFS and some naturemanufacture. I have no idea how to get a build now

should i delete unused shader ? they should not be included anyway as far as i know ?
should i delete "exemple scene" that use shader i dont need ? again, my understanding is if the scene is not included in the build, then the shader used in them shall not be included.

how can i build ?

 

Hiya - was in your same exact boat until about 30 minutes ago (although not quite as bad, I was seeing 2 hour builds). Whats worse, the game would basically immediately consume tons of memory when I ran it (I think trying to load an ungodly amount of shader variants into memory) and crash immediately.

Following advice from some folks earlier in this thread got me back to < 5 minute build times and more importantly a working build.

What ended up working for me was a combination of following Nigey's advice:

1. Removing all entries from Project Settings > Graphics > Always Included Shaders list and only adding back ones I really needed (like UI/Default and UI/Default Font)
2. Creating a ShaderVariant asset via Project Settings > Graphics "save to asset" at the bottom and adding that to my "Preloaded Shaders" list.
3. Setting Project Settings > Graphics > Instancing Variants to "Strip All"

And then I use the Oculus Integration for Desktop package so I also followed fxtentacle's advice about the OVRLint
script and clearing out that stuff:

> "FYI, the "OVR Performance Lint Tool" from the "Oculus Integration" asset keeps a list of FixRecords which contain pointers to game objects, materials and shaders. Apparently, that causes problems, because my build times increased by an hour after I imported the Oculus scripts." -fxtentacle

I'm not sure which of these fixed it but just want to let the thread know these recommended solutions worked for me. I was tearing my hair out. Thank you Nigey and fxtentacle

 

have already done step 2 and 3, ill try number 1 and post back. tried again, left overnight and my shader golder now show 2.2 millions files. rock on unity (my shadervariant file talk about 232 variant)

EDIT : didn't change anything. Final count is 3.3 millions files.... what the h is that

EDIT 2 : despite the shared asset being extremely high in file count once it is created i can produce build in 20 minutes. I still need to understand how to get that file count much much lower.

 

I don't see anyone asking: WHAT PROVOKES THE NEED FOR THIS?

Is there something we can NOT change that will avoid the need to compile these things?

it seems to me that something that has not changed should not require recompilation

tone

 

good question, my limited understanding it that it compile and cache EVERY single shader that are in the project instead of just caching all the variant for the one you actually use. Now why do i get more than 3 millions variant is out of my understanding.... it's pretty dumb for the editor to process everything when not require. It's slow, it crash, it use too much space and having so many file also prevent backup tool to work (sync toy cannot process the folder, the structure is too big, it doesn't need to be backup but microsoft sync toy dont offer to exclude subfolder...).

 

Did anyone take a look at this?
https://portal.productboard.com/uni.../12?utm_medium=social&utm_source=portal_share

"Improvements for shader stripping has landed and is available in beta through 2020.2b and URP v10"

 

Hello, I am trying to build the Oculus VR example for Unity 2019 LTS and am getting this problem.
I have Unity XR package installed
I have disabled Optimise Mesh which helped a bit.
But it is still trying to build 1500 variants of each shader!

Edit: Following another suggestion I did a "replace in files" with
#pragma multi_compile replaced with: #pragma shader_feature
Now only 150 variants.

Seems a little better for the custom shaders. However still building 1200 versions of the standard shaders: Standard - FORWARD, Standard - FORWARD_DELTA, Standard -DEFFERED. Multiple times each.

 

I had similar issues where the compilation of shaders just took way too long. After i moved to URP it took more than 12 hours to compile the shaders when building.

What solved the issue for me was deleting the "ShaderCache" folder inside the [ProjectName]/Library folder. After deletion Unity recreated the shader cache when i ran the build next time, which took around 60 minutes. Once that was done each build completed between 2-4 minutes.

I guess when there have been too many changes in shader code or shaders added/removed from the project the shader cache just gets too large, or it was just corrupted in my case.

As far as i know it is not risky to delete the "ShaderCache" folder inside the "Library" folder, but there might be risks that i am not aware of.

 

Well I built it at last. BTW anyone know the previous version of Unity that doesn't spend so long on shaders? I haven't got the patience to wait that long every time!

 

I'm on 2020.1 and I worked extensively with WebGL for the past year. I must say the compiling shader issue is gone. It might still be there in an initial build but uncompressed builds now take +- 4min for me to make.

I'm talking about a relative - small file sized - project with only code. All assets are loaded externally. Might have something to do with it too.

 

I have a feeling it might be related to the Unity.UI package. It would make sort of sense if it was compiling a shader variant for each individual letter of the alphabet!

Unfortunately the Oculus XR Plugin has a dependency for Unity UI (for some unknown reason) so I can't delete it.

 

If that is true, I am grateful not to be Chinese. Can you image compiling shader variants for an alphabet containing 6500 characters... I imagine there's somewhere a Chinese cloud build server with a project with an average build time of 45 years.

 

Well this just sucks. The reason I liked Unity over Unreal is that Unity didn't compile loads of shaders.

Mind you, just tried one of my own PC games that I updated from Unity 2018 to 2019 LTS and I don't get this problem with millions of standard shader variants. Maybe it's an Android thing.

Probable Solution: I removed Vulkan from the list of graphics APIs under the graphics tab. So now it only has OpenGL ES. I would say that has reduced the number of shader variants to about a quarter of what it was. It can now compile in a few minutes.

Still hundreds of variants though (I don't even know what a variant is for!!!!!).

 

That preloaded shader list is not always a great thing to do. I had done that earlier, but my project changed along the way (making the list irrelevant I guess?).
Removing it cut my build time 10 times!

 

@RoyBarina hmpf... noobs...

 

Unity 2020 LTS
Build Time : 3 minutes 53.69 seconds.
I Disable "OptimiZe Mesh Data" from Player > OtherSettings > Optimisation
Build Time : 15.31 seconds.

 

Did you cleared the shader cache in library before build?

 

No, Why?

I delete it now.
I compile and it took 3 minuts
The second time 11 seconds

 

Well, after a build it caches the variants so in the next build it won't compile them again (if they're the same) thus reducing building time

 

Yeah, this is not going to work. My game is small, I'm using probably 4 or 5 shaders and only building 2 levels.

 

Coming from a full time developer in Unity of 5 years, for what little I know I DEFINITELY know this:

1. Unity cache's your shaders, so you build it once, it's not going to take that long to build again.
2. Use as few of shaders as possible in your project, ESPECIALLY if you're targeting mobile in the future, you're not gonna want all those shaders in your project. Please listen to me when I say this, you do not want a lot of shaders you're not using in your project folder. Also, keep what shaders you do have in your project insanely organized...well...keep everything insanely organized, but especially shaders. You absolutely need to know what shaders your project is using and where at all times if you're planning on porting. Shaders are among one of the #1 causes of a badly optimized port. In Unity's project browser window, under the search bar you can type "t:shader" and you'll see all the shaders in your project. Watch out for downloading any Unity asset store assets as these can add more shaders, and many many times useless ones that do exactly the same thing that you're already doing with other shaders.
3. Shaders aren't materials, so you can have many materials, but make sure to have fewer shaders!
4. Stop using the Resources folder(s)! Use UnityWebRequest and StreamingAssets folder if you're on Windows, and if you're targeting Android create an asset bundle, and download it to your persistentDataPath and use UnityWebRequest to get the asset bundle, then use AssetBundle.Load.

When I first started Unity, I was using Resources.Load a ton, but after I switched to a class I wrote called ExternalResources.Load (that just uses the UnityWebRequest class), after just deleting ALL resources folders and stopping using Resources all together, I can now build a 2GB project in about 60 seconds time, where as before it was taking 10+ minutes to build after shader cacheing, and like 1 hour 10 minutes+ to build before it. Library re-importing was also taking forever, caused by switching graphics APIs in player settings or caused by switching build platforms.

Here's my class for 'ExternalResources' if anyone's interested. With Resources.Load I got used to the format for loading files like 'Resources.Load("System/Clock") without the '.png' and all that, so I kept that format with it. You can change the data path to whatever you'd like.

The class is tested and will work on Android devices, so long as your assets are first in the persistentDataPath folder, this doesn't use AssetBundles. You use the class like this:

Code (CSharp):

1.  
2. using System.Threading.Tasks;
3. using UnityEngine;
4.  
5. public AudioSource audioSource;
6.  
7. IEnumerator GetAudioClipExample() {
8.  
9.    Task<AudioClip> taskAudioClip = ExternalResources.LoadAudioClipAsync("Audio/MyExampleFile"); //relative path without file extension, but this has to be .ogg here.
10.    while (taskAudioClip.Finished == false) {
11.  
12.        Debug.Log("Loading audio clip....");
13.  
14.        yield return new WaitForSeconds(0.05f);
15.     }
16.  
17.      AudioClip myNewAudioClip = taskAudioClip.Result;
18.  
19.      audioSource.PlayOneShot(myNewAudioClip,1.0f);
20.  
21. yield break;
22. }

Above is untested code, below is tested and works on Android.

Code (CSharp):

1.  
2. public class ExternalResources
3. {
4.  
5.     public static string UnityDataPath = Application.persistentDataPath;
6.  
7.     #region PUBLICMETHODS
8.  
9.     public static async Task<AudioClip> LoadAudioClipAsync(string relativePath)
10.     {
11.         string filePath = GetFileFullPath(relativePath, ".ogg");
12.         if (filePath == null)
13.             return null;
14.  
15.         UriBuilder uriBuilder = new UriBuilder(filePath);
16.  
17.         using (UnityWebRequest unityWebRequest = UnityWebRequestMultimedia.GetAudioClip(uriBuilder.Uri, AudioType.OGGVORBIS))
18.         {
19.             unityWebRequest.SendWebRequest();
20.  
21.             while (unityWebRequest.isDone == false)
22.                 await Task.Delay(5);
23.  
24.             if (unityWebRequest.result == UnityWebRequest.Result.ConnectionError || unityWebRequest.result == UnityWebRequest.Result.DataProcessingError)
25.             {
26.                 UnityEngine.Debug.LogError("Data Processing or Connection Error for filePath: " + filePath);
27.                 return null;
28.             }
29.  
30.             return DownloadHandlerAudioClip.GetContent(unityWebRequest);
31.  
32.         }
33.  
34.     }
35.  
36.     public static async Task<Texture2D> LoadTexture2DAsync(string relativePath)
37.     {
38.         string filePath = GetFileFullPath(relativePath, ".png");
39.         if (filePath == null)
40.             return null;
41.  
42.         UriBuilder uriBuilder = new UriBuilder(filePath);
43.  
44.         using (UnityWebRequest unityWebRequest = UnityWebRequestTexture.GetTexture(uriBuilder.Uri, false))
45.         {
46.             unityWebRequest.SendWebRequest();
47.  
48.             while (unityWebRequest.isDone == false)
49.                 await Task.Delay(5);
50.  
51.             if (unityWebRequest.result == UnityWebRequest.Result.ConnectionError || unityWebRequest.result == UnityWebRequest.Result.DataProcessingError)
52.             {
53.                 UnityEngine.Debug.LogError("Data Processing or Connection Error for filePath: " + filePath);
54.                 return null;
55.             }
56.  
57.             return DownloadHandlerTexture.GetContent(unityWebRequest);
58.  
59.         }
60.  
61.     }
62.  
63.     public static Texture2D LoadTexture2D(string relativePath)
64.     {
65.         string filePath = GetFileFullPath(relativePath, ".png");
66.         if (filePath == null)
67.             return null;
68.  
69.         byte[] bytes = File.ReadAllBytes(filePath);
70.  
71.         Texture2D texture2D = new Texture2D(2, 2);
72.         texture2D.LoadImage(bytes);
73.  
74.         SetTexture2DDefaultSettings(ref texture2D, relativePath);
75.  
76.         return texture2D;
77.  
78.     }
79.  
80.     public static TextAsset LoadTextAsset(string relativePath)
81.     {
82.         string filePath = GetFileFullPath(relativePath, ".txt");
83.         if (filePath == null)
84.             return null;
85.  
86.         TextReader textReader   = new StreamReader(filePath, Encoding.UTF8);
87.         string textPulled       = textReader.ReadToEnd();
88.         textReader.Close();
89.  
90.         TextAsset textAsset = new TextAsset(textPulled);
91.  
92.         return textAsset;
93.     }
94.  
95.     #endregion
96.  
97.  
98.     #region PRIVATEMETHODS
99.  
100.     private static void SetTexture2DDefaultSettings(ref Texture2D texture2D, string relativePath)
101.     {
102.         if (relativePath.Contains("AniSets")) {
103.             texture2D.filterMode = FilterMode.Point;
104.             UnityEngine.Debug.Log("aniset type!");
105.         }
106.  
107.     }
108.  
109.  
110.     private static string GetFileFullPath(string relativePath, string extension)
111.     {
112.         string fullPath = UnityDataPath + "/" + relativePath + extension;
113.         if (File.Exists(fullPath) == false)
114.         {
115.             UnityEngine.Debug.LogError("No asset found with path of: " + fullPath);
116.             return null;
117.         } else
118.         {
119.             return fullPath;
120.         }
121.  
122.     }
123.  
124.     #endregion
125.  
126. }

If you're on Windows, this is way easier, just use StreamingAssets and use a methodology like above, but whatever you do never create a 'Resources' folder for your assets, and never ever use Resources.Load. Your build times will EXPONENTIALLY grow.

What most people would recommend is building an asset bundle, on Windows it's really easy you can just put this bundle in your StreamingAssets folder, but this won't work on other devices you port to, so if you're thinking of going multi-platform, just keep this in mind, you want to use persistentDataPath for your external data.

The documentation on asset bundles can be annoying at first, but once you get the hang of it it isn't so bad.

You can't load Scenes and Prefabs externally unless you use asset bundles.

 

@astracat111 Did you try the new Addressable system yet? Apparently it's the new/easier way to deal with asset bundles (but it will still create asset bundles behind the scene if I remember correctly)

 

I did and I like it, but hadn't tested it with Android yet. I really like how they only have asynchronous loading, I think it's important right off the bat to start learning asynchronous code both using IEnumerators and Tasks.

 

Encountering the same issues described here with 2020.3.2f1 currently. ~1 GB sized project is currently building for 40 minutes and still not finished.

Sad to see this has been an ongoing issue for so long. Really hope Unity Team could give a statement on this.

 

I hardly doubt it. I'm on 2021 beta and I build my project about hour(and this is not too bad). When all shader are compiled, and if you don't change any shader after that, next bulds will be lot faster. 5-10 minutes.

 

Also running into some crazy shader variant numbers it after upgrading my project. Already aborted the build yesterday evening after about three+ hours. Restarted this morning.

 

primaerfunktion, have you switch to "strip unused" or "strip all" in the Graphics settings?

 

Yup it is. Followed all the tips people gave on here. Didn't do much. The second time the Build continued to for 5 more hours btw. The project is really not particularly big.

Edit: Looks like some the Shadergraph Shaders I did are very happy to create around a million variants? Even a complete freshly created Lit one. Yolo. Unlit is 113.

Just tested on a fresh Project. A just created shadergraph sits at about 170k variants.

 

My experience with 2020 LTS is that, folowing the advices here, I managed to lower the "1st" build from 60 mins to 15. Following builds take only 2 or 3 mins (simple scene) so everything is ok buuuut, if you do this, then the editor starts to be very laggy (like 1 min to save a scene for example). If i delete the shadercache folder, then the editor goes snappy again, but then, of course the next build is like the first one, again 15 mins to build. So, I lose time on one side, or the other, but the 2 things together don't work

 

Unity might as well remove the build progress bar at this point. It just sits at 99% for hours while it compiles shader variants.

I've tried all the tips in this thread and nothing seems to get the number of shader variants down. I guess we just accept that builds take longer now until Unity devs actually address this.

 

I saw on the Discord that improvements have been made but haven't been pulled into the main branch yet.

 

Some years ago (after first encountering this thread ), in Project Settings -> Graphics, I had changed my project's Shader Stripping settings (Lightmap Modes and Fog Modes) to custom values, and I left Instancing Variants as "Strip Unused".
It did give a nice boost on the builds with already cached variants at the time, but it still seemed a bit slower than it should, as every build seems to make the shader stripping process all over again; sometimes it randomly loses some cached variants and has to compile them again.

Now, I'm using Unity 2020.3.19f1 and URP 10.6.0.
Today, I set both Lightmap Modes and Fog Modes back to Automatic, and I had a HUGE boost on my cached variants builds. The parts where it would always take some time "Compiling Shader Variants" (actually stripping the variants), now they basically complete instantly.

 

Hi all!
Yesterday, I've tried to build my game for Android the first time:

I've implemented all suggested measures in this thread ("strip all", save shadervariants to asset, set asset to preloaded, no "optimize mesh data", no "gpu instancing", emptying shadercache) and migrated the project from Unity 2019.4 to Unity 2021.1, but I was't even able to get at least one build done the whole evening.
Now my question: My project uses the standard render pipeline. Is there a realistic chance, that this problem can be solved by migrating to URP? Or is there any way to reduce the amount of variants?

 

Write shader code instead, use shader features wisely, the problem would be eliminated

 

I had this previously, I added all my shaders to a empty disabled gameobject in my first scene, and then reset the Always inclduded shader settings to default. Problem solved for me went from 50+mins down to under 10.

 

@zhuchun:
Unity offers Shader Graph, but we rather code our own to avoid 3h build time? I'm not sure, if this answer satisfies me.
Using shader features wisely might be a good tip in general regarding performance, but what is wisely here? Most materials in my game have the standard shader with a normal map and a single color (no texture) as base map. Is this already too much?

 

I would argue that it's not a Unity thing, it's a Shader thing, the Material Editor of Unreal Engine has the same problem as well. The more keywords a shader used, the more variation is needed to be compiled, stripping itself is the process to help people get rid of variation they would never use. It's MUCH faster than compiling them.

Using graph tools is a great way to help artists approach those fancy effects, however, engines do NOT know how are people going to use it, so the way they solve this problem is by Template. Each time you drag a master node into a graph, you're choosing a template. In order to meet various needs, templates are needed to be flexible, which means MUCH more keywords than you actually need. As long as stripping is working, things should be fine.

So as you can see, it's not about how people could write shaders "wisely", I mean if you're writing a shader, no doubt you would only use keywords you actually need, right? The problem is that templates are using them "unwisely", Unity is optimizing them, that's cool, but I'm not going to be stuck here for any moment. In case you wanna know how "unwise" it is, IIRC, the early version of the PBR template would generate over 1k variations, for comparison, most well-tailored shaders only generate about 5~30.

In most game studios(well, I guess), we use graph tools to make effects, then, we will redo it again in code for performance-critical shaders. We also customize our own "Standard" shader to make sure that the stripping process is minimal.

I hope this helps.