Shader.parse lag when streaming asset bundles - preloading shaders does not work

Hello,

So I am building a dynamic streaming world in my game that uses asset bundles; at runtime, the bundles are streamed in and out dynamically as the player moves. After a lot of work, I have basically rewritten our entire game's code to use our own IStreamable API to avoid a lot of lag on instantiation. However, one thing seems to still be a problem - when rooms are streamed in, there are noticeable lag spikes as Shader.Parse is called

This seems weird to me because I am preloading the shader variants, as far as I can tell. At build time, all of the streamed assets are loaded into a scene with the Scene view open so that the shaders are all visible. I then create a shader variant asset and place it inside of the preload shaders section in the graphics settings, and then build the game which -should- contain all the shader variants that are inside the assets in the asset bundles



However, when profiling, I notice that I'm still getting a Shader.Parse lag spike for pretty much every asset the first time it is loaded, which makes the game unplayable. For instance, here it is lagging on Bakery/Standard shader


However as you can see here, the variants that the bundles use from Bakery/Standard -should- be inside the shader variant:



We are using a mix of unity standard shaders and custom shaders. This is on Unity 2018.2.20f1 on Windows 64-bit builds.

Is there some weird trick that I have to use to make this work correctly? I would be happy to prewarm all the shader variants that my bundles need in the load phase but I don't know if that will work, since the 'preload shaders' in unity itself isn't working

Asset bundles have been very frustrating to use because of all these undocumented issues that aren't present when you just use scenes. I've already had to write my own version of an addressable system just to make them feasible, and things like this are very frustrating to run into since there is no documentation. I had a similar issue with the new post processing and not knowing that we had to 'always include' the post processing shaders if we were using a camera from an asset bundle.

Thanks

 

So for anyone else who stumbles upon this thread with the same issue, I believe I have solved it

So first, go ahead and download this: https://github.com/Unity-Technologies/AssetBundles-Browser. Download the package directly and put it into your Assets folder, removed the -master from the foldername.

Create an asset bundle that contains the shaders and shader variants you want to use in other bundles. It doesn't matter if you add them in graphics settings or not. Because of how asset bundles work, shaders are packed directly into the bundle. What this means is that even though they are named the same, unity doesn't treat the shaders the same - each bundle will have its own separate shader that needs to be loaded onto the GPU and takes up memory.

This is where the shaders asset bundle comes in. If you add the shaders the other bundles depend on, then unity will NOT pack them separately and instead the other bundles will depend on the shaders bundle. The easiest way is to create a shader variant collection and add it to the shaders asset bundle, then use the browser to see what shaders are included in the shader variant collection. Then add those shaders to the asset bundle as well.

You will need both the shaders and shader variant collection bundled in order to make this work.



You will then need one final step, somewhere in your code you need to load, store, and prewarm the shaders before loading the other assets depending on those shaders.

Here is some example code (AssetBundleFileLoader is just a wrapper I made to async load all the assets given an asset bundle name):

Code (CSharp):

1. AssetBundleFileLoader shaderLoader = new AssetBundleFileLoader("shaders/environment");
2.       while (!shaderLoader.loaded) {
3.         yield return null;
4.       }
5.  
6.       // First load the shaders
7.       environmentShaders = new List<Shader>();
8.       for (int i = 0; i < shaderLoader.loadedAssets.Count; i++) {
9.         if (shaderLoader.loadedAssets[i] is Shader) {
10.           environmentShaders.Add(shaderLoader.loadedAssets[i] as Shader);
11.         }
12.       }
13.       // Now, warm up the shader variants containing them
14.       for (int i = 0; i < shaderLoader.loadedAssets.Count; i++) {
15.         if (shaderLoader.loadedAssets[i] is ShaderVariantCollection) {
16.           ShaderVariantCollection collection = shaderLoader.loadedAssets[i] as ShaderVariantCollection;
17.           collection.WarmUp();
18.         }
19.       }

You want to store the shaders, and warm up all the shader variants.

Oh, one final fun note. This doesn't work for any of unity's built-in shaders (because why would it? That would make too much sense). You'll need to basically create a duplicate of the standard shader that you can put in your Assets folder, and then use that on all the materials that currently use the Standard shader so that you can include it in your shaders bundle.

Really hope that the addressable system does a better job of providing documentation for important things like this, because anyone trying to stream assets without lag is going to run into this if their assets are of any real size and they try to stream them realtime without a loading screen.

 

Awesome, I'm glad to hear that - that would go a long way towards solving these problems! Looking forward to trying out addressables when it's a bit more stable. Thanks for your response!

 

Hey @Ryanc_unity, what would be the best way for me to make a feature request to move Shader.parse to a background thread? It would make streaming worlds a lot faster, and I think is an important feature - is it something that could be easily moved to the job system?

 

@Ryanc_unity I am currently running into large spikes in Shader.Parse on console port in 2017.4.17f1, do you know if there is a better version of Unity that has a better handle on this shader parsing issue?

 

I am using 100% addressable system, Currently running into this issue where no matter how I warm up the shader using the original method, all shader will warm up again on load, took me two days trial and error finally found this thread! thanks for sharing.