Search Unity

Shader.parse lag when streaming asset bundles - preloading shaders does not work

Discussion in 'Asset Bundles' started by zornor90, Dec 22, 2018.

  1. zornor90

    zornor90

    Joined:
    Sep 16, 2015
    Posts:
    126
    Hello,

    So I am building a dynamic streaming world in my game that uses asset bundles; at runtime, the bundles are streamed in and out dynamically as the player moves. After a lot of work, I have basically rewritten our entire game's code to use our own IStreamable API to avoid a lot of lag on instantiation. However, one thing seems to still be a problem - when rooms are streamed in, there are noticeable lag spikes as Shader.Parse is called

    This seems weird to me because I am preloading the shader variants, as far as I can tell. At build time, all of the streamed assets are loaded into a scene with the Scene view open so that the shaders are all visible. I then create a shader variant asset and place it inside of the preload shaders section in the graphics settings, and then build the game which -should- contain all the shader variants that are inside the assets in the asset bundles



    However, when profiling, I notice that I'm still getting a Shader.Parse lag spike for pretty much every asset the first time it is loaded, which makes the game unplayable. For instance, here it is lagging on Bakery/Standard shader


    However as you can see here, the variants that the bundles use from Bakery/Standard -should- be inside the shader variant:



    We are using a mix of unity standard shaders and custom shaders. This is on Unity 2018.2.20f1 on Windows 64-bit builds.

    Is there some weird trick that I have to use to make this work correctly? I would be happy to prewarm all the shader variants that my bundles need in the load phase but I don't know if that will work, since the 'preload shaders' in unity itself isn't working

    Asset bundles have been very frustrating to use because of all these undocumented issues that aren't present when you just use scenes. I've already had to write my own version of an addressable system just to make them feasible, and things like this are very frustrating to run into since there is no documentation. I had a similar issue with the new post processing and not knowing that we had to 'always include' the post processing shaders if we were using a camera from an asset bundle.

    Thanks
     
    Last edited: Dec 22, 2018
  2. zornor90

    zornor90

    Joined:
    Sep 16, 2015
    Posts:
    126
    So for anyone else who stumbles upon this thread with the same issue, I believe I have solved it

    So first, go ahead and download this: https://github.com/Unity-Technologies/AssetBundles-Browser. Download the package directly and put it into your Assets folder, removed the -master from the foldername.

    Create an asset bundle that contains the shaders and shader variants you want to use in other bundles. It doesn't matter if you add them in graphics settings or not. Because of how asset bundles work, shaders are packed directly into the bundle. What this means is that even though they are named the same, unity doesn't treat the shaders the same - each bundle will have its own separate shader that needs to be loaded onto the GPU and takes up memory.

    This is where the shaders asset bundle comes in. If you add the shaders the other bundles depend on, then unity will NOT pack them separately and instead the other bundles will depend on the shaders bundle. The easiest way is to create a shader variant collection and add it to the shaders asset bundle, then use the browser to see what shaders are included in the shader variant collection. Then add those shaders to the asset bundle as well.

    You will need both the shaders and shader variant collection bundled in order to make this work.



    You will then need one final step, somewhere in your code you need to load, store, and prewarm the shaders before loading the other assets depending on those shaders.

    Here is some example code (AssetBundleFileLoader is just a wrapper I made to async load all the assets given an asset bundle name):

    Code (CSharp):
    1. AssetBundleFileLoader shaderLoader = new AssetBundleFileLoader("shaders/environment");
    2.       while (!shaderLoader.loaded) {
    3.         yield return null;
    4.       }
    5.  
    6.       // First load the shaders
    7.       environmentShaders = new List<Shader>();
    8.       for (int i = 0; i < shaderLoader.loadedAssets.Count; i++) {
    9.         if (shaderLoader.loadedAssets[i] is Shader) {
    10.           environmentShaders.Add(shaderLoader.loadedAssets[i] as Shader);
    11.         }
    12.       }
    13.       // Now, warm up the shader variants containing them
    14.       for (int i = 0; i < shaderLoader.loadedAssets.Count; i++) {
    15.         if (shaderLoader.loadedAssets[i] is ShaderVariantCollection) {
    16.           ShaderVariantCollection collection = shaderLoader.loadedAssets[i] as ShaderVariantCollection;
    17.           collection.WarmUp();
    18.         }
    19.       }
    You want to store the shaders, and warm up all the shader variants.

    Oh, one final fun note. This doesn't work for any of unity's built-in shaders (because why would it? That would make too much sense). You'll need to basically create a duplicate of the standard shader that you can put in your Assets folder, and then use that on all the materials that currently use the Standard shader so that you can include it in your shaders bundle.

    Really hope that the addressable system does a better job of providing documentation for important things like this, because anyone trying to stream assets without lag is going to run into this if their assets are of any real size and they try to stream them realtime without a loading screen.
     
    Last edited: Dec 29, 2018
  3. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    165
    Addressables uses the Scriptable Build Pipeline and extracts all built-in shaders into a common asset bundle to reduce duplication, and by extension reduce this parsing. More long term work would be to move the Shader.Parse to background threads so it doesn't stall the rendering or main thread.
     
    zornor90 likes this.
  4. zornor90

    zornor90

    Joined:
    Sep 16, 2015
    Posts:
    126
    Awesome, I'm glad to hear that - that would go a long way towards solving these problems! Looking forward to trying out addressables when it's a bit more stable. Thanks for your response!