Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

Standard Shader duplicated in Asset Bundle Build

Discussion in 'Asset Bundles' started by FallenTreeGames, Dec 5, 2018.

  1. FallenTreeGames

    FallenTreeGames

    Joined:
    Jun 22, 2016
    Posts:
    25
    Hey,

    I've been investigating why our asset bundle builds use more memory than our non-bundle builds, and have narrowed it down to the shaders.

    The detailed memory view shows us that on an asset bundle build, ShaderLab takes up around 350mb, whilst on a non-asset bundle build it uses < 80mb.

    So I dumped out the list of all loaded shaders with Resources.FindObjectsOfTypeAll<Shader>() and the result is that the asset bundle build has numerous copies of some shaders, including 27 copies of the Standard shader, and 31 copies of the Standard (Specular setup) one. (The non-asset bundle build has only one copy of each).

    How do I fix this? Some of the shaders that are duplicated are custom shaders which are not being included in the correct asset bundle (so fair enough) but how do I fix this for the standard shader?
     
  2. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    245
    For built-in shaders, our current recommendation if you are using the BuildPipeline.BuildAssetBundles API is to download the built-in shaders from https://unity3d.com/get-unity/download/archive (click the drop down array next to "Downloads (Win)" or "Downloads (Mac)" to get the "Built in shaders") and copy the ones you want to de-duplicate into your project, renaming the shader internal name to something like "YourProject/Standard". From there you can then assign it to a specific bundle and it will no longer be duplicated. You will need to update all your assets to point to this shader copy instead of the original shader, this can be done with an AssetPostprocessor (https://docs.unity3d.com/ScriptReference/AssetPostprocessor.html) and reimport of your assets.

    In the Scriptable Build Pipeline, we have provided some simple code that will de-duplicate built-in shaders automatically as an example.
     
  3. FallenTreeGames

    FallenTreeGames

    Joined:
    Jun 22, 2016
    Posts:
    25
    Hey,

    Thanks for that. It was the solution I was thinking of, though I assumed it was more of a hack.

    Is the Scriptable Build Pipeline ready for production use?
     
  4. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    245
    Disclaimer: It's still in Preview, so we do not recommend it for production use.

    That being said, if you are not using Asset Bundle Variants and using full Asset Paths to load, you can try out SBP with only a single line code change and switch back if you are having issues. So far, most issues we've run into have been really simple fixes for us to track down and push out a change to fix.
     
  5. FallenTreeGames

    FallenTreeGames

    Joined:
    Jun 22, 2016
    Posts:
    25
    Okay thanks! I'll get sorting this at some point and see if I have any issues :)
     
  6. FallenTreeGames

    FallenTreeGames

    Joined:
    Jun 22, 2016
    Posts:
    25
    Hey,

    So I tried it the first way and we're still getting some duplication, think there's still some things using the default material, but it's a large project so tracking them down is hard.

    Since we're not (currently) using variants and only load scenes from the bundles, I was going to give the SBP a shot. Can you link me to the sample code that de-duplicates the shaders? @Ryanc_unity
     
    Last edited: Mar 6, 2019
  7. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    245
    Code (CSharp):
    1. public static class BuildAssetBundlesExample
    2. {
    3.    public static bool BuildAssetBundles(string outputPath, bool useChunkBasedCompression, BuildTarget buildTarget, BuildTargetGroup buildGroup)
    4.    {
    5.        var buildContent = new BundleBuildContent(ContentBuildInterface.GenerateAssetBundleBuilds());
    6.        var buildParams = new BundleBuildParameters(buildTarget, buildGroup, outputPath);
    7.  
    8.        if (useChunkBasedCompression)
    9.            buildParams.BundleCompression = BuildCompression.DefaultLZ4;
    10.  
    11.        IBundleBuildResults results;
    12.         var tasks = DefaultBuildTasks.Create(DefaultBuildTasks.Preset.AssetBundleBuiltInShaderExtraction);
    13.        ReturnCode exitCode = ContentPipeline.BuildAssetBundles(buildParams, buildContent, out results, tasks);
    14.        return exitCode == ReturnCode.Success;
    15.    }
    16. }
    The key is that this uses the default task list that contains the extra tasks needed for built in shader extraction. If you look at that task source, it will give you an idea on how to make it more generic to apply to more than just built-in shaders if you want to go de-dupe crazy.
     
  8. FallenTreeGames

    FallenTreeGames

    Joined:
    Jun 22, 2016
    Posts:
    25
  9. LightStriker

    LightStriker

    Joined:
    Aug 3, 2013
    Posts:
    2,486
    Tell me there's a way to prevent this from occuring with ALL shaders.
    We simply cannot track all the shaders used in all our assets. We have references towards particles. Having two version of ALL standards shaders in our project is also a huge mess.

    Bundling all our shaders are ALSO not a solution. Tell me there's a way to tell the AssetBundle pipeline to ignore specific asset or even specific type.

    On the same idea, it also duplicate audio asset! It's just insane! I cannot end up putting the whole freaking game in thousand of asset bundle just to prevent everything from duplicating everything else!
     
    Last edited: Sep 11, 2019
    tenderclawsJK likes this.
  10. pdinklag

    pdinklag

    Joined:
    Jan 24, 2017
    Posts:
    40
    Aren't standard shaders always included in the game build anyway? Why are they even duplicated to begin with, why not simply store a reference in the AssetBundle?

    I need some clarification about this: is this doing exactly what I described - replace duplicated shaders by references?

    In any event, I share @LightStriker 's concerns. AssetBundles should not duplicate anything that's contained in the game build, and definitely nothing that Unity ships with (e.g., standard shaders).
     
  11. LightStriker

    LightStriker

    Joined:
    Aug 3, 2013
    Posts:
    2,486
    Ok, this is going beyond my patience.

    We managed to find a way to reroute all the material towards shader downloaded from the Unity website.

    However, once packaged and loaded in game, they ALL are missing Keywords. A standard shader without the _NORMALMAP _METALLICGLOSSMAP and _EMISSION Keywork is very ugly.

    I'm lacking solution here. 1.3Gb Resources folder, or the game simply doesn't work or look like S***.

    [EDIT] Found out the keyword issue was related to the shader being unload (somehow), but still being present on the GPU. Look story short, we figured it out. It's still a mess, but at least it's working.
     
    Last edited: Sep 12, 2019
  12. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    245
    Standard shaders are not always included in the game build as they are pulled in to the first scene that references them and with features enabled based on what was calculated as necessary for the game build.

    The code snippet I provided uses the Scriptable Build Pipeline package to find all built-in shaders that are being referenced in any bundle. Then remaps the build location of those shaders all to a common shared asset bundle. This is similar to what happens when you have loose shaders in your project that you then assign to a common shader bundle, however since built-in shaders are not loose assets that can have bundle meta data associated with it, I had to reassign them during the build process. Thus the extra code.

    The problem with this idea is that you are then tying your game build and your asset bundle build together. So any time you change your game build, you need to rebuild your bundles as the location of where data is located in the game build could be changed.

    In addition, this won't produce correct results when it comes to Shader & Mesh optimization. Shaders compile only the subset of features needed which is calculated based upon the lighting configuration in each scene as well as which materials and their settings that reference that shader. Similarly with meshes which optimize which channels to keep based on the compiled shader features. So this means that any new material settings in bundles would also cause the game build to need to rebuild.
     
  13. Zapan15

    Zapan15

    Joined:
    Apr 11, 2011
    Posts:
    60
    Is there any other way (besides the above mentioned way with downloading the source of buildtin shaders), to use the old BuildPipeline.BuildAssetBundles method and not having the duplication of buildin shaders in each assetbundle? We are developing a plugin and do not want to force our users to use the package manager to install the Scriptable-Buildpipline.

    Using DefaultBuildTasks.Preset.AssetBundleBuiltInShaderExtraction feels also a bit like a hack, as it looks like that the references to the builtin shaders are just removed afterwarts, as the slots in the assetbundle are just set to null. Maybe we can do this also with the old BuildPipeline.BuildAssetBundles?
     

    Attached Files:

  14. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    245
    @Zapan15
    In your provided screen shot, they appear null as they were moved to a different bundle that is not currently loaded in the AssetBundleBrowser when you are displaying the contents of another bundle. It's one of the downsides to that inspect view that it will only load one bundle, so external references appear null even if they are actually valid.

    As for a method using BuildPipeline.BuildAssetBundles, the only other method is really bad and we highly don't recommend it. You can assign built in shaders in the Graphics Settings Always Included shaders list. When a built in shader is assigned to this list, an Asset bundle will reference the shader from player data instead pulling it into the asset bundle. The catch: Shaders in this list build ALL FEATURES! So your project bloat will be huge.
     
  15. Zapan15

    Zapan15

    Joined:
    Apr 11, 2011
    Posts:
    60
    Thank you for the detailed Feedback!

    However, is there any other way of handling this issue, maybe in the future besides DefaultBuildTasks.Preset.AssetBundleBuiltInShaderExtraction?
     
  16. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    245
    Not sure I understand the aversion to using Scriptable Build Pipeline as that is the future. It was designed with the goal of being able to be very explicit on where and how data is built. The extra tasks provided by AssetBundleBuiltInShaderExtraction are designed to show off what is possible by solving a problem users have today.

    Can you provide some additional details as to why you don't want users using SBP?
     
  17. Zapan15

    Zapan15

    Joined:
    Apr 11, 2011
    Posts:
    60
    Sure: The Problem is that we do not want that our plugin has a dependency to other plugins and/or packages like the SRP. This would mean, if a user installs our plugin we, he needs to install this package, too. We want to publish our plugin to the Assetstore, therefor we are using the legancy assetPackages.

    However, maybe you know a way to automatically install dependecy package, maybe this would be a workaround, so that we can use the SRP in our plugin?

    Thank you :)
     
  18. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    245
    Ah, I know the Assetstore will eventually move to the same infrastructure for deploying asset packages as packman. Until that point though I'm not sure. I'll need to ask the asset store team and package manager team about possible workarounds.
     
  19. KBaxtrom-LevelEx

    KBaxtrom-LevelEx

    Joined:
    Feb 5, 2018
    Posts:
    8
    What version of unity was this happening on? We're updating from 2017.3 to 2019.2 and seeing similar issues

    Also, how are we supposed to use this with the normal BuildPipeline.BuildPlayer() funtionality. The method you have provided does not create a manifest file, but the BuildPipeline.BuildPlayer will strip unused files out if a manifest is not provided. The CompatibilityBuildPipeline path returns a manifest, but does not appear to allow us to strip the built-in shaders
     
    Last edited: Sep 30, 2019
  20. LightStriker

    LightStriker

    Joined:
    Aug 3, 2013
    Posts:
    2,486
    Would it be possible to flag objects as always being part of the build, so they could be ignored when making AssetBundles?

    We already have a "Always Included Shaders" list in the player settings.
     
  21. KBaxtrom-LevelEx

    KBaxtrom-LevelEx

    Joined:
    Feb 5, 2018
    Posts:
    8
    Bumping thread any answer?