Search Unity

  1. Unity 2020.1 has been released.
    Dismiss Notice
  2. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Standard Shader duplicated in Asset Bundle Build

Discussion in 'Asset Bundles' started by FallenTreeGames, Dec 5, 2018.

  1. FallenTreeGames

    FallenTreeGames

    Joined:
    Jun 22, 2016
    Posts:
    25
    Hey,

    I've been investigating why our asset bundle builds use more memory than our non-bundle builds, and have narrowed it down to the shaders.

    The detailed memory view shows us that on an asset bundle build, ShaderLab takes up around 350mb, whilst on a non-asset bundle build it uses < 80mb.

    So I dumped out the list of all loaded shaders with Resources.FindObjectsOfTypeAll<Shader>() and the result is that the asset bundle build has numerous copies of some shaders, including 27 copies of the Standard shader, and 31 copies of the Standard (Specular setup) one. (The non-asset bundle build has only one copy of each).

    How do I fix this? Some of the shaders that are duplicated are custom shaders which are not being included in the correct asset bundle (so fair enough) but how do I fix this for the standard shader?
     
  2. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    272
    For built-in shaders, our current recommendation if you are using the BuildPipeline.BuildAssetBundles API is to download the built-in shaders from https://unity3d.com/get-unity/download/archive (click the drop down array next to "Downloads (Win)" or "Downloads (Mac)" to get the "Built in shaders") and copy the ones you want to de-duplicate into your project, renaming the shader internal name to something like "YourProject/Standard". From there you can then assign it to a specific bundle and it will no longer be duplicated. You will need to update all your assets to point to this shader copy instead of the original shader, this can be done with an AssetPostprocessor (https://docs.unity3d.com/ScriptReference/AssetPostprocessor.html) and reimport of your assets.

    In the Scriptable Build Pipeline, we have provided some simple code that will de-duplicate built-in shaders automatically as an example.
     
  3. FallenTreeGames

    FallenTreeGames

    Joined:
    Jun 22, 2016
    Posts:
    25
    Hey,

    Thanks for that. It was the solution I was thinking of, though I assumed it was more of a hack.

    Is the Scriptable Build Pipeline ready for production use?
     
  4. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    272
    Disclaimer: It's still in Preview, so we do not recommend it for production use.

    That being said, if you are not using Asset Bundle Variants and using full Asset Paths to load, you can try out SBP with only a single line code change and switch back if you are having issues. So far, most issues we've run into have been really simple fixes for us to track down and push out a change to fix.
     
  5. FallenTreeGames

    FallenTreeGames

    Joined:
    Jun 22, 2016
    Posts:
    25
    Okay thanks! I'll get sorting this at some point and see if I have any issues :)
     
  6. FallenTreeGames

    FallenTreeGames

    Joined:
    Jun 22, 2016
    Posts:
    25
    Hey,

    So I tried it the first way and we're still getting some duplication, think there's still some things using the default material, but it's a large project so tracking them down is hard.

    Since we're not (currently) using variants and only load scenes from the bundles, I was going to give the SBP a shot. Can you link me to the sample code that de-duplicates the shaders? @Ryanc_unity
     
    Last edited: Mar 6, 2019
  7. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    272
    Code (CSharp):
    1. public static class BuildAssetBundlesExample
    2. {
    3.    public static bool BuildAssetBundles(string outputPath, bool useChunkBasedCompression, BuildTarget buildTarget, BuildTargetGroup buildGroup)
    4.    {
    5.        var buildContent = new BundleBuildContent(ContentBuildInterface.GenerateAssetBundleBuilds());
    6.        var buildParams = new BundleBuildParameters(buildTarget, buildGroup, outputPath);
    7.  
    8.        if (useChunkBasedCompression)
    9.            buildParams.BundleCompression = BuildCompression.DefaultLZ4;
    10.  
    11.        IBundleBuildResults results;
    12.         var tasks = DefaultBuildTasks.Create(DefaultBuildTasks.Preset.AssetBundleBuiltInShaderExtraction);
    13.        ReturnCode exitCode = ContentPipeline.BuildAssetBundles(buildParams, buildContent, out results, tasks);
    14.        return exitCode == ReturnCode.Success;
    15.    }
    16. }
    The key is that this uses the default task list that contains the extra tasks needed for built in shader extraction. If you look at that task source, it will give you an idea on how to make it more generic to apply to more than just built-in shaders if you want to go de-dupe crazy.
     
  8. FallenTreeGames

    FallenTreeGames

    Joined:
    Jun 22, 2016
    Posts:
    25
  9. LightStriker

    LightStriker

    Joined:
    Aug 3, 2013
    Posts:
    2,577
    Tell me there's a way to prevent this from occuring with ALL shaders.
    We simply cannot track all the shaders used in all our assets. We have references towards particles. Having two version of ALL standards shaders in our project is also a huge mess.

    Bundling all our shaders are ALSO not a solution. Tell me there's a way to tell the AssetBundle pipeline to ignore specific asset or even specific type.

    On the same idea, it also duplicate audio asset! It's just insane! I cannot end up putting the whole freaking game in thousand of asset bundle just to prevent everything from duplicating everything else!
     
    Last edited: Sep 11, 2019
    Luke-Lamothe and tenderclawsJK like this.
  10. pdinklag

    pdinklag

    Joined:
    Jan 24, 2017
    Posts:
    76
    Aren't standard shaders always included in the game build anyway? Why are they even duplicated to begin with, why not simply store a reference in the AssetBundle?

    I need some clarification about this: is this doing exactly what I described - replace duplicated shaders by references?

    In any event, I share @LightStriker 's concerns. AssetBundles should not duplicate anything that's contained in the game build, and definitely nothing that Unity ships with (e.g., standard shaders).
     
    Luke-Lamothe likes this.
  11. LightStriker

    LightStriker

    Joined:
    Aug 3, 2013
    Posts:
    2,577
    Ok, this is going beyond my patience.

    We managed to find a way to reroute all the material towards shader downloaded from the Unity website.

    However, once packaged and loaded in game, they ALL are missing Keywords. A standard shader without the _NORMALMAP _METALLICGLOSSMAP and _EMISSION Keywork is very ugly.

    I'm lacking solution here. 1.3Gb Resources folder, or the game simply doesn't work or look like S***.

    [EDIT] Found out the keyword issue was related to the shader being unload (somehow), but still being present on the GPU. Look story short, we figured it out. It's still a mess, but at least it's working.
     
    Last edited: Sep 12, 2019
    Luke-Lamothe likes this.
  12. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    272
    Standard shaders are not always included in the game build as they are pulled in to the first scene that references them and with features enabled based on what was calculated as necessary for the game build.

    The code snippet I provided uses the Scriptable Build Pipeline package to find all built-in shaders that are being referenced in any bundle. Then remaps the build location of those shaders all to a common shared asset bundle. This is similar to what happens when you have loose shaders in your project that you then assign to a common shader bundle, however since built-in shaders are not loose assets that can have bundle meta data associated with it, I had to reassign them during the build process. Thus the extra code.

    The problem with this idea is that you are then tying your game build and your asset bundle build together. So any time you change your game build, you need to rebuild your bundles as the location of where data is located in the game build could be changed.

    In addition, this won't produce correct results when it comes to Shader & Mesh optimization. Shaders compile only the subset of features needed which is calculated based upon the lighting configuration in each scene as well as which materials and their settings that reference that shader. Similarly with meshes which optimize which channels to keep based on the compiled shader features. So this means that any new material settings in bundles would also cause the game build to need to rebuild.
     
  13. Zapan15

    Zapan15

    Joined:
    Apr 11, 2011
    Posts:
    88
    Is there any other way (besides the above mentioned way with downloading the source of buildtin shaders), to use the old BuildPipeline.BuildAssetBundles method and not having the duplication of buildin shaders in each assetbundle? We are developing a plugin and do not want to force our users to use the package manager to install the Scriptable-Buildpipline.

    Using DefaultBuildTasks.Preset.AssetBundleBuiltInShaderExtraction feels also a bit like a hack, as it looks like that the references to the builtin shaders are just removed afterwarts, as the slots in the assetbundle are just set to null. Maybe we can do this also with the old BuildPipeline.BuildAssetBundles?
     

    Attached Files:

  14. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    272
    @Zapan15
    In your provided screen shot, they appear null as they were moved to a different bundle that is not currently loaded in the AssetBundleBrowser when you are displaying the contents of another bundle. It's one of the downsides to that inspect view that it will only load one bundle, so external references appear null even if they are actually valid.

    As for a method using BuildPipeline.BuildAssetBundles, the only other method is really bad and we highly don't recommend it. You can assign built in shaders in the Graphics Settings Always Included shaders list. When a built in shader is assigned to this list, an Asset bundle will reference the shader from player data instead pulling it into the asset bundle. The catch: Shaders in this list build ALL FEATURES! So your project bloat will be huge.
     
  15. Zapan15

    Zapan15

    Joined:
    Apr 11, 2011
    Posts:
    88
    Thank you for the detailed Feedback!

    However, is there any other way of handling this issue, maybe in the future besides DefaultBuildTasks.Preset.AssetBundleBuiltInShaderExtraction?
     
  16. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    272
    Not sure I understand the aversion to using Scriptable Build Pipeline as that is the future. It was designed with the goal of being able to be very explicit on where and how data is built. The extra tasks provided by AssetBundleBuiltInShaderExtraction are designed to show off what is possible by solving a problem users have today.

    Can you provide some additional details as to why you don't want users using SBP?
     
  17. Zapan15

    Zapan15

    Joined:
    Apr 11, 2011
    Posts:
    88
    Sure: The Problem is that we do not want that our plugin has a dependency to other plugins and/or packages like the SRP. This would mean, if a user installs our plugin we, he needs to install this package, too. We want to publish our plugin to the Assetstore, therefor we are using the legancy assetPackages.

    However, maybe you know a way to automatically install dependecy package, maybe this would be a workaround, so that we can use the SRP in our plugin?

    Thank you :)
     
  18. Ryanc_unity

    Ryanc_unity

    Unity Technologies

    Joined:
    Jul 22, 2015
    Posts:
    272
    Ah, I know the Assetstore will eventually move to the same infrastructure for deploying asset packages as packman. Until that point though I'm not sure. I'll need to ask the asset store team and package manager team about possible workarounds.
     
  19. KBaxtrom-LevelEx

    KBaxtrom-LevelEx

    Joined:
    Feb 5, 2018
    Posts:
    23
    What version of unity was this happening on? We're updating from 2017.3 to 2019.2 and seeing similar issues

    Also, how are we supposed to use this with the normal BuildPipeline.BuildPlayer() funtionality. The method you have provided does not create a manifest file, but the BuildPipeline.BuildPlayer will strip unused files out if a manifest is not provided. The CompatibilityBuildPipeline path returns a manifest, but does not appear to allow us to strip the built-in shaders
     
    Last edited: Sep 30, 2019
  20. LightStriker

    LightStriker

    Joined:
    Aug 3, 2013
    Posts:
    2,577
    Would it be possible to flag objects as always being part of the build, so they could be ignored when making AssetBundles?

    We already have a "Always Included Shaders" list in the player settings.
     
  21. KBaxtrom-LevelEx

    KBaxtrom-LevelEx

    Joined:
    Feb 5, 2018
    Posts:
    23
    Bumping thread any answer?
     
  22. LightStriker

    LightStriker

    Joined:
    Aug 3, 2013
    Posts:
    2,577
    After way more working with AssetBundles, I came to the following conclusions:

    1) Two pipeline (scripted and build): It's dumb; they don't talk to each others and there's no way to know if one file is in one pipeline or not, or both.
    2) Black boxes: No way to know what was actually bundled with a file, no output. The references can be manually found, but not once it's in a bundle. We found it often make for bigger bundles then what would be needed, and Unity tends to package stuff we aren't sure if they are really needed.
    3) Duplication of files: Not only shader, but audio mixer, texture, sprite, sprite atlas, animation, animator, etc. Pretty much anything that is reused between bundles get duplicated. It's a massive headache. You have to manually track down those references and manually put them in separated bundles to prevent duplication. (but still duplicate when you build)
    4) Load by Type: Similar to the Ressources issue, loading all object by types pretty much makes Unity load ALL objects in the bundle, look at their type, then discard the wrong ones. This is dumb. The headers should contain the information required to prevent this from occurring. Name, Assembly Qualified Name, GUIDs, whatever!

    Unless flagged "Include in Build", Sprite Atlases do NOT work in AssetBundles. At all. But still get duplicated!
     
  23. Zapan15

    Zapan15

    Joined:
    Apr 11, 2011
    Posts:
    88
    @LightStriker
    Thank you for the detailed feedback.

    @Unity
    Can you/someone give us some insights about the things mentioned above?
     
  24. capyvara

    capyvara

    Joined:
    Mar 11, 2010
    Posts:
    79
    @Ryanc_unity
    How do we avoid custom shaders from being duplicated in the player and into an asset bundle? If we do not put the shader in the Always included shaders it doesn't work correctly with the asset bundle, and if we do put the shader there we get a shader duplication (one in the asset bundle, other in the player).

    Thanks!
     
  25. LightStriker

    LightStriker

    Joined:
    Aug 3, 2013
    Posts:
    2,577
    5) In the AssetBundle pipeline, you cannot reference an asset, you pass a path. If for some reason this path leads to multiple asset (say a FBX with animation and a mesh), everything is pulled, even if you only 1 one asset out of many.

    I'm adding this because I have animation duplication between weapons bundle. I tried to bundle all the animation, and found out I pulled 200Mb of unwanted mesh. Great.
     
  26. LightStriker

    LightStriker

    Joined:
    Aug 3, 2013
    Posts:
    2,577
    6) Sprite Atlas are duplicated. Always. Unless you script something. I'm not joking. SpriteAtlas don't work out of the box. Took us a LONG time to figure it out why we always had a duplicate of all our SpriteAtlas, and if we unchecked "Include in Build", all our sprites would fail. But unchecking it is actually the way to go if you have anything in an AssetBundle referencing a SpriteAtlas. For some reason, something in a bundle is to dumb to figure out the atlas is packed with the game.

    Code (CSharp):
    1.         static AssetBundleManager()
    2.         {
    3.             SpriteAtlasManager.atlasRequested += LoadSpriteAtlas;
    4.         }
    5.  
    6.         private static void LoadSpriteAtlas(string name, Action<SpriteAtlas> callback)
    7.         {
    8.             AssetBundle assetBundle = LoadAssetBundle<SpriteAtlas>();
    9.             callback(assetBundle.LoadAsset<SpriteAtlas>(name));
    10.         }
    That's dumb. I hate Asset Bundles.
     
    oddwhocanfly and stonstad like this.
  27. unity_tGAeeJrY-D2ioQ

    unity_tGAeeJrY-D2ioQ

    Joined:
    Mar 6, 2020
    Posts:
    4
    upload_2020-7-17_14-41-21.png
    Well It is the question I want to ask ,I just rewrite some shader,and what I find is use assetbundle ,in memory profiler I can see more the one reference , and more memory
     
unityunity