Search Unity

  1. Unity 2020.2 has been released.
    Dismiss Notice
  2. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Bug: 4GB limit to Textures in standalone build

Discussion in 'General Graphics' started by ShilohGames, Nov 14, 2016.

  1. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    4,455
    That's very strange that it's crashing. If you could provide us with some crash dumps or logs that would be great. Be nice not to have it crash for you ;)
     
    radiantboy likes this.
  2. dienat

    dienat

    Joined:
    May 27, 2016
    Posts:
    407
    I have the same problem. All game is pink if build is too big for Unity. Actually is not big, i just added a normal scene for standalone, i realized if i removed some parts it worked.
     
    Last edited: Dec 22, 2019
    karl_jones likes this.
  3. dienat

    dienat

    Joined:
    May 27, 2016
    Posts:
    407
    YOu can see it in the build log in C:\Users\nameofyourusername\AppData\LocalLow\Nameofproject\Nameofbuild
     
  4. Crossway

    Crossway

    Joined:
    May 24, 2016
    Posts:
    427
    I get this error.

    sharedassets0.assets'. Serialized files over 4.00 GB

    What should I do? I didn't even created 10% of my game world!!!
     
  5. dienat

    dienat

    Joined:
    May 27, 2016
    Posts:
    407
    I think your scene uses more than 4gb, split it in several scenes. Also for some reason i had also problems with a certain scene and when i moved all the gameobjects to other scene build didnt have any problem. In my case i had all pink and i solved it in that way
     
    Crossway likes this.
  6. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    1,116
    Yea , thanks I worked around this issue.. Streaming solves it pretty good.
     
  7. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    4,455
    You could try the 2020.1 update when it's available, which should solve the issue or re architect your project into multiple scenes or asset bundles.
     
    Crossway likes this.
  8. Crossway

    Crossway

    Joined:
    May 24, 2016
    Posts:
    427
    How can I keep the links between objects if I create multiple scenes?
     
  9. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,474
    Unity Technologies gave a talk about this a while ago:

    Here are the files he was referring to in the talk:
    https://github.com/Unity-Technologies/guid-based-reference
     
    Crossway and karl_jones like this.
  10. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,141
    How do you do your streaming? id like to try if it doesnt involve too much work :)
     
  11. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    4,455
    Look at Asset bundles/Addressables. They can load asynchronously.
     
  12. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,141
    ah yes, to be honest this solution doesnt work for me because its too manual to set up, if it can be automated it would be good, but there is no way i can mark up everything, I have too much in this scene. Ideally there would be a way to auto analyse a scene and generate bundles of graphics, sounds etc for that particular scene. I may write an asset for that sometime next year. For now im just happy with the 4 gig fix so I can soldier on with current resources system, but obviously this will become an issue one day when I need to release a small fix/patch. Thanks.
     
    AcidArrow and Lars-Steenhoff like this.
  13. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    1,116
    If you use steam to sell your game it wont be an issue..
     
  14. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    1,116
    Theres no real tutorial about it.. and streaming does the same.
     
  15. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,141
    why not? What if I fix 1 small bug, wouldnt they need to get the entire game again ?
     
  16. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    1,116
    No, it will only download changed files.
     
    radiantboy likes this.
  17. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,141
    nice, i need to read up on how that works..
     
  18. stonstad

    stonstad

    Joined:
    Jan 19, 2018
    Posts:
    234
    If you have a large Unity project you can tell the bug reporter to *not* include your Unity project in the submission. i.e. I have several gigabytes of assets and I don't want to even try to send that. The Unity team will still get your editor log, crash dump, and versioning in your bug report, which is all useful.
     
    karl_jones likes this.
  19. seansteezy

    seansteezy

    Joined:
    Nov 28, 2013
    Posts:
    115
    Thanks! For anyone else experiencing this issue, on older versions, I did the multiple scene thing and it worked pretty easily. Working with 16gb+ point cloud data files, had to split the assets into 5 or so files of 10 million per scene and then load them async/additive at runtime, grab the assets through a simple script, and combine them. Now the hard part is making it perform fast ;)
     
    PhoenixAdvanced likes this.
  20. cloverme

    cloverme

    Joined:
    Apr 6, 2018
    Posts:
    66
    I can't quite visualize the setup of the original solution, I've been reading the post a few times but it's not quite clicking in my head. Could you maybe provide some more details on the setup?

    Update: I figured it out, it works amazingly well for a solution... prior to when 2020.1.0a19 is released.
     
    Last edited: Jan 16, 2020
  21. Bigpete591

    Bigpete591

    Joined:
    May 7, 2013
    Posts:
    59
    Can anyone confirm this is fixed in 2020.1?
     
  22. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    4,455
    Bigpete591 and richardkettlewell like this.
  23. Zeurin

    Zeurin

    Joined:
    Sep 27, 2018
    Posts:
    5
    First come first, Thanks you for the work on this topic.
    Second, is there a comparative between the loading speed of an asset bundle loaded from assets/bundle folder and the same weight of assets in direct reference way? I am working right now in a project which has Real location photos, text....and I am wondering if it would be a good practice to approach this from an Asset Bundle perspective.

    Thanks.
     
  24. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    4,455
    Its hard to compare. Generally:
    • scene in the build with direct references. data directly referenced loads quickly. has no asset bundle overhead. but ALL data directly referenced must load when scene loads.
    • scene in build, with addressable references. data loading speed may be slightly slower if data is all in one bundle. If data is spread out, it could be much slower. Each bundle will have slight overhead.
    • addressable scene with direct references. again the data could end up in one bundle or many. more efficient to load/unload in one bundle.
    The fact is, having direct references in scenes, and including it all in the build, works pretty well, until it doesn't. If you have a 2gig android project, direct references in the player are somewhat out of the question.

    perhaps another way to phrase it...
    • if "same weight" is 10mb, it's likely slight faster if data is in build.
    • if "same weight" is 1gb, then in-build load time is infinite because no one will have your app.

    If you are using Asset Bundles then you can increase the speed by using uncompressed bundles.
     
  25. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,474
    I wouldn't say that's true always.

    According to my tests, loading speed depends on platform and compression type. For example, I remember a specific console platform loads chunk-based LZ4 compressed bundles significantly faster than any other compression type.

    It's worth to test your game on every target platform with every compression Unity supports. Track how much time each level requires to load. Then pick the compression that works best for you.
     
    karl_jones likes this.
  26. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    1,116
    Hello! when will 2020 be out of beta?
     
  27. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    946
    In 2021 lol
     
  28. KokkuHub

    KokkuHub

    Joined:
    Feb 15, 2018
    Posts:
    604
    Keep in mind LZMA compression will cause the bundle file to be entirely decompressed to memory in one go when it's loaded. This can easily double your memory usage if you keep your bundles around after loading assets from them (which is how Addressables work, if you are using that).
     
    glenneroo likes this.
  29. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    1,116
    lol
     
  30. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    205
    I tried with assetbundles and once i can see the scene Unity freezes like 2 minutes and then show normal fps, which makes it not an option. Scene is already loaded as i can see it but takes fps to almost 1fps for some minutes, which doesnt happen loading a normal scene.
    Tried also addressables but make Unity crash when load scene, and making multiple scenes would complicate much the development as the game is made. Dont know why scenes cant be bigger than just 4gb, that is very small scene
     
  31. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    205
    Same happened to me, after so many hours realized i couldnt make a scene bigger than 4gb, i understood why big companies dont develop with Unity
     
  32. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    205
    Asset bundles dont work, once you can see the scene game freezes for some minutes making AB useless
     
  33. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,474
    The asset bundle system has its flaws, but there are many developers that utilize asset bundles with great success, so I don't think it's entirely broken.

    It could be something with your specific use-case that causes the asset bundle system to under-perform. If you debug/profile what's going on, you might be able to figure out what's causing the freeze and find a workaround

    A common issue for freezes during scene load is scene activation, which I believe is split in at least two phases:
    • Unity is doing something called "Asset Integration" which used to and probably still runs on the main-thread. To workaround this, you can split content across multiple smaller asset bundles, so Unity needs to integrate fewer assets per bundle.
    • Unity calls Awake() on every Component that is in the loaded scene. If you do stuff inside Awake, it can sum up to take quite long. You could split your scene into more smaller ones. Mesh Colliders are also often found to be slow on scene activation.

    This has been fixed/added in Unity 2020.1:
    https://forum.unity.com/threads/bug...n-standalone-build.441116/page-5#post-5300130

    However, putting a lot of assets in a single asset bundle isn't the preferred approach in many cases and can be one reason why the freeze occurs, due to the asset integration step.

    I explained why splitting the content across multiple asset bundles has other advantages too here:
    https://forum.unity.com/threads/bug...n-standalone-build.441116/page-5#post-4473202
     
    Last edited: Mar 7, 2020
    glenneroo and karl_jones like this.
  34. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    946
    Assetbundles and Addressables are a major pain in the butt: they are extremely complicated to design around, not only at the code level, but at the actual gameplay level.
    I know, I have looked into it several times. It's a giant mess.
     
  35. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,474
    What are those pain points? Can you share specifics? I haven't looked at Addressables yet, but need to implement them in the coming weeks, so I'm interested to get some insights.
     
  36. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    205
    I have used successfuly assetbundles in other projects, for Android, with other Unity versions and not loading from hard drive the scene, in this case doesnt work, so its an Unity problem, not me
    The assetbundle is the scene so i cannot split into several assetbundles unless i make several scenes in which case i wouldnt need assetbundles at all. As my game is i cannot split it in several scenes easily, i would have to move objects between scenes and change the code of my app.
    Awake is not the problem otherwise the start of the scene would be sluggish too just loading the scene normally, which doesnt happen
    I knew about 2020.1 but have some problems, its in alpha yet, also i tried 2019.3 and fps drop a lot for VR, VR is broken in 2019 and not sure in 2020
     
  37. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    946
    I looked into addressables with renewed hope when I knew they were releasing them. I hoped it would ease the pain of dealing with assetbundles, but actually it doesn't.
    Addressables are designed to decouple the ACCESS to assetbundles, not to make them any easier to use: addressables is a layer of abstraction between you and the assetbundles to manage downloading them from the web plus loading and unloading them from memory, but there are several big flaws to this:
    You can't use an addressable as a regular asset: you can't use an addressable like a pointer to a texture or to a gameobject. And that's what we would really need: proxy assets that act as pointers to unity objects that can be used in their place and would be loaded or unloaded automatically on demand BY THE ENGINE.

    The current implementation of assetbundles is just a very low level way to import external assets into your project once it has been built, and that was the use cases for what they were designed: to load DLCs or texture packs into your game.
    They were never meant to segment your actual project in different parts but to ADD new parts once the project is done. (hence the big emphasis on network download of assetbundles in the docs)

    This is painfully obvious when you use them and Unity Tech is using them as an excuse to throw at anyone that speaks about the problem of managing big projects in Unity: "Oh, just use AssetBundles, they are meant for that."
    No they aren't.
    Asset management should work just like Async Scene loading and Texture Streaming does: automatically under the hood, were it belongs.
    Using assetbundles or it's extension, addressables, is like using sparse textures to implement texture streaming yourself: it CAN technically be done, yet it's a massive pain in the but.

    Let me be clear: I don't hate assetbundles or addressables. I think they are great for what they were designed to do: load external assets for your built application like texture packs, add-ons or DLCs.
    But they are NOT a good way to manage your project internally.
    It CAN be done and people actually use them to circumvent Unity's limitations, but yet again the fact you can use a fork to cut a steak if you REALLY have to doesn't make it any easier nor the best tool for the job.
     
    Last edited: Mar 8, 2020
  38. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    7,375
    I’m confused, are you saying people don’t download big apps up front and instead prefer asset bundle loading nightmares that never actually load and always interrupt the flow of the game to download some crap?
     
    atomicjoe likes this.
  39. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    946
    Actually, the end user is never the issue here: it's 2020 and we are all used to download GBs worth of updates for literally everything every time.
    The problem is with store fronts like the AppleStore that limit the actual app size to 1GB and force you to deal with it.
     
  40. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    7,375
    AFAIK for iOS the limit is 4GB in the last couple of years (and I think it was 2 before that).
     
  41. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    946
    Yeah sorry, I was thinking about AppleTV ridiculous requirements for app sizes. (a tiny-weeny app size and then you have to download everything yourself)
    I don't actually develop for iOS or TvOS though so it could have changed :p
     
  42. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    7,375
    That changed too btw :) (but we’re getting off topic)
     
    atomicjoe likes this.
  43. KokkuHub

    KokkuHub

    Joined:
    Feb 15, 2018
    Posts:
    604
    I agree 100% with this. If Unity is the only engine you ever worked with it might not sound too bad, but if you have experience with other engines, Unity's run-time (and build-time) asset management is, to be blunt, terrible.

    The biggest flaw is that Unity doesn't automatically know when an asset has become unreferenced, thus being unable to automatically unload them. The only way it has to detect unreferenced assets is by executing Resources.UnloadUnusedAssets(), which triggers a very expensive blocking sweep operation in order to do so. Fine-grained asset unloading is only possible when done manually. Even with addressables, you still need to manually release your AssetReferences, and the code complexity does add up on large projects.

    I began studying Godot lately and I couldn't help but to smile when I saw this in their docs:
    upload_2020-3-8_2-15-36.png
    Then I remembered I have Unity projects to tend to and the smile faded away.

    The second problem is that AssetBundles are so low-level you cannot use the same mechanism to load assets in edit-time as you do at run-time. You have to build a bundle before you can load it, and if you're targeting a different platform than the one you're running the Unity Editor on, it gets even more complicated. The workaround (which is also used by Addressables) is to "mimic" the access by using the editor-only AssetDatabase.

    What is weird is that, before bundles and addressables, there was a system that worked exactly the same in both edit-time and run-time: the resources folder. The problem is that the way Unity packs and loads resources is suboptimal and doesn't scale well, however the Unity team never considered trying to improve the resource system and address its flaws. Sometimes I suspect the code behind the resources system is so arcane there's no one left at Unity who dares modify it.
     
    Fenixake, glenneroo and atomicjoe like this.
  44. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    5,474
    Thanks for the reply!
     
  45. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    946
    I suspect that's what happened with surface shaders too, now that the new and shiny scriptable render pipelines don't support them anymore.
     
    Fenixake likes this.
  46. Fenixake

    Fenixake

    Joined:
    Jan 25, 2014
    Posts:
    139
    Guys wait a bit, make me understand. If I start building my 20gb project now, I will get an error that the size is to big?(I have 2k-4k textures for all of my stuff).Or is something specific to a platform?.
     
  47. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    946
    Read the proposed solution from ccsander here.
     
  48. Fenixake

    Fenixake

    Joined:
    Jan 25, 2014
    Posts:
    139
    I was not looking for that solution, I am trying to understand what the problem is with that 4gb limit, is it per file, is it for assets added together? that type of question my friend.
     
  49. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    946
    Then read Karl_Jones answer here. ;)
     
    karl_jones likes this.
  50. Fenixake

    Fenixake

    Joined:
    Jan 25, 2014
    Posts:
    139
    So you don't know either if you keep redirecting me to what I have read already.Every workaround found here is a bit to much for me, I used plugins and some tools to get my project up and running and in conclusion I cannot use those workarounds or atleast not without spending half of the year dealing with it.
     
unityunity