Search Unity

Bug: 4GB limit to Textures in standalone build

Discussion in 'General Graphics' started by ShilohGames, Nov 14, 2016.

  1. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,389
    2020 crashes instantly for me when I turn on gizmos..
    As for converting to asset bundles, it is still a mistified process, and seems no one has automated it. Doing things one by one is ridiculous. Its still on my todo list to understand the issue and solve it with an asset, but I was waiting for someone else to do it first :) Seems no one has.. or maybe im misunderstanding the problem, I find it too difficult to fully understand how anyone could use resource bundles on average sized projects without dying of old age before they finish. Seems absolutely ripe for automation.
     
  2. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,256
    Assetbundles where never meant to be used like regular assets. They were designed to be used as DLCs and mods for already built projects, not to replace all assets in your project from the ground up.
    The only reason they were pushed on us so much was because of the 4GB limit for assets before Unity 2020.1 : it was a way to deflect criticism, but actually everybody knew they were a pain in the ass.
    Also, addressables don't fix the inherent cumbersomeness of Assetbundles: they only automatize the loading and unloading of them, not the integration with your project. They are great from a programmers point of view for managing packages abstracting from where you source them (local or web) and clearing memory when not needed, but they are an absolute nightmare from an holistic point of view.
    If you have a professional studio divided in designers and programmers, you design your game first and program it later and can afford to have dedicated people whose task is to integrate Assetbundles in the project, then it can work.
    But if you are a one-man-band (like 99% of us indies are) who programs and designs as he goes, Assetbundles are your worst enemy. More details here.
    Your only options are:
    1 - migrating to 2020.1 or
    2 - use the trick described earlier in this thread to make it work with earlier versions.
     
    radiantboy and AcidArrow like this.
  3. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    8,434
    As certain platforms start requiring games be split into smaller chunks, it boggles my mind Unity hasn’t yet provided an easy way to split our games.

    I guess it’s because Unity employees like @karl_jones think people don’t like downloading games up front and instead like having games that trick people into thinking they are a small download and then start downloading a bunch more stuff.

    I have never seen another game engine enforce its views on gaming on its users like that before, especially when their views are so distorted, like insisting that the way to dynamically load an asset instead of another asset is to use a system that was designed for DLC and/or to load stuff from a server.
     
  4. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,256
    On Android builds there is actually a setting to split the build into a small APK and an OBB with all the project ressources (without having to manage it yourself via assetbundles or anything):
    https://docs.unity3d.com/Manual/android-OBBsupport.html

    On iOS it needs you to use AssetBundles though:
    https://docs.unity3d.com/Manual/AppThinning.html
     
  5. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    8,434
    That method for Android is about to be deprecated as Google Play starts requiring everyone uses .aab files which more or less require asset bundles as things stand now.
     
  6. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,256
    OH. GREAT...
     
  7. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,389
    I estimate my final game will be 15 gigs. Id like to make a small fix without saying hey get 6 gig of the same data again. I hope to target pc and ps5/xbox. That would be the dream. it is a huge "open world" game with 3 cities, i dont want to redeploy 16 gigs each time ;) I still dont get asset bundles or addressables, 2020 is not an option as described earlier (been an open bug for 6 months+ with no sign of even trying to fix it)..... plus even if I can the "very simple" navmesh bug kills my game (worldToTile) which kills unity in a heartbeat (unity still refuse to even open a bug on this despite me getting my project to them with instructions). I must say unity used to feel like it was made for us to let us have fun in 3D, now it feels like it was made for us to experience what genuine helll feels like.... take you to the limit of fun and then rip your nuts off before u can feel how good it is to get there. I would rather no feature ehnancements and just a year of debugging goes on.
     
  8. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,093
    Unity 2020 just lifts the 4gb limit, it does not help with smaller patch sizes. The solution is still asset bundles.

    If you plan to release on consoles, read their guidelines and technical requirements documents. There is at least one console where the vendor forces you do use asset bundles, if you don't use them, you're not allowed to release your game on their platform.

    BTW, perhaps this Unite talk is of interest as well. At 25:56 he talks about managing game content:
     
  9. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    949
    A 16GB Unity game that doesn't use bundles? How long are your build times?

    Without bundles, your patches for all consoles are pretty much guaranteed to be almost as big as the entire game, even if your changes are minimal. If you just release on Steam, it doesn't seem that bad because Steam seems to create diff patches at the byte level, but granularity is much lower on consoles (and in two of them the patch is stored alongside the base game, so the installation size can double, or even more).

    Build times are also longer overall. Yes, building bundles from scratch takes longer than building with embedded assets, but Unity can reuse built bundles if their assets didn't change, while embedded resources are always recreated on every build (there's no caching whatsoever).

    Yes, using bundles is a PITA. Addressables reduces some of the friction, but it's still way more complex to use than resources.

    The biggest problem, IMO, is that Unity refuses to improve the resources system and instead keeps making workarounds on top of workarounds.

    Asset bundles themselves are kind of a hack. They use the same resource files Unity uses for embedded scene assets, kinda like "partial" builds without code (which is why than can take longer to build from scratch). Then comes Addressables, which is built on top of asset bundles.

    Here's how Unity could improve the resources system:

    • Implement a caching mechanism so the player resource files aren't regenerated on every build.
    • Generate the indexing data for assets in resources folders at build time, not at load time (where it makes it take longer for the game to boot and is the primary reason Unity tells you to avoid resources).
    • Add means to view and control how assets will be grouped into resource files (kinda like bundles).
    • Make it possible to view and manage all assets in resources folders in the editor, as well as allowing developers to control how they are packed (for example, by making sure each plug-in gets their own resources file).
    • Add support for creating resource patch builds. If you don't know what this is, just look at how Unreal handles pak files: you can create a patch pak file if you preserve the original build of the game (as said before, two consoles require you to keep the first released version of the game around to produce patches), which contains only the assets which are new or modified compared to the base release of the game. When this pak file is present, it's entries override the base pak file's in the runtime asset index, so the patched assets are loaded seamlessly.
    • Work harder to make the whole resource building deterministic. Currently just by building the same project again you get binary differences all over the place even if nothing changed.
     
    Last edited: Nov 5, 2020
    twhittaker and glenneroo like this.
  10. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    8,434
    All this is because Asset Bundles were meant to be for DLC and stuff downloaded from a server, while what most people need is a way to split your game in a reasonable way to be able to comply with platform requirements and be able to produce smaller patches.

    And instead of doing that they keep trying to bend asset bundles to do things they were not meant to do.
     
    Neto_Kokku and atomicjoe like this.
  11. Unplug

    Unplug

    Joined:
    Aug 23, 2014
    Posts:
    256
    see my solution, just put an empty game object with an object array and put large asset you want to import in an empty scene (splash screen, empty async, preloader, etc) so that you cache them in order to reduce your scene size
     
  12. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,389
    Not really sure how that is relevant, most of us have massive scenes, not just some prefabs.
     
  13. Kaldrin

    Kaldrin

    Joined:
    Jul 10, 2018
    Posts:
    36
    Hey!
    I just stumbled upon this issue recently, even though my project is like,barely 900 Mo.
    I see this issue went through the years and I am honestly surprised it wasn't fixed

    I am not a paying using user yet so I won't make any complaints of course, but I'd like to know if there is anyway to bypass this issue without having to spend days or even weeks learning about additive scenes or asset bundles? I need to build, like now, and these things scare me I don't want to have drastic changes made to the project now that's it's finally almost bugfree.

    My game is only 2 scenes, a splash screen and then the main scene, which is barely 10 Mo in itself, but probably references a lot of things. However, it's just not possible for it to reference more than 4gb of assets, as we only have 800Mo of assets

    Unity gives me this error when I'm building



    But the files it's pointing to doesn't exist, I'm confused
     
    Last edited: Dec 25, 2020
  14. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    5,721
    Have you tried 2020.1?
    https://forum.unity.com/threads/bug...n-standalone-build.441116/page-5#post-5300130
     
  15. Kaldrin

    Kaldrin

    Joined:
    Jul 10, 2018
    Posts:
    36
    Thanks a lot for the answer

    We are currently working with 2019.1.1f1, for many reasons we do not want to go through the effort of upgrading the project to 2020, we fear the many issues that upgrading a project might cause with our not very experienced code architecture.

    Also I tried creating scenes with parts of the assets to split better the resources generation upon building and not exceed the 4GB limit (That still makes no sense for a less than 1GB project :c ) but it didn't seem to work, many materials are missing from the build and it doesn't even work properly even though all we added since the last time builds were working is more textures and some adjustments to the code, nothing crazy new.

    Some textures are big animation sprite sheets for some almost make 8192x8192 in size though, in case this might be an issue, but seeing Unity allows for that big textures and it works fine in the editor I don't think it should be a problem, we do not target mobile with this version anyway.
     
  16. Kaldrin

    Kaldrin

    Joined:
    Jul 10, 2018
    Posts:
    36
    I managed to make the build work by reducing some textures quality
    I think I can do an optimization pass but it sucks we still have so much content to add and again it makes no sense the build si'tn even 1Gb nor are the total assets size

    Maybe the mip maps are weighing a lot more ?
     
  17. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,256
    If your build isn't even 1GB, it can not be related to the 4GB ressource limit in Unity before 2020.
    Your issue is somewhere else.
    Try sending a bug report with the project to Unity.
     
    karl_jones likes this.
  18. Kaldrin

    Kaldrin

    Joined:
    Jul 10, 2018
    Posts:
    36
    That is weird indeed the error appearing in the console is this 4GB limit thing, I'll think of doing that yeah, thx
     
  19. Del_Passos

    Del_Passos

    Joined:
    Jan 13, 2021
    Posts:
    3
    I guy's, just to be confirmed here :
    The limit of 4GB is not any more in the 2020 versions... But from wich version ? 2019 not at all ?

    Thanks,

    Del.
     
  20. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    5,721
  21. swantonb

    swantonb

    Joined:
    Apr 10, 2018
    Posts:
    64
    I have this happening to me rn, there's no way one scene has more than 4gb. I use 4 scenes and load them together and the lightmaps of all of them combined are 3gb. It worked a minute ago and now with nothing changed i get this:

    Asset bundle total uncompressed size exceeds maximum size of 4294967295 bytes.

    Why?
     
  22. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,256
    Send a bug report and include your project.
     
  23. swantonb

    swantonb

    Joined:
    Apr 10, 2018
    Posts:
    64
    not sure how to do that i never did
     
  24. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,256
    1- Open your project in Unity
    2- go to HELP, completely on the right of the top menu
    3- click on REPORT A BUG
    4- Fill the bug report following the indications
    (at the bottom of the window you can see what folders will be included with your upload. By default your project folder will be included with the report)

    Once your bug report is sent, you will receive a link to the bug report itself (only the text report. Included projects are not accessible)
    Please share this link here with us so we can follow up on the issue, since we are all interested in this, in this thread. :)

    If your project is too big, you can always make a copy of the project folder, manually delete everything that is not needed for the project to run in that new folder and send this new folder instead.

    Don't worry about sending your project though. It's confidential and Unity will not share it with anyone.
     
    karl_jones and glenneroo like this.
  25. swantonb

    swantonb

    Joined:
    Apr 10, 2018
    Posts:
    64
    Well my project is 35 gb only assets XD, i doubt they will take 1 hour at least to import it...i upgraded to 2020.2 now, didnt get any errors quite yet, i hope i wont encounter any problems generated by the upgrade. I will build in a minute and see if it works now.

    Edit: well, it built, now i hope no errors regarding the upgrade will be encoutnered later
     
    Last edited: Jan 21, 2021
    radiantboy likes this.
  26. AmitloafTotes

    AmitloafTotes

    Joined:
    Jul 10, 2018
    Posts:
    26
    So, we also have that issue, with the sharedassets files. One of them (sharedassets4) exceeds 4GB. It's been said that splitting the scenes could work and it does, but in order to be effective we'll have to understand how to split them.
    So my question is, how are they created? Does each file correspond to a scene? What if there's a model/texture shared between scenes? In that case it appears once, twice or maybe both scenes are merged into the same file? In that case I would have to make sure my split scenes contain no shared asset.
    Anyway, if you can answer that it will be great.
    Thanks, Amit
     
  27. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,256
    As far as I know, the asset will be assigned to the first scene that references it, so if the asset is referenced in multiple scenes, the order of the scenes in the build properties's scene list is what determines to which scene it will be assigned.
    Also, I think that a "sharedasset" file is created by scene. (someone correct me if I'm wrong)
     
  28. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,389
    So I think this now manifests itself in a blackscreen on HDRP unity 2019.4 (only just statrted with HDRP so don't know for sure). I hope so anyway, trying to lower .res now. It's tough teetering on the edge of 4 gig :)
     
  29. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,256
    FWIW Unity just released the new Long Term Support version 2020.3 LTS, maybe you should upgrade... but be VERY careful and BACKUP YOUR PROJECT before doing so! 2019.4 to 2020.3 is NOT a smooth upgrade! (I've been doing it since yesterday and I haven't finished yet...)
     
    Last edited: Mar 24, 2021
    radiantboy likes this.
  30. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,389
    Yeh im going to try this week. I do not think they have fixed my 2 show stoppers though, cant even turn on gizmos in 2020 lol.
     
  31. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,389
    Hmm under 4 gig and still getting black, must be unrelated hdrp bug.
     
  32. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    231
    I have less than 300 mb in resources folder and tells me i supassed the limit using more than 5GB, it is not just for resources folder
     
    Kojote and Kaldrin like this.
  33. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    231
    I used assetbundles and have the same problem, says it can not load files more than 4GB, same happens with addressables, how you load a scene with more than 4GB in Unity ??
     
  34. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    5,721
    Have you tried 2020.1+?
     
  35. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    231
    Does it work for sure in 2020.1 or you mean i could just try? it takes a whole day to change of version
     
  36. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    5,721
  37. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    231
  38. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    5,721
    Yes but the scene won't be a single file now. The assets will be split into multiple files so you should not have any issues with a single file being over 4gb.
     
    NewMagic-Studio and atomicjoe like this.
  39. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    231
    I tried to migrate to 2020.3 and gives me the errors

    Assembly 'Unity.Properties.Codegen' is a CodeGen assembly and cannot be Auto Referenced (Packages/com.unity.properties/Unity.Properties.Codegen/Unity.Properties.Codegen.asmdef)

    Exception: Assembly 'Unity.Properties.Codegen' is a CodeGen assembly and cannot be Auto Referenced
    UnityEditor.Scripting.ScriptCompilation.CustomScriptAssemblyData.ValidateFields () (at <dc14d30a5e0540ee9311ed6df63d9074>:0)
    UnityEditor.AssemblyDefinitionImporterInspector.LoadAssemblyDefinitionState (UnityEditor.AssemblyDefinitionImporterInspector+AssemblyDefinitionState state, System.String path) (at <dc14d30a5e0540ee9311ed6df63d9074>:0)
    UnityEditor.AssetImporters.AssetImporterEditor:InternalSetTargets(Object[])

    Tried to delete the asmdef but keeps creating it, and in package manager don't find the Codegen package
     
  40. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,256
    Seems related to the Entities package.
    You should go to the package manager -> Packages:In Project and update all the packages in your project to the latest versions. This must be done manually.
     
  41. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    231
    Yes, i updated the visual code package and solved it. But still get tons of errors of lot of assets. I get the error :

    Animator' does not contain a definition for 'BindSceneProperty' and no accessible extension method 'BindSceneProperty' accepting a first argument of type 'Animator' could be found (are you missing a using directive or an assembly reference?)"
     
  42. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    231
    Tried to update to 2020 but fails important assets, how can be done before 2020? adding scenes additively?
     
  43. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    231
    But if you make a bundle of the prefab then will include the rest, why make the audio, textures and model bundles?
     
  44. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,093
    I explained it in the "The advantage is" paragraph in my text you quoted.
     
  45. NewMagic-Studio

    NewMagic-Studio

    Joined:
    Feb 25, 2015
    Posts:
    231
    I realized i don't have the sharedresources.asset files in the build, just the sharedresources.resource

    upload_2021-10-19_22-55-46.png
     
unityunity