Search Unity

  1. Unity 2019.1 beta is now available.
    Dismiss Notice
  2. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  3. We're looking for insight from anyone who has experience with game testing to help us better Unity. Take our survey here. If chosen to participate you'll be entered into a sweepstake to win an Amazon gift card.
    Dismiss Notice
  4. Want to provide direct feedback to the Unity team? Join the Unity Advisory Panel.
    Dismiss Notice
  5. Unity 2018.3 is now released.
    Dismiss Notice
  6. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice

Bug: 4GB limit to Textures in standalone build

Discussion in 'General Graphics' started by ShilohGames, Nov 14, 2016.

  1. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    994
    Yea well.. 2d games are not so huge anyways.. Besides thats easy enough to use with bundles..

    But Im making now a topdown survival game, and I understand that bundles is a great way to use dlc or game updates...

    Any toughts on how to make dlc and updates without bundle?
     
  2. MrG

    MrG

    Joined:
    Oct 6, 2012
    Posts:
    261
    I was referring to UI 2D art: icons for buttons and markers and inventory panels and such.
     
  3. glenneroo

    glenneroo

    Joined:
    Oct 27, 2016
    Posts:
    16
    So you're saying it's fixed in 2018.2?
     
  4. MrG

    MrG

    Joined:
    Oct 6, 2012
    Posts:
    261
     
  5. glenneroo

    glenneroo

    Joined:
    Oct 27, 2016
    Posts:
    16
    Thanks I read that and did some more investigating into the new Addressable Asset System but according to this post it requires editing the manifest file before it's available in the Package Manager. Also it's version 0.1.2 i.e. early beta. Instead I just did as recommended elsewhere and moved the bigger resources into separate AssetBundles and left a lot of stuff in the Resources to get it under 4GB.
     
  6. Bevarii

    Bevarii

    Joined:
    Oct 3, 2018
    Posts:
    2
    I just ran into this problem too, I'm glad I found this thread since I've been struggling with this problem for a week and only got the error now.
     
  7. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    994
    I dont understand the asset bundle or how to use it ... easiest is to split the scene in more scenes..
     
    radiantboy and llJIMBOBll like this.
  8. LoekvanKooten

    LoekvanKooten

    Joined:
    Apr 3, 2017
    Posts:
    104
    This is completely unacceptable. It seems I just ran into this limit. Asset Management... all the kind of stuff we *don't* want to do, because all that stuff is exactly what Unity was made for. We want to make games, not manage assets.

    Why is there no standard solution for this yet? I understand the file limit, but why are files not automatically split? Why do I need to read tons of manuals and study asset management features to get a working game in the first place?

    We need a button that splits the files automatically, and we need it now. It's 2019, and apparently Unity still has not caught on with 64 bit.

    I understand the underlying issue. But then at least come with a temporary workaround - completely automated - that splits everything automatically. That can't be too hard, right? Please?
     
    radiantboy likes this.
  9. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,130
    Is this really still a thing? I'm asking, because about in January 2018 (about a year ago at the time of writing), it sounded like they're working on a fix:
    What Unity version do you use?
     
  10. LoekvanKooten

    LoekvanKooten

    Joined:
    Apr 3, 2017
    Posts:
    104
    Hi Peter, the latest version: 2018.2.18f1
     
  11. LoekvanKooten

    LoekvanKooten

    Joined:
    Apr 3, 2017
    Posts:
    104
    And I confirm that this was indeed causing the issue. I am now splitting my scenes. I now got:

    sharedassets1.assets.resS (50,113 kB)
    sharedassets2.assets.resS (4,551,889 kB), which actually looks bigger than my previous file, but it seems the textures are displaying correctly. Unless I run into nasty surprises at the other end of the level which I can't see from here.
     
  12. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    994
    Guys.. Just use Sectr streaming, very easy to set up , it fixes everything
     
  13. LoekvanKooten

    LoekvanKooten

    Joined:
    Apr 3, 2017
    Posts:
    104
    Actually I spoke with the developer of Sectr, and he says this is *not* the case and may introduce lots of other issues.
     
    HakJak likes this.
  14. LoekvanKooten

    LoekvanKooten

    Joined:
    Apr 3, 2017
    Posts:
    104
    By the way, that is Peter right above me :D

    Quote:

    I just read up on that issue, ended up in the long forum thread where you just posted in the end. From how I understood the problem, you can run into that issue either by stuffing up the resources folder with more than 4 GB of stuff, or by having a single scene referencing > 4GB of assets exclusively. SECTR could maybe help you in so far as that it splits up your terrain into multiple scenes, so it might even out the asset file size distribution across those scenes. That would heavily depend on how your assets are distributed across the terrain, and that is usually not the reason why you would add terrain streaming. If you have not used terrain streaming in your project yet, I think it is not a good idea to introduce it just to combat that unity filesize issue, as it might make your overall work more complicated and create different issues with gameplay logic, etc.
     
  15. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    994
    yea but it splits the scene to many scenes wich confuses the resource file limit
     
  16. mmortall

    mmortall

    Joined:
    Dec 28, 2010
    Posts:
    78
    Unity, why not to add an error or popup about this issue during the build? We spend 2 days researching this issue.
    Still, this issue exists in latest Unity without any notifications. We just got several textures corrupted without any reasons. And in Editor all works fine. This is a critical issue that user should be aware of.
     
    radiantboy and mattis89 like this.
  17. Omzy

    Omzy

    Joined:
    Jun 28, 2013
    Posts:
    14
    I spent a month struggling with this issue, blaming the Bakery GPU lightmapper for it, wasting the developer's time with back and forth debugging, redoing and rebaking my scene lighting like 10 times (baking the scene takes hours), and just now I find this thread, describing the exact issue I have. I've had this problem with 2018.3 and the alpha 2019.1. It is 2019 and developers have been on this forum since 2016 asking for a simple error message so that new people like myself encountering this don't waste our project resources trying to figure out what is wrong. What I'm wondering, @karl_jones is if there are plans to at least introduce an error message describing the limitation. It seems obvious that there are no plans to properly fix the issue, so why not at least give your customers that small amount of mercy? Do you not realize how big of a problem the actual debugging process is for people here? If it were an obvious issue, then fine, ignore it, but it takes a lot of time (and money in my case) to figure out what the actual problem is.
     
    radiantboy likes this.
  18. KarelA

    KarelA

    Joined:
    Dec 30, 2008
    Posts:
    412
    Holy crap! Is this issue still a thing? Is there still no documentation or warnings about this problem? Is this thread the only source of information after poor devs have wasted days trying to figure out the problem and stumbled upon here trough luck by googling the problem?

    Unity... what the hell? I remember how mad i was after i found out about this back in August 2017 and rambled here that you have to clearly tell about this limitation to the users. Otherwise you just waste developers time who stumble upon this and are left wondering what is happening. It is now January 2019 and there is still no message whatsoever? And as a result bunch of people have wasted their time and not knowing what is going on.

    For the love of god. Just please put an error or warning message. Do at least something that would let the user know not to go over the limitation. This way we can start look for alternatives and not be left wondering why our builds are looking broken.

    To other devs here. 2 years ago i was in the same situation. Luckily in my case i had a friend who already knew (took him several days to figure it out) about this and pointed me to this thread. I can confirm that SECTR stream is a great solution if you want a woraround. Of course it depends on your game but in my case it worked out perfectly.
     
    radiantboy likes this.
  19. LoekvanKooten

    LoekvanKooten

    Joined:
    Apr 3, 2017
    Posts:
    104
    It's even worse. After an upgrade from 2018.2 to 2018.3, my build size went up from 2 GB to a whopping 6 GB, while nothing changed in the game. I am not happy... how am I ever going to solve this?
     
    radiantboy likes this.
  20. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    737
    I believe if we move to asset bundles this should work as a fix, you can keep them under 4 gig. Also in the end you dont want to ship a 6 gig game, I think its perferable to ship an exe and download bundles that are required during boot up. This obviously makes a patch much easier due to not downloading the same info again.

    The main problem I have is if a game reaches this limit, its surely huge and nightmarish to "convert" to asset bundles (I have over a million objs in my scene). So ultimately I have literally been waiting for years for an asset which can *automatically* change your project to use asset bundles AND keep them small to like 1 gig each or something. Only problem is there is no such asset yet as far as I can see.

    I did try this one and was hoping it would work https://assetstore.unity.com/packages/tools/utilities/autoassetbundles-111942 which I think it did but it did not allow me to specify bundle sizes so the exact same issue still came up. Perhaps someone knows one that can automate the process and allow small bundle sizes? I have toiled over this for years, mostly because im too afraid to dive into doing it manually on such a ridiculously large project :)

    PS, I dont want to split my scenes up, I like my 3 cities in one, its very fast to leap between them and I seem to work better when its all in one scene, not to mention most of the core mechanics would need to be duplicated for each scene which leads to problems. Im sure I could do it (they were seperate but i merged them lol).. but I dont want to, I would love to see auto asset bundle asset above fixed and working.
     
    LoekvanKooten likes this.
  21. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    3,177
    I would look at the Addressables system.
    Its hard to say exactly how to upgrade your project without knowing the details but you should be to write a script to go through your scenes and automatically replace assets with Addressable assets. It may require a bit of tweaking but it should be possible.

    I would do something like this:

    1) Iterate scenes
    2) Find GameObjects in scenes, maybe start with static geometry first. Convert the GameObject to a prefab,
    3) Add the prefab to the Addressable system
    4) Add a new GameObject with a link to the Addressable so it loads at scene start.

    Then to further improve on this you could identify duplicate geometry and reuse the same Addressable asset instead of creating a new Prefab each time. One of the benefits of this should be faster build times as AssetBundles do not need to be rebuilt each time.

    You don't need to convert an entire project. Do it a little at a time to help reduce the size of the serialized files so you can get under the 4gb limit.

    On the topic of an error/warning, we do have some errors that are thrown when the asset is over 4gb, however, there seem to be circumstances when this does not happens so we will further improve this and add to the documentation.
     
    Omzy likes this.
  22. LoekvanKooten

    LoekvanKooten

    Joined:
    Apr 3, 2017
    Posts:
    104
    I've now spread my build over four assets files of 1,5 GB each by instancing prefebs in scenes that are loaded one by one, but still I wonder why 2018.3 needs three times as much space to store the same information as 2018.2.

    And Unity, with all due respect: an export warning is not good enough. We need an asset that spreads your information over files automatically, or we need a bigger file limit. More than half a year has passed and absolutely nothing has been done about this.

    Please put this on the very top of your list.
     
  23. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    3,177
    Please file a bug report regarding the increase.
     
    LoekvanKooten likes this.
  24. LoekvanKooten

    LoekvanKooten

    Joined:
    Apr 3, 2017
    Posts:
    104
  25. LoekvanKooten

    LoekvanKooten

    Joined:
    Apr 3, 2017
    Posts:
    104
    @karl_jones Wait. Not done. After 10 minutes, packing the report is still at 1%.

    Of course I can just zip everything, but this is taking too many resources for too long. Is there another way to send you what you need, without packing and sending the whole 6 GB of data?
     
  26. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    3,177
    You can attach the log files and if we need more we can send you a link to upload the project at a later time if required
     
  27. LoekvanKooten

    LoekvanKooten

    Joined:
    Apr 3, 2017
    Posts:
    104
    Of course. Done!
     
    karl_jones likes this.
  28. truefx001

    truefx001

    Joined:
    Jan 30, 2013
    Posts:
    90
    @karl_jones you said in your previous last year posts that we should look at 2018.2 and this is high priority and should be resolved in coming Unity versions,and here we are in 2019 and the 32bit 4gb resource limit issue is still existing. I don't want to use Asset bundles they are slow and have the same 4gb limit and your explained workaround is a pain to say the least, that's not a good solution and what's your answer to us all waiting for it since this thread opened in 2016 when will you guys address and fix this issue, at this point I really do not care how complicated it is, your engine is 64bit and with that all file systems should be as well, this is just not acceptable from a developer point of view. We don't need a workaround but an ETA when this will be addressed and fixed. Split the resource files even more like it was mentioned before during the build process. We needed a solution the moment you labeled your engine 64bit.
     
    Lex4art likes this.
  29. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    3,177
    I suggested 2018.2 for the Addressables system. 64bit support has now been looked at throughout last year and alternatives such as file splitting have been investigated.
    The only update from this I can provide would be that they are saying this will not change for the existing serialized file system but future systems will support it(I can not offer any ETAs on these). I do not know the full details why.
    So it's not good news for you I'm afraid.
    I will continue to push for an automatic solution internally and will look deeper into why it can not be done but for the moment you should continue to look at alternatives such as Asset Bundles and Addressables.
     
    Peter77 likes this.
  30. LoekvanKooten

    LoekvanKooten

    Joined:
    Apr 3, 2017
    Posts:
    104
    @karl_jones Asset Bundles, Addressables, I don't think anyone cares which provisionary solution Unity proposes, but can the process at least be automated?
     
  31. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    994
    This is just terrible!! Even worse now, build report tool ”Total User Assets” well what the hell counts as that? Everything just popped and took even more space? .. None other game engine have some file limits where you need to starts hassling about stuff you dont understand.. Is it really that hard to make it 8gb or even just remove the limit? What is the problem anyway?

    Still splitting the scenes and streaming is the best solution!
     
  32. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,130
  33. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    3,177
    Anything larger than 4gb requires 64-bit support and not all platforms support 4gb+ files.
    Ultimately It comes down to the large scope of work and limited resources.

    In short
    • Switching all file size and offset values in the editor to 64 bit. (There are a lot of them)
    • Switching all platform virtual file system implementations to 64 bit(Some platforms don't support files over 4gb so we still need to support both 32bit and 64bit!)
    • Writing tests & ensuring all supported platforms work. (A lot of work and people)
    • Supporting upgrade paths for old 32bit files.
    • Supporting these changes until ECS replaces it.

    When we address this we then hit a new issue, all our internal instance ids are 32 bit, so then we need to address this and that is another huge change in itself.

    There are new systems in development to solve these issues. I actually argued that we should still do it and offered to take a look myself. I was given the work done so far and told the issues...Then I realised just how huge and risky this change is :eek:
    Personally, I would still like it to happen ;)
     
    Last edited: Feb 11, 2019 at 11:09 AM
    glenneroo, Omzy and Peter77 like this.
  34. Lex4art

    Lex4art

    Joined:
    Nov 17, 2012
    Posts:
    158
    Better to gather some guts and do it, because alternative - put a head in the sand or rely on "for programmers by programmers" artists-unfirendly hacks like we have now - leads Unity engine into nowere IMHO. My personal case - I can't upgrade my asset in store ("Real Materials Complete") because of this limitation...
     
  35. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    994

    hmm, this dosent work...Cant assign anything to the prefabs to hold slot ..
     
  36. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    994
    Ah okay I see yes, theres alot of work to make it happend! Really appreciate it =) Adnvanced multi scene is something I need to use now and streaming ... but I hit another issue that the File Header takes alot of space, what is that ?
     
    karl_jones likes this.
  37. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    3,177
    What do you mean the file header takes a lot of space? What are you referring to?
     
  38. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    994
    when I build, so the File Header takes 38% of the build? What is File Header?
     
  39. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    3,177
    A scriptable object is a file which contains a file header. The header contains various internal data for the file. All our files require them for us to correctly load them. For example, if you created a class that derived from ScriptableObject with zero data inside it, it would still take up some space when saved to disk just for the file header.

    Maybe you have a lot of small SO files?
     
  40. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    994
    hmm, I dont know .. I have only 16 x 16 km terrain in my scene and vegetation studio pro...
     
  41. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,130
    Easiest way to find out is to fire up a memory profiler and check where it comes from.

    Unity's new Memory Profiler (2018.3 and higher)
    https://forum.unity.com/threads/new...ew-package-available-for-unity-2018-3.597271/

    Unity's old Memory Profiler (5.3 - 2017.4?)
    https://forum.unity.com/threads/weve-started-work-on-a-new-memory-profiler.358963/

    Heap Explorer (2017.4 and higher)
    https://forum.unity.com/threads/wip...ofiler-debugger-and-analyzer-for-unity.527949
     
    glenneroo and karl_jones like this.
  42. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    994

    aaahh perfect! Does it work in build? Didnt find any ”how to” on that, because I need to profile it in build, because even before I see my scene its all black and I get a text in the development console where it says I have exxceeded the 4gb limit...
     
    Last edited: Feb 11, 2019 at 5:40 PM
  43. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    994
    Found my issue... I splitted my 16x16k terrain to 2ks terrain tiles.. And made them all 2k on both texture and height resolutions... That was File Header Size build up..
     
  44. mattis89

    mattis89

    Joined:
    Jan 10, 2017
    Posts:
    994
    and the memory profiler froze unity..
     
  45. tawdry

    tawdry

    Joined:
    Sep 3, 2014
    Posts:
    1,169
    I'm curious if the problem is 4gb limit why can't a second resources folder 3rd fourth etc be implemented that shouldn't need any major restructuring of the unity engine?
     
  46. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    3,177
    The resources folder is only one of the instances where this is an issue. In fact the resources folder is the simplest to work around, switching to Asset Bundles or Addressables is not much of a change when coming from using Resources. Having support for multiple resources folders would be a bad idea since its something we recommend people don't use htps://unity3d.com/learn/tutorials/topics/best-practices/resources-folder.
    It's great for rapid prototyping but for anything more and you should be considered a more manageable approach, as discussed in the link.
     
    Peter77 likes this.
  47. Omzy

    Omzy

    Joined:
    Jun 28, 2013
    Posts:
    14
    @karl_jones I am trying the addressable asset system, but running into a problem. I'll take some images so you can see what I'm talking about. I have a lot of textures in my game, and I intend to add even more, so reducing the number of textures isn't an ideal answer for me, but rather finding ways that Unity can support large numbers of textures. My build report is as follows:

    Build Report
    Uncompressed usage by category (Percentages based on user generated assets only):
    Textures --> 6064.1 mb 88.6% <---
    Meshes 103.2 mb 1.5%
    Animations 13.5 mb 0.2%
    Sounds 1.8 mb 0.0%
    Shaders 98.5 mb 1.4%
    Other Assets 12.0 mb 0.2%
    Levels 92.4 mb 1.4%
    Scripts 2.3 mb 0.0%
    Included DLLs 3.9 mb 0.1%
    File headers 450.8 mb 6.6%
    Total User Assets 6842.6 mb 100.0%
    Complete build size 1174.9 mb

    I have arranged most of my scene asset into addressable asset folders per the 'Trash Dash' video. Here you can see the folders.
    https://gyazo.com/6ddaa6c4651e175771d9d6f36376de9f

    Here you can see the file sizes in my build directory addressables folder: D:\ProjectH\Builds\Build01\Test_Data\StreamingAssets\com.unity.addressables\StandaloneWindows64


    https://gyazo.com/35e094a6e1cbd3cecec5b025933abc46

    So that looks great, the files are all under 4gb. However, my project still has the same size of the sharedasset files as you can see here:


    https://gyazo.com/272d9854ad369a11a415c175047864fd

    I am still encountering the texture glitch bug where my textures are bugging out (the thing that made me and others notice this 4gb limit in the first place). Perhaps I'm using the addressables system incorrectly. I followed the information that was available to me. Any ideas on why the build continues to generate the large sharedassets file? Why has sharedassets1.assets.resS stayed the same size despite reorganizing into addressables? Is this where all the shader variations are compiled? My project seems to generate tons of shader variants, which, from reading on several forums, is normal behaviour. I barely use the Resources folder, there is nearly nothing in there, and I'm using addressables as recommended. I am at my wits end here.
     
  48. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    3,177
    How are you referencing the assets in the scene? If you hold a hard link, that is a script has a variable to it or the thing is in the scene, then it will be pulled in. You need to use the addressable system to load the asset. Can you ask this question in the addressables part of the forum? It's under experimental. The team will then see it and be able to walk through it with you.
     
    Peter77 likes this.
  49. Omzy

    Omzy

    Joined:
    Jun 28, 2013
    Posts:
    14
    Ok, i will ask there. All i have done is place the prefabs into my scene and I'm loading the Scene using a simple load level call. It is all just static level geometry. There are no script calls to load anything. I assumed having prefabs in the Scene serves the same function.
     
  50. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    3,177
    No the prefabs need to be loaded via a script through the Addressables system. If they are in the scene then they will be built into the main scenes data. The simplest way to do this would be with the AssetReference class. Just add a
    AssetReference property to a script and then call the load function in Start.