Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Bug: 4GB limit to Textures in standalone build

Discussion in 'General Graphics' started by ShilohGames, Nov 14, 2016.

  1. Troast

    Troast

    Joined:
    Mar 5, 2017
    Posts:
    11
    I have this problem. My game is over 4gb textures, and will corrupt the graphics if I don't delete something.

    I don't mind waiting for unity to fix it, but for those that have to use a workaround , what is the easiest method?
    Simply of course, it is to just delte my extra textures. But what if I want to keep these?
     
  2. razzraziel

    razzraziel

    Joined:
    Sep 13, 2018
    Posts:
    395
    the thing you need to fix this, is reading the whole topic carefully.
     
  3. Omzy

    Omzy

    Joined:
    Jun 28, 2013
    Posts:
    31
    The easiest workaround is splitting your scene into several scenes and loading the extra ones additively. Preferably with addressables so that things split into different bundles (files less than 4gb). See https://forum.unity.com/threads/problem-with-addressable-prefabs-in-scene.630769/
     
  4. Troast

    Troast

    Joined:
    Mar 5, 2017
    Posts:
    11
    Ok, thankyou. I am glad the work to do this, doesn't seem too troublesome.
     
  5. tawdry

    tawdry

    Joined:
    Sep 3, 2014
    Posts:
    1,356
    I have finally started using asset bundles. Storing them in the streaming folder. So if I have say 5 bundles of 2 gigs each will that be fine in a single folder?So long as each bundle doesn't exceed 4gb ?
     
  6. Omzy

    Omzy

    Joined:
    Jun 28, 2013
    Posts:
    31
    Exactly.
     
  7. Troast

    Troast

    Joined:
    Mar 5, 2017
    Posts:
    11
    Yes assetbundles was my solution. A simple task of getting the assetbundles browser package, draggin my objects to make up a bundle and then building it, and making them instance on scene load. No problems!
     
  8. truefx001

    truefx001

    Joined:
    Jan 30, 2013
    Posts:
    132
    @karl_jones can you tell me what files exactly are put into the first resources.assets.resS I cannot find anywhere what Unity is putting in this first one at point of build. As I understand the other sharedasset0... are resources respectively to each scene. I don't want to use the Assetbundles as they have the same 4gb limitation and are more work (more steps) then resource folders, and sadly I cannot use Addressable system because of some assets that need 2017 and I still have use for the old ui in some parts of my game. This is why I need to have a solution on how to reduce this first resources.assets.resS I have set a lot of textures in my main scene to a lower max resolutions which is frankly just horrible for the look of my game, but better then those clown like stripes that appear on random game objects when one of those resource files gets just a tiny bit over 4gb. Still shaking my head that no solution has come out of this, we are in the year 2019 and not everyone is creating a mobile game. What's the update on a solution to split the resource files per scene to at least fix this limitation for textures. I do not care right now what other things need to be fixed as well as this one is making any game that uses resources (which is the easiest no extra steps) limited to how much textures and resources you can have in the game and with that I see this as lost in profits and reputation not only for my game but Unity as well. Again a 64bit labeled engine should be 64bit all the way since 2015 when it first came out. And it is shameful that Unity leaves it to the developers to find this bug after scratching our heads why we see joker like textures on some game objects. It's just horrible to say the least that you guys didn't fix this yet, without telling developers to do many extra steps because you are not willing to fix your engine and leaving this 32bit limitation as is.
    explorer_HLel03tx4y.png
    this one is 4.03GB meaning so close to be okay, but I need to know what is exactly in this file.
     
  9. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,588
  10. tawdry

    tawdry

    Joined:
    Sep 3, 2014
    Posts:
    1,356
    Well finally read all the posts in this thread.So just to summarize that I know the limitations. Just seperating a bunch of asset bundles into file sizes less than 4gb will not suffice as Unity will still bunch them all together in a scene file if you have only 1 scene(resources.assets.resS). So have to do multiple scenes which will seperate this otherwise large single all encompassing file (resources.assets.resS) into multiple smaller ones each the size of the total assets in each scene.So for instance a game of 15.99gb would need at least 4 scenes with each scene having no more than 4gb worth of content to avoid the issues .?
    Is this right or am I still misunderstanding things?

    If this is right loading one scene of say almost 4gb while unloading another similar sized scene would this cause an issue if a some point the limit of 4gb is exceeded during the process?
    Thx
     
  11. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,588
    I don't believe that's how it works.

    As far as I know, you have to make sure that assets, which are pulled into a scene as dependencies, are stored in separate asset bundles, where each bundle is smaller than 4 GB.

    If you did split the scene content across various asset bundles, then you could also just have a single scene, because the scene asset bundle will only contain the scene file, but not the assets that it's referencing.

    That's how I used the system and actually never had any of those issues.
     
    karl_jones likes this.
  12. tawdry

    tawdry

    Joined:
    Sep 3, 2014
    Posts:
    1,356
    There was mention of uncompressed asset bundles loading faster than nn compressed. Is the compression only for the stored size on disk or does it also effect the ram amount the asset will use?
     
  13. tawdry

    tawdry

    Joined:
    Sep 3, 2014
    Posts:
    1,356
    Hey Peter
    Well that doesn't sound so bad the way some people are going on in this thread it would seem its the end of the world.Totally agree Unity should warn people about the 4gb limit though.
     
  14. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,588
    In my experience, it depends on the target platform. Ultimately you have to profile your game with each compression type to find what compression yields the quickest loading time for your specific game/content on a particular platform.

    For example, LZ4 compressed bundles load significantly faster than uncompressed on some Console for me.
     
  15. radiantboy

    radiantboy

    Joined:
    Nov 21, 2012
    Posts:
    1,633
    I think one of the major problems is, if you exceed that limit you have a huge game. And tagging each asset manually is impossible (my game has 907,000 files). We need some automatic solution, if it exists I cant find it. And then we need a way to specify limit of asset bundle (1 or 2 gig would be good). Then all of our worries would be over :)

    ps, if you ask me asset bundles, addressables etc are still mystified and require scripts and adjustments to existing logic, feels kind of scary when your game works fine currently. Ive watched a ton of videos but it still feels like there should be a place for automation in here, would be a great tool. I say this of course again because my game is insanely huge, would be easy for a pacman clone.
     
    Last edited: Mar 6, 2019
  16. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,588
    You can also tag directories. This makes it easier to add content to asset bundles, unless the project folder is unorganized of course.
     
  17. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,222
    You can automate Addressables. Adding, Removing, Loading etc can all be done through a script. We do it in the Localization package. If you create a new thread in Addressables and explain what you are trying to do then I will be happy to offer some help.
     
    radiantboy and Peter77 like this.
  18. gghitman69

    gghitman69

    Joined:
    Mar 4, 2016
    Posts:
    93
    HI who could do a youtube tutorial on the multi scene please
     
  19. MadeFromPolygons

    MadeFromPolygons

    Joined:
    Oct 5, 2013
    Posts:
    3,967
    loading in multi scenes additively is nothing new, its been around for years and there are plenty of tutorials on this. Look up "load scene additively unity".
     
  20. Zullar

    Zullar

    Joined:
    May 21, 2013
    Posts:
    651
    2018.3.0f2 I have 2 bugs occurring that may be related to this issue.
    1: In my build (not editor) some of my texture sprites get scrambled (3 sprites out of ~1000)
    2: In Unity editor I cannot change the import settings of some textures (i.e. if I change max size to 1024x1024 the texture will remain at 4096x4096). This bug has happened to maybe 20% of my textures.

    Screenshots here:
    https://forum.unity.com/threads/unity-ui-sprite-getting-scrambled.643435/#post-4321876

    There are other people having these exact problems as well.

    These 2 bugs seem unrelated. The strange part however is that they are fixed in the same way. If I copy + paste the texture in windows the newly created copy will work just fine. Then I just delete the old original bugged out texture.

    Unfortunately my project is massive and it would take a long time to repair all the "corrupted" textures in this way. Plus I'm not sure what caused the bug and my project may just get corrupted again.

    My resources folder is 11GB and my build data is 4.9GB

    I'm really panicking about these bugs!
     
  21. Zullar

    Zullar

    Joined:
    May 21, 2013
    Posts:
    651
    My textures are also garbled. Replacing them (a copy + paste + delete the original) fixes the garbling.

    Unfortunately there are 1000's of textures and doing this breaks their links... unsure what to do.
     
  22. Zullar

    Zullar

    Joined:
    May 21, 2013
    Posts:
    651
    Here's some examples of corrupted/garbled textures.

    1: In the build textures get scrambled (they are fine in the editor)
    2: In the editor some textures cannot have "Max Size" set.

    EditorMod.png
    BuildMod.png
    1024 4096.png
     

    Attached Files:

  23. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,222
    Please file a bug report. There were some changes to how the texture import settings were stored that could be the reason the max size is not being honoured.
     
  24. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,588
    Have you checked if it's overridden in the platform tab? If that's the case, any change to "Default" won't change it.
     
    Zullar and karl_jones like this.
  25. Zullar

    Zullar

    Joined:
    May 21, 2013
    Posts:
    651
    I feel dumb. This fixed it (max size was being overridden for PC build). Thanks for the help.

    I'll bug report the scrambled textures.
     
  26. ShilohGames

    ShilohGames

    Joined:
    Mar 24, 2014
    Posts:
    3,015
    When you build your game, check the asset files in your data subfolder of the build. If any single file is over 4GB in size, then you will suffer from the 4GB limit/bug. When that happens, some assets in the build won't be addressable and you will see some garbled textures when playing the build. The same game will run fine in the editor, though. The 4GB file size limit applies only in builds.
     
    Zullar and karl_jones like this.
  27. Huxii

    Huxii

    Joined:
    Sep 15, 2018
    Posts:
    36
    Hello, I have a huge szene and was trying to bake it. It take about 17 hours and then I got the error message, that I have exceeded the 4 GB described above. Would it be possible, that Unity detects on the beginning of the baking process that there is such a huge file an informes me about that at the start of the baking and not at the end? It would save a lot of time.
     
  28. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,222
    Please file a bug report regarding the warning.
     
  29. psych77

    psych77

    Joined:
    Mar 1, 2017
    Posts:
    55
    I just ran into this and was shocked, i spent 2 years with my project, and now when scenes are close to finished this issue happend. Wish i was informed about this ahead... specially now when i spent all resources on unity :(
     
    Velo222 likes this.
  30. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,869
    I can't believe this is still happening in 2019.
    A profesional 64bit game engine should be able to manage more than 32bit worth of ressources.
    This is ridiculous.

    And telling people to use overcomplicated asset management systems because Unity isn't able to do it by itself is much more ridiculous: that's one of the main reasons we use game engines, so we don't have to manage that ourselves.

    Having more than 4gb of assets should be managed by Unity INTERNALLY.
    And don't come now telling people to use assetbundles and addressables (which are assetbundles in disguise) because that's a lot of trouble for the user for something that should be managed by the engine in the first place. (adressables are not really integrated in the engine. They are not treated as regular assets (you can't replace a texture in a material by an addressable) and must be managed by code)

    We just expect Unity to be able to handle more than 4gb of assets by itself just like it manages the first 4gb. We are in 2019 and I think this is a valid expectation for a user of the best game engine in the planet.
    4gb is the amount of RAM memory a Nintendo Switch has nowadays, 32bit address limitations are anachronic in the XXI century.

    We don't care "it's complicated", everything is complicated! Making a game engine is complicated!
    Don't hide a bug behind workarounds, implicitly telling it's the users fault for not doing the engine's work and don't magically knowing this limitation exist in the first place.
     
    Last edited: Mar 25, 2019
    blueivy, MP-ul, radiantboy and 2 others like this.
  31. Huxii

    Huxii

    Joined:
    Sep 15, 2018
    Posts:
    36
    Hello Karl,

    Ok, I will open a bug. One more question. I would like to see, what is so big in my szene, that it causes the file being as big. Unfortunately I cannot open the file to analyse it. Do you have a solution for how I could have a look into the content of this file?
     
  32. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,222
    The build report should contain info on the various asset file sizes. Here is a UI for reading it https://assetstore.unity.com/packages/tools/utilities/build-report-inspector-for-unity-119923
     
  33. tawdry

    tawdry

    Joined:
    Sep 3, 2014
    Posts:
    1,356
    I got a question here if your project is lets say 10 GB big and you don't use either asset bundles or the resources folder is everything lumped into 1 huge file ie 10GB+ or how would unity handle this ?
     
  34. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,222
    It really depends on how you use the assets. They will be put into various different assets files per scene, shared etc. The best way to see is to do a build and see what files are generated. The Build Report tool should also provide more info https://assetstore.unity.com/packages/tools/utilities/build-report-inspector-for-unity-119923
     
  35. tawdry

    tawdry

    Joined:
    Sep 3, 2014
    Posts:
    1,356
    Well if you have just a single scene as in an open world game would the entire scene be saved in to one large file.
     
  36. stonstad

    stonstad

    Joined:
    Jan 19, 2018
    Posts:
    658
    This is a rant and Unity is probably not going to like what I have to say. First, I love Unity. Second, the way this issue is being handled needs to be seriously reconsidered. The bugged behavior plaguing this thread (exceeding a 32-bit pointer) has existed since at least November 2016 and during this time it has clearly impacted tens, if not hundreds of developers who have lost meaningful productivity diagnosing the bugged player. I can't reclaim my three days of lost productivity, therefore I recommend we dock @karl_jones pay until until a simple no-nonsense build warning is sent to the console (/joking... but not really).
     
    Last edited: Apr 15, 2019
    blueivy likes this.
  37. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,222
    I'm just a developer who as well as doing his normal job(fixing bugs etc) also voluntarily helps out on the forums. I have no connection to the 4gb bug, I just feedback to the teams and sometimes offer to help them, when I can. I consider myself a voice for the community, I try to take the communities side on issues and I still often bring up this issue internally, I'm working to find a solution.

    Unity developers participate on the forums voluntarily and as a courtesy, because we want to help. This kind of behaviour dissuades us from doing so, and the assumption that the people trying to help you are somehow personally responsible for the issues facing you is incorrect. Unwarranted personal criticism is unacceptable, please remember to be courteous in the future.
     
  38. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,869
    I think stonstad was just joking, but I understand your concern.
    Shooting the messenger has been an old human tradition since forever but it has never been right.
    I'm sure no one is really directly responsible of a limitation like this except the CTO.
    Any suggestion as how we could get him to understand this is an IMPORTANT issue without pitchforks and torches?
     
    karl_jones likes this.
  39. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,588
    I think the number of bug-reports are used as indicator how much pain an issue causes, which implies how important it's to fix (link).

    If Unity Technologies have no bug-reports for such issue, they are not really (or officially) aware of this problem, as the forum is not where they track issues.

    Everybody who thinks it's a must-fix should submit a bug-report as described in this document:
    https://unity3d.com/unity/qa/bug-reporting

    It's important that you report these issues together with a reproduction project if you want them to get fixed. If you don't do it, it might be a long time until someone else reports them or until Unity Technologies find them.

    Submitting a bug-report is more effort, but a quality bug-report leads to a resolution more often than forum posts.
     
    karl_jones and richardkettlewell like this.
  40. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,222
    He added the joking bit In an edit ;)
    As I said this issue is still being discussed internally and we are still looking for a solution. Ill keep you informed when I have something more concrete.
     
    atomicjoe, JBR-games and Peter77 like this.
  41. Rich_A

    Rich_A

    Joined:
    Nov 22, 2016
    Posts:
    338
    If you want a bug fixed - make noise.

    The noise on the prefab issues was deafening - every time Unity uploaded a Youtube video, the comments were filled with requests - and so the wanted changes were implemented.

    If you really want change, crash a keynote with a sign saying 'fix all bugs before features'.

    Squeaky wheel gets the grease.
     
    Velo222 likes this.
  42. LukeBryan93

    LukeBryan93

    Joined:
    Dec 3, 2018
    Posts:
    17
    I think i may be effected by this bug also. Here is the error im getting. https://imgur.com/tXVCIDm Im only getting it on the seats and no where else in the scene. My project is simple. Literally 1 scene with assets and lights. its nothing crazy. When built its only around 5.5gb. Really need to figure out what is causing this. Any help is greatly appreciated
     
  43. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,222
    If its all in a single scene then that sounds like you are over 4gb to me. Try examing the generated files, are they at 4gb? What does the build report say? Try reducing the size of that one texture that is causing you problems.
     
  44. LukeBryan93

    LukeBryan93

    Joined:
    Dec 3, 2018
    Posts:
    17
    Hi Karl, Thanks for replying so quick

    The build of my project is 6.74gb, the biggest file i have is a stereo video which 6.5gb and is used by AVPro also the video is in my streaming Assets folder. So overall the size of my project without the video is only 196mb's...

    As a test i created a new project and just imported my seats and applied the texture and did a build. and it worked fine. Im baffled
     
  45. stonstad

    stonstad

    Joined:
    Jan 19, 2018
    Posts:
    658
    karl_jones, I wish you no ill will and I appreciate your contributions to the forums and Unity, truly.
     
    karl_jones likes this.
  46. ShilohGames

    ShilohGames

    Joined:
    Mar 24, 2014
    Posts:
    3,015
    One thing Unity should immediately do to address this is simply add a warning message in the editor when the build is affected by this bug. Nobody is expecting Unity to actually fix the underlying bug/limitation in three years time, but we all deserved to at least get a friendly warning message. The lack of a warning message is what causes so many developers to waste a bunch of time on this bug.
     
  47. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,222
    StreamingAssets don't contribute to this issue so I would say something else is going wrong. Perhaps the UVs on the mesh are wrong? Have a look through the log for warnings or errors that may give you a clue.

    :D
     
  48. karl_jones

    karl_jones

    Unity Technologies

    Joined:
    May 5, 2015
    Posts:
    8,222
    We do have a warning message but there are circumstances where it's not being triggered. This is going to be fixed.
     
    Peter77 likes this.
  49. stonstad

    stonstad

    Joined:
    Jan 19, 2018
    Posts:
    658
    Very relieved and glad to know a warning message is planned. I can confirm no helpful warning message is shown in the latest version - 2018.3.12f1 with a vanilla player build.

    To Unity at-large (am not addressing anyone specifically) here's the death toll for this undocumented behavior. There are 40+ other developers who were clearly impacted but they didn't include how long they banged their head trying to diagnosis the cause of texture corruption in built players.

    Textures messed up in Windows Build
    https://forum.unity.com/threads/textures-messed-up-in-windows-build.440675/
    Nov 11 2016 - Nov 14 2016. 4 Days

    Texture problem : Graphical Corruptions - Need Help
    https://forum.unity.com/threads/texture-problem-graphical-corruptions-need-help.395985/
    April 5, 2016 - May 15, 2016. 40 Days

    Graphics Corruption
    https://forum.unity.com/threads/graphics-corruption.659359/#post-4432159
    April 10, 2019. 3 Days

    November 15, 2016. Multiple Days

    April 4, 2017. Multiple Days

    Aug 21, 2017. 3 Days

    August 22, 2017. Multiple Days

    October 30, 2017. 1 Week

    Jan 26, 2018. Multiple Days

    April 23, 2018. Multiple Days

    Dec 17, 2018. 7 Days

    Jan 25, 2019. 2 Days

    Jan 27, 2019. 30 Days

    March 25, 2019. Multiple Days
     
    atomicjoe, Rich_A and karl_jones like this.
  50. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,588