Bug: 4GB limit to Textures in standalone build

Hi Karl

Update. I found the issue causing the strange texture issues i was getting. I only ever get the error when i mark tick static batching in the static drop down menu.

This is a short term fix but realistically i need static batching turn on for better performance in my scene.

Have you heard of this issue before?

 

@karl_jones I appreciate that you talk to the community out of your free will, but we are paying customers and not just a groupie fan base of admirers for Unity. Can you please reach out to the people working on build components or whatever department that is, and tell them to join this forum and answer some questions of paying customers, I'm not happy with a error message but I want a fix for this and not a fix were I now have to do 5 more steps each time, this thread here is since 2016. Unity is promoting that their engine is 64bit since 2015 which is false advertising and not to mention the lost productivity and therefore missing revenue all developers had to endure here. I again have to fight in my game to reduce some textures or god knows what, I'm at the limit where I cannot give my paying customers anymore content because of this and it's a pain in the butt to address this issue every time. Unity should label it's engine 32bit until it has fixed and converted everything to true 64bit. It's a pattern with this company to be honest. This file limitation is one of the most important issues and in 3 years since this thread started you guys could not fix it? We need an ETA when this will be fixed. And to all saying we should write a bug report I have done so last year Feb 28th. 2018 and got an answer from Unity as follows.

Hello,
Thank you for this submission.
As you can see, the development team is aware of this limitation.
The fix is being considered, but not prioritized since we launched the Asset Bundle system and that one is generally better suited for asset handling.
We apologize for this inconvenience, but at this point in time, you will have to find a workaround that suits the needs of your project.

This sums it up, with other words "we do not care fix it yourself, but don't forget to pay your subscription on time "
This is beyond acceptable.

 

Just tried all this again, and re-realised that asset bundles themselves also have the 4gig limit.. so how are you supposed to get around it again?. Isnt there a way to split asset bundles into much smaller ones? Getting pretty rough, not seen a build in years. I really do not want to break my scene up into smaller ones.

[Error] (3) Serialized file size of 5.08 GB (5458044572 bytes) exceeds maximum. File name: 'C:/GD/GD/Temp/StagingArea/Data/BuildPlayer-GD.sharedAssets'. Serialized files over 4.00 GB (4294967295 bytes) cannot be loaded by the player. Some likely ways to reduce this are utilizing asset bundles, re-balancing asset locations, or limiting their serialized size e.g. limiting the maximum texture sizes.
0x00000001414141B8 (Unity) StackWalker::GetCurrentCallstack
0x000000014141A666 (Unity) StackWalker::ShowCallst

But I may split it up in the end, so long as I can keep the main mechanics in a sort of shared bit... I wonder if the game would speed up again too.. just seems inconvenient having them in different scenes to be honest, not to mention load times when going between scenes.
...

 

Split your content into more bundles.

I believe it good practice to group assets into bundles by type and purpose. For example, if I have a "dog" in a game, I would split its content as follows:

Bundles

• dog_audio (no dependency)
  
• dog_textures (no dependency)
  
• dog_model (+animations, +materials, dependency to dog_textures)
  
• dog_prefab (dependency to dog_model (implicitly dog_textures) and dog_audio)

Depending on the number of animations, it can be from advantage to move animations to its own bundle too.

I store every scene in a separate bundle too.

The advantage is:

• Smaller patch sizes. If you're planning to release additional content or patches, this is important. Some console vendors have strict guidelines, it's simply not accepted to build a game with "a single bundle" on some systems.
• Build times reduce significantly. Unity supports incremental building of asset bundles. If you change a dog texture, Unity needs to rebuild the dog_textures asset bundle only, rather than everything. If you work on a larger game than the most trivial stuff, you really want to split your content across multiple bundles to benefit from the incremental build.
• Finer memory management, because you can unload bundles. Think of a menu where you have a prefab of all your 3d characters.

 

thanks, but i dont know how to actually specify the size.. asset bundles confuse me entirely, i dragged in my main scene into the bundle browser, after a long while it added a ton of files, and it says size 71gig. No idea where 71 gig comes from, surely you can drag a scene in like this? And yeh how to specify size?

Getting mighty sick of this whole affair to be honest.

 

I've not used the AssetBundle Browser yet, thus I'm not in the position to provide help for this tool. However, I think it just uses the existing asset bundle system under the hood, so I try to explain how to set those up and perhaps it helps you with that tool.

You can think of an asset bundle like of a container, such as a zip archive. It contains a bunch of files and can be either compressed or uncompressed. What can be confusing is what assets end up in an asset bundle.

If you add a scene to an asset bundle, that is, if you select the scene and specify a name in the "Asset Bundle" drop-down field in the Inspector, Unity creates an asset bundle that contains this scene if you call BuildPipeline.BuildAssetBundles().

By default, Unity also pulls in all the assets that the scene depends on: its dependencies. For example, say you have a scene that contains a model. The model uses a material. The material points to a texture.

If you put this scene into an asset bundle, Unity resolves all its dependencies and includes them into the asset bundle. In this case, the scene asset bundle becomes a container for the scene file, the model, the material and the texture. That is, because the scene uses (depends on) all those assets.

Now you want to make sure that not every single dependency gets pulled into the scene asset bundle, because:

1) The asset bundle can get really big, if the scene contains a lot of different assets.

2) Imagine you have two different scenes/levels, lets say "trainyard.unity" and "harbour.unity", which both share a lot of assets. Since Unity pulls in all the dependencies, all shared assets end up in each scene. Thus, you have a copy of the same shared assets in both scene asset bundles.

This is where storing assets to separate asset bundles comes into play. So rather than relying on Unity pulling in all the dependencies for you, you create separate asset bundles that contain those assets (dependencies) instead. Lets take the example with the cube with material and texture in a scene again.

If you add the texture+cube+material to a separate asset bundle named "stuff", Unity is then smart enough to understand that it must not pull in a copy of those assets into the scene asset bundle again. Instead, it knows "I need those assets, they're in this other asset bundle, so I depend on that bundle then and the game developer makes sure to open this bundle for me".

Now the scene asset bundle depends on the "stuff" asset bundle and it's your responsibility when you load the scene asset bundle, to open the "stuff" asset bundle first, because the scene asset bundle depends on the asset in the "stuff" bundle!

The ugly and difficult part is to untangle all those dependencies. It's very easy in Unity to create assets and asset bundles that depend on "almost the entire project", because there exist no good tools that show those dependencies. This was always the tricky part for me, split content into pieces that have little to no dependencies. I guess that's what the AssetBundle Browser should solve.

I hope I was able to explain how to use asset bundles.
Karl, don't hesitate to correct me if something in my explanation is wrong.

 

Just ran into this problem today. I had no idea it was even a problem or a limitation. I can say, at least, that after the build I was given an error warning -- which is great. At least I know what the problem is now.

However, this seems like a pretty dumb limitation. I get it's a leftover limitation from Unity's past days, but this is a HUGE hurdle in simply getting my large game to build correctly, and I'm not happy about it at all.

I have never used asset bundles, and quite honestly, they scare me a bit. Also, my large scene uses a massive one-piece terrain -- how do I split this into "multi-scene". Or can I keep my large terrain in one scene, and break up my other assets into other scenes? I honestly have no idea how to do that -- and I'll probably have to spend weeks learning and then implementing either a multi-scene solution or an asset bundle solution just to get my game to build...…..this is VERY frustrating!

I'm glad it's a known issue and people might be working on it at Unity, but this seems like a REALLY SILLY limitation in 2019 as other people have eloquently pointed out. This seems borderline unacceptable -- but something we're forced to accept.

 

Can I place textures from my project into asset bundles, load the asset bundle at the very beginning of my game (when the game starts up), and the scenes in my game will recognize them automatically?

For example, say I have a "Starting Screen" (the first scene loaded on initially playing the game), and a main game scene with my large terrain -- with textures on my terrain. If I place the textures in an asset bundle, and simply load the asset bundle from my Starting Screen scene, will the terrain in my other scene be able to auto-recognize the texture references somehow? Or how does that work.

Do I have to somehow tell the terrain in my main scene how and where to find the "loaded from an asset bundle" textures?

 

@Velo222
Hello do this test. add 10 blank scene in the build this will separate your assets and so the build will pass
(I had this problem map giant 40go the project and I have not finished , I divide my card in 65 scene at the beginning I additive load and then I merge them all that gives me a large map and no problem to the build)

 

I'm struggling with the same issue as @Velo222. My scene file is just over 4gb. Luckily for me, we are already using asset bundles, but only for gameobjects we are creating at runtime and so not all the static buildings in our map. What is the recommended workflow to split up a large scene and load it from asset bundles? Can I just pack everything up and throw it in some asset bundles and Unity will do the rest? Or do I actually need a way to spawn the buildings from the asset bundles into an empty scene?

 

We seem to be in the same boat as a lot of users here. We just want to use addressables to reference assets and prefabs in our already existing scenes to split up the resources file sizes. What's the recommended way to do this? We don't really instantiate anything at runtime.

 

@ccsander , your solution from first page link is brilliant. I just use Texture2D array instead of transform array in PrefabHolder script, because textures are the larges assets in my project. I can fill the array using editor script so no drag and drop with mouse . Thank you very much.

 

im still waiting for someone to automate all of this

 

In 2019 and we are limited to 4 GB... c'mon...

And I only find this out after 1000+ hours of dev, and whole weekend troubleshooting extreme fluorescent-gamma build wondering why my game suddenly broken.

Now I know. Great.

 

This is pretty lame... Have been working on this demo for the last 12 hrs and finally got it ready to build and I get the 4gb serialized deal... Why not just make an option to break up assets, it would probably be better for multiple files and threading anyways right?

 

That is exactly the solution ccsander found, and it works like a charm.
I really don't know why Unity doesn't make something like this from the get go
They probably don't care enough about this and think that if you are making something that big, you should use addressables or assetbundles instead.
The thing is, assetbundles are a pain in the butt and addressables only care about HOW and WHERE to download from in the context of online ressources. That's really, REALLY overkill when you only want to have a scene which manages more than 4GB of assets in it.

 

This is just a mess. Still no clear error messages or warning if you hit a limit.
It makes importing assets and managing them in projects extremely tedious when it should be as simple as copy paste.

Using Asset bundles as a fix for this is a bodge - you are working around a 32 bit limitation because its too hard to fix.

In 2019 the old answer as to why you won't move to 64 bit of "We are worried about backwards compatibility" doesn't cut it any more. Unity is showing its age. I would say texture bundles of LESS than 4 GB are going to be the exception rather than the rule unless you're developing for mobile. I'm trying to render and configure planets with very large textures and don't want to be troubleshooting texture limits or having to chop then up when I'm still in the visualisation stage. Try a 86400 x 43200 x 3 = 10.4 GiB
Hardware is there. OSes are there. Other game engines are there. Why isn't Unity there?

 

@karl_jones Hi you do not see the limit to your 32-bit engine (you push the problem later but prendrer you behind on others it's about) why limit the power of labor while the 32-bit doom is to disappear all OS are 64 bits

 

In the Copenhagen road map discussion a slide shows they intend to lift the 4GB asset file limit in Unity 2020.x. This is good. Not sure why it isn't a higher priority, as it is surely a dumpster fire.

 

OMG in 2020 haha it takes them more than 5 years, when it 2020? (last version probably) just wow, they should address the floating point issue which is 32bit as well soon like in 2030 maybe, just imagine before 2050 we will be on par with Unreal 4 engine cannot wait to have my hands on it when I'm retired, fingers crossed I'm still alive.

 

There is no issue: floats are 32bit variables, but maybe you mean using double precision values for everything inside Unity instead of floats? Well, using 64bit variables internally would give way worse performance, specially on mobile and consoles, but also on PC.
Also, consumer GPUs have a pitiful performance with 64bit variables, and that's when it's actually supported, because not all of them do.
So using 64bit values internally would tank performance and break compatibility.

 

I meant the 32 bit floating point limitation of the Terrain and this is an issue, when we are not able to create big open worlds without having our characters and the environment start to act up strangely, from shaking to texture glitches, I do think this is an issue and no the repositioning of the center is not always a solution. A double-precision (64 bit) floating-point data type for terrains would help for sure, performance issues should not happen if used responsibly and it should be a smart approach where there is no limitation in other areas of the engine as well. Apple is dismissing 32bit games now. And lastly my point was the big delay of the 4GB resource file limit fix, a 64bit engine should not have any 32-bit limitations especially when Unity is flashing with new features all the time but not able to deliver a true 64bit engine.

 

Really, if you are using a terrain that big, making it using multiple terrains is a must for performance. And as long as you use multiple smaller terrains instead of a huge one, center repositioning is THE solution.
Actually, every open world you have played uses this technique (except Kerbal space program, but that's because they wanted to maintain astronomical distances acurate)
What I mean is, if you make a large world in a single terrain, you will start having lots of other issues, not just precision ones.
Don't get me wrong, internal 64bit precision would be nice, sure, but you would still need to split huge objects like terrains into smaller ones.
What you should be asking for instead is a native streaming multi-terrain solution to replace the current terrain system. THAT would be nice.

 

THANKS!!!! Im still trying to work asset bundles out LOL.. really should be able to feed a scene in and it auto makes bundles imho.. this 2020 fix will be nice though I havent been able to make a build of my game since about 2015... that's 5 yrs of only testing in the editor.. I hope to see some speed increase in a standalone build over editor It would be great if the res files for each scene could just come out in say multiples of 4 gig chunks, so you could have say 5 x 4gig res files for one scene.. we can but hope, but I appreciate the work guys.

ps. Wow I will be able to send my friends a build 8 yrs of work and no one has played it yet haha.

 

not a single built since 2015? wow, you'll have some suprises when you build

 

Hi, if i understand right, there is a limit of 4GB to each scene, on unity...
Does this limit is still to 4 ? On today 10/10/2019 ?...

Is there any exceptions ? Solutions ? That we could increase this limit ?

weex

 

Yes.

Asset Bundles.

 

Hmm Thanks. But i guess i'm not enaugh good on unity, and my Padawan even... Any link ? Tutorial ? About Asset Bundles... ? Is it a way to permit to avoid this 4GB limit ? That you can have more then 4GB finally in one scene ? By puting several scenes, in a scene ?
Excuse my compare but, a little bit like... Afer Effect ? Nodal approach ?

Thanks for your enlightment !

weex

 

Forget assetbundles, look at this post for a solution.

 

You just can't forget about then
A) If you want to provide content patches for your game and you don't want your users to download most or all of the content again and again, the solution is asset bundles.
B) If you plan to release on (some) console, you must use asset bundles.
C) If you care about build times. Using asset bundles significantly reduces build times, because of its incremental build feature.

 

That's funny because every time I download an update of a game, it weights as much as the whole game, so that doesn't seem to be a concern for anybody...

Now that's interesting.
I know Google and Apple force you to divide your app in small chunks, but didn't know this was the case with consoles.
What console are you talking about?

 

ERM... : it will NOT be for console, ... but cloud gaming. We NO need a download...

So ?

And I would complete my comment here by saying, that this limit of 4GB nowdays is a pure joke, Honeslty ? 4Gb ? Seriously ? ...

We are entering into the era of COULD GAMING, and VR...

What do you to do with 4GB ?...

A bit depressing, Unity, no ? Unreal is the same ? I doubt about it...

 

So the Unity official answer is: use asset bundles and/or adressables
But if you read the docs and find it overkill, you can split your content over several scenes and call it a day.

 

mhmm Sad thread, 25/10/2019 unity is still not able to do this after 5 years , will that thread get fixed one day ?
I have bigs hope with that you said karl_jones hope it will be fixed in 2020, that should even be a priority, but i guess some other stuff are more important.

 

So I am still learning... But from what I am reading Unity cannot handle large scenes with lots of assets. But you can manually split it into smaller bundles and everything will work fine...

WHY COULD THEY NOT HAVE SIMPLY EDITED THE AUTOMATED BUILD OUT TO DO IT FOR YOU?

 

Because they thought 4gb ought to be enough for anyone.
And when faced with the fact that it actually wasn't, they just told everyone to use assetbundles instead and just brushed it off under the carpet.
Assetbundles by the way weren't designed to manage this, but as a way to make on-demand DLCs

 

Just wanted to apply for a place in the "I also spent amounts of time trying to figure out what was glitching graphics in my built" club... As @atomicjoe pointed many times, seems like @ccsander solution -page 1 of this same thread- is a fast patch to it, but from now on I will be in fear that as my project grows, sooner or later this way of doing will turn out as noneffective or even be mark as counter-effective in the big scheme of things...

It is true that sometimes a message appears in Unity console warning about the 4 gb limit; but others don't -it didn't happened to me at first for whatever reason-. Somehow I see it related to the fact of calling many resources through runtime or not; but I am not sure.

 

Well, it's supposed to be fixed sometime in unity 2020, so you shouldn't be that concerned.

 

Indeed. But would not be the first time a date is pointed and nothing significant comes after it. We have examples of that in this same threat. In any case I hope that by the end of the year I have advanced significantly enough in my project with this kind of setting "behind the scenes" -free punt there, huh-; so as many other users here, I can't put me and my project in standby until something finally comes.

In any case it's also true that facing this issues it's always a chance to learn new workarounds about optimizing, which is always good, isn't it?

 

Oh, if that's the case, we're set to learn A LOT using Unity...

 

still happening in Unity 2020.1.0a11 (64-bit) guess it's not implemented yet

 

HERO! Cant thank you and the Unity devs enough Karl, thanks so much, that is the xmas present ive wanted for years!!!! This is going to be AWESOME!!!!!!!!!!!!!.... Thanks again, if you ever pop by Taiwan (where im deving this), I will gladly buy you beers all night.. So happy

 

Thank you for your support and advocacy!

 

Great present thanks!

 

warning guys, upgrading to 2020 alpha 17 crashes my project when it loads.. i was just upgrading to this one before 19. so id advise backing up or not upgrading just yet. im now trying to go back to 2020 v12

appears to be

InvalidOperationException: EnsureRunningOnMainThread can only be called from the main thread
at UnityEngine.Object.EnsureRunningOnMainThread () [0x0000d] in <dd7258062d3a4670ac6a983c4a6697a5>:0
at UnityEngine.Object.GetInstanceID () [0x00001] in <dd7258062d3a4670ac6a983c4a6697a5>:0
at UnityEngine.Object.IsNativeObjectAlive (UnityEngine.Object o) [0x00035] in <dd7258062d3a4670ac6a983c4a6697a5>:0
at UnityEngine.Object.CompareBaseObjects (UnityEngine.Object lhs, UnityEngine.Object rhs) [0x0001d] in <dd7258062d3a4670ac6a983c4a6697a5>:0
at UnityEngine.Object.op_Inequality (UnityEngine.Object x, UnityEngine.Object y) [0x00001] in <dd7258062d3a4670ac6a983c4a6697a5>:0
at UnityEngine.Events.PersistentCall.ValidateTargetAssemblyType () [0x0000e] in <dd7258062d3a4670ac6a983c4a6697a5>:0
at UnityEngine.Events.PersistentCall.OnAfterDeserialize () [0x00001] in <dd7258062d3a4670ac6a983c4a6697a5>:0

 

will try again but I tried 3x, the bug reporter crashes, it nearly always crashes for me (on all versions of unity ive used for last few years) which is why i dont submit bugs anymore. How ironic

edit: sent to the email address instead, cheers.