Feedback Managing the loading of thousand of objects in a large world

• Summary. My world currently sits at 44 terrain tiles by 44 terrain tiles, so that is 1936 "regions". Each region is sure to have hundreds of objects throughout. The challenge is to figure out a decent way to load the appropriate objects. I have already written the code to load in regions (the terrain) around the player, which loads and unloads as the player moves around the world. I could use this region ID with managing the objects as well, associating each object with a region ID.

• Region classes. I do not have 1936 different region classes, so it's not like I could just go into each individual region class and put the objects in there. That would be simple, but then again it wouldn't since I have would to manually go through 1936 classes.

• Easy design for placement. The approach that would be easiest for me from a design perspective would be to manually place objects in Unity and press a GUI button that appends the object info to a master list of objects (just some text or dat file). I then would delete the object in Unity. I would run a program to sort the master list by ID before using it in the game engine.

• Memory considerations. It would be simple to rip through the master list at runtime, organizing all the objects into classes or arrays or whatever method of access might be appropriate to use, then accessing this data with the region ID to get a curated list of the objects that should be loaded into a region, but this presents memory problems with there being so many objects, even if it's just object ID's and location information that would be stored. It also would be unnecessary to be storing all objects across the world when you're just in a minimal number of regions throughout an entire gameplay session. This is why an approach where I read from disk on an as-needed basis would be a better approach.

• Seeking to a specific position. This brings us to the ultimate problem of figuring out how to seek to a specific location in the master list, read x number of lines to extract the objects and their information, and stop reading at a specific line. I'm sure I could do this by pre-calculating a lookup table for region "start" and "end" lines to put that in what amounts to file header data, all done when I sort the master list (as described at the end of ¶3), but I'm wondering if there is a simpler strategy to doing this.

• Split master file up. I could avoid having to worry about file seek start and end points (and creating the lookup table that goes with it) by having 1936 different dat files, each containing the list of objects to load, and each dat file being named according to its region ID. The start and end positions would simply be the beginning and oef. But although this is a well-organized, easy to code approach, having 1936 files seems messy.

I wanted to ask for advice on what you think the best way to solve this problem might be before I go writing all the code for this. Thankyou in advance.

 

you already said the solution, save your stuff to disk and load from disk when needed, do it like minecraft does

dont know why you think its messy to have 2000 files, many games have millions of files

 

Well, I know it is a possible solution, but I wanted to know if my methods of accessing the data on disk might be considered unoptimized, unorganized, or maybe if there is some kind of file structure that could be utilized that would make targeting the region sections easier. Every time a programmer re-invents the wheel with a feature, it ends up making the codebase more verbose than what it needs to be.

As for the 2000 files, many large games pack their files into one. This saves disk space and allows the information to be read from disk contiguously.

 

May I ask how are you unloading objects ?
I got this problem that I have got one big level and I'm reconsidering some options to divide it into chunks.

 

It might make more sense to split the regions into separate files. This is especially true if the player won't have access to the vast majority of the regions at any given time. Load the file for the region the player is in and place all objects in that specific region while at a loadscreen. Load the region files asynchronously for the regions adjacent to the player. Place the objects as the player gets nearer to that region (but far enough away that their LOD culls them).

I'd go with this approach first since it would easier to implement. If you still find that the game isn't performant enough, the majority of the codebase is already written, you'd just have to modify it to accept a single file. Where instead of searching a folder for the .dat that is named the same as the region id, you'd search the master file for the start and stop text of the specific region.

 

to unload objects save their data to json, then clear all references and let the garbage collector clean up the classes that have no reference in the scene

 

What about assets, let just say that you loaded a lot of textures in one zone, if you destroy all objects in that zone on scene are this assets going to be unloaded from ram ? Also do we have to destroy ? Can we just disable GameObjects?

I was trying to find some info about this but failed.

 

use this for removing assets from memory

https://docs.unity3d.com/ScriptReference/Resources.UnloadUnusedAssets.html

you must clear the references to it 1st ttho

 

So there is no way to disable objects clean all resources not used right now and then resources will be loaded again when I enable objects ?

 

No, you either keep them in memory for faster enable or unload them and reload them when the time comes, you need to choose which one is more important for you. What is important that you need to think about dependencies too. You can't unload something which you are depend on at the same time, that wouldn't make any sense.

 

No exactly, mesh and textures on disabled renderers are not used, I don't want to destroy my GameObjects/Renderers when i just want to unload some assets from memory.

It should work like this, we should be able to unload any asset anytime. It will be loaded again the first time it is needed.

 

Technically you don't have to, so if you're hellbent on this, you can do it. The requirement to unload something from memory is not referencing it. Detach it from your game object and you're free to unload. This is how it works in C# and in Unity.

 

So if my material is not on renderer but I keep reference to it somewhere in my code, can I unload textures it is using ?

 

Again: no, if you're referencing something, then it stays. It's not that complicated. If you have a material, which is used by one renderer, if you detach the material from your renderer, then the material can be unloaded.
Same with ScriptableObjects, class instances, etc. The keyword: not referencing >it<. If you keep a reference to its parent and the parent referencing your object, that's still referenced.

BTW, what's your goal with this exactly? Why is it important to unload used textures but not materials? And why don't you cut them loose if you want to unload them? It's not like loading the material itself would take up that much time or something. The big chunk is the textures and you're willing to swallow that loading time, apparently. In the times of Addressables it is extremely easy to handle unload/reload on demand anything.

 

I'm planning to have a big level with quite a lot of graphical sets (regions based - forests deserts dungeons etc.).
I have got many scripts that are making use of materials that are on a renderes, I cannot just remove the material.
I might experiment with addressables a little but I have got already quite established pipeline and no so much time to make this jump.

Using addressables can I unload texture without breaking any references and it is going to be loaded when needed ?

 

No, it solves the problem you have when you have to break reference. It provides an "address" which can be used to call the proper files from the disk. So you can unload the thing without worrying about things, when you need the material, just call the Addressables and load it up.

 

It's something like new "Resources" as I see, this unfortunately don't help me much right now.

 

Sounds like you may need to consider some sort of object pooling system to recycle your GameObject and MonoBehaviour instances.

Aka, rather than unloading or destroying them consider how you might return them to a pool to be reused.

The devils are in the details, ofc, regarding how you hook into the disable and enable events. The Addressable API would be useful in loading and unloading associated references to manage memory.

 

No, pooling is a different thing.
Polling is more about saving time you need to instantiate new object.
My problem was about resources taking a lot of space in memory.

 

Have you considered maintaining your objects list with sqlite or something similar? Also are you really going to edit 1936 regions manually? Maybe you might want to make just a few and use procedural generation for the others?

 

just for reference, i guess mainly uses subscenes to manage big level areas
https://github.com/Unity-Technologies/Megacity-Sample

 

Right, generally they are different. My suggestion is to integrate the two things. That's my plan for my project. When you return to the pool you can release the memory of the associated resources using Addressables.

I haven't tested it yet, so I am curious about what others think.

https://docs.unity3d.com/Packages/com.unity.addressables@1.21/manual/runtime/MemoryManagement.html

 

It's not a good idea, pooled objects should be as ready to use as possible.
If you unload assets, acquiring object from pool will take to much time.