Using Job system for I/O in a chunk/voxel system

I am trying to figure out how to convert my threaded chunking voxel system to ECS and Jobs. Is this possible to use the Job system this way? From what I've seen, every example of the Job System is using pre allocated memory.

Is it possible to use the job system to allocate an undetermined amount of memory? Or do I have to just allocate the max amount of memory per chunk and then maybe jobify the I/O of filling that memory? This is far from ideal but might still be worth the speed up gained.

 

You could have a manager which pre-allocates a few chunks, and maintains a minimum number of spares to send to the jobs whenever you run low. An occasional pre-allocation to top up shouldn't be too much of a problem, I think, unless the player moves through chunks REALLY quickly.

By "occasional" I mean any frequency less than every frame. Probably scaled based on the player's potential movement speed. Could be every second, every other second or rarer.

I think the job system is meant just for the actions, not making the workspace for the actions. One of the goals seems to be keeping memory use relatively flat. Jobs should be ready to go without delay, moved to whichever CPU core is available right away.

 

Okay so after watching a bunch of videos I think I have a better grasp of my problem.

The goal is to create a chunk/voxel system like in Minecraft and by far the biggest bottleneck in any code examples, sample projects, and even vanilla minecraft is the time it takes to deserialize and then allocate memory for a chunk and then to serialize and deallocate memory for unneeded chunks.

I think the ECS system is going to be really helpful in getting meaningful speedups in the allocation and deallocation of memory for the chunks. It sounds like a very daunting task to somehow convert everything in my chunk system to ECS but I guess I just gotta keep fighting it until I can figure it out.

The de/serialization aspect is more interesting to me though. I see two issues with this that are both likely due to my lack of understanding how to use it.

1. I would like to be able to do the RLE and de/serialization on a job, but how would I know how big my output array is? Is this not possible with NativeArrays and instead I just have to allocate the max size array each job?

2. The job system is looking like it wants to be pretty independent in the data or libraries it accesses. Is it going to let me access System.IO and if it does will it terribly break things? All the examples I've seen of the job system are very simple, similar to the CUDA I've seen, but I/O operations are trivial with threads. In line with this, is the goal to never use threads in Unity and only the job system or is the most "optimal" solution to jobify the RLE and de/serialization and then thread the I/O?

I actually already do this in my system, but using traditional Gameobject system imposes some hard limits on view distance, player speed, etc. that I think the ECS and Job system could alleviate.

 

Memory allocation you should use pools. Serialization you can probably get huge gains from just using something better optimized. Read up on varints, that's why protocol buffers is such a win for serialization of lots of small numbers.

ECS/job system is not where you are going to get the biggest wins. It's going to be things like optimizing your data structures and rendering.

 

I would love to figure out how to not have to use pools as per Joachim Ante presented briefly here: Link

I understand that if my main goal was to make the game I could make it work using current tech and lots of pooling, pre-loading, and game design constraints, but my goal is to learn the new ECS and Job system through a system that is "just a bit too complicated." Multithreading chunk de/serialization and de/allocation seems like a cool task to try it on.

You are right that rendering is likely going to be the bottleneck, and I have a lot more work to do to continue to optimize it.

Data structures and data layout on the other hand seem to be exactly why ECS is going to be so important for performance.

I've tried JSON and MessagePack serialization and they didn't work well for me which is partly my fault in not optimizing how I'm using them. Will try Protobuf and read into varints so thanks for that suggestion!

 

I think ECS is going to need a few live sessions, but they've admitted some parts are placeholders until they find better ways, so we'll be learning things which stop being true after a while. At least some of the changes coming will be for the better. For instance, right now there are some declarations and statements the docs/comments say will be going away.

I feel that it's still a tad early to start building anything big with ECS and expect it to last, but experimentation is fine.

 

The first year was rough, but now it's really compact