Performance overhead of DOTS systems

We just started the development of our new game using mostly DOTS. At first I was quite happy with the kind of performance I could have, especially when I had thousands of entities to process data on. But then, after we add more and more systems, a bunch of which only process data on queries of 1 or 2 entities, Something is becoming increasingly clear: there is gonna be a ton of systems, and they all have a non negligeable overhead. To the point where I'm starting to believe it would be more optimal to deal with MonoBehaviours instead.

So my analysis of the issue is this. For a really simple, run-of-the-mill, burst compiled JobComponentSystem that only schedules a job (See spoiler below), I'm getting a 0.007 to 0.01ms overhead on the main thread in a build. I know, it doesn't sound that bad. But I wouldn't be surprised that a finished game could have thousands of these. At 10+ms for the update loop alone, now it does look pretty bad. And that's on a really good CPU. I know that if you organize the queries right and they are empty, the system is not run. But even then, the check alone seems to take around 0.001 to 0.002ms, so kinda significant on a large scale too.

I also have a single query in this example. If I need, for example, to get a singleton or to add/remove components, that makes everything even heavier.

So my question is this: can we hope for that kind of overhead to be lower in the future? Or is this an intrinsic overhead we have to deal with in DOTS? If that's the case, I fear the performance by default claim would only be true for games that often have to deal with tons of similar entities or complex calculation. For the rest, it seems to be quite the opposite.

Spoiler: Example System

Code (CSharp):

1.     public class DoorInteractorIdleDoorAvailabilitySystem : JobComponentSystem
2.     {
3.         private EntityQuery idleEntityQuery;
4.        
5.         [BurstCompile]
6.         private struct SetIdleAvailableJob : IJobForEach<DoorInteractorAvailability>
7.         {
8.             public void Execute(ref DoorInteractorAvailability availability)
9.             {
10.                 availability.IsClockwiseAvailable = true;
11.                 availability.IsCounterClockwiseAvailable = true;
12.             }
13.         }
14.        
15.         protected override void OnCreate()
16.         {
17.             var idleQueryDesc = new EntityQueryDesc
18.             {
19.                 None = new ComponentType[] { typeof(DoorOpened), typeof(DoorAnimation) },
20.                 All = new ComponentType[] { typeof(DoorInteractorAvailability) },
21.             };
22.             idleEntityQuery = GetEntityQuery(idleQueryDesc);
23.         }
24.  
25.         protected override JobHandle OnUpdate(JobHandle inputDeps)
26.         {
27.             var idleJobHandle = new SetIdleAvailableJob().Schedule(idleEntityQuery, inputDeps);
28.             return idleJobHandle;
29.         }
30.     }

 

If you're only acting on 1 or 2 entities, you probably shouldn't schedule a job. When doing something on multiple threads there is always overhead, and it's not always worth it.

Also you shouldn't have thousands of systems in production, I don't think the dependency solver would handle that well, and that kind of granularity would be very hard to follow. You'd have to skip through hundreds of files to follow the logic of what's going on.

And with that many systems you're not getting the benefits of DOTS, because each new system is likely hopping over to a new spot in memory. What's the benefit of tightly packed linearly laid out memory, if you're just going to query for 1 or 2 entities at a time in a thousand different systems? You're essentially just randomly accessing everything at that point.

 

Afaik the system is indeed scaled to have thousands of systems. When there is no work to do the system should have zero overhead, but I suspect what you are pointing out is the overhead when there are some entitites (like a few).

I might be wrong here, so it would be interesting to hear someone from Unity to chime in here?

 

https://forum.unity.com/threads/ecs-code-overhead-for-mobile-games.690265/#post-4840793

 

I'm also kind of curious about this because my primary motivation for working with DOTS is it fits better with how I prefer to have my data and logic broken up, and if you're following the single responsibility design principle, you'll likely have lots of systems. Something to keep in mind though, is even if you have thousands of systems, do you expect all of those to be running every frame? If your EntityQueries don't have any any Entities, the job won't run (simplification of how they decide how they should update, but you get the idea), so I've been trying to be mindful of that when designing systems.

Looking forward to someday being able to see my FixedUpdate systems in the debugger though so I can at a glance see which systems are running and which aren't like I can with my default Simulation systems

 

There is indeed an overhead to an empty query system, as the check to see if the system is empty or not isn't free. But yeah, I do try to be mindful of that.

Also, I won't make systems which make a ton of unrelated stuff, because that will be a pain to maintain (single responsability principle) and I'm pretty sure it won't be more optimal anyway. I also try to schedule burst jobs as much as possible, because it seems to use the main thread less than waiting for the query job to end on the main thread, then executing your work on it.

Edit: The post about mobile does make me more confident about the future performance of DOTS though. I guess I will keep myself up to date about that.

 

What if you could group dots systems by data load and low load systems could be amalgamated, my theory being that the overheads for DOTS could maybe be mitigated or amalgamated for these systems?

Or maybe some of the overheads for DOTS could be reduced e.g. Multi-threading to Single-threading.

After all your systems are probably just a few lines of processing code.

Alternatively can you manually combine systems and just switch/case depending on a flag?

 

Unity has said the will support "many" systems, but I don't think they are talking about thousands. I've never heard anyone mention anything about scaling to thousands of systems. And as far as I'm aware no one using an ECS architecture (whether they are using Unity or not) organizes their game like that.

 

The Unity dev says right there in the above linked thread they expect about 60 fps performance on mobile with 1000 to 2000 systems.

 

You're write. Thanks! I hadn't seen that. I still don't think an architecture with thousands of systems makes sense from an organizational and performance standpoint.

 

Maybe once you get over about 20 systems you should re-analyse what they actually do as I would guess, you will start to see common atomic data operations. After all there are only about 23 basic math and logic operators.

 

First of all the beauty of ECS is that, it is relatively easy to replace multiple system, with one.
So I personally try to go with fine grained systems and only if I can find a measurable overhead, I start thinking about joining the systems.

I know and worked on Games, which used ECS with Single Responsibility Principal in mind and have big number of systems.
It was not Unity ECS, but from what I see and tried, Unity ECS should be at least on par.

From my experience, there are three techniques you can apply to reduce the overhead of a system call:
1. Early exit. The World calling an execute on a system should not be a huge overhead, however in the update it is better if you can avoid performing unnecessary work on every tick and identify if your system can return early.
2. Write better queries. A system iterates on a group of entities, try to design your components in a way that let your systems iterate only on "proper" set of entities / components. Best case scenario, your set is empty and the system has nothing to iterate on. AFAIK in this case the system will be marked as "not running" in Entity Debugger.
3. Think about grouping systems (ComponentSystemGroup) and disabling whole groups in cases where it is clear that in current state those system has nothing to iterate on.

That all sad, take my advice with a grant of salt and always profile .

 

From my experience some systems that that don't take hundreds of entities and are processed in IJobParallelFor (or equivalent), and require a post update command buffer will have enough overhead that they'd be faster in a simple component system on the main thread. But it depends on the use case.

 

That's good news, thanks!

If you need to run commands on the main thread directly after, it may indeed be better to run on the main thread directly. but if you only need to modify values on existing components, it's definitely faster to run a job, since you don't have to wait for the query. The query will be run in parallel and then your job will run, and you'll already be running another system on the main thread while this is happening. However, if you need to change components after your job, you have to wait for your jobs to complete, so it may not be worth it to schedule a job for that.

I already use several groups similar to this:

Spoiler

Code (CSharp):

1. public class InteractGroup : ComponentSystemGroup
2. {
3.     [Preserve] public InteractGroup() {}
4.  
5.     protected override void OnCreate()
6.     {
7.         RequireForUpdate(GetEntityQuery(typeof(Interaction)));
8.     }
9. }

It allows to group every system that work on the same component/query together, you know when all the work on those components is done, and if the query is empty, it culls everything with one check.

 

Sorry for reviving this, but I just tested the same thing with the new Entities release (0.2.0), and it seems that it's basically the same as before for a simple system (about 0.007ms or more). Is there a reason I don't see the improvements that were supposed to be in the release?

I'm kinda starting to fear that the ECS won't be as performant as good old MonoBehaviours for games without a bunch of systems operating on a ton of entities.

 

The above method seems to indeed be much faster, I get 0.01-0.02ms in profiler compared to 0.02-0.03ms of jobified (not sure how to get more granular with the profiler other than profiling manually). Base ComponentSystems show up at about 0.05ms. In development build (not sure in non-dev because profiler) these numbers jumped all over the place but the general trend was still the same.

I find it weird that JobComponentSystem seems to run main thread code more efficiently than ComponentSystem. With 1-entity 1-component the base ComponentSystem is not only slower but creates garbage as well (also in dev build). I assumed this was because of the lambda but JobHandle has the same lambda style and seems to not create any garbage, perhaps because of burst. JobComponentSystem's ForEach also runs faster, perhaps also because burst.

So as of right now it seems like there is no reason to use ComponentSystems, since JobComponentSystems can now run on main thread. I imagine the same optimizations could be applied to ComponentSystems but then there are two ways to basically do the same thing, with JobComponentSystems being more powerful because it can do both (main thread and job). Is there still a use case for them? Is the plan to deprecate them later on?

 

ComponentSystem is still super important tool until there is dots version available to replace all the current oop version features. Besides that, I think also need to do something on ComponentSystem to fully eliminate all the GC caused by lamba expression.

 

In the code posted by Joachim, I don’t see [BurstCompile] anywhere yet the job is apparently bursted. I haven't upgraded yet to 0.2, so is burst on now by default for all jobs or just for the ones with lambda expressions? Is there a way to turn it off for certain jobs or will Unity now automatically figure out what is and what isn't burstable?

Also, I seem to remember from a Unite video that that code with the lambda expression doesn't actually produce garbage, but maybe that's just my memory playing tricks on me. Would be nice to have confirmation.

But anyway, this goes to show that not literally every little thing should be delegated to other cores as that has inherent costs. This has been said many times over these forums and I don't see how Unity can solve this 'automatically'. Scheduling jobs will have overhead no matter what. I don't know if this is possible, but I think the best solution would be a warning from Unity saying that a particular system doesn't have enough work to do to justify being jobified.

I like the idea of being able to configure jobs globally and maybe be able to switch between main thread and parallel at runtime with the debugger in order to see the difference.

 

He is not declaring any jobs in his code above. He just processing the data in the OnUpdate function that runs on the main thread. And I don't think Burst will ever be turned on by default because you are not allowed to run any side effects (such as console logs) for jobs that are burst compiled. So you want it to be configurable by devs.

 

Looks like I was wrong, he is declaring a job in the ForEach function. So Radu, your question still stands.

 

Wow, I'm really happy to hear that you guys are sticking to your 'performance by default' principle without cutting any corners! I was in the middle of refactoring my code when this update hit, so I think I'll have to revert and start over again and use this new way of writing jobs.

But anyway, back on topic. What I was talking about in my third paragraph was that if you only have 1 or 2 operations working on only 1 or 2 entities, then you're much better off at doing this work on the main thread instead of going through the scheduling process, which is what your code posted above does (it's a bursted job running on the main thread). That avoids all the overhead of scheduling which is quite high at the moment. I know you guys plan on further optimizing this scheduling process, but at the end of the day, even after all those optimizations on the scheduling process, will it still be worth it to schedule a job to do a single addition of 2 vectors instead of just doing it on the main thread? Hats off to you if it will be!

 

This is probably a stupid question but would it not it be possible to pre-schedule and pre-calculate the data. We have all these cores with these spare cycles, I think even the Data Oriented Design book mentions doing that. I guess it depends on the data, but if all states are known and fixed then it shouldn't be to difficult, even if there not it might be possible. I've seen google stadia say they are going to use some sort of predictive methods to reduce latency. Jobs are easy to use and create so why not use spare threads to predict and create the data so its ready immediately for when you need it.

 

I was under the impression that scheduling jobs was almost always better, since there may be ongoing jobs on the same data so you shouldn't wait for that data to be ready for further processing, and since you have to wait for the query related jobs (GatherComponentDataJob and so forth) anyway, so might as well schedule your job and go do something else on the main thread. Is that gathering stuff always in jobs like before? If it is, does it not defeat the purpose of not scheduling job, since you have to wait for a scheduled job anyway?

 

Very interesting. Can this be done within a regular Monobehavior Update() as well, since this seems to follow the Task()-syntax?

 

I can give you quite a few reasons actually, although most of them have to do with me trying to associate collections with entities with automatically updating JobHandles (I heavily rely on building lots of acceleration structures).

But for a simple one, what is the intended equivalent of IJFE.ScheduleSingle? I commonly have a bunch of independent job chains scheduled in parallel to each other, each with a temporary NativeList.

 

Is it possible to use EntityQuery with the new JobComponentSystem's Entities.ForEach(), (equivalent to the Entities.With(query).Foreach() in ComponentSystem)? Is WithStoreEntityQueryInField what I need?

This is something I use everywhere, I like to know quickly what I'm working on EntityQuery-wise in a system, so I always create explicit EntityQueries.

 

I am looking into Entities.ForEach, and I am consider convert some IJobForEach. I already converted some IJobChunk to IJobForEach, to reduce amount of boiler plates, when applicable.

However, I looked into samples on github, but somehow I can not buy into the concept with that whole lambda approach.
Maybe if I study it more closely. But somehow for now, it puts me off. So I personally stick with IJobForEach. It may be just me.

 

Ok, so is there a plan to enable something like this at some point?

 

I think I'll try to see if I can write my own With(query) extension method, since I always prefer to create explicit queries in OnCreate instead of the automatically generated ones. It's way more readable than having to guess everytime what the automatic query will be, and I can use the EntityQuery variable in a RequireForUpdate, for CalculateCount or for IsEmptyIgnoreFilter if needed.

Edit: I think it may be harder than I thought, since it seems there's no way to get the ComponentTypes back from the EntityQuery.

 

Seems like SystemBase in 0.6 already make it unnecessary to type out [AlwaysSynchronizeSystem]. How about write code once and then configure globally feature? Will it ship at future version of Entities any time soon?

 

You don't have to use [AlwaysSynchronizeSystem] with SystemBase if you use .Run()? I guess it makes sense, but I just want to make sure.

 

Is there any plan to implement System selection at editor and its APIs to make it only selected system will add into Unity player update loop? So, you can directly remove the system overhead by just not adding the Systems you don't need to run into the update loop. This feature will greatly give benefit to package that you can make it deselect the feature set you don't want then it will just automatically deselect the corresponding Systems. From what I know, DOTS Netcode has the plan to ship different type of Netcode package to solve this problem but having granularity to deselect Systems directly further reduce overhead to the next level.

 

I want to know too. Any news? @Joachim_Ante

 

Before that you mention about to make system configurable at editor to set particular system to run on main thread or across multiple core. Is that in future ScheduleParallel() and Schedule() will be removed and only use Run() API in SystemBase then we configure it at Editor?