[Solved] How automatic is SystemBase's dependency management really?

I saw this drop this morning and decided I had to ask this before I head out.

So the new SystemBase promises automatic dependency management. If this is robust, it is really awesome. However, looking at how it appears to be implemented from SystemBase's perspective, I am skeptical of that. I'm quite suspicious you created an alchemated monster that falls flat on its face the moment you do something other than the bread-and-butter simple jobs that are not enough on their own to build real games (other than tiny games).

Which of these jobs will have their dependencies injected correctly?

• Entities.ForEach
• IJobForEach and variants
• IJobChunk
• IJob and IJobParallelFor and other custom jobs using ComponentDataFromEntity or ArchetypeChunkComponentType
• Job.WithCode
• Any of the above after manually assigning the Dependency property (for example, populating the noise map)

Similar question for automatically writing back to Dependency?

How I would expect this to work:

• I would expect that every ComponentType has a pair of read and write JobHandles associated with it.
• Before OnUpdate in a SystemBase, SystemBase tells the World to register it as the active JobHandle keeper. After OnUpdate, it deregisters.
• Whenever an ECS job (relies on an EntityQuery) from anywhere gets scheduled, it fetches and combines all ComponentType handles it knows about along with an input JobHandle or the active JobHandle from a SystemBase.
• Whenever an ECS job gets scheduled, its output JobHandle gets written back to all ComponentTypes as well as the active JobHandle from SystemBase (if a SystemBase is registered).
• Whenever a ComponentType iterator (ComponentDataFromEntity and friends or ArchetypeChunkComponentType and friends or ArchetypeChunkIterator) is acquired, the JobHandle of the ComponentType is merged into the active JobHandle of SystemBase.
• Whenever a new JobHandle gets written back to Dependency from outside internal code, all JobHandles of the acquired ComponentType iterators are updated as well.
• EntityManager Get/Set completes the JobHandle of the ComponentType in question.
• EntityManager Create/Destroy/Add/Remove/ect only completes the JobHandles of ComponentTypes of the components affected.
• EntityCommandBuffer playback completes all jobs of all ComponentTypes. (Not sure if it makes sense to have a universal JobHandle in addition to per ComponentType for this)
• ExclusiveEntityTransaction acts as an iterator for all ComponentTypes at once and uses the same rules as the other iterators except applied to all ComponentTypes.
• Any non-ECS job could keep the automatic dependency system up to date simply by scheduling the job with Dependency as a parameter and writing back to Dependency.
• Job.WithCode generates code which does the previous point.
• SimpleDependencyMode replaces Dependency to be a property to the World's JobHandle.

 

Hi! You could have a quick look at last ECSSamples. They've updated the sample package this morning as well, and replaced all JobComponentSystem with the new SystemBase. So, it looks they are very sure that SystemBase with automatic dependency management has advantage over 'old' JobComponentSystem. I think in the near future Unity will mark JobComponentSystem as an obsolete class.

 

Ok, I was looking a bit more deeply at the sample package. It seems the new SystemBase is an evolutionary step with the implementation of a more user friendly API. In the samples they use Dependency just like inputDeps from JobComponentSystem. But now you don't need to add Dependency every time for every single job for scheduling, and return it from Update method. The same logic, a slightly different API.

 

Trust me, combining ComponentSystem and JobComponentSystem and using a property instead of a function argument and return value to make it optional is a welcome change. This issue I am concerned with is that apparently you don't need to touch Dependency when using an Entities.ForEach but there is no indication that this works for IJobChunk, Job.WithCode using ComponentDataFromEntity, or some arbitrary combination of jobs. And I really don't want to spend the time testing all the permutations and seeing what breaks.

What I would really like is for there to be one (and only one) rule as to whether Dependency needs to be touched. Here are some options for what I think that rule could be:

• All jobs scheduled require touching Dependency or self-managed JobHandles.
• Only Lambda jobs are not required to touch Dependency explicitly. They always use Dependency as the input JobHandle and write to it on scheduling a job.
• All custom job types from the Entities API automatically gather and write back JobHandles for all required ComponentTypes as well as SystemBase's Dependency if the job is scheduled during the OnUpdate window. (This is what I proposed above).

 

I don't think it's doing anything new in terms of dependency management aside from saving us from having to type inputDeps in the

inputDeps = Entities.ForEach(...).Schedule(inputDeps);

calls. Someone can correct me if I'm wrong, but I think all the same rules apply that applied in a JobComponentSystem, we just have the choice to ignore the job handle if we have no good reason to touch it.

SystemBase also does some extra checks to avoid doing unnecessary work if we don't actually schedule any jobs (like if we only use .Run lambdas), so it seems like it's just better all around really.

Based on what's been said by Unity it seems like we'll eventually have something like

Chunks.ForEach

callable inside a SystemBase as well. I imagine that will do whatever

Entities.ForEach

is doing now and automatically manage the job handle internally so we can just ignore it.

 

entityQuery.Get/AddDependency <- is this method just added in this version?

 

Nope. That was in 5.1. Only new API for EntityQuery according to my diff is the managed extensions and IsCreated.

Having now dug through the diff, I am suspicious that the rule might actually be this based on the lambda code:

I unfortunately need to turn it in for the night, so I will have to write the test cases and check the new DOTS Compiler out tomorrow night, unless someone beats me to it.

 

By the way I have found this error in my system with only 1 ForEach job, that I made sure to not use auto dependency magic and take Dependency in propery, and Dependency = out properly. I think it is impossible to forget assigning anything else to the Dependency when you have only 1 job. Unless I really overlooked something it might be a bug of dependency generator. I have stare at this one system for an hour and there is nowhere else to forget.

 

Awesome! Thank you for the clarification. I am 100% content with this design. I'm really enjoying what 0.6 has to offer. Awesome job!

 

Is SystemBase intended to replace ComponentSystem as well? Or only JobComponentSystem?

 

Both. JobComponentSystem already was doing ComponentSystem-type work better when it got lambdas. The only big advantage ComponentSystem had was the lack of need to use [AlwaysSynchronizeSystem] or handle inputDeps. SystemBase also removes this need over JobComponentSystem. So it really is the best of both.

 

So I'm a bit confused by a couple of cases related to this statement.

1: Do we need to handle the Dependency when Running (not Scheduling) custom jobs? Which ones of these examples of Update methods in a SystemBase system are correct?

Code (CSharp):

1. // Example 1. No dependency management at all
2. protected override void OnUpdate()
3. {
4.     MyCustomJob customJob;
5.     customJob.GetSomething = GetComponentDataFromEntity<Something>(true);
6.     customJob.Run(this);
7. }
8.  
9. // Example 2. Pass the dependency
10. protected override void OnUpdate()
11. {
12.     MyCustomJob customJob;
13.     customJob.GetSomething = GetComponentDataFromEntity<Something>(true);
14.     customJob.Run(this, Dependency);
15. }
16.  
17. // Example 3: Pass and track the dependency
18. protected override void OnUpdate()
19. {
20.     MyCustomJob customJob;
21.     customJob.GetSomething = GetComponentDataFromEntity<Something>(true);
22.     Dependency = customJob.Run(this, Dependency);
23. }

2: As far as I've understood, [AlwaysSynchronizeSystem] is no longer needed on SystemBase systems, due to it being tracked internally somehow, right? But what about when running custom jobs in a SystemBase as in the example above? Is that automatically detected as well? Or do we still need to stick the attribute on the system for these cases?

Thanks in advance!

 

if I understood correctly, but I am still working this out, the automatic dependency system is still "automatic" only to solve dependencies between jobs that use exclusively components. If you need dependency solving based on other kind of datastructures (including native containers) you need to solve it manually.

I am studying the new boid example, I am still trying to understand if this line is necessary:

Code (CSharp):

1. // We pass the job handle and add the dependency so that we keep the proper ordering between the jobs
2.                 // as the looping iterates. For our purposes of execution, this ordering isn't necessary; however, without
3.                 // the add dependency call here, the safety system will throw an error, because we're accessing multiple
4.                 // pieces of boid data and it would think there could possibly be a race condition.
5.                
6.                 m_BoidQuery.AddDependency(Dependency);

 

Dependency is essentially inputDeps from JobComponentSystem. Same rules apply where it has all the ECS dependencies packaged up for you. You still need to manage dependencies for NativeContainers. Lambda jobs generate the code that touches Dependency for you.

I believe the second and third are both valid, but I could be wrong.

 

Thanks @sebas77 and @DreamingImLatios for your feedback, I believe I understand how works now.

I've played around with this and converted a lot of my systems to SystemBase. The main part I had issues with was for systems that previously used [AlwaysSynchronizeSystem] together with custom jobs that used .Run(). When converted to SystemBase, these threw a bunch of dependency issues. I've read through the source code for the base classes and I think I understand how that works now. If anyone knows better, I would appreciate to be corrected. Here's what I learned:

• [AlwaysSynchronizeSystem] is seemingly only handled inside of JobComponentSystem, and does nothing for any of the other system types.
• What happens internally for [AlwaysSynchronizeSystem] in JobComponentSystems is just that they run CompleteDependencyInternal() before updating.
• SystemBase systems that exclusively use Entities.ForEach.Run() gets CompleteDependency() automatically code-generated into their update loop, which in itself runs the previously mentioned CompleteDependencyInternal().
• When having custom jobs that are .Run() in SystemBase systems, I do believe you need to call CompleteDependency() yourself beforehand. This is the part I'm most uncertain about though.

 

Is that SystemBase will become first citizen then JobComponentSystem and ComponentSystem will be deprecated in future version?

 

hmmm this new SystemBase design made me realise something that I didn't fully understand before. The old InputDeps may have made sense ONLY for component based dependencies and wouldn't carry with them dependencies of jobs that do not use components.

the field SytemBase Dependency may be probably used to add dependencies of jobs, but these must still work on Components.

IF my reasoning is correct, it means that dependencies of jobs that do not work on components were never tracked properly by my code as I was assuming they were passed through the inputDeps jobHandle.

Does this mean that once Jobs not using components are used, I have to handle the dependency myself? This would mean that a SystemBase running exclusively jobs that do not run on components do not make any sense, hence I am not sure where I should run them and how to pass their dependency in the system flow.

Any clue for me?

P.S.: not still sure about CompleteDependency and why I should call it explicitly. It seems it should be called a the begin of the update as it's needed only to guarantee that the previously scheduled job is completed?

Edit: after discussing this on Discord it appears that external dependencies are combined as well and carried on in the system as long as the Dependency field is used, which can be used also as input for other jobs, so that something like this makes sense:


protected override void OnUpdate()
{
this.Dependency = JobHandle.CombineDependencies(Dependency, jobHandleWithoutComponents);
}

 

So as long as every system that touches NativeContainerA also touches ComponentA, then you won't have to worry about dependencies for NativeContainerA.

However, if SystemA touches NativeContainerA and ComponentA and SystemB touches NativeContainerA and ComponentB, then SystemB will not have the correct dependencies for NativeContainerA even if you manually wrote in the Dependency in SystemA. You need to keep track of that JobHandle somewhere else.

 

To keep it simple is best to talk about systems that do not use components at all. The question is, will the Dependency field be enough to be sure that the jobs scheduled by the next systems will take the current dependency in consideration? Example:

Code (CSharp):

1. System A                 =>   System B
2. Schedule a job                schedules another job without components
3. without components            but needs the job of system A to be done.
4. updates Dependency            It doesn't need to call complete, but I need a dependency
5. with the JobHandle         concatenation. So the current Job is sheduled with Dependency as input

.


will this ensure the correct job concatenation? This will happen if Dependency actually carries the previous jobHandle to the next System Update.

 

No. Only simple dependency mode works that way and I haven't tested if that works with SystemBase.

If you want containers to be tracked, you either need to make sure they are always used with the same component or handle the JobHandles yourself. I do have a solution in the works for associating containers with entities like components and automatically updating Dependency with the extra JobHandles and vice-versa if you are interested.

 

So are you saying that I have to pass the job handle from the other job in other ways? This is not what I have been told but obviously next week I have to do some serious tests

 

For NativeContainers and not ECS components, yes. Only ECS components are tracked.

I'm not sure who told you otherwise, but this rule has been in place even with JobComponentSystem. The ECS API has no way of knowing what your NativeContainers are for it to automatically track them.

 

Is this talking about dependencies BETWEEN systems only? Because within a single system it seems clear that jobs without ECS components are scheduled correctly just by passing a JobHandle, otherwise I'd expect several of my systems not to work correctly.

 

Yes. This is between systems.

 

You can see from my first post that your understanding was mine too, there must be some confusion out there and I ought to make some proper tests. To be fair UECS doen't do a good job in being self explanatory (although it's surely improving) and the Dependency field is very ambiguous, as I can combine in it external dependencies, I would assume those dependencies are carried between systems. However I would understand the case of tracking just components too, it's just that it's not clear as I can do Dependency = JobHandle.Combine(whateva).

Edit: the documentation is actually clear on this and it does say that the Dependency jobhandle tracks only component dependencies.

 

One question. If I use "Systembase", one behaviour changed. When I use a simple foreach, like so:
Entities.ForEach((int entityInQueryIndex, ref Translation translation) => {
...
}
And there is no entity with a translation, everything else I do "onUpdate()", like a Debug.Log("test"), doesn't get fired.
Once I create an entity, the Debug.Log fires.
With JobComponentSystem, I didn't need an actual entity to do stuff outside of a "Entities.Foreach". I think I missed something else, but if not, this feels like a bug. Does anyone know about this?

Edit:
Here we go:
Entities 0.7.0 Preview 19
The system invokes OnUpdate() once per frame on the main thread when any of this system's EntityQueries match existing entities, the system has the [AlwaysUpdateSystem] attribute, or the system has no queries at all.

 

Hey, guys.

Any summary of how to use all of that in the right way (SystemBase, systems dependencies, etc)?
Do you have some links or videos?
I am new to all of that and it is quite mess what to study and what is in the latest state

 

When it comes to dependency management nothing has really changed since the introduction of SystemBase.

There's plenty of general DOTS learning resources if you search the forums and YouTube (we haven't gotten a DOTS update in over a year so stuff in mid-to-late 2020 is usually up-to-date). If those aren't answering your questions, it is best to ask specific questions as to what you are confused about.

 

Thank you. Recently Unity released0.50, so I will wait for the updated DOTS Samples to check how they propose to leverage the framework