Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice
  2. Enter the 2020.2 Beta Sweepstakes for a chance to win an Oculus Quest 2.
    Dismiss Notice

[Solved] How automatic is SystemBase's dependency management really?

Discussion in 'Data Oriented Technology Stack' started by DreamingImLatios, Feb 17, 2020.

  1. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    1,782
    I saw this drop this morning and decided I had to ask this before I head out.

    So the new SystemBase promises automatic dependency management. If this is robust, it is really awesome. However, looking at how it appears to be implemented from SystemBase's perspective, I am skeptical of that. I'm quite suspicious you created an alchemated monster that falls flat on its face the moment you do something other than the bread-and-butter simple jobs that are not enough on their own to build real games (other than tiny games).

    Which of these jobs will have their dependencies injected correctly?
    • Entities.ForEach
    • IJobForEach and variants
    • IJobChunk
    • IJob and IJobParallelFor and other custom jobs using ComponentDataFromEntity or ArchetypeChunkComponentType
    • Job.WithCode
    • Any of the above after manually assigning the Dependency property (for example, populating the noise map)
    Similar question for automatically writing back to Dependency?

    How I would expect this to work:
    • I would expect that every ComponentType has a pair of read and write JobHandles associated with it.
    • Before OnUpdate in a SystemBase, SystemBase tells the World to register it as the active JobHandle keeper. After OnUpdate, it deregisters.
    • Whenever an ECS job (relies on an EntityQuery) from anywhere gets scheduled, it fetches and combines all ComponentType handles it knows about along with an input JobHandle or the active JobHandle from a SystemBase.
    • Whenever an ECS job gets scheduled, its output JobHandle gets written back to all ComponentTypes as well as the active JobHandle from SystemBase (if a SystemBase is registered).
    • Whenever a ComponentType iterator (ComponentDataFromEntity and friends or ArchetypeChunkComponentType and friends or ArchetypeChunkIterator) is acquired, the JobHandle of the ComponentType is merged into the active JobHandle of SystemBase.
    • Whenever a new JobHandle gets written back to Dependency from outside internal code, all JobHandles of the acquired ComponentType iterators are updated as well.
    • EntityManager Get/Set completes the JobHandle of the ComponentType in question.
    • EntityManager Create/Destroy/Add/Remove/ect only completes the JobHandles of ComponentTypes of the components affected.
    • EntityCommandBuffer playback completes all jobs of all ComponentTypes. (Not sure if it makes sense to have a universal JobHandle in addition to per ComponentType for this)
    • ExclusiveEntityTransaction acts as an iterator for all ComponentTypes at once and uses the same rules as the other iterators except applied to all ComponentTypes.
    • Any non-ECS job could keep the automatic dependency system up to date simply by scheduling the job with Dependency as a parameter and writing back to Dependency.
    • Job.WithCode generates code which does the previous point.
    • SimpleDependencyMode replaces Dependency to be a property to the World's JobHandle.
     
    bobbaluba likes this.
  2. cjxm

    cjxm

    Joined:
    Feb 1, 2016
    Posts:
    6
    Hi! You could have a quick look at last ECSSamples. They've updated the sample package this morning as well, and replaced all JobComponentSystem with the new SystemBase. So, it looks they are very sure that SystemBase with automatic dependency management has advantage over 'old' JobComponentSystem. I think in the near future Unity will mark JobComponentSystem as an obsolete class.
     
    bobbaluba likes this.
  3. cjxm

    cjxm

    Joined:
    Feb 1, 2016
    Posts:
    6
    Ok, I was looking a bit more deeply at the sample package. It seems the new SystemBase is an evolutionary step with the implementation of a more user friendly API. In the samples they use Dependency just like inputDeps from JobComponentSystem. But now you don't need to add Dependency every time for every single job for scheduling, and return it from Update method. The same logic, a slightly different API.
     
    PhilSA likes this.
  4. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    1,782
    Trust me, combining ComponentSystem and JobComponentSystem and using a property instead of a function argument and return value to make it optional is a welcome change. This issue I am concerned with is that apparently you don't need to touch Dependency when using an Entities.ForEach but there is no indication that this works for IJobChunk, Job.WithCode using ComponentDataFromEntity, or some arbitrary combination of jobs. And I really don't want to spend the time testing all the permutations and seeing what breaks.

    What I would really like is for there to be one (and only one) rule as to whether Dependency needs to be touched. Here are some options for what I think that rule could be:
    • All jobs scheduled require touching Dependency or self-managed JobHandles.
    • Only Lambda jobs are not required to touch Dependency explicitly. They always use Dependency as the input JobHandle and write to it on scheduling a job.
    • All custom job types from the Entities API automatically gather and write back JobHandles for all required ComponentTypes as well as SystemBase's Dependency if the job is scheduled during the OnUpdate window. (This is what I proposed above).
     
  5. Sarkahn

    Sarkahn

    Joined:
    Jan 9, 2013
    Posts:
    280
    I don't think it's doing anything new in terms of dependency management aside from saving us from having to type inputDeps in the
    inputDeps = Entities.ForEach(...).Schedule(inputDeps);
    calls. Someone can correct me if I'm wrong, but I think all the same rules apply that applied in a JobComponentSystem, we just have the choice to ignore the job handle if we have no good reason to touch it.

    SystemBase also does some extra checks to avoid doing unnecessary work if we don't actually schedule any jobs (like if we only use .Run lambdas), so it seems like it's just better all around really.

    Based on what's been said by Unity it seems like we'll eventually have something like
    Chunks.ForEach
    callable inside a SystemBase as well. I imagine that will do whatever
    Entities.ForEach
    is doing now and automatically manage the job handle internally so we can just ignore it.
     
    Last edited: Feb 18, 2020
    vestigial likes this.
  6. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,510
    entityQuery.Get/AddDependency <- is this method just added in this version?
     
    sebas77 likes this.
  7. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    1,782
    Nope. That was in 5.1. Only new API for EntityQuery according to my diff is the managed extensions and IsCreated.

    Having now dug through the diff, I am suspicious that the rule might actually be this based on the lambda code:
    I unfortunately need to turn it in for the night, so I will have to write the test cases and check the new DOTS Compiler out tomorrow night, unless someone beats me to it.
     
  8. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,510
    By the way I have found this error in my system with only 1 ForEach job, that I made sure to not use auto dependency magic and take Dependency in propery, and Dependency = out properly. I think it is impossible to forget assigning anything else to the Dependency when you have only 1 job. Unless I really overlooked something it might be a bug of dependency generator. I have stare at this one system for an hour and there is nowhere else to forget.

     
  9. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,130
    Only lambda based jobs track dependencies automatically.

    Internally this code:
    Code (CSharp):
    1. Entities.ForEach(...).Schedule();
    is equivalent to this:
    Code (CSharp):
    1. Dependency = Entities.ForEach(...).Schedule(Dependency);
    It's basically just a shortcut for having to type the exact same pattern every time. Internally works the same as what JobComponentSystem did with input dependency and return value.

    when using IJobChunk or any custom job types, it is required to handle the dependencies yourself with the same pattern as the manual Entities.ForEach version.
     
  10. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    1,782
    Awesome! Thank you for the clarification. I am 100% content with this design. I'm really enjoying what 0.6 has to offer. Awesome job!
     
  11. LastResortMatthew

    LastResortMatthew

    Joined:
    Apr 10, 2019
    Posts:
    65
    Is SystemBase intended to replace ComponentSystem as well? Or only JobComponentSystem?
     
  12. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    1,782
    Both. JobComponentSystem already was doing ComponentSystem-type work better when it got lambdas. The only big advantage ComponentSystem had was the lack of need to use [AlwaysSynchronizeSystem] or handle inputDeps. SystemBase also removes this need over JobComponentSystem. So it really is the best of both.
     
  13. Zec_

    Zec_

    Joined:
    Feb 9, 2017
    Posts:
    105
    So I'm a bit confused by a couple of cases related to this statement.

    1: Do we need to handle the Dependency when Running (not Scheduling) custom jobs? Which ones of these examples of Update methods in a SystemBase system are correct?
    Code (CSharp):
    1. // Example 1. No dependency management at all
    2. protected override void OnUpdate()
    3. {
    4.     MyCustomJob customJob;
    5.     customJob.GetSomething = GetComponentDataFromEntity<Something>(true);
    6.     customJob.Run(this);
    7. }
    8.  
    9. // Example 2. Pass the dependency
    10. protected override void OnUpdate()
    11. {
    12.     MyCustomJob customJob;
    13.     customJob.GetSomething = GetComponentDataFromEntity<Something>(true);
    14.     customJob.Run(this, Dependency);
    15. }
    16.  
    17. // Example 3: Pass and track the dependency
    18. protected override void OnUpdate()
    19. {
    20.     MyCustomJob customJob;
    21.     customJob.GetSomething = GetComponentDataFromEntity<Something>(true);
    22.     Dependency = customJob.Run(this, Dependency);
    23. }

    2: As far as I've understood, [AlwaysSynchronizeSystem] is no longer needed on SystemBase systems, due to it being tracked internally somehow, right? But what about when running custom jobs in a SystemBase as in the example above? Is that automatically detected as well? Or do we still need to stick the attribute on the system for these cases?

    Thanks in advance!
     
    Last edited: Feb 22, 2020
  14. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,222
    if I understood correctly, but I am still working this out, the automatic dependency system is still "automatic" only to solve dependencies between jobs that use exclusively components. If you need dependency solving based on other kind of datastructures (including native containers) you need to solve it manually.

    I am studying the new boid example, I am still trying to understand if this line is necessary:

    Code (CSharp):
    1. // We pass the job handle and add the dependency so that we keep the proper ordering between the jobs
    2.                 // as the looping iterates. For our purposes of execution, this ordering isn't necessary; however, without
    3.                 // the add dependency call here, the safety system will throw an error, because we're accessing multiple
    4.                 // pieces of boid data and it would think there could possibly be a race condition.
    5.                
    6.                 m_BoidQuery.AddDependency(Dependency);
     
  15. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    1,782
    Dependency is essentially inputDeps from JobComponentSystem. Same rules apply where it has all the ECS dependencies packaged up for you. You still need to manage dependencies for NativeContainers. Lambda jobs generate the code that touches Dependency for you.
    I believe the second and third are both valid, but I could be wrong.
     
  16. Zec_

    Zec_

    Joined:
    Feb 9, 2017
    Posts:
    105
    Thanks @sebas77 and @DreamingImLatios for your feedback, I believe I understand how works now.

    I've played around with this and converted a lot of my systems to SystemBase. The main part I had issues with was for systems that previously used [AlwaysSynchronizeSystem] together with custom jobs that used .Run(). When converted to SystemBase, these threw a bunch of dependency issues. I've read through the source code for the base classes and I think I understand how that works now. If anyone knows better, I would appreciate to be corrected. Here's what I learned:
    • [AlwaysSynchronizeSystem] is seemingly only handled inside of JobComponentSystem, and does nothing for any of the other system types.
    • What happens internally for [AlwaysSynchronizeSystem] in JobComponentSystems is just that they run CompleteDependencyInternal() before updating.
    • SystemBase systems that exclusively use Entities.ForEach.Run() gets CompleteDependency() automatically code-generated into their update loop, which in itself runs the previously mentioned CompleteDependencyInternal().
    • When having custom jobs that are .Run() in SystemBase systems, I do believe you need to call CompleteDependency() yourself beforehand. This is the part I'm most uncertain about though.
     
    Last edited: Feb 24, 2020
    DreamingImLatios likes this.
  17. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,130
    This is accurate. For the same reason [AlwaysSynchronizeSystem] doesn't make sense to use on a SystemBase anymore.
     
    optimise, Jes28, Zec_ and 1 other person like this.
  18. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    613
    Is that SystemBase will become first citizen then JobComponentSystem and ComponentSystem will be deprecated in future version?
     
  19. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,130
    Yes thats the idea. But of course we will keep both around for an extended period of time once we mark them deprecatd warning.
     
    bobbaluba, charleshendry and optimise like this.
  20. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,222
    hmmm this new SystemBase design made me realise something that I didn't fully understand before. The old InputDeps may have made sense ONLY for component based dependencies and wouldn't carry with them dependencies of jobs that do not use components.

    the field SytemBase Dependency may be probably used to add dependencies of jobs, but these must still work on Components.

    IF my reasoning is correct, it means that dependencies of jobs that do not work on components were never tracked properly by my code as I was assuming they were passed through the inputDeps jobHandle.

    Does this mean that once Jobs not using components are used, I have to handle the dependency myself? This would mean that a SystemBase running exclusively jobs that do not run on components do not make any sense, hence I am not sure where I should run them and how to pass their dependency in the system flow.

    Any clue for me?

    P.S.: not still sure about CompleteDependency and why I should call it explicitly. It seems it should be called a the begin of the update as it's needed only to guarantee that the previously scheduled job is completed?

    Edit: after discussing this on Discord it appears that external dependencies are combined as well and carried on in the system as long as the Dependency field is used, which can be used also as input for other jobs, so that something like this makes sense:


    protected override void OnUpdate()
    {
    this.Dependency = JobHandle.CombineDependencies(Dependency, jobHandleWithoutComponents);
    }
     
    Last edited: Feb 29, 2020
    mkracik likes this.
  21. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    1,782
    So as long as every system that touches NativeContainerA also touches ComponentA, then you won't have to worry about dependencies for NativeContainerA.

    However, if SystemA touches NativeContainerA and ComponentA and SystemB touches NativeContainerA and ComponentB, then SystemB will not have the correct dependencies for NativeContainerA even if you manually wrote in the Dependency in SystemA. You need to keep track of that JobHandle somewhere else.
     
  22. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,222
    To keep it simple is best to talk about systems that do not use components at all. The question is, will the Dependency field be enough to be sure that the jobs scheduled by the next systems will take the current dependency in consideration? Example:

    Code (CSharp):
    1. System A                 =>   System B
    2. Schedule a job                schedules another job without components
    3. without components            but needs the job of system A to be done.
    4. updates Dependency            It doesn't need to call complete, but I need a dependency
    5. with the JobHandle         concatenation. So the current Job is sheduled with Dependency as input
    .


    will this ensure the correct job concatenation? This will happen if Dependency actually carries the previous jobHandle to the next System Update.
     
  23. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    1,782
    No. Only simple dependency mode works that way and I haven't tested if that works with SystemBase.

    If you want containers to be tracked, you either need to make sure they are always used with the same component or handle the JobHandles yourself. I do have a solution in the works for associating containers with entities like components and automatically updating Dependency with the extra JobHandles and vice-versa if you are interested.
     
  24. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,222
    So are you saying that I have to pass the job handle from the other job in other ways? This is not what I have been told but obviously next week I have to do some serious tests
     
  25. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    1,782
    For NativeContainers and not ECS components, yes. Only ECS components are tracked.

    I'm not sure who told you otherwise, but this rule has been in place even with JobComponentSystem. The ECS API has no way of knowing what your NativeContainers are for it to automatically track them.
     
  26. LastResortMatthew

    LastResortMatthew

    Joined:
    Apr 10, 2019
    Posts:
    65
    Is this talking about dependencies BETWEEN systems only? Because within a single system it seems clear that jobs without ECS components are scheduled correctly just by passing a JobHandle, otherwise I'd expect several of my systems not to work correctly.
     
  27. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    1,782
    Yes. This is between systems.
     
    LastResortMatthew likes this.
  28. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,222
    You can see from my first post that your understanding was mine too, there must be some confusion out there and I ought to make some proper tests. To be fair UECS doen't do a good job in being self explanatory (although it's surely improving) and the Dependency field is very ambiguous, as I can combine in it external dependencies, I would assume those dependencies are carried between systems. However I would understand the case of tracking just components too, it's just that it's not clear as I can do Dependency = JobHandle.Combine(whateva).

    Edit: the documentation is actually clear on this and it does say that the Dependency jobhandle tracks only component dependencies.
     
    Last edited: Mar 2, 2020
  29. Zyblade

    Zyblade

    Joined:
    Jul 1, 2014
    Posts:
    125
    One question. If I use "Systembase", one behaviour changed. When I use a simple foreach, like so:
    Entities.ForEach((int entityInQueryIndex, ref Translation translation) => {
    ...
    }
    And there is no entity with a translation, everything else I do "onUpdate()", like a Debug.Log("test"), doesn't get fired.
    Once I create an entity, the Debug.Log fires.
    With JobComponentSystem, I didn't need an actual entity to do stuff outside of a "Entities.Foreach". I think I missed something else, but if not, this feels like a bug. Does anyone know about this?

    Edit:
    Here we go:
    Entities 0.7.0 Preview 19
    The system invokes OnUpdate() once per frame on the main thread when any of this system's EntityQueries match existing entities, the system has the [AlwaysUpdateSystem] attribute, or the system has no queries at all.
     
    Last edited: May 9, 2020
    KAV2008 likes this.
unityunity