Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Feedback CombineDependencies is slow / schedules batched jobs?

Discussion in 'Entity Component System' started by amarcolina, Apr 9, 2021.

  1. amarcolina

    amarcolina

    Joined:
    Jun 19, 2014
    Posts:
    65
    I've been running some tests on the Schedule performance, and came to the surprising result that CombineDependencies is slower than I expected. It seems to be even slower than calling ScheduleBatchedJobs after every Schedule. I have two tests below, that schedules 200 jobs in a chain. One uses CombineDependencies, and another does not.

    Code (CSharp):
    1.  
    2. EmptyJob emptyJob = new EmptyJob();
    3. JobHandle dependency1 = default;
    4. JobHandle dependency2 = default;
    5.  
    6. Profiler.BeginSample("Total Cost");
    7. Profiler.BeginSample("Schedule Cost");
    8. for (int i = 0; i < 100; i++) {
    9.     var combined = JobHandle.CombineDependencies(dependency1, dependency2);
    10.  
    11.     dependency1 = emptyJob.Schedule(combined);
    12.     dependency2 = emptyJob.Schedule(combined);
    13. }
    14. Profiler.EndSample();
    15. JobHandle.CombineDependencies(dependency1, dependency2).Complete();
    16. Profiler.EndSample();
    17.  

    Code (CSharp):
    1.  
    2. EmptyJob emptyJob = new EmptyJob();
    3. JobHandle dependency1 = default;
    4. JobHandle dependency2 = default;
    5.  
    6. Profiler.BeginSample("Total Cost");
    7. Profiler.BeginSample("Schedule Cost");
    8. for (int i = 0; i < 100; i++) {
    9.     dependency1 = emptyJob.Schedule(dependency2);
    10.     dependency2 = emptyJob.Schedule(dependency1);
    11. }
    12. Profiler.EndSample();
    13. JobHandle.CombineDependencies(dependency1, dependency2).Complete();
    14. Profiler.EndSample();
    15.  

    The examples are not exactly equivalent of course, but I tried to get them as close as possible without using CombineDependencies in one of them. The timing numbers were very surprising!

    With Combine Dependencies:
    Schedule Cost: 1.25ms
    Total Cost: 1.26ms

    Without Combine Dependencies:
    Schedule Cost: 0.12ms
    Total Cost: 0.51ms

    I see two things from these results. First is that using CombineDependencies when scheduling increases my main-thread schedule cost by around 10x, which I thought was very extreme. There is another interesting result, when using CombineDependencies, note that the total cost is basically exactly equal to the schedule cost. Looking into the profiler, when using CombineDependencies jobs seem to start executing right away as they are scheduled. Is CombineDependencies calling ScheduleBatchedJobs under the hood? Is it possible this is part of the reason for the dramatic performance difference? Is there possibly something I am doing wrong with CombineDependencies? I'd rather not be in a situation where I try to avoid depending on more than one job at a time.
     
  2. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,753
    So yes combine dependencies is not fast BUT...

    Do you have jobs debugger enabled? It makes them A LOT slower.
     
  3. amarcolina

    amarcolina

    Joined:
    Jun 19, 2014
    Posts:
    65
    I do not unfortunately :(

    Job Debugger is disabled, Leak detection is off, I also turned off safety checks too just to be safe, although I doubt it makes any different in this situation. These numbers are also identical in a release build. I am running Unity 2020.3.3f1
     
  4. Mortuus17

    Mortuus17

    Joined:
    Jan 6, 2020
    Posts:
    105
    Sorry that I don't contribute to the discussion but I love this kinda stuff :D
     
    amarcolina likes this.
  5. sngdan

    sngdan

    Joined:
    Feb 7, 2014
    Posts:
    1,154
    Have you tried combinedependencies with an array of job handles instead of each loop? I have not been using unity for a long time but I recall using it this way
     
  6. amarcolina

    amarcolina

    Joined:
    Jun 19, 2014
    Posts:
    65
    That only works if you want to combine all of the dependencies of independent jobs, but the point is for situations where you have a sequence of jobs that depend on each other one after the other.

    Also this post is less about trying to optimize a specific situation, and more about the specific performance characteristics of CombineDependencies. Also the weird fact that it seems to schedule batched jobs is weird?
     
  7. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,223
  8. Timboc

    Timboc

    Joined:
    Jun 22, 2015
    Posts:
    238
    I have also found it frustrating and is the main bottleneck for me in practice with anything less than 1000 entities. I was surprised to find there doesn't seem to be an .IsEmpty check when combining dependencies so if you have a long chain where not necessarily all jobs run, you can get quite a big gain from doing an if-check around them: https://forum.unity.com/threads/using-query-isempty-to-reduce-scheduling-cost.1078688/#post-6955079. Other than that I guess we're still waiting for optimisations around this to become higher priority. I had high hopes for ISBs but it doesn't seem they've yet made a difference.
     
    amarcolina likes this.
  9. amarcolina

    amarcolina

    Joined:
    Jun 19, 2014
    Posts:
    65
    Seems like that linked thread directly confirms that CombineDependencies calls ScheduleBatchedJobs under the hood :( At least it's mentioned that this shouldn't be happening, and so there could be hope for potentially big savings if that kind of behavior can be changed? It feels like this would be a huge win, both for regular jobs and ECS, since System jobs seem like the kind of jobs that would regularly have more than one dependency.
     
  10. amarcolina

    amarcolina

    Joined:
    Jun 19, 2014
    Posts:
    65
    Wanted to bump this thread, hoping to get a response from Unity specifically. It seems like the linked thread confirms that CombineDependencies schedules batched jobs, is this still considered a Known Issue, or is it just Working As Intended? Seems like it could be a big performance win, as well as being low-hanging fruit.
     
    Egad_McDad and Lukas_Kastern like this.
  11. amarcolina

    amarcolina

    Joined:
    Jun 19, 2014
    Posts:
    65
    Bumping this again, this time with a ticket number too: 1395198

    Just tested this again using the latest beta, 2022.1.0b3.2528, and saw much the same results. In a build I was actually seeing potentially even a larger delta, up to a 16x-18x schedule cost with using CombineDependencies compared to without.

    The reason I keep focusing on this, is that it seems like it could be a significant performance barrier for larger games with many systems. If you have 200 or so systems/jobs, and enough dependencies that most systems have a dependency on 2 or more jobs, you are going to be spending more than 1ms just for _scheduling_ alone.

    The other reason I keep coming back to this is the fact that CombineDependencies seems to call ScheduleBatchedJobs. From the other forum threads this has sounded like an oversight, and thus potentially a very easy fix? If not, and there is an important reason CombineDependencies also schedules batched jobs, I'd love to understand why. Maybe an overload that allows us to bypass scheduling batched jobs would be in order?

    Alternatively, is it possible some of the secret work on the new Entities package addresses this in a different way? I have to imagine this effects ECS performance directly.
     
    Baggers_ likes this.
  12. s_schoener

    s_schoener

    Unity Technologies

    Joined:
    Nov 4, 2019
    Posts:
    81
    We're aware of the pretty high costs around all of these things. The problem with CombineDependencies is that it might also end up scheduling a job, which is pretty costly but unfortunately sometimes necessary. There is a lot of work going into optimizing the job system right now, and it shouldn't be too far off into the future. Sorry to keep you waiting, and thanks for bumping this: Jobs have been a bit slow for too long :)
     
    Last edited: Jan 16, 2022
    forestrf, DrBoum, Timboc and 4 others like this.
  13. amarcolina

    amarcolina

    Joined:
    Jun 19, 2014
    Posts:
    65
    Thanks so much for the reply! It's great to hear that this is being worked on. It's fascinating to me that calling CombineDependencies can sometimes _need_ to schedule a job, is that an implementation detail? Or is there some aspect of safety that makes it required?