IJobParallelFor jobs tend to not use the main thread

I have a bunch of IJobParallelFor jobs (a fluid simulation), where each next job depends on the previous one. Like this:

Code (CSharp):

1. var job1 = new Job1 { ... };
2. var handle = job1.Schedule(count, BATCH_SIZE);
3.  
4. var job2 = new Job2 { ... };
5. handle = job2.Schedule(count, BATCH_SIZE, handle);
6.  
7. var job3 = new Job3 { ... };
8. handle = job3.Schedule(count, BATCH_SIZE, handle);
9.  
10. var job4 = new Job4 { ... };
11. handle = job4.Schedule(count, BATCH_SIZE, handle);
12.  
13. var job5 = new Job5 { ... };
14. handle = job5.Schedule(count, BATCH_SIZE, handle);
15.  
16. handle.Complete();

I've noticed that most of the time only the fist job uses all the 4 cores my CPU has. The second job prefers to use 3 cores (but sometimes uses 4), the other jobs use 3 cores almost always, while the main thread is waiting doing nothing.

Tested in Unity 2019.1, both editor and dev builds.

Common case:


Less common:


Very rare (two final jobs still run on 3 cores, but I don't care since they are very short):


The empirical rule is the following: When handle.Complete() is called, the first of the jobs, that have been scheduled by that time, tend to use all the threads, and the following jobs tend to not occupy the main thread (they still may use it, but with rapidly decreasing probability).

If I want all the jobs to use all the threads all the time, I have to "complete" each job right after it has been scheduled, introducing extra synchronization points. It doesn't matter in this particular case, but still the solution is neither elegant nor obvious. Are there any other solution to force the jobs to use all the cores?

 

Doesn't look so.

However, I'm not sure the modern task manager can be trusted. On the first picture it shows that 1/4 of my quad-core CPU is idle. On the second picture it shows that almost 100% of CPU is occupied by Unity Editor process.

By the way, I set the process priority from normal to high to see if it would change anything. It didn't.




PS
On standalone builds (both dev and normal) the task manager shows the same, the process takes about 75% of the CPU and the rest 25% is idle.

PPS
Here is the build: https://dl.dropbox.com/s/flay945gn4ifwpr/Fluids.zip (Win x64)
I would expect it to consume all the CPU available, but it doesn't. In case of a Threadripper, you can always add more particles ("1" - remove 1000 particles, "2" - add 1000 particles).

 

count is the amount of particles in my simulation. The last two screenshots were taken with count = 40000 or something like that. On the screenshots in the original post the value, I think, was about 10000.

The BATCH_SIZE values I have tried are from 64 to 256. There is no noticeable difference.

 

Here is a simple repro:

Code (CSharp):

1. using Unity.Collections;
2. using Unity.Jobs;
3. using UnityEngine;
4.  
5. public class Test : MonoBehaviour
6. {
7.     private NativeArray<float> array;
8.  
9.     private void OnEnable()
10.     {
11.         array = new NativeArray<float>(100_000, Allocator.Persistent);
12.     }
13.  
14.     private void OnDisable()
15.     {
16.         array.Dispose();
17.     }
18.  
19.     private void Update()
20.     {
21.         const int BATCH_SIZE = 200;
22.  
23.         var job1 = new DummyJob { array = array };
24.         var handle = job1.Schedule(array.Length, BATCH_SIZE);
25.  
26.         var job2 = new DummyJob { array = array };
27.         handle = job2.Schedule(array.Length, BATCH_SIZE, handle);
28.  
29.         var job3 = new DummyJob { array = array };
30.         handle = job3.Schedule(array.Length, BATCH_SIZE, handle);
31.  
32.         var job4 = new DummyJob { array = array };
33.         handle = job4.Schedule(array.Length, BATCH_SIZE, handle);
34.      
35.         var job5 = new DummyJob { array = array };
36.         handle = job5.Schedule(array.Length, BATCH_SIZE, handle);
37.  
38.         handle.Complete();
39.     }
40. }
41.  
42. public struct DummyJob : IJobParallelFor
43. {
44.     public NativeArray<float> array;
45.  
46.     public void Execute(int index)
47.     {
48.         array[index] = Mathf.Sqrt(array[index]);
49.     }
50. }

Most of the time the profiler shows this:

 

Much clearer repro than mine (https://forum.unity.com/threads/mai...ting-for-jobs-completion.691597/#post-4628221). This repro eliminates ecs from the picture and shows with clarity an issue with the jobs. My repro has the same result, but is set up via ECS.

 

Well whether it's classified as a bug or not is of less interest to me, I would just say that the 'after' pic is a very sexy core utilisation result ! I will definitely be keeping my eye out on alpha releases for this change! I think it's going to give a lot of people who have well-jobified code a very welcome performance boost!

Thanks for the improvement!

 

Will this change eventually be backported to 19.1/19.2?

 

Alpha 10 has landed!
"Kernel: Improve main thread utilisation for IJobParallelFor jobs"

I just left for work so can't test now. But looking forward to seeing the difference in my own project.

 

Can we have a way to explicitly state, that we want the job to be working NOT on main thread? Long background jobs is a thing in my project. If Job Scheduler will decide to throw such a thing onto main thread, there will be hell of a a hiccup.

 

It should only throw work on the main thread if the main thread is locked waiting for jobs to complete.

 

Just noticed this thread after posting a new one regarding long jobs. Yep! They definitely cause hiccups
Would definitely love a way to add a note to the scheduler "Hey, don't run this on main"

 

But hey, jobs don't run on the main thread, unless you explicitly ask them to, calling jobHandle.Complete().

 

MMmm the profiler says differently.

Edit: I think I know why it is now: 2019.3a11

 

Jobs should only run on main thread if the main thread is locked waiting for the job to complete due to dependencies.

You can force this by calling Complete() but the dependency system may eventually require certain jobs to complete before a system can run etc.

If I simply schedule a long running IJobParallelFor with no dependencies on it then it won't ever run on main thread. This is easily shown calculating a bunch of prime numbers.



Never touches main thread. Maxing my cpu usage but I'm still pumping out 100s of fps.

Now the issue is all your worker threads are at capacity so if I wanted to schedule any other job it will have to be scheduled on main thread. So if you're running a multi frame job I'd highly recommend keeping it to a single thread.

 

Your last paragraph.

Not sure if this is possible, but if the scheduler could accept a prio (ASAP vs long running) and then only run the long running, when not busy with ASAP

 

This could be a very nice feature.

 

Uhm, no. Even without dependency locking, my job is still being sent to main thread periodically. There needs to be a way to specify if a job should be allowed to run on main thread instead of hoping magic will suffice the scheduler

 

Show us your code because as I said and demonstrated it seems to work for me fine

 

Hi guys. I am facing a kind of same issue. I am scheduling collision Jobs and simple IJob, with a schedule call but they are running on main thread. Aren't they need to be run on worker thread?
Currently I don't have screenshot of profiler but I will share it ASAP.

Edit:
The Default World Collision is the system and it is running on main thread.

Code (CSharp):

1. [BurstCompile]
2.     struct CollisionFall : ICollisionEventsJobBase
3.     {
4.         public ComponentDataFromEntity<BucketComponent> bucket;
5.         public ComponentDataFromEntity<BlastComponent> component;
6.         public ComponentDataFromEntity<PhysicsGravityFactor> gravity;
7.         public ComponentDataFromEntity<PhysicsMassOverride> physicMass;
8.         public BufferFromEntity<BufferComponent> buffer;
9.  
10.         public void Execute(CollisionEvent collisionEvent)
11.         {
12.             Entity entityA = collisionEvent.EntityA;
13.             Entity entityB = collisionEvent.EntityB;
14.  
15.             if (component.HasComponent(entityA) && bucket.HasComponent(entityB))
16.             {
17.                 if (!component[entityA].isBlasted)
18.                 {
19.                     physicMass[entityA] = new PhysicsMassOverride
20.                     {
21.                         IsKinematic = 0
22.                     };
23.  
24.                     gravity[entityA] = new PhysicsGravityFactor
25.                     {
26.                         Value = 1
27.                     };
28.  
29.                     BlastComponent blastComponent = component[entityA];
30.                     blastComponent.isBlasted = true;
31.                     component[entityA] = blastComponent;
32.                 }
33.             }
34.  
35.             if (component.HasComponent(entityB) && bucket.HasComponent(entityA))
36.             {
37.                 if (!component[entityB].isBlasted)
38.                 {
39.                     physicMass[entityB] = new PhysicsMassOverride
40.                     {
41.                         IsKinematic = 0
42.                     };
43.                     gravity[entityB] = new PhysicsGravityFactor
44.                     {
45.                         Value = 1
46.                     };
47.                     component[entityA] = new BlastComponent
48.                     {
49.                         isBlasted = true
50.                     };
51.                 }
52.             }
53.         }
54.     }

This is the collision Job

Code (CSharp):

1.   protected override JobHandle OnUpdate(JobHandle inputDeps)
2.     {
3.         var job = new CollisionFall();
4.         job.component = GetComponentDataFromEntity<BlastComponent>(false);
5.         job.bucket = GetComponentDataFromEntity<BucketComponent>(false);
6.         job.gravity = GetComponentDataFromEntity<PhysicsGravityFactor>(false);
7.         job.physicMass = GetComponentDataFromEntity<PhysicsMassOverride>(false);
8.  
9.         JobHandle jobHandle = job.Schedule(m_stepPhysicsWorld.Simulation, ref m_BuildPhysicsWorld.PhysicsWorld, inputDeps);
10. return jobHandle;

My Main System

 

If you call Complete() early whatever left of the job will complete on the main thread. I have set up all my "long" jobs to use the IsComplete-check and will at the earliest complete on the next frame. I don't know if ECS/DOTS have a concept of a LateUpdate() so that's why I am doing the check.

 

This is also a 1 year necro as well as a double post. Your new thread was sufficient, not to mention your claim that you are facing a "kind of same issue" is not true. This thread was about poor utilisation of the main thread while waiting for jobs completion despite many jobs still waiting to be executed, while you are asking about how to schedule jobs on worker threads.

I'm not typically a necro cop but this is a little silly ^^;

 

I thought it's related to this post rather than creating a new thread, after that I thought I should create a new thread as well.