Question How to force heavy scheduled job to run first before small jobs?

Pre-1.x it was enough to just run system first, and the scheduled job would match execution order with system.
In 1.x this happens:

Since systems OnUpdate are now way faster, they generate smaller jobs faster.
Jobs that are smaller will run first delaying heavy jobs later into the frame, causing a stall as a result on the first read / write attempt. Small jobs aren't dependent on the large ones.

I could make something like a fake dependency chain to enforce smaller jobs to run after heavy ones, but it would be a major pain to manage. And well, groups does not affect scheduling order of jobs now as it seems if they do not depend on each other.

Any other option?

 

Interesting, I haven't had this problem but it sounds like something I'll be dealing with sooner than later.

Have you tried

JobHandle.ScheduleBatchedJobs

?

 

Still the same delay. One more thing I'm thinking about.

It seems like since 2022.2 parallel jobs have higher priority.
Small jobs are pushed ahead of parallel jobs which results in a stall.

Basically:
1. Small jobs; (Ignores priority due to queue changes and tries to snatch as much as possible)
2. Parallel job; (Higher priority)
3. Single job; (Gets lower priority as a result)

Since parallel job has to finish on all threads where there are extra small jobs before the parallel one - it just stalls more than it should.

There doesn't seems to be a way to set force priority since internally its basically high or low. A bummer.

 

Even without parallel job it looks weird, like this:

Two jobs scheduled from the same system gets separated to the different threads.
Which causes a different case stall.

 

Even if its scheduled as a one single job other small jobs "steal" queue and try to push their small jobs.
E.g. with IJob + ArchetypeChunk iteration.

Gathering chunk job is expected, but as soon as that completes - job from the different system executes:




Then there's this case where job queue steals into same heavy job:


Pure chaos.

I guess job system needs some kind of a guide to mark a job like:
- "Hey this job chain is heavy, don't try to steal from other queues";
- "Hey, this job chain was already heavy, do not execute heavy jobs again";

Or some kind of priority system.
Idk. Right now it looks like single long running jobs is a no.

 

Is this happens in the one of the first frames? Or later at runtime?
Job scheduler does something smart with jobs, which may reorganise jobs, as they run, over duration of frames.
Not sure how smart it is tho.

Unless something changed with jobs scheduling, as long you didn't queued jobs together, you could just create jobs, which run independently from each other. And then they should have be assigned in own threads.

Be in mind, I haven't tested this approach for a long time now. So don't know if this is still valid behaviour of jobs scheduling.

 

If there is a thread idling before the big job gets consumed by a worker thread, then you have an accidental job dependency somewhere. What you describe about prioritization concerns would only be valid concerns if your worker threads were swamped with jobs. But in your captures, most of your worker threads are idling, so this is not the issue you are having.

The two most common causes of the issue you are having:
1) You forgot to set readonly to true for a lookup type on the main thread.
2) Unity ordered your systems (not jobs, systems) in a really dumb way that lengthens your dependency chain.

 

Later into [all] frames at random.

Pretty much this happens:
https://blog.unity.com/engine-platform/improving-job-system-performance-2022-2-part-2
(see how queue / stealing is handled now)

I've tried detaching job from the chain, and even adding extra. But it doesn't seems to matter.

1. Could be. But I don't think this is the case. Even without extra dependencies (e.g. last screenshot), job system attempts to jam heavy job into the same heavy job completely ignoring other threads. It happens more rarely, but causes a spike. Dependencies that read from the data structure are added later into the frame. These ones are the ones that fill the octree (write) first.

Overall, I was thinking out dropping this kind of spatial partitioning for a different structure that supports parallel insertion. So that job wouldn't be this long. However, might just try to optimize container first. Got a few ideas.

2. Its manually grouped / ordered against. Was working prior in 0.51, but now jobs tend to run whenever they want to.

 

In 2022, jobs don't get scheduled targeting any particular thread (well, they do, but it is always the main thread for user jobs). Instead, idle worker threads just try to steal whatever they can when they have nothing to do. So if a worker thread is idling, that means that thread believes there is nothing for it to steal to work on in that moment. You need to ask yourself why that is.

There's a very slim chance that there is a bug in Unity where the worker threads are missing a signal. But I think what is much more likely is that there's something you are overlooking. Perhaps there's a job that only gets scheduled on very few frames that causes a dependency to exist between the two big jobs?

If you don't believe me, there is a blog post about the changes to the job system scheduling for 2022. I suggest reading it.

 

Pushing more work off main thread is wip™. Its just hybrid things.

I've tried test setup with jobs constantly running. And it just happens.
Sometimes job system schedules heavy jobs near instantly. Sometimes it gets delayed way later into the frame because something else is already running e.g. some kind of parallel job. Or just another job. Which causes heavy job to get delayed. And while its delayed something else gets stolen and cycle repeats until main thread requests complete without heavy job even starting. Which is suboptimal.

Yeah I believe you, I've literally posted link to it in the same post

It would be great to able to specify priority of the jobs.
Plus it seems priority scheduling already exists for native components like Animator jobs etc, just not exposed. Something like an attribute or as a "Hint" during scheduling would be great to see.

 

This is possible, and requires a few more tricks to fix. But none of the screenshots you've shared actually show this. All of your screenshots show at least one worker thread idling.

Indeed you did. Sorry, I skimmed over that part and missed that detail. And also, the way you were reasoning about things in the post seemed to contradict the blog post.

Anyways, my point stands that if you have a worker thread idling and your big job isn't being picked up right away, you have dependency issues. And if you have your worker threads filled to the brim up until the big job starts running, then share a profiler capture of that full frame along with some dependency requirements, and I can propose a couple of ways to deal with it.

 

So, I can't reproduce it anymore for some reason. I've added more load to the main thread, and now jobs are executed near instantly. Either slowing down main thread helped, or something else was running in the background messing around with multithreading. I don't even know at this point.

If it happens again, I'll post a profiler capture.

 

Restarting Unity