IJobForEach only uses one core?

I have a job using IJobForEach that is trivially parallel across entities and when I look in the debugger I see this:



I was under the impression that IJobForEach supported parallelism across cores. If that is not the case, what is the intended workflow for implementing such behavior?

I've also unable to remove parallel write restrictions from BufferFromEntity<T> when using it in multiple IJob's I manually schedule in a loop. Is this intended behavior or a bug?

 

Are you using .Schedule() or ScheduleSingle() on your job?
Are you using a command buffer? If so is it marked as concurrent?

 

It’s parallelize work per chunk.

 

IJobForEach runs parallel on chunks, how many chunks are you processing?

Parallel write from multiple Jobs is not supported as far as I can tell (concurrent versions allow parallel from 1 job)

edit: sorry, forgot to press send and in the meanwhile there are already 2 replies...

 

@siggigg I'm using Schedule and no command buffers.

I probably do not have enough entities to break across a chunk border in this example. Having chunks be the only determining factor for IJobForEach parallelism isn't great if that's true. ECS principles aren't about how many entities you can shove on screen at once, but how efficiently you can split up work across them.

Parallel write to most containers is definitely supported via the `NativeDisableParallelForRestriction` and `NativeDisableContainerSafetyRestriction` attributes. If you know what you are doing it is perfectly reasonable and expected that you need this functionality.

 

Also if the per-chunk concurrency is true and not just a current limitation that will be enhanced in the future, it would be great if the docs could mention this. Currently they say `If you used the Schedule() method instead, the system uses parallel jobs to process the entities.`

 

you can always use the generic IJobParallelFor to go parallel

to which container have you been able to write to from multiple parallel scheduled IJobs --- I recall that I had trouble doing this a few ECS version back (I have not used ECS in a bit)

 

If I have a regular job containing a mutable BufferFromEntity<T> that is marked to disable safety checks, when I actually pull a buffer out of it inside of a job the safety check system blows up and complains that concurrent writes are not allowed.

I haven't had issue with other containers so far.

Also an issue with IJobParallelFor is you cannot schedule it and express to the job system your dependency on an EntityQuery. So if I want to use BufferFromEntity within it I cannot because it would be unsafe for the job system to schedule.

 

I did extensive performance tests on a custom collision system a few months back - initially i tried to do parallelize as much as possible, but parallel writes to containers are costly (you gain much less than scheduling independent jobs parallel) - but I don't know your setup.

I think I used a Buffer with interlock to write in parallel to it (worked, but was not worth it in the end) --- let me see if I find it

 

I'm not writing to the same DynamicBuffer at once. I have N entities, each of which I am doing a CPU intensive task on, and individually they write to their own buffers/component data. It's a perfect use case for disabling parallel safety checks.

Basically I'm going to need to allocate a bunch of intermediate result containers via Allocator.Temp, have my IJobParallelFor write to those instead of a DynamicBuffer, and then have another job at the end run via a IJobForEach and copy the contents into the per-entity DynamicBuffer. I then also need another system at the end of the frame to force complete the job and make a callback to deallocate the temp data.

This is the third time I've ran into this problem on my project, so it's not a terribly uncommon use-case if you have some demanding CPU tasks unfortunately.

 

maybe this helps - old API

The SpriteToGridBufferParallelInterlockedJob is of your interest

Spoiler: update - job is scheduled here

Code (CSharp):

1.         JobHandle MyUpdateJobified(JobHandle jobHandle)
2.         {
3.             //sampler.Begin();
4.          
5.             ClearNativeContainers();
6.          
7.             var collisionBufferCandiateArray = collisionCandidateKeys_Group.GetBufferArray<CollisionInfoBuffer>();
8.             var collisionBufferPairArray = collisionPairKeys_Group.GetBufferArray<CollisionPairBuffer>();
9.  
10.             // Clear spatial grid - ParallelFor
11.             jobHandle = new ClearBufferDictionaryJob
12.             {
13.                 collisionCandidateKeyArray                = collisionBufferCandiateArray,
14.                 collisionPairKeyArray                    = collisionBufferPairArray
15.             }.Schedule(collisionBufferCandiateArray.Length, 1, jobHandle);
16.          
17.             // Assign sprites to spacial grid - IJobProcessComponentDataWithEntity
18.             var bufferLocks = new NativeArray<int>(collisionBufferCandiateArray.Length, Allocator.TempJob);
19.             jobHandle = new SpriteToGridBufferParallelInterlockedJob
20.             {
21.                 grid                                    = myGrid,
22.                 keyArray                                = collisionBufferCandiateArray,
23.                 bufferLocksArray                        = bufferLocks
24.             }.ScheduleGroup(sprite_Group, jobHandle);
25.          
26.             // Check collisions per grid - ParallelFor
27.             jobHandle = new AABBCollisionBufferToBufferJob
28.             {
29.                 collisionCandidates             = collisionBufferCandiateArray,
30.                 collidingEntities                = collisionBufferPairArray
31.             }.Schedule(collisionBufferCandiateArray.Length, 1, jobHandle);
32.          
33.             // Merge collisions per grid into a unique collision pair / collision entity hashmap - ParallelFor
34.             jobHandle = new MergeCollisionsPerGridDistinctFromBufferJob
35.             {
36.                 InputBuffer                        = collisionBufferPairArray,
37.                 DistinctCollisionPairHashMap    = distinctCollisionPairHashMap.ToConcurrent(),
38.                 DistinctCollisionEntityHashMap  = distinctCollisionEntityHashMap.ToConcurrent()
39.             }.Schedule(collisionBufferPairArray.Length, 1, jobHandle);
40.      
41.             // Color Colliding Entities - IJobProcessComponentDataWithEntity
42.             jobHandle = new ColorCollidingEntitiesJob
43.             {
44.                 DistinctCollisionEntityHashMap        = distinctCollisionEntityHashMap,
45.                 hitColor                            = Settings.hitColor,
46.                 normColor                            = Settings.normColor
47.             }.Schedule(this, jobHandle);
48.          
49.             // start processing all scheduled jobs
50.             JobHandle.ScheduleBatchedJobs();
51.          
52.             //jobHandle.Complete();
53.             //sampler.End();
54.          
55.             return jobHandle;
56.         }

Spoiler: SpriteToGridBufferParallelInterlockedJob

Code (CSharp):

1.     [BurstCompile]
2.     public struct SpriteToGridBufferParallelInterlockedJob : IJobProcessComponentDataWithEntity<Box>
3.     {
4.         [ReadOnly] public ColGrid grid;
5.         [NativeDisableParallelForRestriction, WriteOnly] public BufferArray<CollisionInfoBuffer> keyArray;
6.         [NativeDisableParallelForRestriction, DeallocateOnJobCompletion] public NativeArray<int> bufferLocksArray;
7.          
8.         public void Execute(Entity e, int i, [ReadOnly] ref Box box)
9.         {  
10.             var boxMinGrid = (int2) ((box.Center - box.Extends - grid.Min) * grid.OneOverCellSize);
11.             var boxMaxGrid = (int2) ((box.Center + box.Extends - grid.Min) * grid.OneOverCellSize);
12.  
13.             for (int x = boxMinGrid.x; x <= boxMaxGrid.x; x++)
14.             {
15.                 if (x >= 0 && x < grid.Dim.x)
16.                 {
17.                     for (int y = boxMinGrid.y; y <= boxMaxGrid.y; y++)
18.                     {
19.                         if (y >= 0 && y < grid.Dim.y)
20.                         {
21.                             var key = x + y * grid.Dim.x;
22.                             unsafe
23.                             {
24.                                 while(Interlocked.CompareExchange(ref ((int*)bufferLocksArray.GetUnsafePtr())[key], -1, 0) != 0) {}  
25.                             }
26.                              
27.                             keyArray[key].Add(new CollisionInfoBuffer{entity = e, box = box});
28.                             bufferLocksArray[key] = 0;
29.                         }
30.                     }
31.                 }
32.             }
33.         }
34.     }

 

That's using a deprecated API. ArchetypeChunk has a GetBufferAccessor on it now, but that seems usable only within an IJobChunk -- which has the same issue as IJobForEach, I need maximum per-entity parallelism.

 

Yes, I wrote it uses old API, actually I just saw that I updated it at the time...maybe this is a bit newer but also used deprecated API (i.e. the job rename, etc.)

Spoiler: Update, scheduling job

Code (CSharp):

1.         JobHandle MyUpdateJobified(JobHandle jobHandle)
2.         {
3.             //Profiler.BeginSample("COLLISION");
4.             ClearNativeContainers();
5.            
6.             var collisionBufferCandiateFromEntity = GetBufferFromEntity<CollisionInfoBuffer>();
7.    
8.             // Clear spatial grid - ParallelFor
9.             jobHandle = new ClearCandidateBufferFromEntityDictionaryParallelForJob
10.             {
11.                 collisionBufferEntityArray                = collisionBufferEntityArray,
12.                 collisionBufferCandiateFromEntity        = collisionBufferCandiateFromEntity
13.             }.Schedule(collisionBufferEntityArray.Length, 1, jobHandle);
14.            
15.             // Assign sprites to spacial grid - IJobProcessComponentDataWithEntity
16.             var bufferLocks = new NativeArray<int>(myGrid.CellCount, Allocator.TempJob);
17.             jobHandle = new SpriteToGridBufferNewApiParallelInterlockedJob
18.             {
19.                 grid                                    = myGrid,
20.                 keyArray                                = collisionBufferCandiateFromEntity,
21.                 keyIndexArray                            = collisionBufferEntityArray,
22.                 bufferLocksArray                        = bufferLocks
23.             }.ScheduleGroup(sprite_Group, jobHandle);
24.            
25.             // Check collisions per grid - ParallelFor
26.             jobHandle = new AABBCollisionBufferToHashMapNewApiJob
27.             {
28.                 collisionCandidates                     = collisionBufferCandiateFromEntity,
29.                 DistinctCollisionPairHashMap            = distinctCollisionPairHashMap.ToConcurrent(),
30.                 DistinctCollisionEntityHashMap          = distinctCollisionEntityHashMap.ToConcurrent()
31.             }.ScheduleGroup(collisionCandidateKeys_Group, jobHandle);
32.        
33.             // Color Colliding Entities - IJobProcessComponentDataWithEntity
34.            
35.             jobHandle = new ColorCollidingEntitiesJob
36.             {
37.                 DistinctCollisionEntityHashMap            = distinctCollisionEntityHashMap,
38.                 hitColor                                = Settings.hitColor,
39.                 normColor                                = Settings.normColor
40.             }.Schedule(this, jobHandle);
41.            
42.            
43.             // start processing all scheduled jobs
44.             JobHandle.ScheduleBatchedJobs();
45.             //jobHandle.Complete();
46.             //    Profiler.EndSample();
47.             return jobHandle;
48.         }

Spoiler: job (newer api)

Code (CSharp):

1.     [BurstCompile]
2.     public struct SpriteToGridBufferNewApiParallelInterlockedJob : IJobProcessComponentDataWithEntity<Box>
3.     {
4.         [ReadOnly] public ColGrid grid;
5.         [NativeDisableParallelForRestriction, WriteOnly] public BufferFromEntity<CollisionInfoBuffer> keyArray;
6.         [ReadOnly] public NativeArray<Entity> keyIndexArray;
7.         [NativeDisableParallelForRestriction, DeallocateOnJobCompletion] public NativeArray<int> bufferLocksArray;
8.            
9.         public void Execute(Entity e, int i, [ReadOnly] ref Box box)
10.         {  
11.             var boxMinGrid = (int2) ((box.Center - box.Extends - grid.Min) * grid.OneOverCellSize);
12.             var boxMaxGrid = (int2) ((box.Center + box.Extends - grid.Min) * grid.OneOverCellSize);
13.    
14.             for (int x = boxMinGrid.x; x <= boxMaxGrid.x; x++)
15.             {
16.                 if (x >= 0 && x < grid.Dim.x)
17.                 {
18.                     for (int y = boxMinGrid.y; y <= boxMaxGrid.y; y++)
19.                     {
20.                         if (y >= 0 && y < grid.Dim.y)
21.                         {
22.                             var pos = x + y * grid.Dim.x;
23.                             var key = keyIndexArray[pos];
24.                             unsafe
25.                             {
26.                                 while(Interlocked.CompareExchange(ref ((int*)bufferLocksArray.GetUnsafePtr())[pos], -1, 0) != 0) {}  
27.                             }
28.                            
29.                             keyArray[key].Add(new CollisionInfoBuffer{entity = e, box = box});
30.                             bufferLocksArray[pos] = 0;
31.                         }
32.                     }
33.                 }
34.             }
35.         }
36.     }

 

I'm not sure I see how this is different though. IJobProcessComponentData == IJobForEach and IJobForEach apparently only supports concurrency per-chunk. So same issue with it not allowing for per-entity concurrency.

Regardless, thanks for the suggestions. I'm waiting to see a Unity response to this for some additional clarification.

 

Per entity parallelism it’s not the best choise. This is why unity use per chunk logic for IJFE. You get overhead of scheduling job per entity. If you still want it, only way (better of all) is use IJobParallelFor with low batch count.

You can. Of course if you write safely.

 

Hold on. You had 2 issues

- IJobForEach -> I said use IJobParallelFor if you do not want the per chunk parallelism of IJobForEach

- parallel write to buffer -> the above examples with the old api (although I am not sure I understood you correctly)

 

I cannot access DynamicBuffer in this job type safely (as far as I'm aware, please point me to an example if it exists!). So then I must externally allocate some temp data array storage. But then I cannot provide an array of native arrays (or whatever, could be NativeQueue) for the IJobParallelFor to lookup into by index. So right back to using N * IJob.

Those examples above use either an old API, or the IJobForEach equivalent that has issues outlined in my previous post.

 

If no one sent example to you before tomorrow, I’ll show you tomorrow, cos now I’m in bed

 

I disagree. The assumption here is that you have so many entities that "of course" you'd want to avoid job scheduling overhead by batching them. But in my example I am running an RVO algorithm that is very expensive per-entity. If I only ever have as many entities that fit in a single chunk (or two) I am hitting a major bottleneck.

In my post above I outline why this isn't an option due to the need for a re-sizable native container per-entity.

I have not found an example that shows how to express an EntityQuery dependency in a normal job type. Every example I've seen writes to pre-allocated native containers and then eventually writes back into the ECS data.

 

Yes please! It's using the most recent API right?

 

I just implemented the method I described above and here is what the profiler looks like now:



I've gone from 0.6ms to update 20 entities down to under 0.075ms to update them all via 20 IJob's.

 

Following example should work if you can use
IJobProcessComponentDataWithEntity and IBufferElementData

 

Yep in this case it’s fine.

And buffers cover that without problem

It’s only one case. You always can use GetBufferFromEntity, GetComponentDataFromEntity, and schedule parallel job by count of entities from EQ, and many other possibilities

 

I always on latest API version

 

IJobForEachWithEntity IJPCDWE deprecated name

 

No it doesn't, again it has the same per-chunk concurrency issue I have been describing.

 

Is IJobProcessComponentDataWithEntity now depreciated with 2019.x?

 

Oh well, I tried

 

Keep calm, I’ll show you tomorrow with IJobParallelFor

 

Name - yes. Now it’s called IJobForEachWithEntity

 

Oh well, there will be a big catch up for me, when I update from 2018.3 to 2019.x. Still need finish phase of my dev, before transition. So I feel now I bit outdated already

 

I don't see any verification this is true. If you look inside the source for JobForEachExtensions there is a lot of read/write safety registration going on when you schedule those types of jobs by passing either an EntityQuery or a ComponentSystem reference to Schedule().

Let's say within a system I schedule dependent jobs A -> B (regular IJobParallelFor, has BufferFromEntity<T>) -> C. At the same time job D (IJobForEach) is scheduled within the system for running and references the same BufferFromEntity<T>. If B and D are scheduled to run at the same time by the job scheduler because it only understands that job D is using the ECS safety system, you have a race issue and the safety system will yell at you.

Again, I would love some Unity clarification on this because without access to the job scheduler system source I can't directly verify these assumptions.

 

Parallelism runs on a per chunk basis to avoid multiple threads accessing data on the same cache line and thereby leading to unexpected performance degradation via false sharing.

For your use case I would suggest using a ISharedComponentData on each entity with a value 0 -> n-1 (n being the number of threads on your platform) to force entities to be split across a minimum number of chunks. With this you could also then switch to IJobChunk iteration, play nicely within Unity’s recommended restrictions and give yourself some more flexibility on accessing the buffers directly without the BufferFromEntity lookup.

 

I access NativeArrays and BufferArrays in parallel all the time. They work fine in different cases.
Unless OP is discussing specific job type. But I don't see issue there.
Unless something has changed in 2019.x in that department.

 

There is nothing wrong with scheduling a few IJobs in parallel, either. It seems this works for your case?

Your use case is a bit too abstract for me to further contribute - I seem to miss the point.

 

You shouldn’t process same data within different jobs in the same time, it’s race condition by definition, if you can’t guarantee safety (for example if you sure that your jobB write to something in range 0-10 and jobD in range 11-20, it’s completely fine and you can disable safety restrictions) or you not use concurrent containers.

Edit: Ooops my bad, after night I look at this more clear, different jobs in parallel process one container not possible by safety restriction (even if you can guarantee safety) only in dependency chain. But you still can process one container in parallel inside one parallel job (IJobParallelFor for example)

 

So am I. If multiple IJob fine for him, than IJobParallelFor completely cover his case.

 

Nothing changed around OP case.

 

I had thought of this, but it's a bit of a short-sighted optimization. You would be optimizing based on the usage pattern of one system and then causing other systems that would benefit from per-chunk access to suffer.

 

Well my main issue has been that it's very clunky to implement what I've ended up needing to do. It seems like something that should be much easier given how smooth IJobForEach/IJobChunk are, and with some small API additions it could be.

 

By definition you have need to have some sort of default behavior and for IJobForEach it follows the general use case. In a few months time, as the preview of this package matures, there may be a more specific solution. You do however already have other mechanisms to augment or create behavior that matches your requirements, like the one I provided above. Not sure what more we can otherwise reasonably expect.

 

Well my use case would be immediately satisfied if IJobForEach.Schedule had the option to explicitly specify the amount of concurrency you need. Be it per-entity, per-chunk, or like IJobParallelFor where you can specify the amount of work a single job thread can steal.

 

Wouldn't be easier to create a NativeArray with the entities using EntityQuery.ToEntityArray(Allocator.TempJob), combining IJobParallelFor with innerloopBatchCount set to 1 (or something like NativeArray.Length / ProcessorThreadCount) and use ComponentData/BufferFromEntity to access the data?

If it's really a few entities the NativeArray creation should be fast and the lookup in ComponentData/BufferFromEntity should be minimal compared to your algorithm, right?

[]'s

 

Yea the NativeArray creation is very fast, and I use it in my solution using N IJob's.

I can't however seem to get mutable access with the [DisableNative*] attributes to a DynamicBuffer from BufferFromEntity inside any job type except a IJobForEach/IJobChunk. Despite ensuring I am never concurrently reading/writing to the same entity's buffer, the safety system blows up on me. I posted about this elsewhere because it feels like a bug. I don't have the same problem with ComponentDataFromEntity.

 

As I promised. Here is fast sample. I create 6 entities, every antity has buffer with length 2, in OnCreate I set values in buffer equal "i" index. Inside job I set value 100 to every buffer item attached to every entity, every entity processed on different thread. It's your case?

Code (CSharp):

1. [InternalBufferCapacity(2)]
2. public struct SomeBuffer : IBufferElementData
3. {
4.     public int Value;
5. }
6.  
7. public class Sample : JobComponentSystem
8. {
9.     private EntityQuery _query;
10.  
11.     protected override void OnCreate()
12.     {
13.         _query = GetEntityQuery(typeof(SomeBuffer));
14.      
15.         var archetype = EntityManager.CreateArchetype(typeof(SomeBuffer));
16.         var entitiesArr = new NativeArray<Entity>(7, Allocator.TempJob);
17.      
18.         EntityManager.CreateEntity(archetype, entitiesArr);
19.         Debug.Log("-----------------On Create----------------");
20.         for (int i = 0; i < entitiesArr.Length; i++)
21.         {
22.             var buffArr = EntityManager.GetBuffer<SomeBuffer>(entitiesArr[i]);
23.             for (int j = 0; j < 2; j++)
24.             {
25.                 buffArr.Add(new SomeBuffer()
26.                 {
27.                     Value = i
28.                 });
29.             }
30.             Debug.Log($"{entitiesArr[i].Index} - [{buffArr[0].Value},{buffArr[1].Value}]");
31.         }
32.         Debug.Log("-----------------On Create----------------");
33.         entitiesArr.Dispose();
34.     }
35.  
36.     private struct ParallelJob : IJobParallelFor
37.     {
38.         [NativeDisableParallelForRestriction] public BufferFromEntity<SomeBuffer> buffer;
39.         [DeallocateOnJobCompletion] public NativeArray<Entity> ents;
40.         public void Execute(int index)
41.         {
42.             var e = ents[index];
43.          
44.             Debug.Log($"{e.Index} - [{buffer[e][0].Value},{buffer[e][1].Value}] Inside Job");
45.          
46.             var buffArr = buffer[e];
47.             for (int j = 0; j < buffArr.Length; j++)
48.             {
49.                 var buffElement = buffArr[j];
50.                 buffElement.Value = 100;
51.                 buffArr[j] = buffElement;
52.             }
53.         }
54.     }
55.  
56.     protected override JobHandle OnUpdate(JobHandle inputDeps)
57.     {
58.         inputDeps = new ParallelJob()
59.         {
60.             buffer = GetBufferFromEntity<SomeBuffer>(),
61.             ents = _query.ToEntityArray(Allocator.TempJob, out var arrDeps)
62.         }.Schedule(_query.CalculateLength(), 1, JobHandle.CombineDependencies(arrDeps, inputDeps));
63.  
64.     return inputDeps;
65. }
66. }

 

Yep right, missed this when thought about OP case, edited my post Thanks!

 

I was write two NativeArray<> ,but when i run the procedures it error. why? and how to dispose it(this two was used for save the child data, just like a chain, and I use the BaseBode to record all the childNode information )? thank you for your help;

 

You can't use native containers in IComponentData. Use dynamic buffer instead.