Looping over two separate entity types

Suppose I have a large number of entities with one set of components and a small number of entities with a different set of components - let's call them "Subjects" and "Observers", or something. I want each Observer to keep track of all the Subjects in a scene, do some math based on the values of each Subject's components and its own distance from that Subject, and write the final result to a value in its Observer component.

Sounds easy enough. I've done it before without ECS by just making a script for it and attaching that to every Observer object. Could also do it in a single script with a nested loop. What I am having trouble with is wrapping my head around how to do this in a JobComponentSystem. If I were to write an IJobForEach looking for the Subject and Observer components, as I understand it, the job would try to loop over every entity that has both the Subject and the Observer components, which is not what I want. And we can't schedule jobs from within jobs. So, while this feels like a stupid question, how can I implement what is essentially a nested for loop over two sets of entities in a system?

 

So admittedly I haven't tested any of these personally but here are some things to try:
1) Nest an Entities.ForEach inside an Entities.ForEach, with the inner ForEach using WithoutBurst() and Run().
2) Nest an IJobChunk.RunWithoutJobs() inside a Entities.ForEach.
3) Get the NativeArray<ArchetypeChunk> of both queries and manually iterate chunks.

As an aside, this might sound strange, but there's a decent chance you will get better performance making the observers be the inner loop if your job is single-threaded. The reason for that is the small number of observers can likely fit and hang around in cache the whole time which saves you bandwidth of having to reload the subjects for each observer.

 

Somehow, I hadn't thought of just nesting ForEach in the main thread. That does achieve the functionality I want. I would really like to be able to do it using jobs, though, since it is math being done on a whole lot of entities, multiple times each. My first thought was to schedule jobs in ForEach, but since the OnUpdate function in JobComponentSystem has to return a JobHandle, I don't think that is actually possible? It would only return the handle for one job.

I do not strictly need every single Observer to update every frame, so I could theoretically stagger them such that a different Observer runs its calculation each frame, but that seems like an inelegant workaround for something that feels like it should be simple. Maybe my intuition is just wrong for multithreading?

 

I haven't tested this other than checking that it compiles and runs, but you should be able to do something like this:

Code (CSharp):

1. public class TestSystem: SystemBase
2. {
3.     private EntityQuery subjectQuery;
4.     protected override void OnCreate()
5.     {
6.         this.subjectQuery = GetEntityQuery(
7.             ComponentType.ReadOnly(typeof(Subject)),
8.             ComponentType.ReadOnly(typeof(Translation))
9.         );
10.     }
11.  
12.     protected override void OnUpdate()
13.     {
14.         var subjectEntities = this.subjectQuery.ToEntityArrayAsync(Allocator.TempJob, out JobHandle subjectHandle);
15.         Dependency = JobHandle.CombineDependencies(Dependency, subjectHandle);
16.  
17.         Dependency = Entities.ForEach((ref Observer observer, in Translation translation) =>
18.         {
19.             var averageDist = 0.0f;
20.             for (int i = 0; i < subjectEntities.Length; i++)
21.             {
22.                 var subjectData = GetComponent<Subject>(subjectEntities[i]);
23.                 var subjectTranlation = GetComponent<Translation>(subjectEntities[i]);
24.                 var diff = translation.Value - subjectTranlation.Value;
25.                 var dist = math.sqrt(math.dot(diff, diff));
26.                 averageDist += dist;
27.             }
28.             averageDist = averageDist / subjectEntities.Length;
29.             observer.AverageDistance = averageDist;
30.         }).ScheduleParallel(Dependency);
31.        
32.         Dependency = subjectEntities.Dispose(Dependency);
33.     }
34. }

The idea being only going parallel on the outer loop and processing the inner loop sequentially. You might want to experiment with inverting your inner and outer loops if the calculation allows depending on which component has greater numbers, or using chunk iteration for the inner loop. But the basic premise seems to work anyway without over-complicating things too much

 

I finally got around to working on this again and made some progress. I did not manage to get the above example to work; Unity wants me to run that ForEach with .Run() and .WithoutBurst(), which I believe would negate the multithreading. I did get the following to run:

Code (CSharp):

1.    protected override void OnCreate()
2.     {
3.         m_Group = GetEntityQuery(typeof(Translation), typeof(Subject));    // Query for all entities with Translation and Subject components
4.     }
5.  
6.     protected override JobHandle OnUpdate(JobHandle inputDeps)
7.     {
8.         ObserverJob observerJob = new ObserverJob
9.         {
10.            // Pass the Subject and Translation components into the job as NativeArrays
11.             subjectComponents = m_Group.ToComponentDataArray<Subject>(Allocator.TempJob),
12.             subjectTranslations = m_Group.ToComponentDataArray<Translation>(Allocator.TempJob)
13.         };
14.  
15.         return observerJob.Schedule(this, inputDeps);
16.     }
17.  
18.     [BurstCompile]
19.     struct ObserverJob : IJobForEach<Translation, Observer>
20.     {
21.         [DeallocateOnJobCompletion] [ReadOnly] public NativeArray<Subject> subjectComponents;
22.         [DeallocateOnJobCompletion] [ReadOnly] public NativeArray<Translation> subjectTranslations;
23.  
24.         public void Execute(ref Translation observerTrans, ref Observer o)
25.         {
26.             o.Value = 0;
27.             for (int i = 0; i < subjectComponents.Length; i++)
28.             {
29.                 Subject s = subjectComponents[i];
30.                 Translation subjectTrans = subjectTranslations[i];
31.  
32.                 if (s.Active)
33.                 {
34.                     float3 distance = subjectTrans.Value - observerTrans.Value;
35.                     float R = math.dot(distance, distance);
36.                     // blah blah more math
37.                 }
38.             }
39.         }
40.     }

I am querying for all Subjects and passing their Translations into an IJobForEach which iterates over every Observer. For each Observer, it looks at all Subjects and performs its calculations.

This code runs successfully and seems to do what I want. I believe it would be most efficient when the number of entities you want to write to is large and the number of entities you only have to read from is small, but unfortunately this is the opposite of my use case.
I was concerned that this approach would perform poorly, since passing large NativeArrays into and out of jobs proved to be a bottleneck when I was implementing jobs in MonoBehaviour, but it works faster than I expected. As a very loose benchmark, it hovers a little above 75 FPS in the editor (acknowledging that profiling in the editor is inaccurate) while handling 100,000 subjects and 50 observers.

I think it should be possible to improve this by making some structural changes, but I found something in the profiler that puzzles me:



It looks like the job is not being distributed properly among the worker threads. A different worker thread is being used each frame, but only one actually executes the job while the others twiddle thumbs.
Could it be my use of an actual for loop in the job that is causing this? It makes perfect sense that the entire for loop has to happen on a single thread, but I would expect multiple observers to be split up between threads somewhat evenly.

What do you all make of this?

 

Entities get split up across threads by chunks. If all your observers are in the same chunk, then your application will run single-threaded.

I'm still not sure why the Entities.ForEach is not working for you.

 

Did you try using SystemBase? You are still using JobComponentSystem in your example. JobComponentSystem and IJobForEach will both be removed in a future release.

 

Using latest API and Chunk iteration for inner loop.
Also avoiding the extra allocs from ToComponentDataArray

Code (CSharp):

1. public class TestSystem : SystemBase
2. {
3.     private EntityQuery m_Group;
4.      
5.     protected override void OnCreate()
6.     {
7.         m_Group = GetEntityQuery(
8.         ComponentType.ReadOnly<Subject>(),
9.         ComponentType.ReadOnly<Translation>());
10.     }
11.  
12.     protected override void OnUpdate()
13.     {
14.         var subjectType = GetArchetypeChunkComponentType<Subject>();
15.         var translationType = GetArchetypeChunkComponentType<Translation>();
16.         var chunks = m_Group.CreateArchetypeChunkArray(Allocator.TempJob);
17.  
18.         Entities
19.             .WithReadOnly(subjectType)
20.             .WithReadOnly(translationType)
21.             .WithDeallocateOnJobCompletion(chunks)
22.             .ForEach((ref Observer observer, in Translation translation) => {
23.  
24.                 observer.Value = 0;
25.  
26.                 for (int c = 0; c < chunks.Length; c++)
27.                 {
28.                     var chunk = chunks[c];
29.  
30.                     var subjects = chunk.GetNativeArray(subjectType);
31.                     var translations = chunk.GetNativeArray(translationType);
32.  
33.                     for (int i = 0; i < chunk.Count; i++)
34.                     {
35.                         if (subjects[i].Active)
36.                         {
37.                             float3 distance = translations[i].Value - translation.Value;
38.                             float R = math.dot(distance, distance);
39.                             // blah blah more math
40.                         }
41.                     }
42.                 }
43.  
44.             }).ScheduleParallel();
45.     }
46. }

 

Aha, that makes sense. I tried reducing the number of Subjects and increasing the number of Observers by a lot, and sure enough, once I create enough Observers, they are split up among the worker threads. 10,000 Observers monitoring 1,000 subjects perform noticeably better than 1,000 Observers monitoring 10,000 Subjects, which just confirms that the ideal use case is having a small number of read-only components and a large number of components to write to.
Is it possible to force the machine to split a smaller number of entities into multiple chunks? But probably it would be more effective to just change my method, maybe to have each Subject do the math for each Observer that sees it and write to the Observers in the main thread.

I did use SystemBase to test thebanjomatic's code, but you raise a good point in that I'd forgotten JobComponentSystem is to be deprecated because I've been in a hole tinkering with this for a while. So it's very possible I did something wrong due to unfamiliarity. I'll have to get familiar with the new workflow and try again, and update my other systems in any case.

Thank you, this is going on my list to try!

 

It turns out the problem was one of versioning. After making sure I had everything updated to the latest versions, I tried these two approaches and both worked. Chris-Herold's code in particular performs much faster than mine.

I think I have a pretty good idea of how to go forward from here. Thanks!

 

Number 1 does not work. I tried it. The compiler gets angry.

 

Yes it wouldn't work as nested ForEach not supported, and Unity throwing error for that (but currently only when you capturing any local variable for ForEach lambda, if you capturing nothing it wouldn't complain but will behave wrong, they already have a ticket for that)
As Joel Pryde said:

We check for nested Entities.ForEach and typically throw an error.  It looks like we don't in the case that the parent Entities.ForEach does not capture a variable however. I added a ticket to fix this case.

 

I get an error DC0012: WithDeallocateOnJobCompletion requires its argument to be a local variable that is captured by the lambda expression.

Entities preview.4 - 0.11.1

 

Found an error: I did not use the variable chunks inside the ForEach

 

Here is my solution:

Code (CSharp):

1.  
2. using Unity.Transforms;
3. using Unity.Mathematics;
4. using Unity.Entities;
5. using Unity.Collections;
6. using Unity.Jobs;
7.  
8. namespace Game
9. {
10.     public class DestructionSystem : SystemBase
11.     {
12.         EndSimulationEntityCommandBufferSystem endSimulationEcbSystem;
13.         private EntityQuery bulletQuery;
14.         float thresholdDistance = 2f;
15.  
16.         protected override void OnCreate()
17.         {
18.             base.OnCreate();
19.             endSimulationEcbSystem = World.DefaultGameObjectInjectionWorld.GetOrCreateSystem<EndSimulationEntityCommandBufferSystem>();
20.             bulletQuery = GetEntityQuery(
21.                 ComponentType.ReadOnly<BulletTag>(),
22.                 ComponentType.ReadOnly<Translation>());
23.         }
24.  
25.         protected override void OnUpdate()
26.         {
27.             var ecb = endSimulationEcbSystem.CreateCommandBuffer().ToConcurrent();
28.             var bulletPos = bulletQuery.ToComponentDataArrayAsync<Translation>(Allocator.TempJob, out var jobHandle1);
29.             Dependency = JobHandle.CombineDependencies(Dependency, jobHandle1);
30.  
31.             float3 playerPosition = (float3)GameManager.GetPlayerPosition();
32.             float dstSq = thresholdDistance * thresholdDistance;
33.  
34.             Dependency = Entities
35.                 .WithAll<EnemyTag>()
36.                 .WithDeallocateOnJobCompletion(bulletPos)
37.                 .ForEach((Entity enemy, int entityInQueryIndex, in Translation enemyPos) =>
38.                 {
39.                     playerPosition.y = enemyPos.Value.y;
40.  
41.                     if (math.distancesq(enemyPos.Value, playerPosition) <= dstSq)
42.                     {
43.                         ecb.DestroyEntity(entityInQueryIndex, enemy);
44.                     }
45.                     else
46.                     {
47.                         for (int i = 0; i < bulletPos.Length; i++)
48.                         {
49.                             if (math.distancesq(bulletPos[i].Value, enemyPos.Value) < dstSq)
50.                             {
51.                                 ecb.DestroyEntity(entityInQueryIndex, enemy);
52.                             }
53.                         }
54.                     }
55.  
56.                 }).ScheduleParallel(Dependency);
57.  
58.             endSimulationEcbSystem.AddJobHandleForProducer(Dependency);
59.         }
60.     }
61. }
62.  

 

This is exactly what I was looking for, next question is how to trigger that the player has been hit. How do I capture the result of line 43. ecb.DestroyEntity(... and make the player die, or some other result after the ForEach has completed?

 

Are you sure that updating resolved this issue? I still have the same as you have in the screenshot.

And why is this happening? All threads execute their jobs in 0.001-0.003 ms, while 1 random thread uses all the rest of the processing time.

 

Every 55 observers, a new parallel job is created. Accordingly, if I create 250 observers, then 8 threads are executed in approximately the same time (1ms), and one in half the time (0.5ms). The remaining 7.5 threads are idle. It doesn't seem to work as I expected, but that's okay.

 

Late to the party here but was experimenting with the ECS system and decided to implement a flocking boid system.
Without a spatial partitioning system, this requires that I do an N^2 algorithm iterating all other boids for each boid. That seemed like a pretty good test of the ECS system's power.

Anyway, I have everything setup for this but I am running into the nitty gritty with the details of building my SystemBase class. I have be scouring the internet looking for documentation and examples and it seems that this stuff has changed a lot over the years so one person or another's example doesn't work with the latest flavor of the Entities package.

I have this SystemBase derived class and (ignoring the fact that my boids might not behave great yet) I am mainly just trying to get a working ECS system example going without errors.

The error that I am running into is this:
IndexOutOfRangeException: Index 0 is out of restricted IJobParallelFor range [0...-1] in ReadWriteBuffer.
ReadWriteBuffers are restricted to only read & write the element at the job index. You can use double buffering strategies to avoid race conditions due to reading & writing in parallel to the same elements from a job.
Unity.Collections.NativeArray`1[T].FailOutOfRangeError (System.Int32 index) (at <ae72fc958ab44fdbb61b9d8c36cf141e>:0)
Unity.Collections.NativeArray`1[T].CheckElementReadAccess (System.Int32 index) (at <ae72fc958ab44fdbb61b9d8c36cf141e>:0)
Unity.Collections.NativeArray`1[T].get_Item (System.Int32 index) (at <ae72fc958ab44fdbb61b9d8c36cf141e>:0)
SBoidSystem+SBoidSystem_LambdaJob_0_Job.OriginalLambdaBody (CBoidData& data, Unity.Transforms.Translation& myPos) (at Assets/Code/ECS/System/SBoidSystem.cs:56)

SBoidSystem.cs:56 is this line from the code below.

Code (CSharp):

1. var chunk = chunks[i];
2.  

Looking for advice on what I am doing wrong here. My chunks NativeArray should be completely read-only so not sure where it is getting classified as a ReadWriteBuffer

Code (CSharp):

1. using Unity.Entities;
2. using Unity.Transforms;
3. using Unity.Mathematics;
4. using Unity.Jobs;
5. using Unity.Collections;
6.  
7. public partial class SBoidSystem : SystemBase
8. {
9.     [ReadOnly]
10.     private EntityQuery m_BoidQuery;
11.  
12.     protected override void OnCreate()
13.     {
14.         base.OnCreate();
15.  
16.         m_BoidQuery = GetEntityQuery(
17.             ComponentType.ReadOnly<Rotation>(),
18.             ComponentType.ReadOnly<Translation>(),
19.             ComponentType.ReadOnly<CBoidTag>()
20.             );
21.     }
22.  
23.     protected override void OnUpdate()
24.     {
25.         float dT = Time.DeltaTime;
26.         EntityManager mgr = World.DefaultGameObjectInjectionWorld.EntityManager;
27.  
28.         // Fetch read only arrays of boid translations and rotations
29.         var typeRot = mgr.GetComponentTypeHandle<Rotation>(true);
30.         var typeTrans = mgr.GetComponentTypeHandle<Translation>(true);
31.  
32.         var chunks = m_BoidQuery.CreateArchetypeChunkArray(Allocator.TempJob);
33.  
34.         Entities.
35.             WithReadOnly(typeTrans).
36.             WithReadOnly(typeRot).
37.             WithNativeDisableParallelForRestriction(typeTrans).
38.             WithNativeDisableParallelForRestriction(typeRot).
39.             ForEach((ref CBoidData data, in Translation myPos) =>
40.         {
41.             data.m_Alignment = math.float3(0.0f);
42.             data.m_AlignmentCount = 0;
43.  
44.             data.m_Cohesion = math.float3(0.0f);
45.             data.m_CohesionCount = 0;
46.  
47.             data.m_Separation = math.float3(0.0f);
48.             data.m_SeparationCount = 0;
49.  
50.             for ( int i = 0; i < chunks.Length; i++)
51.             {
52.                 var chunk = chunks[i];
53.                 var translations = chunk.GetNativeArray(typeTrans);
54.                 var rotations = chunk.GetNativeArray(typeRot);
55.                 for (int j = 0; j < chunk.Count; j++)
56.                 {
57.                     float3 delta = myPos.Value - translations[j].Value;
58.                     float distSq = math.lengthsq(delta);
59.                     if (distSq > 0 && distSq < (2 * 2))
60.                     {
61.                         data.m_Alignment += math.forward(rotations[j].Value);
62.                         data.m_AlignmentCount++;
63.  
64.                         data.m_Cohesion += translations[j].Value;
65.                         data.m_CohesionCount++;
66.                      
67.                         data.m_Separation += delta;
68.                         data.m_SeparationCount++;
69.                     }
70.                 }
71.             }
72.  
73.         }).ScheduleParallel();
74.  
75.         // Average the results when all that is done and combine
76.         JobHandle finalize = Entities.
77. //            WithDeallocateOnJobCompletion(chunks).
78.             ForEach((ref CBoidData data, ref Rotation rot, in Translation pos ) =>
79.         {
80.             if ( data.m_AlignmentCount > 0 )
81.                 data.m_Alignment /= data.m_AlignmentCount;
82.             if ( data.m_CohesionCount > 0 )
83.                 data.m_Cohesion /= data.m_CohesionCount;
84.             if ( data.m_SeparationCount > 0 )
85.                 data.m_Separation /= data.m_SeparationCount;
86.  
87.             float3 direction = (data.m_Alignment + data.m_Cohesion + data.m_Separation) - pos.Value;
88.  
89.             if (math.lengthsq(direction) > 0)
90.                 rot.Value = quaternion.LookRotation(direction, math.up());//  quaternion. Slerp(transform.rotation, quaternion.LookRotation(direction), 5.0f * dT);
91.  
92.         }).ScheduleParallel(this.Dependency);
93.  
94.  
95.         finalize.Complete();
96.  
97.         chunks.Dispose();
98.     }
99. }
100.  

 

Well you didn't call WithReadOnly on it like you did for typeTrans and typeRot, so the job system thinks otherwise.

 

Aha! I see. I am also a newbie with respect to lamba functions in C#. Thank you for pointing that out. That resolved my issue.