Help with SharedComponent job design

I need some help on how to process this system if anyone has the fortitude to understand it.
Basically I have cargo ships (tankers) with cargo containers on them.
I need to run a bunch of systems on the containers but at the ship level. E.g.
- Total mass/centre of mass of the containers
- Total fuel in fuel containers
- Find the highest container on the ship.

To do this, I just add a component to the ships that need those systems, e.g. FuelManagerComponent, MassManagerComponent etc that stores the relevant information.
I went with grouping containers to their ships via a SharedComponent (ParentShip) as I thought this would be the easiest approach but I'm struggling with the syntax.
I've pasted a mass summing system below that sums the mass of the containers on the ship. In pseudo, it does this:

Code (csharp):

1. shipGroup (MassManager)
2. containerGroup (Mass, ParentShip)
3.  
4. OnUpdate()
5.     foreach shipChunk
6.         foreach ship
7.             containerGroup.setFilter(ship.id)
8.             create new ComputeMassJob with filtered containerGroup chunks
9.  
10. ComputeMassJob : IJob - one per ship
11.     float totalMass = 0
12.     foreach containerChunk
13.         foreach container
14.             totalMass += container.mass
15.     parentShip.mass = totalMass

Actual Code

Code (CSharp):

1. public class MassManagerSystem : JobComponentSystem
2. {
3.     ComponentGroup _shipGroup;
4.     ComponentGroup _containerGroup;
5.  
6.     protected override void OnCreateManager()
7.     {
8.         _shipGroup = GetComponentGroup(
9.             ComponentType.Create<MassManager>()
10.         );
11.  
12.         _containerGroup = GetComponentGroup(
13.             ComponentType.ReadOnly<Mass>(),
14.             ComponentType.ReadOnly<ParentShip>()    // SharedComponent
15.         );
16.     }
17.  
18.     // Sum the mass for all containers on a single ship
19.     struct ComputeMassJob : IJob
20.     {
21.         public Entity parentShipEntity;
22.         [DeallocateOnJobCompletionAttribute]
23.         public NativeArray<ArchetypeChunk> containerChunks;
24.  
25.         [ReadOnly]
26.         public ArchetypeChunkComponentType<Mass> massType;
27.         [NativeDisableParallelForRestriction]
28.         public ComponentDataFromEntity<MassManager> massManagers;
29.  
30.         public void Execute()
31.         {
32.             float totalMass = 0;
33.  
34.             for (int chunkIndex = 0; chunkIndex < containerChunks.Length; chunkIndex++)
35.             {
36.                 ArchetypeChunk containerChunk = containerChunks[chunkIndex];
37.                 NativeArray<Mass> masses = containerChunk.GetNativeArray<Mass>(massType);
38.                 for (int i = 0; i < containerChunk.Count; i++)
39.                 {
40.                     totalMass += masses[i].mass;
41.                 }
42.             }
43.  
44.             massManagers[parentShipEntity] = new MassManager{totalMass = totalMass};
45.         }
46.     }
47.  
48.     protected override JobHandle OnUpdate(JobHandle inputDeps)
49.     {
50.         if (_shipGroup.CalculateLength() == 0)
51.             return inputDeps;
52.  
53.         // Ship Chunk Types
54.         ArchetypeChunkEntityType entityType = GetArchetypeChunkEntityType();
55.         ArchetypeChunkComponentType<MassManager> massManagerType = GetArchetypeChunkComponentType<MassManager>(false);
56.  
57.         // Container Chunk Types
58.         ArchetypeChunkComponentType<Mass> massType = GetArchetypeChunkComponentType<Mass>(true);
59.         ComponentDataFromEntity<MassManager> massManagersCDFE = GetComponentDataFromEntity<MassManager>(false);
60.  
61.         NativeArray<ArchetypeChunk> shipChunks = _shipGroup.CreateArchetypeChunkArray(Allocator.TempJob);
62.         for (int chunkIndex = 0; chunkIndex < shipChunks.Length; chunkIndex++)
63.         {
64.             ArchetypeChunk shipChunk = shipChunks[chunkIndex];
65.             NativeArray<Entity> shipEntities = shipChunk.GetNativeArray(entityType);
66.             NativeArray<MassManager> massManagers = shipChunk.GetNativeArray<MassManager>(massManagerType);
67.  
68.             for (int shipIndex = 0; shipIndex < shipChunk.Count; shipIndex++)
69.             {
70.                 massManagers[shipIndex] = new MassManager(); // reset MassManager on ship
71.                 Entity shipEntity = shipEntities[shipIndex];
72.  
73.                 _containerGroup.SetFilter<ParentShip>(new ParentShip(shipEntity.Index));
74.  
75.                 ComputeMassJob job = new ComputeMassJob
76.                 {
77.                     parentShipEntity = shipEntity,
78.                     containerChunks = _containerGroup.CreateArchetypeChunkArray(Allocator.TempJob),
79.                     massType = massType,
80.                     massManagers = massManagersCDFE
81.                 };
82.  
83.                 inputDeps = job.Schedule(inputDeps);
84.             }
85.         }
86.  
87.         shipChunks.Dispose();      
88.         return inputDeps;
89.     }
90. }

I'm clearly way off the path as there's only about three lines in this mess related to the actual mass computation. I think the SharedComponent grouping is fine but I don't know the best way syntactically to process it.

 

I kinda lost at your code because you was reusing code from old ECS samples.

You don't have to GetChunk from ComponentGroup then get Entities, Data from that chunk.
Just use IJobChunk or IJobComponentData whatever to simplify thing. Make sure OnUpdate() clean, readable as possible.

And for SharedComponentData filter. All you have to do is this:

Code (CSharp):

1. public class MassManagerSystem : JobComponentSystem
2. {
3.     ComponentGroup _shipGroup;
4.     ComponentGroup _containerGroup;
5.  
6.     protected override void OnCreateManager()
7.     {
8.         _shipGroup = GetComponentGroup(ComponentType.Create<MassManager>());
9.  
10.         _containerGroup = GetComponentGroup(ComponentType.ReadOnly<Mass>(),
11.                                             ComponentType.ReadOnly<ParentShip>() // SharedComponent
12.         );
13.     }
14.  
15.     protected override JobHandle OnUpdate(JobHandle inputDeps)
16.     {
17.         // Get Parent Ship id from _shipGroup ???
18.         // Foreach ships ??
19.         _containerGroup.SetFilter(new ParentShip(){Id = "Your Ship Id or data"});
20.         // Schedule Job
21.     }
22. }

 

Me too, that's the problem. It's not using old ecs though, it's all chunk iteration.

My point is that I don't know how to do that with my current situation. It's not a simple scenario. I can't use IJobChunk or IJobProcessComponentData because the container chunks on a single ship can't write to that ship in parallel.
There are n chunks per ship, I need to iterate the entities in those and accumulate a value, then write it to the parent ship. What I'm doing is to jobify that process for each ship but somethings not right as it keeps complaining about parallel writes with the CDFE even though it can't happen. I don't know what the equivalent of [NativeDisableParallelForRestriction] is in an IJob.

What you've written in your OnUpdate() is exactly what I'm doing. It's just not easy to decipher because there's so much boilerplate.

 

I might have read this too quickly
- using shared component might or might not be a good choice, rationale unclear
- can you not just aggregate your data in hashmaps with the shipID as key?

 

Ok I finally got a version working using a hashmap.
Full code pasted at bottom though you'd have to be a masochist to want to look at it.
In pseudo, all it's doing is this:

Code (CSharp):

1. OnUpdate() {
2.     foreach ship in shipGroup
3.         containerGroup.SetFilter(ship)
4.         ComputeLoadoutMassJob // sums container masses into threadSafeArray
5.         CombineLoadoutMassJob // sums threadSafeArray into hashmap[shipEntity]
6.  
7.     AssignLoadoutMassJob // write hashmap values to ships
8. }
9.  
10. ComputeLoadoutMassJob : IJobChunk
11. CombineLoadoutMassJob : IJob
12. AssignLoadoutMassJob : IJobProcessComponentData

So there's two jobs per ship to sum the container masses down to a single mass value in a hashmap, and then one final job that applies the mass to each ship.
This works but it's ludicrously complicated and error prone for the simple act of summing child entities into a parent. The equivalent of OO

Code (CSharp):

1. foreach(ship in ships)
2.     foreach(container in ship)
3.         ship.totalMass += container.mass

I'm starting to agree though not sure how else to do it.
I have a bunch of ships all carrying a bunch of containers. I need to accumulate varying types of data about the containers on a ship and store the results on the ship. Not all ships have all systems though, hence the top down approach of "for each ship with componentX, for each container".
I'm surely making this harder than it needs to be but I'm not familiar enough yet with the job system and tools.

Actual code

Code (CSharp):

1. public class LoadoutMassJobSystem : JobComponentSystem
2. {
3.     int _threadCount;
4.     NativeHashMap<Entity, float> _shipMassHashmap;
5.  
6.     ComponentGroup _shipGroup;
7.     ComponentGroup _containerGroup;
8.  
9.     protected override void OnCreateManager()
10.     {
11.         _shipGroup = GetComponentGroup(
12.             ComponentType.Create<LoadoutMassManager>()
13.         );
14.  
15.         _containerGroup = GetComponentGroup(
16.             ComponentType.ReadOnly<Mass>(),
17.             ComponentType.ReadOnly<ParentShip>()    // SharedComponent
18.         );
19.  
20.         _threadCount = Unity.Jobs.LowLevel.Unsafe.JobsUtility.MaxJobThreadCount;
21.     }
22.  
23.     protected override void OnDestroyManager()
24.     {
25.         if (_shipMassHashmap.IsCreated)
26.             _shipMassHashmap.Dispose();
27.     }
28.  
29.  
30.     // For all container chunks belonging to a single ship,
31.     // sum the mass per chunk into the thread safe array loadoutMassThreadArray. I.e.
32.     // loadoutMassThreadArray = [thread1Mass, thread2Mass, thread3Mass, thread4Mass]
33.     struct ComputeLoadoutMassJob : IJobChunk
34.     {
35.         [ReadOnly] public ArchetypeChunkComponentType<Mass> massChunkType;
36.         [NativeSetThreadIndex] private int threadIndex;
37.         public NativeArray<float> loadoutMassThreadArray;
38.  
39.         public void Execute(ArchetypeChunk chunk, int chunkIndex, int firstEntityIndex)
40.         {
41.             NativeArray<Mass> masses = chunk.GetNativeArray<Mass>(massChunkType);
42.  
43.             float totalMass = 0;
44.             for (int i = 0; i < chunk.Count; i++)
45.             {
46.                 totalMass += masses[i].mass;
47.             }
48.             loadoutMassThreadArray[threadIndex] += totalMass;
49.         }
50.     }
51.  
52.     // Sum the masses from ComputeLoadoutMassJob into hashmap. I.e.
53.     // shipMassHashmap[shipEntity] = sum(loadoutMassThreadArray)
54.     struct CombineLoadoutMassJob : IJob
55.     {
56.         public Entity shipEntity;
57.         [ReadOnly][DeallocateOnJobCompletion]
58.         public NativeArray<float> loadoutMassThreadArray;
59.         [WriteOnly]
60.         public NativeHashMap<Entity, float>.Concurrent shipMassHashmap;
61.  
62.         public void Execute()
63.         {
64.             float totalMass = 0;
65.             for (int i = 0; i < loadoutMassThreadArray.Length; i++)
66.             {
67.                 totalMass += loadoutMassThreadArray[i];
68.             }
69.  
70.             shipMassHashmap.TryAdd(shipEntity, totalMass);
71.         }
72.     }
73.  
74.     // Write mass value in shipMassHashmap to Ship. I.e.
75.     // ship.LoadoutMassManager.totalMass = shipMassHashmap[shipEntity]
76.     struct AssignLoadoutMassJob : IJobProcessComponentDataWithEntity<LoadoutMassManager>
77.     {
78.         [ReadOnly]
79.         public NativeHashMap<Entity, float> shipMassHashmap;
80.  
81.         public void Execute(Entity entity, int index, ref LoadoutMassManager loadoutMassManager)
82.         {
83.             float totalMass = 0;
84.             shipMassHashmap.TryGetValue(entity, out totalMass);
85.             loadoutMassManager.totalMass = totalMass;
86.         }
87.     }
88.  
89.  
90.     protected override JobHandle OnUpdate(JobHandle inputDeps)
91.     {
92.         if (_shipMassHashmap.IsCreated)
93.             _shipMassHashmap.Dispose();
94.  
95.         if (_shipGroup.CalculateLength() == 0)
96.             return inputDeps;
97.  
98.         _shipMassHashmap = new NativeHashMap<Entity, float>(100, Allocator.TempJob);
99.  
100.         // Archetype Chunk Types
101.         var entityType = GetArchetypeChunkEntityType();
102.         var massManagerType = GetArchetypeChunkComponentType<LoadoutMassManager>(false);
103.         var massChunkType = GetArchetypeChunkComponentType<Mass>(true);
104.         var shipChunks = _shipGroup.CreateArchetypeChunkArray(Allocator.TempJob);
105.  
106.         // For each Ship Chunk
107.         for (int shipChunkIndex = 0; shipChunkIndex < shipChunks.Length; shipChunkIndex++)
108.         {
109.             var shipChunk = shipChunks[shipChunkIndex];
110.             var shipEntities = shipChunk.GetNativeArray(entityType);
111.             var massManagers = shipChunk.GetNativeArray<LoadoutMassManager>(massManagerType);
112.  
113.             // For each Ship Entity
114.             for (int shipIndex = 0; shipIndex < shipChunk.Count; shipIndex++)
115.             {
116.                 massManagers[shipIndex] = new LoadoutMassManager(); // reset MassManager on ship
117.  
118.                 // Filter to containers for this ship
119.                 Entity shipEntity = shipEntities[shipIndex];
120.                 _containerGroup.SetFilter<ParentShip>(new ParentShip(shipEntity.Index));
121.                 if (_containerGroup.CalculateLength() == 0)
122.                     continue;
123.  
124.                 // Schedule Jobs for Ship
125.                 var loadoutMassThreadArray = new NativeArray<float>(_threadCount, Allocator.TempJob, NativeArrayOptions.ClearMemory);
126.  
127.                 var computeLoadoutMassJob = new ComputeLoadoutMassJob {
128.                     massChunkType = massChunkType,
129.                     loadoutMassThreadArray = loadoutMassThreadArray };
130.  
131.                 var combineLoadoutMassJob = new CombineLoadoutMassJob {
132.                     shipEntity = shipEntity,
133.                     shipMassHashmap = _shipMassHashmap.ToConcurrent(),
134.                     loadoutMassThreadArray = loadoutMassThreadArray };
135.  
136.                 var computeLoadoutMassJobHandle = computeLoadoutMassJob.Schedule(_containerGroup, inputDeps);
137.                 var combineLoadoutMassJobHandle = combineLoadoutMassJob.Schedule(computeLoadoutMassJobHandle);
138.                 inputDeps = combineLoadoutMassJobHandle;
139.             }
140.         }
141.         inputDeps = new AssignLoadoutMassJob{shipMassHashmap = _shipMassHashmap}.Schedule(this, inputDeps);
142.      
143.         shipChunks.Dispose();
144.         return inputDeps;
145.     }
146. }

 

I did not look at your code. There are many variables , we don’t know, that could influence a recommendation (I.e. how many ships, containers, change frequency, etc.)

But let’s say you have a container with
-shipID
-weight

If ships don’t get created / destroyed after start, shipID could be the entiyId. But let’s assume they do, the you need to maintain a lookup table, associating your custom shipID to the ship entityId

You run an ijobprocesscomponentdata job on all containers , summing up their weight. You could store this in a concurrent hashmap.

Then you run a dependent iprocesscomponentdara job on all ships with your “hasweight component” and lookup the corresponding weight in the hash table

 

Thanks for your input sngdan, it's helped get me to a cleaner solution.
The problem I have with IJobProcessComponentData is that not every container needs to be processed.
E.g. there are ten 10 ships with 100 containers each but only two of those ships have a LoadoutMassManager component on them. It doesn't make sense to iterate all 1000 containers asking each one if it's parent ship has that component and if so, adding the mass.
There are many similar aggregation systems like this so ship 1 might have management systems A and B, ship 2 might have A, D and E etc. The inefficiency of iterating all containers would compound.

This is my attempt using IJobProcessComponent data with 'Job 1' iterating containers to copy their masses to a hashmap[ShipEntity], and 'Job 2' iterating the ships to sum those masses together and store on the ship.

Code (CSharp):

1. // Job 1
2. // For each Container whose parent has a LoadoutMassManager component, copy its mass to a hashmap
3. struct PrepareLoadoutMassJob : IJobProcessComponentData<Mass, ShipRef>
4. {
5.     [ReadOnly] public ComponentDataFromEntity<LoadoutMassManager> loadoutMassManagers;
6.     [WriteOnly] public NativeMultiHashMap<Entity, float>.Concurrent loadoutMassHashmap;
7.  
8.     public void Execute(ref Mass mass, ref ShipRef shipRef)
9.     {
10.         if (! loadoutMassManagers.Exists(shipRef.entity))
11.             return;
12.  
13.         loadoutMassHashmap.Add(shipRef.entity, mass.mass);
14.     }
15. }
16.  
17.  
18. // Job 2 - depends on Job 1
19. // For each ship, sum the masses from loadoutMassHashmap[ShipEntity] and write to LoadoutMassManager
20. struct ComputeLoadoutMassJob : IJobProcessComponentDataWithEntity<LoadoutMassManager>
21. {
22.     [ReadOnly] public NativeMultiHashMap<Entity, float> loadoutMassHashmap;
23.  
24.     public void Execute(Entity entity, int index, ref LoadoutMassManager loadoutMassManager)
25.     {
26.         float totalMass = 0;
27.         float mass = 0;
28.         NativeMultiHashMapIterator<Entity> it;
29.         if (loadoutMassHashmap.TryGetFirstValue(entity, out mass, out it))
30.         {
31.             totalMass += mass;
32.             while (loadoutMassHashmap.TryGetNextValue(out mass, ref it))
33.             {
34.                 totalMass += mass;
35.             }
36.         }
37.  
38.         loadoutMassManager.totalMass = totalMass;
39.     }
40. }

This is probably the simplest approach from a visual code perspective but is highly inefficient for my scenario. I could optimise this a lot though using chunk iteration with SharedComponent to filter out whole chunks. It gets very verbose and error prone though for such a simple task.
I'm not sure I'm using the hashmap in the correct way either or if there's actually a way to sum this data in Job 1 without having to copy every mass value into the hashmap.

 

Hashmap is not required but it seems in general OK for what you try to do(i.e. you could also use 2 arrays, with pre-allocated length so maybe wasting some memory). As I said the design depends on many factors, but if you enable burst on your jobs, you will see that looping through a few entities does not come at such a high cost.

Job1: possibly better to do IJob, iterate over all containers and sum up their weight directly into the Hashmap (if you dont go parallel, you can use trygetvalue and sum directly into the hashmap) - I would simply sum all containers, even if some are on ships, where you dont need the weight

edit: not sure, maybe you can keep it as IJobProcessComponentDataWithEntity and sum directly into the hashmap --- i would have to check this at the computer - it might work parallel, if forgot

Job2: keep IJobProcessComponentDataWithEntity, and have it iterate all ships that need a weight. set the Hashmap readonly, they can then directly pick up the weight. (You could also do this in a second loop in Job 1, but it wont be parallel - you have to test which is faster)

You can look into optimization later - this depends really on too many unknowns to give any advise here...

 

So, I just came home and created a small test case with 1000 ships, each 10 containers (i.e. 10,000) and 100 of those ships that actually need to know their weight. I just noted at the end that your weight was a float and I used an int.

Editor profiling (not build) - completing the job after scheduling and using the entity debugger
- Parallel system (with 2 jobs, as yours) takes 0.16-0.17ms
- Simple IJob with 2 loops takes 0.17ms and likely does not scale up as well

Note, those are using [BurstCompile] - without the parallel system takes 0.57ms.

In my equivalent to your Job1, I also have both components as readonly, i.e. Execute([ReadOnly] ref Mass mass, [ReadOnly] ref ShipRef shipRef)

Spoiler: IJob

Code (CSharp):

1.     [BurstCompile]
2.     struct WeightPerShipJob : IJob
3.     {
4.         [ReadOnly] public ComponentDataArray<ShipParent> parentCDA;
5.         [ReadOnly] public ComponentDataArray<ContainerWeight> containerWeightCDA;
6.  
7.         [ReadOnly] public EntityArray shipEntities;
8.         public ComponentDataFromEntity<ShipWeight> shipWeightCDFE;
9.  
10.         public void Execute()
11.         {
12.             for (int i = 0; i < shipEntities.Length; i++)
13.             {
14.                 shipWeightCDFE[shipEntities[i]] = new ShipWeight{Value = 0}; // this could also be cleared in a previous parallel job
15.             }
16.            
17.             for (int i = 0; i < containerWeightCDA.Length; i++)
18.             {
19.                 var parentID = parentCDA[i].Value;
20.                
21.                 if (shipWeightCDFE.Exists(parentID))
22.                 {
23.                     var weight = shipWeightCDFE[parentID];
24.                     weight.Value += containerWeightCDA[i].Value;
25.                     shipWeightCDFE[parentID] = weight;
26.                 }
27.             }
28.         }
29.     }

 

Thanks sngdan, that's very useful.
Were you using the same hashmap approach in the parallel tests? That certainly makes me less worried about any performance issues but it still seems like a wasteful approach.

I hadn't thought of that non-parallel IJob approach you've used there. That's definitely an easy option for the case where I don't use a SharedComponent to group containers to their ship.
Isn't ComponentDataArray deprecated? I noticed in the latest boids version, they're using ComponentGroup.ToComponentDataArray now.

I just managed to get my original one-job-per-ship version working in parallel.
I find this single job approach more manageable as all the computation is done in one function. The mass scenario is simple so works ok with IJPCD but I have more complex systems that require intermediate working variables, like finding the topmost container on the ship. Things like that get a bit messy using IJPCD or IJobChunk and multiple passes.

Ah yes, I simply forgot about adding those. I'm testing about a dozen different job scenarios.

 

- yes, same multihashmap
- yes, deprecated (as you noted the closest equivalent seems to be tocomponentdataarray, which copies to nativearray - works similar, but is slower because of the copy or I did something wrong)
- they will also allow nativearray allocation in jobs, this can help with storing intermediate calculations

All in all, my approach would be to get things working first (with burst, new math library) and then optimize (ie chunk level logic) / parallel

 

Thanks, I didn't know that.

Yes you're right. I just needed a working solution and can move on now. Ecs is changing so rapidly there'll probably be cleaner code approaches down the track.