Efficient Hashmap with jobs?

Hey there everyone, ive been messing around with the wonders of ECS but I cant seem to figure out how, in any way to do this. The current way im doing this may make you laugh as well as feel free to destroy me and open my eyes into the vastness of ECS and how its really done.

In my previous project, the one im trying to port, you would get input, the input would contain an int referenceing the "ID" of the set of object that the player wants to move, and the Vector3s of positions that the player moved to. You would then set up a formation for the units with the positions and would be able to reference all the objects to move into the formation with list[ID].List<Transform>.position you get the idea. In the big problem is I must process multiple of these.

In ECS I keep the same input with the 2 positions and the IDs but I cant seem to find a way to get all the positions or whatever component on all the entities with the ID efficiently. Because the formation calculation from the many different inputs come in can be expensive I made a job containing the results. The problem is, because there can be 4 inputs in 1 update, I batch the formation calculations and the outputed float3's in an IParallelFor together. Here is my main problem: I make multiple float3[] for the positions of these formations inside the job. One I am not sure how to output float[] because you cant use a NativeHashMap<int NativeArray<float3>> inside the job. There must be a way to get all the positions even if it doesnt use NativeHashMap, as long as it is effiecent.

The main reason for this though is the second job im queueing, this job is a simple job that should get all the entities with the same ID as the input and then set there position accordingly to the position inside the first job.

Just to point out, I understand that NativeHashMap is not Blittable, hopefull that just serves as an example of what I was trying to do. I also could used IJobProcessComponentData for both Jobs but this doesnt change the fact that I am unsure how to achieve my purpose. One other idea I had but I dont think its possible was to make a component group and sort of all the IDs from the NewDestinations to find all the corrent units but then I had a bunch of ComponentDataArrays and you cant ComponentDataArrays[] into a job.

Code (CSharp):

1.  
2.  
3. [Serializable]
4. public struct FormationSetNewDestination : IComponentData
5. {
6.     public float3 startPosition;
7.     public float3 endPosition;
8.     public int formationID;
9. }
10.  
11. [Serializable]
12. public struct UnitFormationID : IComponentData
13. {
14.     public int Value;
15. }
16. [Serializable]
17. public struct UnitIndex : IComponentData
18. {
19.     public int Value;
20. }
21.  
22. [Serializable]
23. public struct UnitDestination : IComponentData
24. {
25.     public float3 Value;
26. }
27.  
28. public class FormationSetDestinationSystem : JobComponentSystem
29. {
30.  
31. public struct NewDestinationsInjection
32.     {
33.         public ComponentDataArray<FormationSetNewDestination> newDestinations;
34.         public readonly int Length;
35.     }
36. [Inject] public NewDestinationsInjection newDestinations;
37.  
38. public struct DestinationsInjection
39.     {
40.         public ComponentDataArray<UnitFormationID> unitFormationIDs;
41.         public ComponentDataArray<UnitIndex> unitIndexs;
42.         public ComponentDataArray<UnitDestination> unitDestinations;
43.     }
44.  
45. [Inject] public DestinationsInjection destinations;
46.  
47. public struct GetUnitPositionsJob : IJobParallelFor
48.     {
49.  
50. public ComponentDataArray<FormationSetNewDestination> formationDestinations;
51.         public NativeHashMap<int, NativeArray<float>> unitPositions;
52.  
53. public void Execute(int i)
54.         {
55.             NativeArray<float> a = new NativeArray<float>(240, Allocator.TempJob);
56.  
57.             float3[] positions = CaluclateUnitDestinations(0.5f, 0.5f, 0.5f, 0.5f, 240, formationDestinations[i].startPosition, formationDestinations[i].endPosition);
58.  
59.             for (int j = 0; j < positions.Length; j++)
60.             {
61.                 a[j] = positions[j].x;
62.             }
63.             unitPositions.TryAdd(formationDestinations[i].formationID, a);
64.         }
65. }
66.  
67. public struct SetUnitPositionsJob : IJobParallelFor
68.     {
69.  
70. [ReadOnly] public NativeHashMap<int, NativeArray<float>> unitPositions;
71.         [ReadOnly] public ComponentDataArray<UnitFormationID> unitFormationIDs;
72.         [ReadOnly] public ComponentDataArray<UnitIndex> unitIndexs;
73.         public ComponentDataArray<UnitDestination> unitDestinations;
74.  
75. for (int formationIndex = 0; formationIndex < unitFormationIDs.Length; formationIndex++)
76.             {
77.  
78.                 if(unitPositions.TryGetValue(unitFormationIDs[formationIndex].Value, out a))
79.                 {
80.                     unitDestinations[formationIndex] = new UnitDestination { Value = new float3(a[unitIndexs[formationIndex].Value], 0, 0) };
81.                 }
82. }
83. }
84. }
85.  
86. protected override JobHandle OnUpdate(JobHandle inputDeps)
87. {
88. NativeHashMap<int, NativeArray<float>> unitPositions = new NativeHashMap<int, NativeArray<float>>(1, Allocator.TempJob);
89.  
90. var unitGetPositionJob = new GetUnitPositionsJob
91.         {
92.             unitPositions = unitPositions,
93.             formationDestinations = newDestinations.newDestinations
94.         };
95.         var unitGetPositionDependency = unitGetPositionJob.Schedule(newDestinations.Length, 1);
96.  
97.         var unitSetPositionJob = new SetUnitPositionsJob
98.         {
99.             unitPositions = unitPositions,
100.             unitFormationIDs = destinations.unitFormationIDs,
101.             unitIndexs = destinations.unitIndexs,
102.             unitDestinations = destinations.unitDestinations
103.         };
104.         var unitSetPositionDependency = unitSetPositionJob.Schedule(newDestinations.Length, 1, unitGetPositionDependency);
105.  
106. unitPositions.Dispose();
107.  
108. }
109. }

My unit count is around 500,000 so I value efficiency over anything.

This is one of my first post on any forum, this one really had me stuck with the lack of examples and info on the current ECS. Anything helps and thank you for your responses.

 

Use NativeMultiHashMap for storing multiple floats per single ID and process them in IJobParallelFor, IJobNativeMultiHashMapVisitKeyValue, IJobNativeMultiHashMapMergedSharedKeyIndices etc.
BTW I recommend you use actual API not deprecated Inject, CDA etc.

 

I had a use case where a "constructed" MultiHashMap performed better, this was due to
- I needed to access the values stored per key by index (multiple iterations)
- I used a large amount of values stored and clearing the NMHM per frame took a toll

The "constructed" MultiHashMap is an entity for each key with a IBufferElementData for the values
- allows to cast the buffer to an array .AsNativeArray()
- clears fast, i.e. length = 0
- my buffer data fitted into chunks (set with Buffer Capacity)

I had a chat here somewhere with @eizenhorn and following that i benchmarked against IJobNativeMultiHashMapVisitKeyValue (which was not available originally) - still not faster.

 

Yep it’s fine to all depends on context actually I’m using NHM<key, Entity> and then get buffer by this entity and doing things

 

What is the data in question? What are those 500,000 entities representing? Maybe there is some indirect creative approach to this.

Second thought it what do your groups look like. What are the constraints on them. Because storing them in a NativeArray and indexing into that would seem like the best approach assuming most are either static or at least within some reasonable range. Like if you could alter the design so there are only so many group sizes.

I'm just thinking 500,000 of anything, you likely have room to use creative approaches because the player won't notice at that number. Just a hunch.

 

I apologize for the lack of clarity in my question @snacktime These entities are representing little boids that will move towards the positions given to them in the struct UnitDestination. The 1 constraint I can think of is that the boids are attached to a parent object that does path finding for them while the boids just move locally into the formation around the parent. The main problem is is the player can decide to move 10 of them 1 time, 3 of them another time, etc, so the job had to somehow be able to iterate over no fixed count efficently.

@eizenhorn I did not realize there were those types of jobs. This is a game changer! Is there any documentation that shows me all the different interfaces that i can implement with the job system. Also is NativeMultiHashMap blittable inside a job because if so I never got it to work. Maybe it was because I was storing NativeArray<float3> and NativeArrays may not be able to store those.

Regarding the inject, could you further explain why it is recommend to use the API instead of an inject.

Lastly, @sngdan that sounds really interesting, but what did you benchmark against IJobNativeMultiHashMapVisitKeyValue and would half a million make a big difference using it.

Thank you for your time and replies. I cant wait to get crack-a-lacking!

 

As @eizenhorn said, the optimal approach is very dependent on the use case. With 500,000 entities you will have to profile and play a little bit around with various approaches and eliminate bottlenecks.

I benchmarked NMHM (Visite key value) vs the buffer approach that I described above (for each “key” you have an entity that holds the “values” in a buffer)

 

Because Inject (ComponentDataArray, EntityArray etc.) deprecated.
https://github.com/Unity-Technologi...b/master/Documentation~/ecs_best_practices.md

 

If that is the case how would I pass in Components into a job that the IJobProcessComponentData components woud need in order to execute the job? So far i have either used Inject or a ComponentGroup.

 

You can use chunk iteration, still can use CG, GetComponentDataFromEntity, ComponentDataArray replaced to NativeArray, which can be getted by ToComponentDataArray from CG etc. Just inspect Unity ECS Samples, and how they work.

 

When looking through the unity examples I think I have found an answer to my question. According to the documentation on NativeMultiHashMaps: It has special rules for allowing safe and deterministic write access from ParallelFor jobs. The NativeHashMap.Concurrent method lets you add items in parallel from IJobParallelFor. This is what I needed. I am not sure what exactly happenes when using .Concurrent but now I can use the NativeHashMaps. Thank you for your help.

If you dont mind, I couldnt find any information on IJobNativeMultiHashMapVisitKeyValue, IJobNativeMultiHashMapMergedSharedKeyIndices except when they are used in the unity examples. If you could give me a brief explanation of their functionality or a link to there explanation that would be great.

 

Use forum search There is many info, and just try it and learn how it work, and look at source code, it’s better “documentation”

 

Thanks for the continued support.

 

IJobNativeMultiHashMapVisitKeyValue - iterates through all values stored in the NMHM (provides you the key & associated values)

IJobNativeMultiHashMapMergedSharedKeyIndices - I never used this, boid documentation video explains it, what I understood from the video is: it is a specific job that splits access to first key-value pair from following key-values and therefore allows you to write to the first key-value pair while iterating through the rest (i.e. to sum up data, etc) - but better check it out in source

 

I know this thread is a bit dated, but I hope it may be useful for someone in future.

I have tested it and what it returns, are values.
Instead of typical job Execute, it uses ExecuteFirst and ExecuteNext.

Code (CSharp):

1. public void ExecuteFirst ( int firstValueOfUniqueKey )

returns first value of iterated unique key.
Second part

Code (CSharp):

1. public void ExecuteNext ( int firstValueOfUniqueKey , int nextValueOfUniqueKey)

this returns as in previous case, value of iterated unique key and any next values, contained in current unique key.

So for example, having multiple Boids in hashed cells, using this job, you can get all boids per each individual cell.

I use it for example:
First random generate lifeform instances on the ma, made of tiles.
I set in NativeMultiHashMap, keys per tile index and value per entity index.
Then in the job, I populate DynamicBuffer's of my tiles with lifeforms, which occupy each corresponding tile.

Now I can use desire tile, to retrieve occupied lifeforms using tile's DynamicBuffer.

Linking these also for reference
https://forum.unity.com/threads/how-does-ijobnativemultihashmapmergedsharedkeyindices-work.535104/
Depreciation?
https://forum.unity.com/threads/rep...apmergedsharedkeyindices.882019/#post-5894366


Edit:
Regarding potential solution for depreciation, please see this thread.
unique keys from NativeMultiHashMap?