Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Looking for a Job Interface for NativeMultiHashMap

Discussion in 'Entity Component System' started by AndesSunset, Apr 3, 2019.

  1. AndesSunset

    AndesSunset

    Joined:
    Jan 28, 2019
    Posts:
    60
    Hi! I’m wondering if there’s a Job type interface to do the following: I’d like a different job for each key in a NativeMultHashMap.

    *Not* for every Key/Value pair entry in a NativeMultiHashMap.

    So for example: if I had 3 entries in a NativeMultiHashMap which each had the same key, Unity would only start one Job. It’s Execute method would look something like this:

    Code (CSharp):
    1. public override void Execute(TKey key, NativeSlice<TValue> values)
    2. {
    3. }
    Does anything like this currently exist, or should I stick to a custom solution?

    Thank you for any info. :)
     
    Last edited: Apr 3, 2019
  2. Abbrew

    Abbrew

    Joined:
    Jan 1, 2018
    Posts:
    417
    A partial solution is to use IJobNativeMultiHashMapVisitKeyValue, which provides a public void ExecuteNext(Key key, Value value) method. Unfortunately this means you can only process one K-V pair at a time independent of each other.
     
  3. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,683
    IJobParalelFor iterating through NMHM.GetKeyArray() and every Execute is Key -> Values[] coz you NMHM.TryGetFirstValue by this Key and inside iterating through TryGetNextValue which mean you can access all Values for current key in one Execute.
     
    AndesSunset and Abbrew like this.
  4. AndesSunset

    AndesSunset

    Joined:
    Jan 28, 2019
    Posts:
    60
    Thanks, that sounds like a great solution.

    Would I run into trouble if I passed a GetBufferFromElement to that IJobParallelFor, then tried to write to different Buffers associated with different (Entity) keys?
     
  5. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,683
    If you sure in safety (you no read\write to the same buffer at the same time)
     
    Last edited: Sep 3, 2019
  6. PublicEnumE

    PublicEnumE

    Joined:
    Feb 3, 2019
    Posts:
    729
    Thanks for posting this solution. Is there any way to do this in a deferred manner, so that the IJobParallelFor iterates over each key in the NativeMultiHashMap, after previous jobs have added entries to it?

    So far, I've only seen ways to do this with NativeList, which of course can't be written to from an IJobForEach (or anything that runs in parallel).
     
    Last edited: Sep 2, 2019
  7. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,753
    I actually wrote a custom job interface to do just this which I discussed over here: https://forum.unity.com/threads/tip-avoid-allocating-per-entity.726275/

    I called it IJobNativeMultiHashMapVisitUniqueKey<TKey>

    I think I only posted code for my custom iterator as that was all that was requested but I can post code for custom job when I get some time later.
     
    PublicEnumE and S_Darkwell like this.
  8. PublicEnumE

    PublicEnumE

    Joined:
    Feb 3, 2019
    Posts:
    729
    Hi @tertle, I’ve been out of commission for a few days. If you ever have time to post your custom job type, I would really appreciate seeing what you’re doing there.

    I’ve written some custom job interfaces on occasion (just to experiment). But nothing that mimics the ‘deferred’ behavior of IJobParallelForDefer. I would love to learn more about how that works, from your example.

    Is it right to assume you based this on your custom NativeHash/MultiHashMap imposters you set up a whole back?

    Thanks for posting,
    E
     
  9. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,753
    Oh totally forgot about this. Cheers for alert.

    Yep uses my my imposters to get internal access. Doesn't really work like IJobParallelForDefer.
    Basically just works like and is based off Unitys IJobNativeMultiHashMapVisitKeyValue or whatever it is called except only does the key part.

    Ok let me see if I can find all the code

    IJobNativeMultiHashMapVisitUniqueKey / JobNativeMultiHashMapVisitUniqueKey

    Code (CSharp):
    1. // <copyright file="IJobNativeMultiHashMapVisitUniqueKey.cs" company="BovineLabs">
    2. //     Copyright (c) BovineLabs. All rights reserved.
    3. // </copyright>
    4.  
    5. namespace BovineLabs.Common.Collections
    6. {
    7.     using System;
    8.     using Unity.Collections;
    9.     using Unity.Collections.LowLevel.Unsafe;
    10.     using Unity.Jobs;
    11.     using Unity.Jobs.LowLevel.Unsafe;
    12.  
    13.     [JobProducerType(typeof(JobNativeMultiHashMapVisitUniqueKey.NativeMultiHashMapVisitUniqueKeyJobStruct<,,>))]
    14.     public interface IJobNativeMultiHashMapVisitUniqueKey<TKey, TValue>
    15.         where TKey : struct, IEquatable<TKey>
    16.         where TValue : struct
    17.     {
    18.         void ExecuteNext(TKey key);
    19.     }
    20.  
    21.     public static class JobNativeMultiHashMapVisitUniqueKey
    22.     {
    23.         internal struct NativeMultiHashMapVisitUniqueKeyJobStruct<TJob, TKey, TValue>
    24.             where TJob : struct, IJobNativeMultiHashMapVisitUniqueKey<TKey, TValue>
    25.             where TKey : struct, IEquatable<TKey>
    26.             where TValue : struct
    27.         {
    28.             [ReadOnly]
    29.             public NativeMultiHashMapImposter<TKey, TValue> HashMap;
    30.  
    31.             public TJob JobData;
    32.  
    33.             // ReSharper disable once StaticMemberInGenericType
    34.             private static IntPtr JobReflectionData;
    35.  
    36.             public static IntPtr Initialize()
    37.             {
    38.                 if (JobReflectionData == IntPtr.Zero)
    39.                 {
    40.                     JobReflectionData = JobsUtility.CreateJobReflectionData(
    41.                         typeof(NativeMultiHashMapVisitUniqueKeyJobStruct<TJob, TKey, TValue>),
    42.                         typeof(TJob),
    43.                         JobType.ParallelFor,
    44.                         (ExecuteJobFunction)Execute);
    45.                 }
    46.  
    47.                 return JobReflectionData;
    48.             }
    49.  
    50.             public delegate void ExecuteJobFunction(
    51.                 ref NativeMultiHashMapVisitUniqueKeyJobStruct<TJob, TKey, TValue> fullData,
    52.                 IntPtr additionalPtr,
    53.                 IntPtr bufferRangePatchData,
    54.                 ref JobRanges ranges,
    55.                 int jobIndex);
    56.  
    57.             public static unsafe void Execute(
    58.                 ref NativeMultiHashMapVisitUniqueKeyJobStruct<TJob, TKey, TValue> fullData,
    59.                 IntPtr additionalPtr,
    60.                 IntPtr bufferRangePatchData,
    61.                 ref JobRanges ranges,
    62.                 int jobIndex)
    63.             {
    64.                 while (true)
    65.                 {
    66.                     if (!JobsUtility.GetWorkStealingRange(ref ranges, jobIndex, out var begin, out var end))
    67.                     {
    68.                         return;
    69.                     }
    70.  
    71.                     var buckets = (int*)fullData.HashMap.Buffer->Buckets;
    72.                     var keys = fullData.HashMap.Buffer->Keys;
    73.  
    74.                     for (int i = begin; i < end; i++)
    75.                     {
    76.                         int entryIndex = buckets[i];
    77.  
    78.                         if (entryIndex != -1)
    79.                         {
    80.                             var key = UnsafeUtility.ReadArrayElement<TKey>(keys, entryIndex);
    81.                             fullData.JobData.ExecuteNext(key);
    82.                         }
    83.                     }
    84.                 }
    85.             }
    86.         }
    87.  
    88.         public static unsafe JobHandle Schedule<TJob, TKey, TValue>(
    89.             this TJob jobData,
    90.             NativeMultiHashMap<TKey, TValue> hashMap,
    91.             int minIndicesPerJobCount,
    92.             JobHandle dependsOn = default)
    93.             where TJob : struct, IJobNativeMultiHashMapVisitUniqueKey<TKey, TValue>
    94.             where TKey : struct, IEquatable<TKey>, IComparable<TKey>
    95.             where TValue : struct
    96.         {
    97.             var imposter = (NativeMultiHashMapImposter<TKey, TValue>)hashMap;
    98.  
    99.             var fullData = new NativeMultiHashMapVisitUniqueKeyJobStruct<TJob, TKey, TValue>
    100.             {
    101.                 HashMap = imposter,
    102.                 JobData = jobData,
    103.             };
    104.  
    105.             var scheduleParams = new JobsUtility.JobScheduleParameters(
    106.                 UnsafeUtility.AddressOf(ref fullData),
    107.                 NativeMultiHashMapVisitUniqueKeyJobStruct<TJob, TKey, TValue>.Initialize(),
    108.                 dependsOn,
    109.                 ScheduleMode.Batched);
    110.  
    111.             return JobsUtility.ScheduleParallelFor(
    112.                 ref scheduleParams,
    113.                 imposter.Buffer->BucketCapacityMask + 1,
    114.                 minIndicesPerJobCount);
    115.         }
    116.     }
    117. }
    The imposters,

    NativeHashMapDataImposter

    Code (CSharp):
    1. // <copyright file="NativeHashMapDataImposter.cs" company="BovineLabs">
    2. //     Copyright (c) BovineLabs. All rights reserved.
    3. // </copyright>
    4. // <auto-generated>It's not, just don't want analyzers working on this file atm.</auto-generated>
    5.  
    6. namespace BovineLabs.Common.Collections
    7. {
    8.     using System.Runtime.InteropServices;
    9.     using Unity.Jobs.LowLevel.Unsafe;
    10.  
    11.     [StructLayout(LayoutKind.Explicit)]
    12.     internal unsafe struct NativeHashMapDataImposter
    13.     {
    14.         [FieldOffset(0)]
    15.         public byte* Values;
    16.  
    17.         [FieldOffset(8)]
    18.         public byte* Keys;
    19.  
    20.         [FieldOffset(16)]
    21.         public byte* Next;
    22.  
    23.         [FieldOffset(24)]
    24.         public byte* Buckets;
    25.  
    26.         [FieldOffset(32)]
    27.         public int Capacity;
    28.  
    29.         [FieldOffset(36)]
    30.         public int BucketCapacityMask; // = bucket capacity - 1
    31.  
    32.         [FieldOffset(40)]
    33.         public int AllocatedIndexLength;
    34.  
    35.         [FieldOffset(JobsUtility.CacheLineSize < 64 ? 64 : JobsUtility.CacheLineSize)]
    36.         public fixed int firstFreeTLS[JobsUtility.MaxJobThreadCount * IntsPerCacheLine];
    37.  
    38.         // 64 is the cache line size on x86, arm usually has 32 - so it is possible to save some memory there
    39.         public const int IntsPerCacheLine = JobsUtility.CacheLineSize / sizeof(int);
    40.     }
    41. }
    NativeMultiHashMapImposter

    Code (CSharp):
    1. // <copyright file="NativeMultiHashMapImposter.cs" company="BovineLabs">
    2. //     Copyright (c) BovineLabs. All rights reserved.
    3. // </copyright>
    4. // <auto-generated>It's not, just don't want analyzers working on this file atm.</auto-generated>
    5.  
    6. namespace BovineLabs.Common.Collections
    7. {
    8.     using System;
    9.     using System.Runtime.InteropServices;
    10.     using Unity.Collections;
    11.     using Unity.Collections.LowLevel.Unsafe;
    12.  
    13.     [StructLayout(LayoutKind.Sequential)]
    14.     internal unsafe struct NativeMultiHashMapImposter<TKey, TValue>
    15.         where TKey : struct, IEquatable<TKey>
    16.         where TValue : struct
    17.     {
    18.         [NativeDisableUnsafePtrRestriction]
    19.         internal NativeHashMapDataImposter* Buffer;
    20.  
    21. #if ENABLE_UNITY_COLLECTIONS_CHECKS
    22.         private AtomicSafetyHandle Safety;
    23.  
    24.         [NativeSetClassTypeToNullOnSchedule]
    25.         private DisposeSentinel DisposeSentinel;
    26. #endif
    27.  
    28.         private Allocator AllocatorLabel;
    29.  
    30.         public static implicit operator NativeMultiHashMapImposter<TKey, TValue>(NativeMultiHashMap<TKey, TValue> hashMap)
    31.         {
    32.             var ptr = UnsafeUtility.AddressOf(ref hashMap);
    33.             UnsafeUtility.CopyPtrToStructure(ptr, out NativeMultiHashMapImposter<TKey, TValue> imposter);
    34.             return imposter;
    35.         }
    36.     }
    37. }
    I think that is all that is required.

    Create a job like this

    Code (CSharp):
    1. [BurstCompile]
    2. public unsafe struct HashMapJob : IJobNativeMultiHashMapVisitUniqueKey<int, Data>
    3. {
    4.     [ReadOnly]
    5.     public NativeMultiHashMap<int, Data> HashMap;
    6.  
    7.     /// <inheritdoc />
    8.     public void ExecuteNext(int key)
    9.     {
    10.         this.HashMap.TryGetFirstValue(key, out var data, out var it); // will always be true
    11.         this.DoSomething(key, data);
    12.  
    13.         while (this.HashMap.TryGetNextValue(out data, ref it))
    14.         {
    15.             this.DoSomething(key, data);
    16.         }
    17.     }
    18. }
    Schedule like this

    Code (CSharp):
    1. handle = new HashMapJob
    2.     {
    3.         HashMap = this.hashMap,
    4.     }
    5.     .Schedule(this.hashMap, 1, handle);
     
    Last edited: Sep 4, 2019
    PublicEnumE likes this.