Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

ConvertExistingDataToNativeArray and AtomicSafetyHandle

Discussion in 'Entity Component System' started by arkano22, Apr 28, 2020.

  1. arkano22

    arkano22

    Joined:
    Sep 20, 2012
    Posts:
    1,891
    Hi there,

    Here's my use case: I have a pointer to unmanaged memory, basically a buffer of floating point data. I just want to wrap a NativeArray over it to be able to use this data in a Job.

    So I use NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray(), passing it my pointer, the size of the buffer, and Allocator.None since I manage the memory buffer myself.

    Now, If I try to execute any Job using that array with the Jobs Debugger enabled I get:
    Everything works just fine with the debugger disabled.

    Dwelving into this a bit, my conclusion is that the array requires to have an AtomicSafetyHandle, and that I must Release() that handle manually when the array is no longer valid, even though I specified Allocator.None because I'm in charge of managing the data that the NativeArray is wrapping for me.

    Is this assumption correct? Is there no way to simply wrap a NativeArray over custom data without worrying about releasing the NativeArray or its safety handle?
     
  2. ejl103

    ejl103

    Joined:
    Dec 21, 2021
    Posts:
    6
    I seem to be running into the same issue, it seems that if you use Allocator.None the handle isn't setup properly causing things to break - say when you use GetUnsafePtr(), but if you add one - because it is Allocator.None, it won't clear it up in the Dispose for you - makes it very hard to use it safely, unless I'm missing something
     
  3. elliotc-unity

    elliotc-unity

    Unity Technologies

    Joined:
    Nov 5, 2015
    Posts:
    230
    Yeah, it's a little weird. This is the common pattern:

    var myarray = NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray<T>(ptr, Allocator.Invalid);
    #if ENABLE_UNITY_COLLECTIONS_CHECKS
    NativeArrayUnsafeUtility.SetAtomicSafetyHandle(ref myarray, AtomicSafetyHandle.
    GetTempUnsafePtrSliceHandle
    ());
    #endif


    [Edited to switch from
    GetTempMemoryHandle to
    GetTempUnsafePtrSliceHandle, which people tell me is preferable.]
     
    Last edited: Mar 18, 2022
    joshuacwilde, Mockarutan and arkano22 like this.
  4. ejl103

    ejl103

    Joined:
    Dec 21, 2021
    Posts:
    6
    Thank you, that does seem to fix the issue.

    Is there any chance the documentation on using ConvertExistingDataToNativeArray() can be improved as it is very sparse, also would be nice if it was made more friendly by default so you don't have to do this.
     
  5. ejl103

    ejl103

    Joined:
    Dec 21, 2021
    Posts:
    6
    Oh and I'm using Allocator.None rather than Allocator.Invalid - does that make any difference? Looking at the source of NativeArray I don't think so, but might have missed something
     
  6. arkano22

    arkano22

    Joined:
    Sep 20, 2012
    Posts:
    1,891
    Doesn't work for me. When attempting to use the resulting native array in a job, I get:
    Which makes sense since the documentation explicitly says the handle returned by GetTempUnsafePtrSliceHandle cannot be used in a job:
    https://docs.unity3d.com/ScriptRefe...e.AtomicSafetyHandle.GetTempMemoryHandle.html

    What good is an AtomicSafetyHandle that cannot be used in a job? Aren't these handles specifically designed to safeguard against race conditions/concurrent writes and such? Maybe I'm misunderstanding them...
     
    Last edited: Oct 26, 2022
    burningmime likes this.
  7. burningmime

    burningmime

    Joined:
    Jan 25, 2014
    Posts:
    845
    Bump. There seems to be no way to allocate something externally (not via the job system) on the main thread and then pass that pointer to a job as a NativeArray, even if you're 100% sure it's safe. You can pass it as a
    T*
    only.
     
    arkano22 likes this.
  8. burningmime

    burningmime

    Joined:
    Jan 25, 2014
    Posts:
    845
    Ah, OK I think I got it, although it was a pain in the arsebiscuits...

    Code (CSharp):
    1. // converting pointer
    2. NativeArray<T> nativeArray = NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray<T>(ptr, length, Allocator.None);
    3. AtomicSafetyHandle atomicSafetyHandle = AtomicSafetyHandle.Create();
    4. NativeArrayUnsafeUtility.SetAtomicSafetyHandle(ref nativeArray, atomicSafetyHandle);
    5.  
    6. // disposing atomic safety
    7. AtomicSafetyHandle.CheckDeallocateAndThrow(atomicSafetyHandle);
    8. AtomicSafetyHandle.Release(atomicSafetyHandle);
    For the particular case of pinning a managed array...

    Code (CSharp):
    1. unsafe struct ManagedToNativeArrayPin : IDisposable
    2. {
    3.     private GCHandle _gcHandle;
    4. #if ENABLE_UNITY_COLLECTIONS_CHECKS
    5.     private AtomicSafetyHandle _atomicSafetyHandle;
    6. #endif
    7.     private enum PinMode { NONE, EMPTY_ARRAY, PINNED }
    8.     private PinMode _mode;
    9.     public bool IsAllocated => _mode != PinMode.NONE;
    10.  
    11.     public static (NativeArray<T> nativeArray, ManagedToNativeArrayPin pin) Alloc<T>(T[] managedArray, int length = -1) where T : unmanaged
    12.     {
    13.         if(managedArray == null)
    14.             throw new Exception("Array must be non-null");
    15.         if(length < 0)
    16.             length = managedArray.Length;
    17.         if(length > managedArray.Length)
    18.             throw new Exception($"Requested length {length} was greater than array length {managedArray.Length} for type {typeof(T)}[]");
    19.        
    20.         T* ptr;
    21.         ManagedToNativeArrayPin pin;
    22.         if(length > 0)
    23.         {
    24.             pin._mode = PinMode.PINNED;
    25.             pin._gcHandle = GCHandle.Alloc(managedArray, GCHandleType.Pinned);
    26.             ptr = (T*) pin._gcHandle.AddrOfPinnedObject();
    27.         }
    28.         else
    29.         {
    30.             pin._mode = PinMode.EMPTY_ARRAY;
    31.             pin._gcHandle = default;
    32.             ptr = null;
    33.         }
    34.        
    35.         NativeArray<T> nativeArray = NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray<T>(ptr, length, Allocator.None);
    36.        
    37.     #if ENABLE_UNITY_COLLECTIONS_CHECKS
    38.         AtomicSafetyHandle atomicSafetyHandle = AtomicSafetyHandle.Create();
    39.         NativeArrayUnsafeUtility.SetAtomicSafetyHandle(ref nativeArray, atomicSafetyHandle);
    40.         pin._atomicSafetyHandle = atomicSafetyHandle;
    41.     #endif
    42.        
    43.         return (nativeArray, pin);
    44.     }
    45.  
    46.     public void Dispose()
    47.     {
    48.         if(_mode == PinMode.PINNED)
    49.             _gcHandle.Free();
    50.        
    51.     #if ENABLE_UNITY_COLLECTIONS_CHECKS
    52.         if(_mode != PinMode.NONE)
    53.         {
    54.             AtomicSafetyHandle.CheckDeallocateAndThrow(_atomicSafetyHandle);
    55.             AtomicSafetyHandle.Release(_atomicSafetyHandle);
    56.         }
    57.     #endif
    58.        
    59.         _mode = PinMode.NONE;
    60.     }
    61. }
    You can manage a list of pins somewhere like this...

    Code (CSharp):
    1.  
    2.         private readonly List<ManagedToNativeArrayPin> _pins = new();
    3.         public NativeArray<T> pin<T>(T[] managedArray, int length = -1) where T : unmanaged
    4.         {
    5.             (NativeArray<T> nativeArray, Pin pin) = ManagedToNativeArrayPin.Alloc(managedArray, length);
    6.             _pins.Add(pin);
    7.             return nativeArray;
    8.         }
    And free them all at once each frame:

    Code (CSharp):
    1.             foreach(ManagedToNativeArrayPin pinned in _arrayHandles)
    2.                 pinned.Dispose();
    This was enough for me, since a lot of important graphics APIs you'll need when writing an SRP still require only managed arrays and not native arrays, even in the 2023 alpha.

    EDIT 2: OK; now this should work!
     
    Last edited: Dec 4, 2022