Search Unity

Copy any NativeArray<T> into a Byte Array and back?

Discussion in 'Data Oriented Technology Stack' started by Quatum1000, Sep 2, 2018.

  1. Quatum1000

    Quatum1000

    Joined:
    Oct 5, 2014
    Posts:
    874
    Hi everyone,

    I'd like to copy a NativeArray into a byte array and back. Would that generally possible?
    I tested with some unsafe copy and pointer but it crash the compiler and unity completely.

    I tried by pointers and UnsafeUtility.MemCpy, but this crash sir unity a lot!
    Perhaps is crash because of use the un-managed and managed memory at the same time.

    Code (CSharp):
    1.  
    2. using Unity.Mathematics;
    3. using Unity.Collections.LowLevel.Unsafe;
    4.  
    5. void unsafe Test() {
    6.  
    7.   var size_e = 10;
    8.   var float4_NA = new NativeArray<float4>(size_e, Allocator.Persistent);
    9.   float4_NA[0] = 1.11f;
    10.   float4_NA[9] = 9.99f;
    11.  
    12.   var sizeCopy = UnsafeUtility.SizeOf<float4>() * size_e;
    13.   byte[] b_array = new byte[sizeCopy];
    14.   byte* srcPtr = (byte*)float4_NA.GetUnsafePtr();
    15.   byte* dstPtr = (byte*)b_array[0];
    16.  
    17. // from float4 to b_array
    18.         UnsafeUtility.MemCpy(srcPtr, dstPtr, sizeCopy);
    19.  
    20. // from b_array to float4_NA
    21.         UnsafeUtility.MemCpy(dstPtr, srcPtr, sizeCopy);
    22. }
    23.  

    I'd like to use it to save/load a NativeArray<float4> persistence.
    Does anyone have a solution for this?
    Thank you.
     
    Last edited: Sep 2, 2018
  2. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,180
    Code (CSharp):
    1. // This performs no copying...
    2. var byteSlice  = new NativeSlice<float4>(myArray).SliceConvert<byte>();
    3.  
    4. // This copies using memcpy
    5. byteSlice.CopyTo(managedByteArray);
    6.  
    Overall this is both very fast and safe.
     
    jdtec and Quatum1000 like this.
  3. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,180
    Oops. Sorry. NativeSlice doesn't take Memcpy fast path. We'll need to make it do that. Sorry.

    So for now you need to use the low level API's...
    This is the implementation of the CopyTo, you can probably adapt it from this.

    Code (CSharp):
    1.  
    2.         public static void Copy(NativeArray<T> src, int srcIndex, T[] dst, int dstIndex, int length)
    3.         {
    4. #if ENABLE_UNITY_COLLECTIONS_CHECKS
    5.             AtomicSafetyHandle.CheckReadAndThrow(src.m_Safety);
    6.             if (dst == null)
    7.                 throw new ArgumentNullException(nameof(dst));
    8.             if (length < 0)
    9.                 throw new ArgumentOutOfRangeException(nameof(length), "length must be equal or greater than zero.");
    10.             if (srcIndex < 0 || srcIndex > src.Length || (srcIndex == src.Length && src.Length > 0))
    11.                 throw new ArgumentOutOfRangeException(nameof(srcIndex), "srcIndex is outside the range of valid indexes for the source NativeArray.");
    12.             if (dstIndex < 0 || dstIndex > dst.Length || (dstIndex == dst.Length && dst.Length > 0))
    13.                 throw new ArgumentOutOfRangeException(nameof(dstIndex), "dstIndex is outside the range of valid indexes for the destination array.");
    14.             if (srcIndex + length > src.Length)
    15.                 throw new ArgumentException("length is greater than the number of elements from srcIndex to the end of the source NativeArray.", nameof(length));
    16.             if (dstIndex + length > dst.Length)
    17.                 throw new ArgumentException("length is greater than the number of elements from dstIndex to the end of the destination array.", nameof(length));
    18. #endif
    19.             var handle = GCHandle.Alloc(dst, GCHandleType.Pinned);
    20.             var addr = handle.AddrOfPinnedObject();
    21.             UnsafeUtility.MemCpy(
    22.                 (byte*)addr + dstIndex * UnsafeUtility.SizeOf<T>(),
    23.                 (byte*)src.m_Buffer + srcIndex * UnsafeUtility.SizeOf<T>(),
    24.                 length * UnsafeUtility.SizeOf<T>());
    25.             handle.Free();
    26.         }
    27.  
     
    jdtec and Quatum1000 like this.
  4. jooleanlogic

    jooleanlogic

    Joined:
    Mar 1, 2018
    Posts:
    401
    Are you missing an 'address of' operator or do you not need this in c#? (byte*)&b_array[0];?
     
  5. Quatum1000

    Quatum1000

    Joined:
    Oct 5, 2014
    Posts:
    874
    ... double post ...
     
    Last edited: Sep 3, 2018
  6. Quatum1000

    Quatum1000

    Joined:
    Oct 5, 2014
    Posts:
    874
    Does not work because of a protection level and not nullable issue. I'd like to use it in monobehavior.

    Code (CSharp):
    1. CopyToByteArray(ref float4_NA, 0, ref b_array, 0, b_array.Length);
    2.  
    3. public unsafe void CopyToByteArray<T>(ref NativeArray<T> src, int srcIndex, ref byte[] dst, int dstIndex, int length) {
    4.        AtomicSafetyHandle.CheckReadAndThrow(src.m_Safety); //  .m_Safety Protection level issue
    5.        var handle = GCHandle.Alloc(dst, GCHandleType.Pinned);
    6.        var addr = handle.AddrOfPinnedObject();
    7.        UnsafeUtility.MemCpy(
    8.            (byte*)addr + dstIndex * UnsafeUtility.SizeOf<T>(), // <T> Not nullable issue
    9.            (byte*)src.m_Buffer + srcIndex * UnsafeUtility.SizeOf<T>(), // <T> Not nullable issue
    10.            length * UnsafeUtility.SizeOf<T>()); // <T> Not nullable issue
    11.        handle.Free();
    12.    }
    How can I solve this protection level and not nullable issue?
    Thank you.


    Thank you. This is the unsafe and pointer adventure! & is a pointer operator for single objects in a managed environment.
    eg works:
    float float1 = 1;
    byte* dstPtr = (byte*) &float1;

    But does not for an array.
     
    Last edited: Sep 3, 2018
  7. elcionap

    elcionap

    Joined:
    Jan 11, 2016
    Posts:
    135
    You're converting a byte value to an address instead of getting the pointer. Also you need to use a fixed block, otherwise the object could be moved in the middle of the copy.

    Also, UnsafeUtility.MemCpy parameters order is DESTINATION then the SOURCE.

    Code (CSharp):
    1.  
    2. using Unity.Mathematics;
    3. using Unity.Collections.LowLevel.Unsafe;
    4.  
    5. void unsafe Test() {
    6.  
    7.   var size_e = 10;
    8.   var float4_NA = new NativeArray<float4>(size_e, Allocator.Persistent);
    9.   float4_NA[0] = 1.11f;
    10.   float4_NA[9] = 9.99f;
    11.  
    12.   var sizeCopy = UnsafeUtility.SizeOf<float4>() * size_e;
    13.   byte[] b_array = new byte[sizeCopy];
    14.   byte* srcPtr = (byte*)float4_NA.GetUnsafePtr();
    15.  
    16.   fixed (byte* dstPtr = b_array) {
    17.  
    18. // from float4 to b_array
    19.          UnsafeUtility.MemCpy(dstPtr, srcPtr, sizeCopy);
    20.  
    21. // from b_array to float4_NA
    22.          UnsafeUtility.MemCpy(srcPtr, dstPtr, sizeCopy);
    23.  }
    24. }
    25.  
    []'s
     
    Opeth001 and Quatum1000 like this.
  8. Quatum1000

    Quatum1000

    Joined:
    Oct 5, 2014
    Posts:
    874
    Cool thank you!

    Are you able to solve this too? Would be great to have the <T> parameter

    Code (CSharp):
    1.     CopyToByteArray(ref float4_NA, 0, ref b_array, 0, b_array.Length);
    2.    
    3.     public unsafe void CopyToByteArray<T>(ref NativeArray<T> src, int srcIndex, ref byte[] dst, int dstIndex, int length) {
    4.            AtomicSafetyHandle.CheckReadAndThrow(src.m_Safety); //  .m_Safety Protection level issue
    5.            var handle = GCHandle.Alloc(dst, GCHandleType.Pinned);
    6.            var addr = handle.AddrOfPinnedObject();
    7.            UnsafeUtility.MemCpy(
    8.                (byte*)addr + dstIndex * UnsafeUtility.SizeOf<T>(), // <T> Not nullable issue
    9.                (byte*)src.m_Buffer + srcIndex * UnsafeUtility.SizeOf<T>(), // <T> Not nullable issue
    10.                length * UnsafeUtility.SizeOf<T>()); // <T> Not nullable issue
    11.            handle.Free();
    12.        }
    13.  
     
  9. elcionap

    elcionap

    Joined:
    Jan 11, 2016
    Posts:
    135
    Code (csharp):
    1.  
    2.     public unsafe void CopyToByteArray<T>(ref NativeArray<T> src, int srcIndex, ref byte[] dst, int dstIndex, int length) where T : struct
    3.     {
    4.         AtomicSafetyHandle.CheckReadAndThrow(NativeArrayUnsafeUtility.GetAtomicSafetyHandle(src)); //  .m_Safety Protection level issue
    5.  
    6.         var size = UnsafeUtility.SizeOf<T>();
    7.         var srcAddr = (byte*)src.GetUnsafeReadOnlyPtr();
    8.  
    9.         fixed (byte* dstAddr = dst)
    10.         {
    11.             UnsafeUtility.MemCpy(&dstAddr[dstIndex * size], &srcAddr[srcIndex * size], length);
    12.         }
    13.     }
    14.  
    I'm not sure about the AtomicSafetyHandle line. Is it correct? Should have a check for some "safe define" to skip it in the build?
    Also the length parameters was using the size in bytes. I kept it for compatibility with your code but, maybe using the size of the T would be better for reading. I'm not sure.

    Anyway this only answer the error in your code, not your problem. It's ok if it was only curiosity but what would be the best for your case will be hard to say without more information. What are you trying to achieve?
     
    WildStyle69 and Quatum1000 like this.
  10. elcionap

    elcionap

    Joined:
    Jan 11, 2016
    Posts:
    135
    Just one more thing. This code isn't safe at all. Any srcIndex, dstIndex or length out of the bounds will crash your editor.
     
    Quatum1000 likes this.
  11. Quatum1000

    Quatum1000

    Joined:
    Oct 5, 2014
    Posts:
    874
    Thanks a lot. I'm impressed, great.

    I wrote 2 functions, MoveTo and MoveFrom complete length. and should be safe.

    Code (CSharp):
    1. // Description:
    2. // MemoryCopy the contents of a NativeArray to a ByteArray and back.
    3. //
    4.     public unsafe void MoveToByteArray<T>(ref NativeArray<T> src, ref byte[] dst) where T : struct
    5.        {
    6. #if ENABLE_UNITY_COLLECTIONS_CHECKS
    7.        AtomicSafetyHandle.CheckReadAndThrow(NativeArrayUnsafeUtility.GetAtomicSafetyHandle(src));
    8.        if (dst == null)
    9.            throw new ArgumentNullException(nameof(dst));
    10. #endif
    11.        var size = UnsafeUtility.SizeOf<T>() * src.Length;
    12.        var srcAddr = (byte*)src.GetUnsafeReadOnlyPtr();
    13.        if (dst.Length != size)
    14.            dst = new byte[size];
    15.  
    16.        fixed (byte* dstAddr = dst) {
    17.            UnsafeUtility.MemCpy(&dstAddr[0], &srcAddr[0], size);
    18.        }
    19.    }
    20.  
    21.    public unsafe void MoveFromByteArray<T>(ref byte[] src, ref NativeArray<T> dst) where T : struct
    22.        {
    23. #if ENABLE_UNITY_COLLECTIONS_CHECKS
    24.        AtomicSafetyHandle.CheckReadAndThrow(NativeArrayUnsafeUtility.GetAtomicSafetyHandle(dst));
    25.        if (src == null)
    26.            throw new ArgumentNullException(nameof(src));
    27. #endif
    28.        var size = UnsafeUtility.SizeOf<T>();
    29.        if (src.Length != (size * dst.Length)) {
    30.            dst.Dispose();
    31.            dst = new NativeArray<T>(src.Length / size, Allocator.Persistent);
    32. #if ENABLE_UNITY_COLLECTIONS_CHECKS
    33.            AtomicSafetyHandle.CheckReadAndThrow(NativeArrayUnsafeUtility.GetAtomicSafetyHandle(dst));
    34. #endif
    35.        }
    36.  
    37.        var dstAddr = (byte*)dst.GetUnsafeReadOnlyPtr();
    38.        fixed (byte* srcAddr = src) {
    39.            UnsafeUtility.MemCpy(&dstAddr[0], &srcAddr[0], src.Length);
    40.        }
    41.    }
    Would it possible to get the Allocator type from the NativeArray and apply to the new declaration directly?
    getType() returns the field typ.

    Thank you.
     
    Last edited: Sep 3, 2018
  12. Quatum1000

    Quatum1000

    Joined:
    Oct 5, 2014
    Posts:
    874
    Code (CSharp):
    1. AtomicSafetyHandle.CheckReadAndThrow(NativeArrayUnsafeUtility.GetAtomicSafetyHandle(src));
    Is this used only for editor code?
     
unityunity