Question Does copying from a Native container change the order?

EDIT: The thread title really should have been better and asked "Does using CopyFrom() on a Native container change the order of items". Apologies for the crappy title.

I have a Parallel job that I run which puts a bunch of data into a NativeList. Then on the conclusion of all the jobs I copy the data back out of that list into may main data structures. If I do this using a CopyFrom it is quite fast (about 5 seconds faster) but the order of the items seems to get scrambled. Either that or I have another bug somewhere. So tl;dr what I am trying to determine is if the order is guaranteed to be stable during this operation or might it be changing?

 

What order are you expecting when you have a bunch of different threads writing to a NativeList at the same time?

The copy functions do not change order. They use memcpy internally.

 

The order isn't important just that it doesn't change when copied from the container. If that isn't happening then I guess I have a bug somewhere else.

Basically I run a parallel job that populates a Native list with some data objects, and one of the properties of those objects is the index it is in the list. But after copying those to another list after the job runs, their internal index doesn't seem to match the original index.

My only other thought is the way I am determining the index inside the job doesn't work?

Basically I don't care what order they end up, I just need to know what it is Right now I'm spending a good 5-10 seconds additional time AFTER the job running back through the list and reindexing them which seems like a huge waste.

Here is the job code. Maybe nextPolyIndex isn't working like I expect it to?

Code (CSharp):

1.  
2. public struct GenerateHexMeshJob : IJobParallelFor
3. {
4.     [ReadOnly]
5.     public NativeList<float3> vertices;
6.     [ReadOnly]
7.     public NativeList<Polygon> polygons;
8.     [ReadOnly]
9.     public NativeParallelHashMap<Edge, int> midPointCache;
10.  
11.     public NativeList<Polygon>.ParallelWriter newPolygons;
12.     public int nextPolyIndex;
13.  
14.     public void Execute(int index)
15.     {
16.         var polygon = polygons[index];
17.  
18.         var centroidIndex = polygon.CenterIndex;
19.         var neighbors = polygon.NeighborIndicesAsArray();
20.         foreach (var neighborIndex in neighbors)
21.         {
22.             var neighbor = polygons[neighborIndex];
23.             var neighborCentroidIndex = neighbor.CenterIndex;
24.             var midpoint = midPointCache[new Edge(centroidIndex, neighborCentroidIndex)];
25.             var sharedEdge = polygon.GetNeighborEdge(neighborIndex);
26.  
27.             var newPoly = new Polygon(centroidIndex, midpoint, sharedEdge.A, vertices, nextPolyIndex);
28.             newPolygons.AddNoResize(newPoly);
29.             nextPolyIndex++;
30.             newPoly = new Polygon(centroidIndex, midpoint, sharedEdge.B, vertices, nextPolyIndex);
31.             newPolygons.AddNoResize(newPoly);
32.             nextPolyIndex++;
33.         }
34.     }
35. }
36.  
37.  

 

Yeah. That's not thread-safe. You have no idea what index NativeList is actually putting your elements at when you call AddNoResize().

 

Well that sucks. Guess I'm stuck reindexing them after which takes nearly as long as the job itself

 

You might try an extension method from this thread:
https://forum.unity.com/threads/fea...oresize-return-new-length-of-the-list.913412/

Unfortunately Unity has not adopted that feature.

Edit: In your case, it may help to have two unsafe extension methods: one to increment the length of the NativeList in a thread-safe way, another to write your data using

UnsafeUtility.WriteArrayElement

Maybe similar to this:

Code (CSharp):

1. public unsafe static int UnsafeAddNoResize<T>(this NativeList<T>.ParallelWriter writer)
2.             where T : unmanaged
3. {
4.             var idx = Interlocked.Increment(ref writer.ListData->m_length) - 1;
5.             CheckSufficientCapacity(writer.ListData->Capacity, idx + 1);
6.             return idx;
7. }
8.  
9. public unsafe static void UnsafeWrite<T>(this NativeList<T>.ParallelWriter writer, int idx, T value)
10.             where T : unmanaged
11. {
12.           CheckSufficientCapacity(writer.ListData->Capacity, idx + 1);
13.           UnsafeUtility.WriteArrayElement(writer.ListData->Ptr, idx, value);
14. }

 

I thought about this some more and it bugged me that

UnsafeWrite

(above) allows you to write to arbitrary indexes. Clearly that is unsafe compared to the usual native collections. So I wrote this extension method that gives you a ref to the element located at the new index. This means you can write to that particular index, but not others, preserving

ParallelWriter

safety.

Code (CSharp):

1. public static unsafe ref T NextElementNoResize<T>(this NativeList<T>.ParallelWriter list, out int index) where T : unmanaged
2. {
3. #if ENABLE_UNITY_COLLECTIONS_CHECKS
4.     AtomicSafetyHandle.CheckWriteAndThrow(list.m_Safety);
5. #endif
6.  
7.     index = Interlocked.Increment(ref list.ListData->m_length) - 1;
8.     CheckSufficientCapacity(list.ListData->Capacity, index + 1);
9.  
10.     return ref list.ListData->ElementAt(index);
11. }
12.  

You need to add the ref keyword, so it is a bit more verbose:

Code (CSharp):

1. ref var element = ref writer.NextElementNoResize(out var index);
2. element = value;