Unsafe containers

Hi! I need help with UnsafeList. I have a system that needs to use UnsafeList but the behaviour is very strange.
The code is here:

Code (CSharp):

1. public struct MyCollider : IComponentData
2. {
3.     public AABB localAABB;
4.     public AABB worldAABB;
5. }
6. public struct CellData
7. {
8.     public MyCollider collider;
9.     public Entity entity;
10. }
11. //system variables used for creating multiple grids but stored in the same arrays
12. NativeArray<UnsafeList<CellData>> cellArray;
13.     NativeArray<UnsafeList<CellData>.ParallelWriter> cellArrayParallelWriter;
14.     NativeArray<float3> cellSize;
15.     NativeArray<int> gridStartIndex;
16.     NativeArray<int3> numberOfCells;
17.     NativeArray<float3> inverserCellSize;
18.     public int numberOfCellsX = 16;
19.     public int numberOfCellsY = 16;
20.     public int numberOfCellsZ = 16;
21.     public int cellSizeX = 1;
22.     public int cellSizeY = 1;
23.     public int cellSizeZ = 1;
24.     public float hierarchyFactor = 2f;
25.     public int numberOfHierarchies = 4;
26.  
27. protected unsafe override void OnCreate() {
28.  
29. cellArray = new NativeArray<UnsafeList<CellData>>(100000, Allocator.Persistent);
30.         cellArrayParallelWriter = new NativeArray<UnsafeList<CellData>.ParallelWriter>(100000, Allocator.Persistent);
31.         totalNumberOfCells = 0;
32.         for ( int i = 0; i < numberOfHierarchies; i++)
33.         {
34.             float factor = math.pow(hierarchyFactor, i);
35.             float3 size = new float3(cellSizeX * factor, cellSizeY * factor, cellSizeZ * factor);
36.             int cellsX = (int)(numberOfCellsX / factor);
37.             int cellsY = (int)(numberOfCellsY / factor);
38.             int cellsZ = (int)(numberOfCellsZ / factor);
39.            
40.             for (int j = 0, length = cellsX * cellsY * cellsZ; j < length; j++)
41.             {
42.                 cellArray[totalNumberOfCells + j] = new UnsafeList<CellData>(25, Allocator.Persistent);
43.                 cellArrayParallelWriter[totalNumberOfCells + j] = cellArray[totalNumberOfCells + j].AsParallelWriter();
44.             }
45.            
46.             cellSize[i] = size;
47.             inverserCellSize[i] = 1 / size;
48.             gridStartIndex[i + 1] = gridStartIndex[i] + cellsX * cellsY * cellsZ;
49.             numberOfCells[i] = new int3(cellsX, cellsY, cellsZ);
50.             totalNumberOfCells += cellsX * cellsY * cellsZ;
51.         }
52.         cellArrayParallelWriter[0].AddNoResize(new CellData { entity = new Entity(), collider = new MyCollider { localAABB = new AABB { Center = new float3(1, 0, 1) } } });
53.         cellArray[0].AddNoResize(new CellData { entity = new Entity(), collider = new MyCollider { localAABB = new AABB { Center = new float3(2, 0, 2) } } });
54.         Debug.Log(cellArray[0].Ptr[0].collider.localAABB.Center);
55.         Debug.Log(cellArray[0].Ptr[1].collider.localAABB.Center);
56.         Debug.Log(cellArray[0].Length);
57.  
58.  
59. }
60.  

The console:

Code (CSharp):

1. float3(2f, 0f, 2f)
2. float3(-6.360683E-12f, 5.885454E-43f, 1.842707E-42f)
3. 0

So the remarks are that for some reason the list is not adding but actually replacing just the 1st item and why is the length just 0? (really strange)
Let's do this test in the OnUpdate method:

Code (CSharp):

1. Debug.Log(cellArray[0].Capacity);
2.         Debug.Log(cellArray[0].Length);
3.         cellArray[0].AddNoResize(new CellData { entity = new Entity(), collider = new MyCollider { localAABB = new AABB { Center = new float3((float)Time.ElapsedTime, 0, (float)Time.ElapsedTime) } } }); ;
4.         Debug.Log(cellArray[0].Ptr[0].collider.localAABB.Center);

The console:
Every frame Capacity is 32 (why 32!??! I wanted it to be 25)
Every frame Length is 0(again why is it not increasing?)
Every frame the item Debug is the real elapsed time.
(I did this test with Capacity set to 1 and that behaviour still repeated, so I didn't get any errors)

I did all these tests with a UnsafeList<CellData> (outside of an array) and everything worked as expected (tho capacity still had a bigger number than what I set).

What is happening? Does this mean that I cannot use NativeArray<UnsafeList<>>? If that is the case, why does ECS not stop me?

 

NativeArray's indexer returns by value. In your case it means that the elements of UnsafeList will get updated (since you're accessing them through a pointer) but Length/Capacity will not. There are two solutions:

Code (CSharp):

1. // 1: Write list back after you modify it
2. var list = cellArray[i];
3. list.AddNoResize(/*...*/);
4. cellArray[i] = list;
5.  
6. // 2: Use a reference or pointer to list
7. ref var list = ref UnsafeUtility.ArrayElementAsRef<UnsafeList</*...*/>>(cellArray.GetUnsafePtr(), i);
8. list.AddNoResize(/*...*/);

 

God damn forgot about the value return of arrays... Thank you!

 

In terms of performance, is 1 going to be slower than 2 for many lists with many elements?

 

The reason why you suspect that is typically optimized out by Burst. I suspect both methods will boil down to Burst loading the UnsafeList into registers, modifying it, and finally writing it back to the array memory address.

 

I need help with parallel writing now. I made an IJobChunk and schedule it as parallel but I will post just the relevant section

Code (CSharp):

1. //hashCode is a calculated value that is within bounds
2. //cellArray and cellArrayParallelWriter are those from the OnCreate that I posted at the beginning of this thread, basically creating for each element of the array an UnsafeList<CellData> and parallelWriter will hold the UnsafeList<CellData>.ParallelWriter
3. Debug.Log(cellArray[hashCode].Capacity);
4.                 Debug.Log(cellArray[hashCode].Length);
5.                 Debug.Log(cellArray[hashCode].IsCreated);
6.              
7. //newCell is just a CellData
8.                 var cellParallelWriter = cellArrayParallelWriter[hashCode];
9.                 cellParallelWriter.AddNoResize(newCell);
10.                 cellArrayParallelWriter[hashCode] = cellParallelWriter;

when I try to run the job I get these errors

Code (CSharp):

1. Exception: AddNoResize assumes that list capacity is sufficient (Capacity 217, Length 1601556639), requested length 1601556635!
2. Unity.Collections.LowLevel.Unsafe.UnsafeList.CheckNoResizeHasEnoughCapacity (System.Int32 length, System.Int32 index) (at

Capacity is 32 every time, length is 0 and IsCreated is true. I think there's something wrong with how I'm using the ParallelWriter. How should I use ParallelWriter? Shouldn't it be the same as NativeList<>.ParallelWriter?

 

1. You can not cache ParallelWriter in a array. when the UnsafeList reallocate the interal listData is reallocated, so the pointer to listdata in ParallelWriter will be pointing to recycled memory location.
2. Unsafe Container doesn't have safety checks, which throws error when jobs are writing/reading in parallel. you are responsible for all the safety check. Any parallel read/write could easlly corrupt your data.

3. NativeArray<UnsafeList<CellData>> cellArray;

   this need a pointer struct, so you don't lost the internal pointer when realloc happens. or just dont use NativeArray for at top level. Bacause pointer is not a type in c# it can not be used as a genereic type paramter so

   NativeArray<UnsafeList<CellData>*>

   is not valid.

IMO You should use something like:

Code (CSharp):

1. var listDataSize= UnsafeUtility.SizeOf<UnsafeList<CellData>>();
2. int totalSize=listDataSize*100000;
3. UnsafeList<CellData>* cellArray = (UnsafeList<CellData>*)UnsafeUtility.Malloc(totalSize,JobsUtility.CacheLineSize,Allocator.Persistent);
4. for ( int i = 0; i < 100000; i++)
5. {
6.     cellArray[i]=  new UnsafeList<CellData>(25, Allocator.Persistent);
7. }

 

Thank you for your answer! Is this valid inside the job?

Code (CSharp):

1. var cellParallelWriter = cellArray[index].AsParallelWriter();
2. cellParallelWriter.AddNoResize(item);

instead of caching the ParallelWriter?
Also I would really appreciate if you could point me a resource for learning UnsafeUtility, I've not found anything yet.

 

Well, it's unsafe anyway. so it's okay. But better to call AsParallelWriter outside of the job, and pass it in as fields.

 

Hi everyone,
I need help on combining native arrays in to a one big array. The scenario is like that.

I need to create some mesh data in different jobs (verts, normals, uvs etc.) There is no problem to create them as I create Nativearray list and send each list item (Nativearray) to jobs to make calculations)

After receiving the data i can apply them to the dynamically created meshes.

But the problem starts here. Assigning the arrays to mesh data is very CPU expensive. I also try to combine these arrays in main thread and then assign to one mesh is also CPU expensive. Combining arrays in main thread taking to much time, even more than assigning separately.

My intention is to add all native arrays into a one big array in another job after completion of mesh data calculation jobs.
I, first created an unsafe list
Then create different jobs (one fore each native array) and in these jobs, add each native array item to big unsafe list. to do this i send same big unsafe list to each job.

When i debugged i saw that the native array items are added to the unsafe list inside the jobs. But when i try to get the unsafe list back in main thread it seems emtpy.

Briefly: I need to combine native array lists in to a big list/array/queue inside jobs.

All comments on this will be appreciated

Thanks in advance.

 

I Know this is an old reply and my answer is no use. But still. I think you need to pass NativeLust.ParallelWriter into jobs, not the unsafe list itself. Only that way you can write in original list