Use NativeQueue as a NativeArray?

is it possible to access the data from a NativeQueue like from a NativeArray or NativeList, without first getting it out with Dequeue()?

Alternatively is there an other way to Add() elements in parallel to something like a [WriteOnly] NativeList? I know about IJobParallelForFilter but it is currently single threaded.

 

Directly - No, Queue (not here but abstract, like Stack, Linked List and other) this is one of the basic algorithms for structuring data, and it works like this by design. You can try use NativeHashMap.Concurent or NativeMultiHashMap.Concurent for writing data multithreaded safely.

 

I looked a bit at the NativeHashMap but I'm not sure how to iterate over all the elements. Could you please provide a small code example?

Thinking about it, NativeHashMap might not be that great for my use case anyway... I should probably give a few more details of what I'm trying to do...

I have a sort of custom particle system, with a set maximum capacity using NativeArrays. To "kill" the dead particles I get their specific indices inside multi threaded jobs (right now using NativeQueue). Then in a single thread I rearrange the array (alive particles at the start, dead ones at the end), so this requires looping over the deadParticles Queue multiple times. Right now I have to first Dequeue() them all in a separate array.

One other thing that might help.... is there a faster way to copy all the data from a NativeQueue?

 

I am guessing you are using NativeQueue because you want the Concurrent Add possibility, and then the trade off is that the queue is not indexable which cause problem on modification and it is a hassle to delete a specific item from a queue?

I would do it like this :

NativeArray<Particles> particles;
Instantiate to the size of maximum possible number of particles, should be able to calculate concretely from spawn rate and life time.

NativeList<int> aliveList;
A list of index of the prior array that are currently alive and usable/allowing render.
To get all alive particles you can iterate on this list.

NativeList<int> vacantList;
A list of index that is not alive. We maintain this to be the inverse of `aliveList`

Spawn :
1. Single threaded job retrieve free slots from vacantList according to a number you would like to spawn, removing them then add them to aliveList.
2. Multithreaded job IJobParallelFor with dependency from the prior job. Use [NativeDisableParallelForRestriction] on particles NativeArray to set starting data according to those retrieved slots.

Update/Render :
Multithreaded jobs work on each element in [ReadOnly] aliveList and update its corresponding index in the particles NativeArray

Remove :
An inverse of Spawn. You left the dead data in NativeArray because the aliveList will tell the update logic not to update them.

 

Yes... but I don't even need to access a specific item, I just need to loop through it multiple times, like Peek() but also go further, something like a PeekNext().

This would obviously require testing with the final use case data (the specific particle effect) on the target hardware, but it seems like in general this approach might have more slow-down points:

1. Spawn: Single threaded loop through the NativeLists.
2. Spawn: Random access into the particles NativeArray.
3. Update: Random access into the NativeArray.
4. Remove: Single threaded loop through the NativeLists. (this happens with my approach too)
5. Render: Random access into the NativeArray.
6. Render: Packing together the active particles data before it is sent to the GPU.

In some cases it might actually be more efficient to not delete or move any data, and just have an extra if(particles.isAlive) check in Update and Render, including the shader.

 

I've seen other posts for a while, of people requesting a similar thing. NativeList<>.Concurrent write access, or multi threaded IJobParallelForFilter.

It was suggested that they can write it themselves. So if anyone did, it would be very appreciated if they post their solution! Something like this is useful in a lot of situations.

 

I wonder if you the deterministic ordering concept from EntityCommandBuffers could be applied to a NativeList<T>.Concurrent implementation.
NativeQueue<T> would still be useful when order doesn't matter.

Right now, maybe you could take it by making a structure that contains your particle data and the IJobParralelFor index. Then you could dequeue and reconstitute the order into a NativeList on an IJob. It has a resize function that doesn't just alter capacity, but resizes length and leaves you to set the unassigned space. You can use the queue's length to determine the proper size if your sure it is done being written to.

 

I actually haven't really used the Entity System much, so far... just custom basic arrays and the Jobs System... so I'm not sure what you mean about the EntityCommandBuffers. But I though you can't Dequeue() inside a job. Isn't that a race condition?

 

You can deque on a job if you are only dequeuing on a single job. So you'd split out the work to be queued across multiple parallel for or other jobs then have a single IJob to dequeue and process or linearize it before doing other jobs that rely on the transformed data.

You can also call things like the sort extension functions on the data after you have moved it into a NativeList<T> if you use the same or another IJob.

 

Ooh sorry, you meant single threaded job... Yes, that is exactly what I'm doing right now. This is the step I'm trying to avoid:

To be able to loop through the NativeQueue multiple times without emptying it. Skip the step of dequeuing it to a NativeList... or at least faster way of copying the data.

Or allow parallel Add() into a [WriteOnly] NativeList, just like a NativeQueue.

 

Here is one way I have gotten around this predicament. I whipped this up pretty quick, but i think i got everything.

Spoiler: Example

Code (CSharp):

1. public unsafe class YourSystem : ComponentSystem
2.     {
3.         [BurstCompile]
4.         private struct YourJob : IJobParallelFor
5.         {
6.             [NativeDisableUnsafePtrRestriction] public int* LastIndex;
7.  
8.             public NativeArray<int> Input;
9.             public NativeArray<int> Output;
10.  
11.             public void Execute(int index)
12.             {
13.                 var outputIndex = Interlocked.Increment(ref *LastIndex) - 1;
14.  
15.                 Output[outputIndex] = Input[index];
16.             }
17.         }
18.  
19.         protected override void OnUpdate()
20.         {
21.             var lastIndex = 0;
22.             var input     = new NativeArray<int>(50, Allocator.TempJob);
23.             var output    = new NativeArray<int>(100, Allocator.TempJob);
24.  
25.             new YourJob
26.                 {
27.                     LastIndex = &lastIndex,
28.                     Input     = input,
29.                     Output    = output
30.                 }
31.                .Run(input.Length);
32.  
33.             for (var i = 0; i < lastIndex; i++)
34.             {
35.                 var o = output[i];
36.             }
37.  
38.             input.Dispose();
39.             output.Dispose();
40.         }
41.     }