Array reduction and IJobParallelForFilter

First of all, ECS, the job system and Burst are an amazing collection of technology that I can only praise highly.
Just today I noticed that I forgot to put the ComputeJobOptimization tag on one of my jobs and almost couldn't believe it as I watched the job vanish in the profiler timeline.

Now to my question:
My problem is as follows; I use one IJobParallelFor to fill a NativeArray with what is basically a bool1 for each Entity in an ECS system. For each bool1 in that array that is equal to true, some Entities need to be created and destroyed, which is done in a separate IJob because EntityCommandBuffer is not (yet?) supported by Burst.

The IJob takes up a lot of time because it iterates over all entries (even if they don't need to be processed).
So how would one approach reducing the array to only the relevant entries in a (burst, parallel for) job in an efficient way?

Some things I tried:
- Using an IJobParallelForFilter (of which I don't clearly understand how it works) to filter for the relevant indices (?). But that wasn't faster than the other IJob and using Burst on it only crashed my editor.
- Doing some (naive) sorting but that obliterated the performance.

Edit:
Maybe there is a solution which involves avoiding creating redundant entries. If so I would be glad to hear it, though it would probably need to avoid issues with parallel adding into a list.

 

May, just add a component to the entities who needs to get updated?

So the idea is to use components as a tag to signal some changes.
I don't know for what you need this bool flag... But as I understood... this is used to signal an update.

It would be just more efficient if you would remove that bool and add an extra component to each entity which needs to be updated.

Systems should only handle entities that are importent to them.

 

Yes, that is basically what I'm already doing one step earlier. Entities are tagged with components to signal that they need to be processed by the system that emits the boolean value. Based on that I need to create and destroy some entities.

I could do the creating and destroying in the same job (or create another tag component), which would eliminate my problem, but the difficulty with this approach is that those operations are all hard sync points (according to the documentation). You can get around that with an EntityCommandBuffer, but this is not yet supported by Burst.

And therein lies my problem. The system that does the processing needs to do some moderately expensive operations on a lot of entities, so it is crucial that the job can use Burst.

I guess this is a broader problem than the title may hint at but I hope this clears some things up.

 

I don't know if this will work...

Hold an NativeArray<Entity> in your job for all entities which should be destroyed and process this array outside your job in the system.

 

I would have to test it but I guess the problem again is that I don't know how big that array needs to be before I have processed all the entities and so would have to allocate redundant entries that just signal: 'no change'.

It all comes down to parallel writing problems.
However, I've just now successfully used NativeQueue.Concurrent to be able to write concurrently from an IJobParallelFor.
With this method, I don't have to generate any redundant entries and performance increased significantly. The processing of that queue is still the bottleneck because it can use neither an IJobParallelFor nor Burst, but this will only be a problem if a lot of entries are enqueued at the same time or until Burst supports EntityCommandBuffer.

So for anyone who has a similar problem, NativeQueue.Concurrent might be worth trying.