Feedback Approach for collecting output from jobs? (Parallel writing)

Hello folks! Just started playing with ECS this weekend. Really enjoying it. Think I've got a decent feel for the basics in terms of executing some simple behaviors. However at the moment, I'm struggling with the data containers and trying to generate some kind of output from a batch of jobs. Been Googling on this for a day or two and have yet to find a good example so figured I'd try the message board for feedback. Currently fighting InvalidOperationExceptions with NativeList and NativeArrays.

A simple example of what I'm trying to do is event processing. I've got a system that is generating event entities and I would like to do something like iterate through all of these entities to create an array of strings that I could feed to the console log in the main thread. I know the simplest way to do this would be on the main thread, especially if there aren't many entities but I still would like to know how to approach parallel writing for future implementations.

Thanks!

 

Note that string is a reference type and won't work with the burstcompiler, if you want to use it (and you should, for the sake of performance). https://docs.unity3d.com/Packages/com.unity.burst@0.2/manual/index.html

To collect data from a job, create a (temp. or persistent, as you like) nativearray in the mainthread, pass it to the job, write to it. Complete the job and use the nativearray as you need to in the mainthread or in other jobs.

Also mark all nativearrays as [Readonly] in the job, which you don't need to be written to, to save some performance.

 

Ahhhh, ok. Thank you for the clarification! This is the route I'm going however I was trying to use a NativeList (swapped out a string with an int just for figuring out the parallel writing for now). This approach though means you have to pre-allocate the array. What if a job had a variable output (One to many per queried/processed entity)? Is that currently feasible?

Is NativeArray the only data structure we can write to? Are there any concurrent data structure classes for this?

 

using the new mathematics lib there are some nativestrings that seems to work with jobs/burst, atleast it didnt give me any errors while I was messing with them.

Did someone already try using them to check if burst really works?

 

Not quite sure what you mean, but for ParallelForJobs, you can tag nativecollections with

[NativeDisableParallelForRestriction]

within the job to turn off thread saftey checks. Be careful doing that, you have to guarantee thread saftey yourself and that only one thread / for loop bit writes to an item at the same time.

 

Oh, now I gottcha. well I think you can use a nativelist for that too, or a nativehashmap ? (keyvalue based). You can't create persistent (only tmp) nativecollections wihtin the job itself, afaik. So if you need output from a job, always pass pre allocated native collections to it.

 

If i am in case, when i need to collect data from a IJobParallelFor, then the only way is to use concurrent containers, which is only NativeQueue and NativeMultiHashMap. And both of this can't be dispose without forcing Complete() on job that access to it. Am i right?

 

Some containers have the ability to be deallocated when the job is finished via an attribute where as others do need to be done on the main thread. I know NativeLists got a new dispose method that accepts a job handle for determining when a list should be disposed so you may want to check the queue to see if it got such a benefit.

If that is not the case my recommendation is to store a job handle and a queue and on the start of the update complete said handle then clear the queue.


Alternatively there is also a ParallelForFilter which may provide you some solutions.

 

Unfortunately NativeQueue has no Dispose method with JobHandle parameter like NativeList<T>.Dispose(JobHandle). What i want is to collect entities to container inside IJobForEach and then pass container to another job. I see no way except using NativeQueue, but it goes to need to force Complete() on my jobs.

 

You just create a persistent queue in your system and pass it between jobs. It's a very common pattern. You don't need to create and dispose it every frame and you should not need to call a complete.

 

I just realized that i have no need to call clear() on my queue, cause it's clean after dequeuing all elements. Feel so dumb %).