Job system can't tell slices don't overlap

Consider two jobs, and one native array. The first job writes to the first half of the native array, and the second half writes to the second half. Naturally, the job system is going to complain that the two jobs interfere, and that a dependency should be created. The job system has no idea that one job will only write to certain indexes, and so it naturally complains because safety is not provable.

This makes sense, but a natural workaround does not. Instead of having the jobs write directly to the native array, how about they each get a slice of the native array to write to? In that way, one declares specifically which indexes the jobs can write to. They can only write into the slice they are given, and even if that slice is backed by the same native array as another slice another job is writing to, there should not be any issue if the slices do not contain any of the same elements.

Unfortunately, the job system still complains in this situation. The job system seems to *always* consider it unsafe if the slices are backed by the same array, even if none of the slices overlap with any of the other slices. The information is there inside of the slice declarations, I think it would be extremely useful for the job safety system to take advantage of that.

 

That is unfortunate, since that leaves me with basically 2 solutions. Either I turn of the safety system altogether, or I am forced to effectively single thread my code. I suppose for now I will have to just live without the safety system so that I can get the needed parallelism.

 

just don't be stupid and don't brake rules or be stupid and cry that something don't work

 

Your mapping logic needs to be done outside of the jobs that work on the data. I would say prefer mapping data to separate collections per job. If for some reason that has a huge performance hit, then at least put an abstraction over your single collection and hand that abstraction to jobs with some type of id for accessing the underlying data partition that job should be working on.

If you start creating leaky abstractions in concurrent code, god help you.

 

To me a NativeSlice<T> fits that definition pretty well, which is why I was hoping it could be the method I used for this. It perfectly describes the whole 'can only write to this part of the array' constraint that is required. I don't want to create a separate array for each job, that is unnecessary overhead when each job could easily be writing data directly into the required destination instead of into a temp array that is copied over later.

Creating a new abstraction I do not think would be possible anyway, without again disabling safety features. I could use unsafe pointers instead of slices of course, but then I'm basically replicating the Slice api without the safety. I can't embed a NativeArray inside another struct, since the job system still sees it and still complains about possible dependency issues.

And like I said, this is something that already works perfectly as long as I turn the safety system off. I can have N jobs, each working on their own little problem, writing their results directly into their unique slice of the destination without any additional copying needed. It's incredibly useful, it's just a shame that I have to go without all of the other safety systems in order to actually get the system to not complain.

 

Why would you create two separate jobs to fill the same data in one NativeArray? That's what the IJobParallelFor is for, no?

 

IJobParallelFor is wonderful when you want to use the same logic to write to every element in the destination array, but that is not exactly what I am running into right now, and doesn't cover every use case.

A simple example would be building a vertex list for a procedural mesh. This hypothetical mesh is composed of two independent pieces that do not share vertices. What one might want to do is have one job that computes the vertices for the first piece, and a second job computes the vertices for the second piece. IJobParallelFor wouldn't really help in this situation, since you have two completely independent pieces of logic.

 

Just define two NativeArrays for the two jobs then and when you load into the mesh, you just copy it one after the other. You have to copy them currently, regardless.
Or if your meshes are deterministic, just generate one after the another, both with IJobParallelFor (if you just apply a deterministic mathematical function)

Although I understand the idea what you're asking for, but I think it's not impossible (most of the time) to solve with safe functions and within the restrictions.

 

Just because the mesh api currently requires me to make a copy doesn't mean that it always will, or that this problem will never manifest in another way. As the job system develops, we are going to be seeing more and more methods pop up across the engine that take in NativeArrays as arguments. Furthermore, I only used the mesh example because it was easy to describe, it would be very easy to come up with a situation specific to a project that had the same requirements. Maybe another job is going to consume the results, and so no copy is required or should be needed. Maybe the results are being piped out to a native plugin.

I want to reiterate, I understand this problem is solvable if I want to make concessions. It is very easy to create multiple arrays, and then combine them after the jobs are complete. It is also very easy to create dependencies between the jobs so they have no chance of interfering. Both of these options have negative performance implications. All I want to do is propose that it might be useful to allow two separate jobs to write into the same array. The job safety system currently complains about something that is provably safe, and I think improving that would be really useful for a number of different situations.

 

Sorry for necro, but has this yet been corrected in more recent versions?

 

Yeah, just [NativeDisableParallelForRestriction] or something