Dependencies between systems

So, i need to keep information that i can access by index (a flattened 2d grid), so it's not really good enough to put it in IComponentData and have to iterate them all looking for the one with the right index every time i need to get a value. I need to store it in a NativeArray so i can take my worldposition, convert it to an index and just access the values directly.

What I've been doing is putting the nativeArray in the system that writes the information.
Then other systems that need to read from it inject that system and grab the array for their own jobs
The reading systems are all set to UpdateAfter the writing system.

Writing System -----> Reading system A
Writing System -----> Reading system B
Writing System -----> etc

but the writing system isn't included in their dependency anyway, so it just throws errors because they're scheduling jobs that read from the array when the writing systems job isn't guaranteed to be complete.

Here's a very minimal version of what i mean:
WritingSystem keeps an array, does some work on it in a job
ReadingSystem takes the array from SystemA, reads some values from it
Both systems are Attributed so WritingSystem updates before ReadingSystem
No dependency is being made between the systems, so it won't work.

Code (CSharp):

1. using Unity.Entities;
2. using Unity.Jobs;
3. using Unity.Collections;
4.  
5. [UpdateBefore(typeof(ReadingSystem))]
6. class WritingSystem : JobComponentSystem
7. {
8.     public NativeArray<int> testArray;
9.  
10.     protected override void OnCreateManager()
11.     {
12.         testArray = new NativeArray<int>(100, Allocator.Persistent);
13.     }
14.     protected override void OnDestroyManager()
15.     {
16.         testArray.Dispose();
17.     }
18.  
19.     protected override JobHandle OnUpdate(JobHandle dependsOn)
20.     {
21.         WritingJob writingJob = new WritingJob
22.         {
23.             testArray = this.testArray
24.         };
25.         return writingJob.Schedule(testArray.Length, 1, dependsOn);
26.     }
27. }
28.  
29. [UpdateAfter(typeof(WritingSystem))]
30. class ReadingSystem : JobComponentSystem
31. {
32.     [Inject] WritingSystem writingSystem;
33.  
34.     protected override JobHandle OnUpdate(JobHandle dependsOn)
35.     {
36.         ReadingJob readingJob = new ReadingJob
37.         {
38.             testArray = writingSystem.testArray
39.         };
40.         return readingJob.Schedule(writingSystem.testArray.Length, 1, dependsOn);
41.     }
42. }
43.  
44. struct WritingJob : IJobParallelFor
45. {
46.     [WriteOnly] public NativeArray<int> testArray;
47.  
48.     public void Execute(int index)
49.     {
50.         testArray[index] = index * index;
51.     }
52. }
53.  
54. struct ReadingJob : IJobParallelFor
55. {
56.     [ReadOnly] public NativeArray<int> testArray;
57.  
58.     public void Execute(int index)
59.     {
60.         int a = testArray[index];
61.     }
62. }
63.  

 

Instead of using IComponentData, you could use a Buffer. That way, the data and its dependencies are nicely handled by ECS.

 

For some things i could, but i actually use NativeHashMap and NativeMultiHashMap like this more so than arrays.

I just feel like when i explictly mark a system as needing to UpdateAfter another system, it should put that system in as a dependency. Since if it's gaurenteed to not update until after the previous system has updated, conflicts of write/reading the same arrays shouldn't be possible.

 

So im refactoring a 2D project for ECS and ive come across this issue as well. One of the issues you may have might actually have move to do with the jobs than the order of the systems. Even if Writing System comes before Reading system A the actual jobs might not run in that order. Its possible that a job from one system trys to access the data from that array while another is as well.

one of my work arounds is to find a better solution for grouping with component data then in the reading system only read data that is needed in the proper group. I think this would probably be more performant and not need to have weird system dependencies.

One thing that i have not tried that might help (if you dont want to go the whole grouping route) is to try some of the things listed here.

Code (CSharp):

1.     [Test]
2.     public void ResizedListToDeferredJobArray([Values(0, 1, 2, 3, 4, 5, 6, 42, 97, 1023)]int length)
3.     {
4.         var list = new NativeList<int> (Allocator.TempJob);
5.  
6.         var setLengthJob = new SetListLengthJob { list = list, ResizeLength = length };
7.         var jobHandle = setLengthJob.Schedule();
8.  
9.         var setValuesJob = new SetArrayValuesJobParallel { array = list.ToDeferredJobArray() };
10.         setValuesJob.Schedule(list, 3, jobHandle).Complete();
11.        
12.         Assert.AreEqual(length, list.Length);
13.         for (int i = 0;i != list.Length;i++)
14.             Assert.AreEqual(length, list[i]);
15.  
16.         list.Dispose ();
17.     }

Not sure if this will work for what you need but something to look into maybe?

 

Isn't that the whole point of UpdateBefore and UpdateAfter? to say that 'this systems work needs to be done before that systems work'. Because their usefulness goes down dramatically if they don't affect the order of jobs scheduled by the systems. In all the experimenting i've done the order that the profiler shows jobs running in has always matched the UpdateBefore and UpdateAfter tags i set for the systems that scheduled them.

I'm not sure what you mean by this. list -> deferred array is for scheduling paralel jobs when you don't know how many items there will be at the time you're scheduling it. So you can have one job set up a nativelist with data, and then a 2nd parallel job that consumes that data as an array with the correct number of execute calls/index numbers. It doesn't really relate to sharing access to native collections between systems with the correct dependencies.

I don't think it's a wierd dependency thing to say "systemB uses data calculated by systemA, ensure it runs after it", Which is what the attributes do. But the jobhandle dependencies given to the 2 systems don't respect the attributes so you get access errors about potential conflicts even tho you've already explicitly said B shouldn't be done until after A is done.

 

from my understanding yes and no. I think inputDependancies you get from the update has to do with jobs related to systems that work on the same groups of components not all jobs scheduled prior to that system running. I could be wrong but thats just how i took it.
For example if SystemB works on CompB why should it have to wait for SysA to finish its work on CompA? It shouldnt and both jobs should run at the same time on separate threads. The only order that should be cared about is the order in which data is accessed via its components or the jobs that have explicitly been assigned to run in a specific order. Other wise you might as well run all of your code on the main thread with the exception of ParrallelFor Jobs no?

In your example there really is no order to your systems jobs since they dont have Component groups so it will try and run both jobs at the same time. ReadingSystem i dont think gets the jobs from the WritingSystem because they dont depend on the same data (ie Component Group).

This is why I suggested not passing around systems (as it was also suggested to me when i went the same route). Its not very ECS like. The better solution would be to find a way to configure your data for other systems to work on then allow they systems to self arrange jobDep order based on the groups of data they work with.

If you find something that suggests otherwise id be interested in listening.

 

I think you misunderstand the purpose of that attribute in relation to JobComponentSystems. In a ComponentSystems you would be correct but all a JobComponentSystems does is schedule jobs to be worked on at a later data. When SystemB is done there is no work done. Just a schedule of jobs to be done later. and because the jobs have unrelated data explained above they will attempt to run at the same time.

 

You're talking about a case where 2 systems are accessing 2 different sets of information. Of course you don't want a dependency there, why would you.
I'm talking about a case where 2 systems are accessing the same set of information and it matters which system accesses it first (since A is setting up data and the others are consuming it. You need a dependency here. the ECS should pick up on the fact that I've explicitly marked B as requiring A to have completed first, and provide that dependency.

In an framework where you're supposed to be putting as much as possible into jobs, what's the point in even having an [UpdateBefore] or [UpdateAfter] attribute if they don't 1) ensure that systemA jobs execute before SystemB jobs and 2) result in systemB being given a jobhandle that is aware that systemA jobs will have completed, allowing threadsafe access to shared resources.

 

Its not the system that is accessing the data it is the jobs. The jobs dependencies are defined by relevant data first then system order (assuming its using the same component data). Without relevant data(components) system order means nothing. Your system are working in the correct order and so are the jobs. Your writing system does not pass its jobs to the reading system because they dont access the same data(components). Yes the same native array but thats not what the component systems look at. It looks at defined component groups.

To keep it short your component systems see each other as unrelated. SystemA has no reason to pass its jobs to SystemB which is why you get the errors.

 

A quote from Joachim_Ante here thats relevant to your problem:
The JobComponentSystem.Update() method handles all job dependency setup for you. So that should be totally fine.
"
If you use JobComponentSystem correctly then it is expected that any dependency chains will be setup correctly independent of the order in which you call JobComponentSystem.Update();

Thats the fundamental design. Now if you have shared containers that are not IComponentData etc.
Eg. some NativeArray<> being passed around then you have to also pass around the job handles manually."

Further down he also says this which might help you: "Usually for modularity reasons you have some kind of class / struct that is referenced by all 3 systems that stores the active JobHandle so you essentially use it as dependency then change the dependency to the just scheduled job. Thus creating a chain with every step that you introduce."

 

That's pretty much what i was doing to work around it but it felt like a bit of a hack togethor solution. But if that's the workaround they use themselves then i guess i'll stick with it. thanks for your help