Search Unity

Can't share containers between systems

Discussion in 'Data Oriented Technology Stack' started by GilCat, May 12, 2019.

  1. GilCat

    GilCat

    Joined:
    Sep 21, 2013
    Posts:
    355
    I have 2 systems (let's call them A and B) where System A processes some data and stores it on a native array that will be accessible from other systems (System B in this example). System B will update after System A and reads from System A native array.
    I'm getting an exception complaining about the need of calling complete on the jobs that read from the native array.
    Code (CSharp):
    1. InvalidOperationException: The previously scheduled job SystemB:ReadJob reads from the NativeArray ReadJob.Data. You must call JobHandle.Complete() on the job SystemB:ReadJob, before you can deallocate the NativeArray safely.
    If i'm correct about the error is that System A is deallocating the native array but that only happens on the next frame and feels like there shouldn't be a problem.
    Is it possibile to achieve this without calling complete() on those jobs that read from the System A native array?

    Here are my systems:
    System A
    Code (CSharp):
    1. public class SystemA : JobComponentSystem {
    2.   public NativeArray<int> ProcessedValues {
    3.     get;
    4.     private set;
    5.   }
    6.   public JobHandle InputDeps {
    7.     get;
    8.     private set;
    9.   }
    10.  
    11.   Random m_random;
    12.  
    13.   protected override void OnCreate() {
    14.     base.OnCreate();
    15.     m_random = new Random((uint)System.DateTime.Now.Millisecond);
    16.   }
    17.  
    18.   protected override void OnStopRunning() {
    19.     base.OnStopRunning();
    20.     if (ProcessedValues.IsCreated)
    21.       ProcessedValues.Dispose();
    22.   }
    23.  
    24.   [BurstCompile]
    25.   struct ProcessJob : IJobParallelFor {
    26.     [WriteOnly]
    27.     public NativeArray<int> Data;
    28.     public int Seed;
    29.     public void Execute(int index) {
    30.       var rnd = new Random((uint)(Seed + index));
    31.       Data[index] = rnd.NextInt();
    32.     }
    33.   }
    34.  
    35.   protected override JobHandle OnUpdate(JobHandle inputDeps) {
    36.     if (ProcessedValues.IsCreated)
    37.       ProcessedValues.Dispose();
    38.     ProcessedValues = new NativeArray<int>(1000, Allocator.TempJob);
    39.     inputDeps = new ProcessJob {
    40.       Data = ProcessedValues,
    41.       Seed = m_random.NextInt()
    42.     }.Schedule(ProcessedValues.Length, 64, inputDeps);
    43.     InputDeps = inputDeps;
    44.     return inputDeps;
    45.   }
    46. }
    System B
    Code (CSharp):
    1. [UpdateAfter(typeof(SystemA))]
    2. public class SystemB : JobComponentSystem {
    3.  
    4.   SystemA m_systemA;
    5.  
    6.   protected override void OnCreate() {
    7.     base.OnCreate();
    8.     m_systemA = World.GetOrCreateSystem<SystemA>();
    9.   }
    10.  
    11.   [BurstCompile]
    12.   struct ReadJob : IJobParallelFor {
    13.     [ReadOnly]
    14.     public NativeArray<int> Data;
    15.     public void Execute(int index) {
    16.       // Do something
    17.     }
    18.   }
    19.  
    20.   protected override JobHandle OnUpdate(JobHandle inputDeps) {
    21.     inputDeps = JobHandle.CombineDependencies(inputDeps, m_systemA.InputDeps);
    22.     inputDeps = new ReadJob {
    23.       Data = m_systemA.ProcessedValues
    24.     }.Schedule(m_systemA.ProcessedValues.Length, 64, inputDeps);
    25.     // Calling complete here fixes the problem
    26.     //inputDeps.Complete();
    27.     return inputDeps;
    28.   }
    29. }
    Thanks
     
    Last edited: May 12, 2019
  2. siggigg

    siggigg

    Joined:
    Apr 11, 2018
    Posts:
    110
    It might not be relevant, but there seems to be some confusion between similar variable names and you keep overwriting the input param of the OnUpdate method

    Try this on SystemB:

    Code (CSharp):
    1.  protected override JobHandle OnUpdate(JobHandle inputDeps) {
    2.     //inputDeps.Complete(); // Why do you need this?
    3.     if (ProcessedValues.IsCreated)
    4.       ProcessedValues.Dispose();
    5.     ProcessedValues = new NativeArray<int>(1000, Allocator.TempJob);
    6.     InputDeps = new ProcessJob {
    7.       Data = ProcessedValues,
    8.       Seed = m_random.NextInt()
    9.     }.Schedule(ProcessedValues.Length, 64, inputDeps);
    10.     return InputDeps;
    11.   }
    Also in SystemA:
    Code (CSharp):
    1. protected override JobHandle OnUpdate(JobHandle inputDeps) {
    2.     var combinedInputDeps = JobHandle.CombineDependencies(inputDeps, m_systemA.InputDeps);
    3.     var jobHandle = new ReadJob {
    4.       Data = m_systemA.ProcessedValues
    5.     }.Schedule(m_systemA.ProcessedValues.Length, 64, combinedInputDeps);
    6.     // Calling complete here fixes the problem
    7.     //inputDeps.Complete();
    8.     return jobHandle;
    9.   }
     
  3. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    210
    System A needs the JobHandle of System B in OnStopRunning so that you don't destroy the array while System B is still reading it.

    It may be worth developing a custom manager to keep track of inter-system dependencies like this.
     
    siggigg likes this.
  4. GilCat

    GilCat

    Joined:
    Sep 21, 2013
    Posts:
    355
    Sorry.
    You don't, it got hanging around from testing. Edited the system above.
     
  5. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,279
  6. GilCat

    GilCat

    Joined:
    Sep 21, 2013
    Posts:
    355
    Thanks a lot. This is just what i was looking for.
    The solution is to mark the read only array with [NativeDisableContainerSafetyRestriction]
    Code (CSharp):
    1.   [BurstCompile]
    2.   struct ReadJob : IJobParallelFor {
    3.     [ReadOnly]
    4.     [NativeDisableContainerSafetyRestriction]
    5.     public NativeArray<int> Data;
    6.     public void Execute(int index) {
    7.       // Do something
    8.     }
    9.   }
     
    siggigg likes this.
  7. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,575
    That is not a real solution. The safety system is telling you that there is a race condition.
    Disabling the safety system so it stops telling you about it doesn't solve the problem.

    If you have a manager that has a native array that is used by other systems. Then you have to register the job handle of every job that is writing / reading from it. And when destroying / accessing it on the main thread depend on all of those jobs. And when scheduling a job that writes to it, depend on all of those jobs.
     
    GilCat likes this.
  8. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,170
    I made up a simple abstraction for managing shared dependencies. The native containers are declared and created/destroyed in the concrete implementation. With a dedicated ComponentSystem that creates/destroys those implementations and makes them available via public fields.

    So as per @Joachim_Ante 's suggestion in another post, you call Combine before you schedule the job that uses a shared dependency, and after it call Set. Complete is for main thread access.

    Code (csharp):
    1.  
    2. public abstract class SharedDependency
    3.     {
    4.         private JobHandle JobHandle;
    5.  
    6.         public void Complete()
    7.         {
    8.             if (!JobHandle.IsCompleted)
    9.             {
    10.                 JobHandle.Complete();
    11.             }
    12.         }
    13.  
    14.         public JobHandle Combine(JobHandle other)
    15.         {
    16.             return JobHandle.CombineDependencies(JobHandle, other);
    17.         }
    18.  
    19.         public void Set(JobHandle other)
    20.         {
    21.             JobHandle = other;
    22.         }
    23.  
    24.         public abstract void OnCreate();
    25.         public abstract void OnDestroy();
    26.     }
    27.  
     
    GilCat likes this.
  9. Sarkahn

    Sarkahn

    Joined:
    Jan 9, 2013
    Posts:
    86
    Another solution is just to schedule all your dependent jobs in one place, that way you can easily pass in whatever they need as you schedule them without passing your data all over the place. For my game I have a single "board" (NativeArray<Entity>) and I was originally trying to pass it between a bunch of systems along with the job handle. It never really felt right.

    Eventually I realized (in my case at least) it was pointless to have all these separate systems operating on the same data. I broke out all the jobs from each system into their own file to keep things clean then I just schedule them all in order in a single BoardSystem:

    Code (CSharp):
    1.  
    2.    protected override JobHandle OnUpdate(JobHandle inputDependencies)
    3.     {
    4.         // Clear the board
    5.         for (int i = 0; i < board_.Length; ++i)
    6.             board_[i] = Entity.Null;
    7.         for (int i = 0; i < heightMap_.Length; ++i)
    8.             heightMap_[i] = 0;
    9.  
    10.         var boardJob = inputDependencies;
    11.  
    12.         boardJob = new InitializeBoardJob
    13.         {
    14.             board = board_,
    15.             childLookup = GetBufferFromEntity<Child>(true),
    16.             tilesLookup = GetBufferFromEntity<PieceTiles>(true),
    17.         }.Schedule(this, boardJob);
    18.  
    19.         boardJob = new BuildHeightmapJob
    20.         {
    21.             heightMap = heightMap_,
    22.             tilesLookup = GetBufferFromEntity<PieceTiles>(true),
    23.         }.Schedule(this, boardJob);
    24.  
    25.         And so on...
    26.  
     
    Last edited: May 13, 2019
    GilCat likes this.
  10. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    210
    EntityManager.LockChunkOrder is for this purpose. It lets you build your array right into chunk iteration.
     
    Sarkahn and GilCat like this.