Job Writing to NativeArray throwing Error

I'm still working on my spawning system, and I _think_ I've got it working (at least the entity debugger works correctly). But, I'm getting some errors in my log about thread safety.

2 each of these (I have two teams in the current setup):

Code (CSharp):

1. InvalidOperationException: The previously scheduled job SpawnAssignmentSystem:GatherEntitiesJob writes to the NativeArray GatherEntitiesJob.Iterator. You are trying to schedule a new job SpawnAssignmentSystem:GatherEntitiesJob, which writes to the same NativeArray (via GatherEntitiesJob.Iterator). To guarantee safety, you must include SpawnAssignmentSystem:GatherEntitiesJob as a dependency of the newly scheduled job.
2. Unity.Entities.JobForEachExtensions.Schedule (System.Void* fullData, Unity.Collections.NativeArray`1[T] prefilterData, System.Int32 unfilteredLength, System.Int32 innerloopBatchCount, System.Boolean isParallelFor, System.Boolean isFiltered, Unity.Entities.JobForEachExtensions+JobForEachCache& cache, System.Void* deferredCountData, Unity.Jobs.JobHandle dependsOn, Unity.Jobs.LowLevel.Unsafe.ScheduleMode mode) (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/IJobForEach.cs:451)
3. Unity.Entities.JobForEachExtensions.ScheduleInternal_EC[T] (T& jobData, Unity.Entities.ComponentSystemBase system, Unity.Entities.EntityQuery query, System.Int32 innerloopBatchCount, Unity.Jobs.JobHandle dependsOn, Unity.Jobs.LowLevel.Unsafe.ScheduleMode mode) (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/IJobForEach.gen.cs:1607)
4. Unity.Entities.JobForEachExtensions.Schedule[T] (T jobData, Unity.Entities.EntityQuery query, Unity.Jobs.JobHandle dependsOn) (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/IJobForEach.gen.cs:1154)
5. LSS.DCHL.Systems.SpawnAssignmentSystem.OnUpdate (Unity.Jobs.JobHandle inputDependencies) (at Assets/LSS/DCHL/Systems/SpawnAssignmentSystem.cs:77)
6. Unity.Entities.JobComponentSystem.InternalUpdate () (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ComponentSystem.cs:951)
7. Unity.Entities.ComponentSystemBase.Update () (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ComponentSystem.cs:287)
8. Unity.Entities.ComponentSystemGroup.OnUpdate () (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ComponentSystemGroup.cs:451)
9. UnityEngine.Debug:LogException(Exception)
10. Unity.Debug:LogException(Exception) (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/Stubs/Unity/Debug.cs:25)
11. Unity.Entities.ComponentSystemGroup:OnUpdate() (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ComponentSystemGroup.cs:455)
12. Unity.Entities.ComponentSystem:InternalUpdate() (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ComponentSystem.cs:818)
13. Unity.Entities.ComponentSystemBase:Update() (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ComponentSystem.cs:287)
14. Unity.Entities.DummyDelegateWrapper:TriggerUpdate() (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ScriptBehaviourUpdateOrder.cs:135)

Code (CSharp):

1. InvalidOperationException: The previously scheduled job SpawnAssignmentSystem:RemoveUnusedSpawnsJob writes to the NativeArray RemoveUnusedSpawnsJob.Iterator. You are trying to schedule a new job SpawnAssignmentSystem:GatherEntitiesJob, which writes to the same NativeArray (via GatherEntitiesJob.Iterator). To guarantee safety, you must include SpawnAssignmentSystem:RemoveUnusedSpawnsJob as a dependency of the newly scheduled job.
2. Unity.Entities.JobForEachExtensions.Schedule (System.Void* fullData, Unity.Collections.NativeArray`1[T] prefilterData, System.Int32 unfilteredLength, System.Int32 innerloopBatchCount, System.Boolean isParallelFor, System.Boolean isFiltered, Unity.Entities.JobForEachExtensions+JobForEachCache& cache, System.Void* deferredCountData, Unity.Jobs.JobHandle dependsOn, Unity.Jobs.LowLevel.Unsafe.ScheduleMode mode) (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/IJobForEach.cs:451)
3. Unity.Entities.JobForEachExtensions.ScheduleInternal_EC[T] (T& jobData, Unity.Entities.ComponentSystemBase system, Unity.Entities.EntityQuery query, System.Int32 innerloopBatchCount, Unity.Jobs.JobHandle dependsOn, Unity.Jobs.LowLevel.Unsafe.ScheduleMode mode) (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/IJobForEach.gen.cs:1607)
4. Unity.Entities.JobForEachExtensions.Schedule[T] (T jobData, Unity.Entities.EntityQuery query, Unity.Jobs.JobHandle dependsOn) (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/IJobForEach.gen.cs:1154)
5. LSS.DCHL.Systems.SpawnAssignmentSystem.OnUpdate (Unity.Jobs.JobHandle inputDependencies) (at Assets/LSS/DCHL/Systems/SpawnAssignmentSystem.cs:75)
6. Unity.Entities.JobComponentSystem.InternalUpdate () (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ComponentSystem.cs:951)
7. Unity.Entities.ComponentSystemBase.Update () (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ComponentSystem.cs:287)
8. Unity.Entities.ComponentSystemGroup.OnUpdate () (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ComponentSystemGroup.cs:451)
9. UnityEngine.Debug:LogException(Exception)
10. Unity.Debug:LogException(Exception) (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/Stubs/Unity/Debug.cs:25)
11. Unity.Entities.ComponentSystemGroup:OnUpdate() (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ComponentSystemGroup.cs:455)
12. Unity.Entities.ComponentSystem:InternalUpdate() (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ComponentSystem.cs:818)
13. Unity.Entities.ComponentSystemBase:Update() (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ComponentSystem.cs:287)
14. Unity.Entities.DummyDelegateWrapper:TriggerUpdate() (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/ScriptBehaviourUpdateOrder.cs:135)

And then 2 of these:

Code (CSharp):

1. A Native Collection has not been disposed, resulting in a memory leak. Enable Full StackTraces to get more details.

Here's my code, I'm not sure where I'm not including the proper job dependencies (I believe I'm tracking the handles appropriately), or where I'm failing to clean up the NativeArrays (which are marked in the last job that uses them to be deallocated with DeallocateOnJobCompletion) -- my assumption is that I don't need to track them so they can be disposed once I've done that). I do see this issue being looked into internally (https://github.com/Unity-Technologies/EntityComponentSystemSamples/issues/116), so might or might not be an issue.

On another note, I was trying to use EntityQuery.ToEntityArray rather than rolling my own gathering job, and the JobHandle parameter doesn't seem to be working as a dependency.

Code (CSharp):

1. using System.Collections.Generic;
2. using LSS.DCHL.Components;
3. using Unity.Collections;
4. using Unity.Entities;
5. using Unity.Jobs;
6.  
7.  
8. namespace LSS.DCHL.Systems
9. {
10.  
11. [UpdateInGroup( typeof( InitializationSystemGroup ) )]
12. public sealed class SpawnAssignmentSystem : JobComponentSystem
13. {
14.     protected override void OnCreate()
15.     {
16.         m_UnassignedPlayers = GetEntityQuery( new EntityQueryDesc
17.         {
18.             All = new[]
19.             {
20.                 ComponentType.ReadOnly< PlayerDataComponent >(),
21.                 ComponentType.ReadOnly< TeamAssignment >(),
22.             },
23.             None = new[]
24.             {
25.                 ComponentType.ReadOnly< SpawnAssignment >(),
26.             },
27.         } );
28.  
29.         m_UnassignedSpawns = GetEntityQuery( new EntityQueryDesc
30.         {
31.             All = new[]
32.             {
33.                 ComponentType.ReadOnly< CharacterSpawn >(),
34.                 ComponentType.ReadOnly< TeamAssignment >(),
35.             },
36.             None = new[]
37.             {
38.                 ComponentType.ReadOnly< AssignedSpawn >(),
39.             },
40.         } );
41.  
42.         m_CommandBuffer = World.GetOrCreateSystem< BeginInitializationEntityCommandBufferSystem >();
43.     }
44.  
45.     protected override JobHandle OnUpdate( JobHandle inputDependencies )
46.     {
47.         EntityManager.GetAllUniqueSharedComponentData( m_AvailableTeams );
48.  
49.         JobHandle _merged = inputDependencies;
50.         for ( var _index = 1; _index < m_AvailableTeams.Count; _index++ )
51.         {
52.             TeamAssignment _assignment = m_AvailableTeams[_index];
53.  
54.             m_UnassignedSpawns.SetFilter( _assignment );
55.             m_UnassignedPlayers.SetFilter( _assignment );
56.  
57.             int _spawnCount = m_UnassignedSpawns.CalculateLength();
58.             int _playerCount = m_UnassignedPlayers.CalculateLength();
59.  
60.             if ( _spawnCount == 0 ) continue;
61.  
62.             if ( _playerCount == 0 )
63.             {
64.                 JobHandle _removeUnusedSpawnsJobHandle = new RemoveUnusedSpawnsJob
65.                 {
66.                     CommandBuffer = m_CommandBuffer.CreateCommandBuffer().ToConcurrent(),
67.                 }.Schedule( m_UnassignedSpawns, inputDependencies );
68.  
69.                 m_CommandBuffer.AddJobHandleForProducer( _removeUnusedSpawnsJobHandle );
70.  
71.                 _merged = JobHandle.CombineDependencies( _merged, _removeUnusedSpawnsJobHandle );
72.  
73.                 continue;
74.             }
75.  
76.             var _spawns = new NativeArray< Entity >( _spawnCount, Allocator.TempJob, NativeArrayOptions.UninitializedMemory );
77.  
78.             JobHandle _toArrayJob = new GatherEntitiesJob
79.             {
80.                 Entities = _spawns,
81.             }.Schedule( m_UnassignedSpawns, inputDependencies );
82.  
83.             JobHandle _handle = new AssignSpawnsToPlayersJob
84.             {
85.                 CommandBuffer = m_CommandBuffer.CreateCommandBuffer().ToConcurrent(),
86.                 Spawns = _spawns,
87.             }.Schedule( m_UnassignedPlayers, _toArrayJob );
88.  
89.             m_CommandBuffer.AddJobHandleForProducer( _handle );
90.  
91.             _merged = JobHandle.CombineDependencies( _merged, _handle );
92.         }
93.  
94.         m_AvailableTeams.Clear();
95.         m_UnassignedPlayers.ResetFilter();
96.         m_UnassignedSpawns.ResetFilter();
97.  
98.         return _merged;
99.     }
100.  
101.     private struct GatherEntitiesJob : IJobForEachWithEntity< CharacterSpawn >
102.     {
103.         [WriteOnly]
104.         public NativeArray< Entity > Entities;
105.  
106.         public void Execute( Entity entity, int index, ref CharacterSpawn c0 )
107.         {
108.             Entities[index] = entity;
109.         }
110.     }
111.  
112.     private struct AssignSpawnsToPlayersJob : IJobForEachWithEntity< PlayerDataComponent >
113.     {
114.         public EntityCommandBuffer.Concurrent CommandBuffer;
115.  
116.         [DeallocateOnJobCompletion]
117.         public NativeArray< Entity > Spawns;
118.  
119.         public void Execute( Entity entity, int index, ref PlayerDataComponent playerData )
120.         {
121.             CommandBuffer.AddComponent( index, entity, new SpawnAssignment {Spawn = Spawns[index]} );
122.             CommandBuffer.AddComponent( index, Spawns[index], new AssignedSpawn() );
123.         }
124.     }
125.  
126.     private struct RemoveUnusedSpawnsJob : IJobForEachWithEntity< CharacterSpawn >
127.     {
128.         public EntityCommandBuffer.Concurrent CommandBuffer;
129.  
130.         public void Execute( Entity entity, int index, ref CharacterSpawn spawn )
131.         {
132.             CommandBuffer.DestroyEntity( index, entity );
133.         }
134.     }
135.  
136.     private EntityQuery m_UnassignedPlayers;
137.     private EntityQuery m_UnassignedSpawns;
138.  
139.     private EntityCommandBufferSystem m_CommandBuffer;
140.  
141.     private readonly List< TeamAssignment > m_AvailableTeams = new List< TeamAssignment >();
142. }
143.  
144. }

Any ideas?

 

You are scheduling your jobs in parallel rather than having a dependency on the previous job.

you're dependency tree looks like this

inputDependencies => _toArrayJob (GatherEntitiesJob) => _handle (AssignSpawnsToPlayersJob) }
inputDependencies => _toArrayJob (GatherEntitiesJob) => _handle (AssignSpawnsToPlayersJob) } => _merged
inputDependencies => _toArrayJob (GatherEntitiesJob) => _handle (AssignSpawnsToPlayersJob) }

You can't have your GatherEntitiesJob jobs running at the same time

 

hmm. The goal of my setup was to have the exact tree that you show, such that the jobs are as wide as possible. Each GatherEntitiesJob was intended to have an independent NativeArray to work from. This is also modelled on the Boids system, which does something similar with the filters and types. The only piece I can see that would force the jobs into a fully serial run (and this only for the _toArrayJob, is if the SetFilter on the queries isn't evaluated until the job is actually scheduled (which I could see leading to some weird determinism issues).

That said, I am still brand new to Unity's ECS, so it's fully possible I've not worked out all of the nuances yet . The big one I can think of is if the jobs are being reused under the hood when I call for a new job, instead of placing another job struct in memory.

Placing the dependencies as you suggest seems (to me) like I'd effectively be converting this structure to serial rather than parallel, which (aside from pushing the calculation off the main thread) kind of defeats the purpose. Now, with that said, this system is a run-once kind of thing working with a very small number of objects (roughly 32 at the high end), so it partially was an exercise in identifying what conceptually was independent and splitting it out -- this process could likely be achieved quite handily on the main thread with minimal overhead, and it's not intended to be run per-frame. But, solving it in parallel seemed a good exercise to prep me for the rest of the game

Conceptually, there shouldn't be any dependencies on prior jobs executing for the loop. It's just a case of filtering the query at the top of the loop occurring prior to the job scheduling. As I said, I also tried using a ToEntityArray from the filtered EntityQuery, outputting a JobHandle to pass to the AssignPlayersToSpawn job as a dependency. In that case, the AssignPlayersToSpawnJob was receiving a null reference for the array unless I explicitly called Complete() on the ToEntityArray handle, so something strange seemed to be happening there.

With all that said, I'm assuming that there's some processing on the EntityQuery that is being missed, and I'm not seeing any way to set a dependency on that processing other than I have done.