Using EntityCommandBuffer.Concurrent but not completing the jobs it depends on in one frame

I have a JobComponentSystem running a very resource intensive IJobProcessComponentDataWithEntity job which takes much more than one frame to complete and requires adding/removing components. Here's an example of my system:

Code (CSharp):

1. // using
2.  
3. [UpdateAfter(typeof(DoStuffSystem))]
4. public class BufferSystem : EntityCommandBufferSystem { }
5.  
6. [UpdateAfter(typeof(AnotherSystem))]
7. public class DoStuffSystem : JobComponentSystem
8. {
9.     BufferSystem commandBufferSystem;
10.    
11.     protected override void OnCreateManager()
12.     {
13.         commandBufferSystem = World.Active.GetOrCreateManager<BufferSystem>();
14.     }
15.  
16.     protected override JobHandle OnUpdate(JobHandle inputDependencies)
17.     {
18.         DoStuffJob doStuffJob = new DoStuffJob{
19.             // stuff
20.         };
21.  
22.         JobHandle handle = doStuffJob.Schedule(this, inputDependencies);
23.         commandBufferSystem.AddJobHandleForProducer(handle);
24.  
25.         return handle;
26.     }
27.  
28.     public struct DoStuffJob : IJobProcessComponentDataWithEntity<AComponent, Counts>
29.     {
30.         public EntityCommandBuffer.Concurrent commandBuffer;
31.  
32.         public void Execute(Entity entity, int jobIndex, ref AComponent aComponent, ref Counts counts)
33.         {
34.              //  I need this to be added here to indicate that the next system in the chain can process the data in this command buffer
35.             DynamicBuffer<BufferComponent> aBuffer = commandBuffer.AddBuffer<BufferComponent>(jobIndex, entity);
36.             aBuffer.ResizeUninitialized(aComponent.lengthOfData);
37.  
38.             DataGeneratorStruct meshGenerator = new DataGeneratorStruct{
39.                 dynamicBuffer    = aBuffer,
40.             };
41.  
42.             for(int i = 0; i < aComponent.lengthOfData; i++)
43.             {
44.                 meshGenerator.GenerateData(i);
45.             }
46.  
47.             commandBuffer.RemoveComponent<Counts>(jobIndex, entity);
48.         }
49.     }
50. }

Currently, my EntityCommandBufferSystem halts everything while it completes the scheduled IJobProcessComponentDataWithEntity so it can play back the EntityCommandBuffer.Concurrent (I think). This causes one very long frame. I want to free up my JobComponentSystem from this behaviour, allowing it to work independently from the main thread in some way.

I assume that I can't make IJobProcessComponentDataWithEntity only process a few entities per frame as opposed to processing all matching entities. That would certainly solve my problem. I'm guessing I can use [DisableAutoCreation] with the EntityCommandBuffer system then update it manually every few frames.

Would that be the best way to free-up a JobComponentySystem using an EntityCommandBufferSystem from needing to complete every frame? Is there a better method I'm overlooking?

 

Probably you can do this with ComponentSystem:

Code (CSharp):

1.     public class DoStuffSystem : ComponentSystem
2.     {
3.         EntityCommandBuffer CommandBuffer;
4.         JobHandle jobHandle;
5.  
6.         protected override void OnUpdate()
7.         {
8.             if (jobHandle.IsCompleted)
9.             {
10.                 CommandBuffer.Playback(EntityManager);
11.                 var job = new DoStuffJob() { commandBuffer = CommandBuffer.ToConcurrent() };
12.                 jobHandle = job.Schedule(this, default);
13.             }
14.         }
15.     }

 

Thanks, that makes sense. So perhaps this is just not a fitting use case for JobComponentSystem. I wonder if I could achieve a similar behaviour but using JobComponentSystem and whether I would benefit from it:

Code (CSharp):

1. // using
2.  
3. [DisableAutoCreation]
4. [UpdateAfter(typeof(DoStuffSystem))]
5. public class BufferSystem : EntityCommandBufferSystem { }
6. [UpdateAfter(typeof(AnotherSystem))]
7. public class DoStuffSystem : JobComponentSystem
8. {
9.     BufferSystem commandBufferSystem;
10.     JobHandle currentHandle;
11.  
12.     protected override void OnCreateManager()
13.     {
14.         commandBufferSystem = World.Active.GetOrCreateManager<BufferSystem>();
15.     }
16.     protected override JobHandle OnUpdate(JobHandle inputDependencies)
17.     {
18.         if(currentHandle.isComplete)
19.         {
20.             commandBufferSystem.Update();
21.  
22.             DoStuffJob doStuffJob = new DoStuffJob{
23.                 commandBuffer = commandBufferSystem.CreateCommandBuffer.ToConcurrent();
24.             };
25.    
26.             currentHandle = doStuffJob.Schedule(this, inputDependencies);
27.             commandBufferSystem.AddJobHandleForProducer(handle);
28.         }
29.  
30.         return inputDependencies;
31.     }
32.  
33.     ...

The fact that I'm just returning the original inputDependencies implies that, again, JobComponentSystem just isn't fitting for this task. Also it seems like an unusual use of EntityCommandBufferSystem to update it manually. Interested to know if anyone has thoughts on the above.

 

JobComponentSystem did now work. I think what I learned today is that using ECS with the Job System effectively doesn't necessarily mean using JobComponentSystem. Although I feel I might still be missing something. Here's the system I made to process entities independently of the main thread with EntityCommandBuffer used in the job.

https://gist.github.com/sSaltyDog/3ed9cbf9416f02755b4e7000c62fa5a4

Code (CSharp):

1. public class ExampleSystem : ComponentSystem
2. {
3.     int batchSize = 32;
4.  
5.     EntityManager entityManager;
6.  
7.     ComponentGroup exampleGroup;
8.  
9.     //  Currently runnning Job member variables
10.     public JobHandle runningJobHandle;
11.     EntityCommandBuffer runningCommandBuffer;
12.  
13.     protected override void OnCreateManager()
14.     {
15.         entityManager = World.Active.GetOrCreateManager<EntityManager>();
16.  
17.         squareWidth = TerrainSettings.mapSquareWidth;
18.  
19.         EntityArchetypeQuery exampleQuery = new EntityArchetypeQuery{
20.             All     = new ComponentType[] { typeof(AComponent) }
21.         };
22.         exampleGroup = GetComponentGroup(exampleQuery);
23.  
24.         //  Initialise handle and command buffer for first iteration
25.         runningJobHandle = new JobHandle();
26.         runningCommandBuffer = new EntityCommandBuffer(Allocator.TempJob);
27.     }
28.  
29.     protected override void OnDestroyManager()
30.     {
31.         runningCommandBuffer.Dispose();
32.     }
33.  
34.     protected override void OnUpdate()
35.     {
36.         //  If job is done, play it back then start more
37.         if(runningJobHandle.IsCompleted)
38.         {
39.             runningJobHandle.Complete();
40.  
41.             runningCommandBuffer.Playback(entityManager);
42.             runningCommandBuffer.Dispose();
43.         }
44.         //  Job is still running, do nothing this frame
45.         else
46.         {
47.             return;
48.         }
49.  
50.         EntityCommandBuffer         commandBuffer   = new EntityCommandBuffer(Allocator.TempJob);
51.         NativeArray<ArchetypeChunk> chunks          = exampleGroup.CreateArchetypeChunkArray(Allocator.TempJob);
52.  
53.         ArchetypeChunkEntityType                        entityType      = GetArchetypeChunkEntityType();
54.         ArchetypeChunkBufferType<Thing>                 thingType      = GetArchetypeChunkBufferType<Thing>(true);
55.  
56.         //  Store previous handle and all handles combines here
57.         JobHandle                   allHandles      = new JobHandle();
58.         JobHandle                   previousHandle  = new JobHandle();
59.  
60.         for(int c = 0; c < chunks.Length; c++)
61.         {
62.             ArchetypeChunk chunk = chunks[c];
63.  
64.             //  Get chunk data
65.             NativeArray<Entity>             entities        = chunk.GetNativeArray(entityType);
66.             BufferAccessor<Thing>           thingAccessor   = chunk.GetBufferAccessor(thingType);
67.  
68.             for(int e = 0; e < entities.Length; e++)
69.             {
70.                 Entity entity = entities[e];
71.  
72.                 //  If you are handing the job a reference type from a chunk array,
73.                 //  use a copy! Otherwise other systems cause errors while the job runs
74.                 DynamicBuffer<Thing> thingBuffer   = thingAccessor[e];
75.                 NativeArray<Thing> thingArray = new NativeArray<Thing>(thingsBuffer.Length, Allocator.TempJob);
76.                 blocksArray.CopyFrom(thingsBuffer.AsNativeArray());
77.                
78.                 FacesJob job = new FacesJob(){
79.                     commandBuffer = commandBuffer,
80.  
81.                     entity = entity,
82.                    
83.                     //  Use [DeallocateOnJobCompletion] in the IJob struct
84.                     things  = thingsArray,
85.                 };
86.                
87.                 //  Schedule the job without completing, and add it's dependency to the master dependency
88.                 //  Use the previous job's handle to prevent command buffer errors
89.                 JobHandle thisHandle = job.Schedule(previousHandle);
90.                 allHandles = JobHandle.CombineDependencies(thisHandle, allHandles);
91.                 previousHandle = thisHandle;
92.             }
93.         }
94.         //  Store the new dependencies and command buffer to check next frame
95.         runningCommandBuffer = commandBuffer;
96.         runningJobHandle = allHandles;
97.  
98.         chunks.Dispose();
99.     }
100. }