Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Scheduling long-running jobs from Systems

Discussion in 'Entity Component System' started by Srokaaa, Feb 14, 2019.

  1. Srokaaa

    Srokaaa

    Joined:
    Sep 18, 2018
    Posts:
    169
    How would I go about scheduling long-running (1-10 seconds) job from a system and how can I obtain a result from it? The job I want to run is procedurally generating new levels for my game and based on its result I'll spawn about 10k entities. Is this a good candidate for running in a separate World? And if so, are there any examples on how to do this? Currently my game just stalls until a Job finishes and I guess this is not acceptable.
     
  2. GilCat

    GilCat

    Joined:
    Sep 21, 2013
    Posts:
    676
    For long running jobs i think you must use old school threading. In Unity jobs must start and end in the same thread.
    That is a fairly low value, it should do just fine in 1 frame. I'm doing 80k without any hassle. Maybe there is something else to it.
     
    Srokaaa likes this.
  3. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    19.1 supports long running threads. (18.3 gives warnings in this case, which can be safely ignored)
     
    FROS7 likes this.
  4. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    You can just EntityManager.Instantiate and fill the data but that is not the most efficient approach. (Which is what we do for loading scenes in the megacity demo as well)

    1. Create a dedicated world for streaming / loading / proc-genning entities into
    2. Fill it out on a C# job that takes however long you like
    3. MoveAll entities from the streaming world into the main world when the job completes

    This way all data is fully prepared, the only cost that can affect framerate is the moving of entities into the live world, which is heavily optimized. This approach is proven to be scalable to what we do in megacity which is loading tiles / sections of streamed data in blocks of 80k - 200k entities while maintaining 60 FPS. (Framerate spike in megacity was ~1ms when moving the 200k entities)

    Documentation on how to do this exactly is somewhat sparse at the moment.

    https://github.com/Unity-Technologi...ocumentation~/exclusive_entity_transaction.md
     
    JesOb, recursive and GilCat like this.
  5. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,683
    They don’t. You can complete job in any other frame.
     
  6. GilCat

    GilCat

    Joined:
    Sep 21, 2013
    Posts:
    676
    How would I do that?
    Store the job handle and call the complete in any other frame?
     
  7. sngdan

    sngdan

    Joined:
    Feb 7, 2014
    Posts:
    1,154
    I have not done it, but the api has a check if the job is completed (I think jobhandle.isComplete) I guess you would just check this and once it’s done, you call complete and schedule the next job
     
  8. Attatekjir

    Attatekjir

    Joined:
    Sep 17, 2018
    Posts:
    23
    You can schedule a job for more than one frame if there are no other systems dependent on the data that you pass to the job. If you know this to be true then you can simply not pass the jobhandle of the long running job back to other systems.

    An example system:

    Code (CSharp):
    1. using System.Collections.Generic;
    2. using Unity.Collections;
    3. using Unity.Entities;
    4. using Unity.Mathematics;
    5. using Unity.Rendering;
    6. using Unity.Transforms;
    7. using UnityEngine;
    8. using Unity.Jobs;
    9. using Unity.Burst;
    10. using UnityEngine.Experimental.PlayerLoop;
    11.  
    12. class TestSystem : JobComponentSystem
    13. {
    14.  
    15.     private JobHandle jobhandle;
    16.  
    17.     private enum SystemStatus
    18.     {
    19.         Idle,
    20.         Busy,
    21.     }
    22.     private SystemStatus systemstatus = SystemStatus.Idle;
    23.  
    24.     private NativeArray<float> storeresult;
    25.  
    26.     protected override JobHandle OnUpdate(JobHandle inputDeps)
    27.     {
    28.  
    29.         //Debug.Log("A game update in which this system:");
    30.  
    31.         if (systemstatus == SystemStatus.Busy && jobhandle.IsCompleted)
    32.         {
    33.  
    34.             Debug.Log("Completes the Job");
    35.  
    36.             jobhandle.Complete();
    37.  
    38.             //Now we are free to do anything we want with the results from the job
    39.            
    40.             systemstatus = SystemStatus.Idle;
    41.         }
    42.         else if (systemstatus == SystemStatus.Idle)
    43.         {
    44.  
    45.             Debug.Log("Schedules the Job");
    46.  
    47.             if (storeresult.IsCreated)
    48.             {
    49.                 storeresult.Dispose();
    50.             }
    51.             storeresult = new NativeArray<float>(1000, Allocator.Persistent);
    52.  
    53.             var AJob = new AJob
    54.             {
    55.  
    56.                 storeresult = storeresult,
    57.  
    58.             };
    59.             jobhandle = AJob.Schedule(inputDeps); // note: not returning this
    60.  
    61.             systemstatus = SystemStatus.Busy;
    62.         }
    63.         else
    64.         {
    65.             Debug.Log("Still works on the job");
    66.         }
    67.  
    68.         return inputDeps;
    69.     }
    70.  
    71.     struct AJob : IJob
    72.     {
    73.  
    74.         public NativeArray<float> storeresult;
    75.  
    76.         public void Execute()
    77.         {
    78.  
    79.             for (int i = 0; i < storeresult.Length; i++)
    80.             {
    81.                 storeresult[i] = 50f % 23 * 2;
    82.             }
    83.  
    84.             //Prolong this job...
    85.             for (int i = 0; i < 99999999; i++)
    86.             {
    87.                 float test = 59f * 93f / 444f % 2;
    88.             }
    89.  
    90.         }
    91.     }
    92.  
    93.     protected override void OnDestroyManager()
    94.     {
    95.         base.OnDestroyManager();
    96.  
    97.         jobhandle.Complete();
    98.         if (storeresult.IsCreated)
    99.         {
    100.             storeresult.Dispose();
    101.         }
    102.     }
    103.  
    104. }
     
    GilCat likes this.
  9. Srokaaa

    Srokaaa

    Joined:
    Sep 18, 2018
    Posts:
    169
    @Joachim_Ante I think i nearly got it working. I am able to spawn create a new World, spawn Jobs generating the new level and spawn entities in this new World. I am failing at MoveEntitiesFrom though. It throws a following error at me and I wasn't able to figure out whats the cause event though the error seems descriptive enough:


    Code (CSharp):
    1. ArgumentException: EntityManager.MoveEntitiesFrom failed - All ISharedComponentData references must be from EntityManager. (For example ComponentGroup.SetFilter with a shared component type is not allowed during EntityManager.MoveEntitiesFrom)
    2. Unity.Entities.EntityManager.MoveEntitiesFrom (Unity.Entities.EntityManager srcEntities, Unity.Collections.NativeArray`1[T] entityRemapping) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/EntityManager.cs:1044)
    3. Unity.Entities.EntityManager.MoveEntitiesFrom (Unity.Entities.EntityManager srcEntities) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/EntityManager.cs:1030)
    4. Arc.LevelGeneration.CopyEntitiesToGameWorldSystem.OnUpdate () (at Assets/Scripts/Arc/LevelGeneration/GenerateLevelSystem.cs:176)
    5. Unity.Entities.ComponentSystem.InternalUpdate () (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ComponentSystem.cs:464)
    6. Unity.Entities.ScriptBehaviourManager.Update () (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ScriptBehaviourManager.cs:83)
    7. Unity.Entities.ScriptBehaviourUpdateOrder+DummyDelagateWrapper.TriggerUpdate () (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ScriptBehaviourUpdateOrder.cs:706)
    8.  
    EDIT: Uh, I got it working by properly filtering only the entities I wanted to move. Now I am fighting with Unity.Rendering:


    Code (CSharp):
    1. ArgumentException: All entities passed to EntityManager must exist. One of the entities has already been destroyed or was never created.
    2. EntityCommandBuffer was recorded in Unity.Rendering.CreateMissingRenderBoundsFromMeshRenderer using PostUpdateCommands.
    3.   at Unity.Entities.EntityDataManager.AssertEntitiesExist (Unity.Entities.Entity* entities, System.Int32 count) [0x0005a] in /Users/maciejsrokowski/Unity/arc/Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/EntityDataManager.cs:347
    4.   at Unity.Entities.EntityManager.AddComponent (Unity.Entities.Entity entity, Unity.Entities.ComponentType type) [0x00008] in /Users/maciejsrokowski/Unity/arc/Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/EntityManager.cs:474
    5.   at Unity.Entities.EntityCommandBuffer.PlaybackChain (Unity.Entities.EntityManager mgr, Unity.Entities.ECBSharedPlaybackState& playbackState, Unity.Collections.NativeArray`1[T] chainStates, System.Int32 currentChain, System.Int32 nextChain) [0x00246] in /Users/maciejsrokowski/Unity/arc/Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/EntityCommandBuffer.cs:1059
    6.   at Unity.Entities.EntityCommandBuffer.Playback (Unity.Entities.EntityManager mgr) [0x0025d] in /Users/maciejsrokowski/Unity/arc/Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/EntityCommandBuffer.cs:907
    7.   at Unity.Entities.ComponentSystem.AfterOnUpdate () [0x0000f] in /Users/maciejsrokowski/Unity/arc/Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ComponentSystem.cs:431
    8. Unity.Entities.ComponentSystem.AfterOnUpdate () (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ComponentSystem.cs:437)
    9. Unity.Entities.ComponentSystem.InternalUpdate () (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ComponentSystem.cs:471)
    10. Unity.Entities.ScriptBehaviourManager.Update () (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ScriptBehaviourManager.cs:83)
    11. Unity.Entities.ScriptBehaviourUpdateOrder+DummyDelagateWrapper.TriggerUpdate () (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ScriptBehaviourUpdateOrder.cs:706)
    12.  
    I guess this have something to do with update order.
     
    Last edited: Feb 15, 2019
  10. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Not sure. Could be an internal corruption. We did a bunch of fixes for next build in that area.

    MoveEntitiesFrom with filter is not as heavily used internally as moving all entities...

    Can you make a repro project folder?