Instantiating Entities in a Job

tl:dr: I need help to solve a performance problem with instantiating new entities via a job and command buffer

Hello,

first some things to the scenario and the problem i am facing at the moment. I am writing a space strategy game where the stations and space ships have many independent turrets that are firing kind of rapidly. I am using a hybrid approach where i have a pretty clear separation of my "ecs world" and the "monobehavior world" and few sync points between those. but like i said i am now facing a problem, where i could need some help for a good solution.

so i have a JobComponentSystem where i spawn projectiles for each "spawner entity" i have in the world. This system is using the "spawner"-Entities to define a proxy/offset location with the Turret-Entity as a parent to have a typical transform hierarchy. These spawner entities also have a component that is holding the projectile prefab to instantiate. In my current test scene i have arround 150 turrets, each firing 2 bullets every 0.2 seconds. so in one burst all together producing (just) 300 projectile entities. but this gives a spike in the profiler of about 6ms. that is way to much time for what i have expected.

i can't batch instantiate a native array because i am in a job and have to use command buffers that seem to work serially and of course are only working in the main thread. the only things i can think of is that i use other worlds to create these entities in parallel and copy those into the main world somehow. at least this is what i have found so far. but i don't know how to actually do this jet.
another thing i could think of is to spread the instantiation process over more frames what means either don't use the command buffer, so i would have to write my own kind of command buffer, or to offset the timing of every spawner in a way that i have the fewest ammount of overlapping spawns possible.
i think an integrated pooling mechanism for entities would be nice so you could trade memory space for processing power.

so my question is if some of you already faced a similar problem and/or have a solution or hint how to reduce the time needed to instantiate many entities at once from inside a system.

screenshot of the profiler:



and here is my spawner system code, maybe this helps:

Code (CSharp):

1. private struct ECSTurretFireSystemJob : IJobForEachWithEntity<LocalToWorld, Translation, Rotation, ECSTurretMuzzleComponent>
2.     {
3.         [ReadOnly]
4.         public float DeltaTime;
5.  
6.         [ReadOnly]
7.         public ComponentDataFromEntity<ECSVelocityComponent> m_AllVelocities;
8.  
9.         public EntityCommandBuffer.Concurrent ECSCommandBuffer;
10.  
11.         public void Execute( [ReadOnly]Entity entity, [ReadOnly]int index, [ReadOnly] ref LocalToWorld p_LtoW, [ReadOnly] ref Translation p_Translation, [ReadOnly] ref Rotation p_Rotation, ref ECSTurretMuzzleComponent p_Muzzle )
12.         {
13.             p_Muzzle.TimeAccumulator += this.DeltaTime;
14.  
15.             if ( p_Muzzle.TimeAccumulator >= p_Muzzle.FireRate )
16.             {
17.                 p_Muzzle.TimeAccumulator = 0;
18.  
19.                 quaternion SpawnRotation = quaternion.LookRotation( p_LtoW.Forward, p_LtoW.Up );
20.                
21.                 Entity ProjectileInstance = this.ECSCommandBuffer.Instantiate( index, p_Muzzle.Projectile_Prefab );
22.                 this.ECSCommandBuffer.SetComponent<Translation>( index, ProjectileInstance, new Translation() { Value = math.mul( p_LtoW.Value, math.float4(0,0,0,1 ) ).xyz } );
23.                 this.ECSCommandBuffer.SetComponent<Rotation>( index, ProjectileInstance, new Rotation() { Value = SpawnRotation } );
24.                 this.ECSCommandBuffer.SetComponent<ECSVelocityComponent>( index, ProjectileInstance, m_AllVelocities[p_Muzzle.Velocity_EntityLink] );
25.                
26.             }
27.         }
28.     }
29.  
30.     protected override JobHandle OnUpdate( JobHandle inputDependencies )
31.     {
32.         ECSTurretFireSystemJob job = new ECSTurretFireSystemJob()
33.         {
34.             m_AllVelocities = this.GetComponentDataFromEntity<ECSVelocityComponent>( true ),
35.             ECSCommandBuffer = this.ECSCommandBuffer.CreateCommandBuffer().ToConcurrent(),
36.             DeltaTime = UnityEngine.Time.deltaTime
37.         };
38.  
39.         inputDependencies = job.Schedule( this, inputDependencies );      
40.         ECSCommandBuffer.AddJobHandleForProducer( inputDependencies );
41.  
42.         return inputDependencies;
43.     }

 

Which part is causing the spike? The job, or the EntityCommandBuffer playback.

If it is the job, the next Entities package is supposed to allow Burst to work with ECBs.

If it is the playback, then things get a little more complicated. Instead of using the ECB, you want to write out all the data (Translation, Rotation, ECSVelocityComponent, and Projectile_Prefab) into some data structure. If you can schedule the job single, that could be a NativeList of a custom struct type. Otherwise that can be a NativeStream or a NativeMultiHashMap. You will then want a second job to reorder those outputs so that the same prefab is grouped together and you have indexes and counts of each unique prefab. After that, you can use EntityManager to instantiate all the entities. And finally, you can put all those entities into an array and send them off to a job with ComponentDataFromEntity to populate their values. It seems like a lot of work, but all the jobs can be Bursted, so this scales well at high entity counts. However, at 300 entities, I wouldn't expect a 6 ms spike.

 

If you can estimate the count before running the jobs, I did get very good results batch instantiating entities directly in the jobcomponentSystem ' s update on main thread and then passing the array of entities to the job.

 

first of all, thanks for your replies!

it is the EntityCommandBuffer playback.

this is a very nice info!

i decided to implement something like you both have mentioned. i did it like this:
first, in the job, i fill a NativeQueue<>.ParallelWriter with all the data i need to initialize each single projectile entity. then in the system's OnUpdate() i use this filled queue to have a count to initialize a NativeArray<Entity> with the correct size in one step. i also create another NativeArray<> and copy all data from the queue to that array, to still work in parallel in a following job. unfortunatly i have no idea how to copy the data from the queue to the array in an elegant way so i had to dequeue each element one by one and fill the array step by step in the main thread. after that is done, i start a new IJobParallelFor to copy the data from the array to the correct ComponentDataFromEntity with [NativeDisableParallelForRestriction].

well. it was an improvement performance wise. the time needed to instantiate 300 projectiles is now reduced from 6ms to 0.75ms. but its a bit much to do and to code for such a simple task. also i am still a bit confused why 300 projectiles had such an impact in the first place.

i also tried to use the nativelist but it seems with that i am not able to write to it in parallel when i dont know how many items there will be in the first place. then i tried a nativestream, but i wasn´t able to write to it from inside a job.

the updated code:

Code (CSharp):

1. public class ECSTurretFireSystem : JobComponentSystem
2. {
3.  
4.     struct SpawnData
5.     {
6.         public Entity prefab;
7.         public Translation position;
8.         public Rotation rotation;
9.         public ECSVelocityComponent velocity;
10.     }
11.  
12.     [BurstCompile]
13.     private struct ECSTurretFireSystemJob : IJobForEachWithEntity<LocalToWorld, Translation, Rotation, ECSTurretMuzzleComponent>
14.     {
15.         [ReadOnly]
16.         public float DeltaTime;
17.  
18.         [ReadOnly]
19.         public ComponentDataFromEntity<ECSVelocityComponent> m_AllVelocities;
20.  
21.         [WriteOnly]
22.         public NativeQueue<SpawnData>.ParallelWriter m_AllSpawnData;
23.  
24.         public void Execute( [ReadOnly]Entity entity, [ReadOnly]int index, [ReadOnly] ref LocalToWorld p_LtoW, [ReadOnly] ref Translation p_Translation, [ReadOnly] ref Rotation p_Rotation, ref ECSTurretMuzzleComponent p_Muzzle )
25.         {
26.             p_Muzzle.TimeAccumulator += this.DeltaTime;
27.  
28.             if ( p_Muzzle.TimeAccumulator >= p_Muzzle.FireRate )
29.             {
30.                 p_Muzzle.TimeAccumulator = 0;
31.  
32.                 quaternion SpawnRotation = quaternion.LookRotation( p_LtoW.Forward, p_LtoW.Up );//math.mul( quaternion.LookRotation( p_LtoW.Forward, p_LtoW.Up ), p_Rotation.Value );
33.  
34.                 SpawnData spawndata = new SpawnData()
35.                 {
36.                     prefab = p_Muzzle.Projectile_Prefab,
37.                     position = new Translation() { Value = math.mul( p_LtoW.Value, math.float4(0,0,0,1 ) ).xyz },
38.                     rotation = new Rotation() { Value = SpawnRotation },
39.                     velocity = m_AllVelocities[p_Muzzle.Velocity_EntityLink]
40.                 };
41.  
42.                 m_AllSpawnData.Enqueue( spawndata );
43.             }
44.         }
45.     }
46.  
47.     [BurstCompile]
48.     private struct ECSTurretFireSystemSetValuesJob : IJobParallelFor
49.     {
50.         [ReadOnly]
51.         public NativeArray<Entity> m_SpawnedProjectiles;
52.  
53.         [ReadOnly]
54.         public NativeArray<SpawnData> m_AllSpawnData;
55.  
56.         [NativeDisableParallelForRestriction]
57.         [WriteOnly]
58.         public ComponentDataFromEntity<Translation> m_AllTranslations;
59.  
60.         [NativeDisableParallelForRestriction]
61.         [WriteOnly]
62.         public ComponentDataFromEntity<Rotation> m_AllRotations;
63.  
64.         [NativeDisableParallelForRestriction]
65.         [WriteOnly]
66.         public ComponentDataFromEntity<ECSVelocityComponent> m_AllVelocities;
67.  
68.         public void Execute( int index )
69.         {
70.             SpawnData data = m_AllSpawnData[index];
71.             m_AllTranslations[m_SpawnedProjectiles[index]] = data.position;
72.             m_AllRotations[m_SpawnedProjectiles[index]] = data.rotation;
73.             m_AllVelocities[m_SpawnedProjectiles[index]] = data.velocity;
74.         }
75.     }
76.  
77.     protected override JobHandle OnUpdate( JobHandle inputDependencies )
78.     {
79.         NativeQueue<SpawnData> AllSpawnData = new NativeQueue<SpawnData>(Allocator.TempJob);
80.        
81.         ECSTurretFireSystemJob job = new ECSTurretFireSystemJob()
82.         {
83.             m_AllVelocities = this.GetComponentDataFromEntity<ECSVelocityComponent>( true ),
84.             m_AllSpawnData = AllSpawnData.AsParallelWriter(),
85.             DeltaTime = UnityEngine.Time.deltaTime
86.         };
87.        
88.         job.Schedule( this, inputDependencies ).Complete();
89.  
90.         if ( AllSpawnData.Count > 0 )
91.         {
92.             NativeArray<Entity> AllSpawnedEntities = new NativeArray<Entity>(AllSpawnData.Count, Allocator.TempJob);
93.             NativeArray<SpawnData> AllSpawnDataArray = new NativeArray<SpawnData>(AllSpawnData.Count, Allocator.TempJob);
94.  
95.             // would be great if i could skip this loop somehow
96.             for ( int i = 0; i < AllSpawnDataArray.Length; i++ )
97.             {
98.                 AllSpawnDataArray[i] = AllSpawnData.Dequeue();
99.             }
100.  
101.             this.EntityManager.Instantiate( AllSpawnDataArray[0].prefab, AllSpawnedEntities );
102.            
103.             ECSTurretFireSystemSetValuesJob job2 = new ECSTurretFireSystemSetValuesJob()
104.             {
105.                 m_SpawnedProjectiles = AllSpawnedEntities,
106.                 m_AllSpawnData = AllSpawnDataArray,
107.                 m_AllRotations = GetComponentDataFromEntity<Rotation>(),
108.                 m_AllTranslations = GetComponentDataFromEntity<Translation>(),
109.                 m_AllVelocities = GetComponentDataFromEntity<ECSVelocityComponent>()
110.             };
111.  
112.             job2.Schedule( AllSpawnedEntities.Length, 64, inputDependencies ).Complete();
113.  
114.             AllSpawnDataArray.Dispose();
115.             AllSpawnedEntities.Dispose();
116.         }
117.  
118.         AllSpawnData.Dispose();
119.  
120.         return inputDependencies;
121.     }
122. }

 

Make sure that the 6 ms spike wasn't just completing job dependencies. If you are not profiling using the timeline view, I suggest you start. You can move the Dequeueing to an IJob and make use of Burst. This is one of those moments where that job would be better using Run() than Schedule().

And yes, it is a lot of work. But much of that work is adding knowledge about how things are spawned and adding parallelism support. This solution might be overkill for your problem, but if you want to add an order of magnitude more turrets or drastically increase the fire rate or making a "shotgun" effect, this solution will scale up to it quite efficiently.

 

I am curious where you put the script in the project, and if something calls it or activates it, or if it is supposed to be on during the lifetime of the scene or session.

I have a burst compiled job that (doesn't work in WebGL yet, but...) handles 9.1 million instances and I get at most 1.2 ms on the main thread primarily because of the job creation. It's getting called on demand upon camera change with an additional delay of 0.2 to 1 second. However, I did everything of the object creation at the start so the job only handles an existing array of objects in a pool.

 

I have tried that now and it absolutely helps. I extended my testing scenario a little bit so that i can see the performance impacts much more clear again. I am now spawning 1600 projectiles in one burst. with that new setting in mind i ran some testes. when i used my code from yesterday with the dequeueing on the main thread i need about 2.5ms. i also had one test with a nativehashmap instead of a queue where i copied the value-set to a nativearray in the main thread which gave me 2ms, but i dropped that again because i needed to pre define the hashmap size, but i dont have a count at that moment and i didn't wanted to roughly estimate a size. but with your single burst compiled job solution and just Run() to dequeue the queue to a nativearray the time needed is now about 1.5ms. where just the EntityManager.Instantiate( prefab, AllSpawnedEntities ) already takes about 1.3ms.

so what i am now thinking about is that i will build a pooling mechanism in the future, where i always hold a pre instantiated ammount of projectile entities and only set the component values with the spawner system, because the "only set component values"-part is lightning fast. maybe reusing the projectile entities that i am Destroy() at the moment when they collide with something. but this will be done in some time in the future when i feel the need to do that.

the system itself is running all the time. but i deactivate the entities when they dont need to fire so the system skips these entities and only processes the activated ones. the activation/deactivation of the entities is for now done from the mono-behaviour side where i just hold references to the spawner entities and only call something like that:

Code (CSharp):

1. public void EnableShooting()
2.     {
3.         foreach ( Entity Muzzle in this.ECSMuzzleProxies )
4.         {
5.             ECSManager.SetEnabled( Muzzle, true );
6.         }
7.     }
8.  
9.     public void DisableShooting()
10.     {
11.         foreach ( Entity Muzzle in this.ECSMuzzleProxies )
12.         {
13.             ECSManager.SetEnabled( Muzzle, false );
14.         }
15.     }

so in fact i have "ships" that store the "muzzle" entities.
i maybe change that in the future, because i am facing some other issues where i more and more think of a full conversion to ECS and only use monobehaviours for my UI.

in fact i have another question about the unity.physics package. i am at the moment using the old physX. i tried the new physics package but have dropped it again, because i didn't saw a overlapsphere test in the api. this is something i will use many times in the future. but recently it came to my mind that maybe i could do a collider-cast with "no destination" to mimic the functionality of the old physics.overlappsphere? or is there something else that i haven't found yet?

edit: well i think for the overlappsphere thing i could also live with the AABB-Region query and use the results to check against a radius

 

That seems really slow to me. And I still suspect there's a job complete wrapped up in that. But .2ms for all the other Burst jobs is really good.

Either approach should work for you.

 

edit: so for everyone who stumble on this. createentity and instantiate are roughly the same speed. instantiate only has some more instructions that are executed when in editor and these additional instructions slow the instantiate down. when you make the real build, those instructions are skipped when compiled, so everithing will be fine.

hello,

i made some tests today i wanted to share. so i came back to the problem and solution i had a week ago and i thought i could live with it. but like always i wanted a little more. more ships, turrets and therefore more projectile spawns.

i changed my code to use CreateEntity(), instead of Instantiate(). and the results are roughly 10x faster with this approach. so i had to declare an archetype for my projectile prefab manually via code and use the GameObjectConversionUtility.ConvertGameObjectHierarchy( GameObjectPrefab, World.Active ) as a template to copy all the prefab data with a parallel job. i think its clearer when you see my example test code.

Code (CSharp):

1. using Unity.Burst;
2. using Unity.Collections;
3. using Unity.Entities;
4. using Unity.Jobs;
5. using Unity.Mathematics;
6. using Unity.Rendering;
7. using Unity.Transforms;
8. using UnityEngine.Profiling;
9. using static Unity.Mathematics.math;
10.  
11. public class ECSSpawnTest : JobComponentSystem
12. {
13.     private EntityArchetype m_TestArchetype;
14.     private Entity m_TestPrefab;
15.  
16.     protected override void OnCreate()
17.     {
18.  
19.         m_TestArchetype = EntityManager.CreateArchetype( new ComponentType[] {
20.                 typeof(LocalToWorld),
21.                 typeof(NonUniformScale),
22.                 typeof(Rotation),
23.                 typeof(Translation),
24.                 typeof(RenderBounds)
25.         } );
26.  
27.         m_TestPrefab = EntityManager.CreateEntity( m_TestArchetype );
28.     }
29.  
30.     [BurstCompile]
31.     private struct ECSManualInstantiationJob : IJobParallelFor
32.     {
33.         [ReadOnly] public NativeArray<Entity> m_AllCreatedEntities;
34.  
35.         [ReadOnly] public LocalToWorld m_CopyFromLocalToWorld;
36.         [ReadOnly] public NonUniformScale m_CopyFromNonUniformScale;
37.         [ReadOnly] public Rotation m_CopyFromRotation;
38.         [ReadOnly] public Translation m_CopyFromTranslation;
39.         [ReadOnly] public RenderBounds m_CopyFromRenderBounds;
40.  
41.         [NativeDisableParallelForRestriction][WriteOnly] public ComponentDataFromEntity<LocalToWorld> m_AllLocalToWorlds;
42.         [NativeDisableParallelForRestriction][WriteOnly] public ComponentDataFromEntity<NonUniformScale> m_AllNonUniformScales;
43.         [NativeDisableParallelForRestriction][WriteOnly] public ComponentDataFromEntity<Rotation> m_AllRotations;
44.         [NativeDisableParallelForRestriction][WriteOnly] public ComponentDataFromEntity<Translation> m_AllTranslations;
45.         [NativeDisableParallelForRestriction][WriteOnly] public ComponentDataFromEntity<RenderBounds> m_AllRenderBounds;
46.  
47.         public void Execute( int index )
48.         {
49.             m_AllLocalToWorlds[m_AllCreatedEntities[index]] = m_CopyFromLocalToWorld;
50.             m_AllNonUniformScales[m_AllCreatedEntities[index]] = m_CopyFromNonUniformScale;
51.             m_AllRenderBounds[m_AllCreatedEntities[index]] = m_CopyFromRenderBounds;
52.             m_AllRotations[m_AllCreatedEntities[index]] = m_CopyFromRotation;
53.             m_AllTranslations[m_AllCreatedEntities[index]] = m_CopyFromTranslation;
54.         }
55.     }
56.  
57.  
58.  
59.     protected override JobHandle OnUpdate(JobHandle inputDependencies)
60.     {
61.         inputDependencies.Complete();
62.  
63.         NativeArray<Entity> TestEntities1 = new NativeArray<Entity>(100000, Allocator.TempJob);
64.         NativeArray<Entity> TestEntities2 = new NativeArray<Entity>(100000, Allocator.TempJob);
65.  
66.         Profiler.BeginSample( "Create with archetype and copy values" );
67.      
68.         this.EntityManager.CreateEntity( m_TestArchetype, TestEntities1 );
69.  
70.         new ECSManualInstantiationJob()
71.         {
72.             m_AllCreatedEntities = TestEntities1,
73.             m_CopyFromLocalToWorld = EntityManager.GetComponentData<LocalToWorld>( m_TestPrefab ),
74.             m_CopyFromNonUniformScale = EntityManager.GetComponentData<NonUniformScale>( m_TestPrefab ),
75.             m_CopyFromRenderBounds = EntityManager.GetComponentData<RenderBounds>( m_TestPrefab ),
76.             m_CopyFromTranslation = EntityManager.GetComponentData<Translation>( m_TestPrefab ),
77.             m_CopyFromRotation = EntityManager.GetComponentData<Rotation>( m_TestPrefab ),
78.             m_AllLocalToWorlds = GetComponentDataFromEntity<LocalToWorld>( false ),
79.             m_AllNonUniformScales = GetComponentDataFromEntity<NonUniformScale>( false ),
80.             m_AllRotations = GetComponentDataFromEntity<Rotation>( false ),
81.             m_AllTranslations = GetComponentDataFromEntity<Translation>( false ),
82.             m_AllRenderBounds = GetComponentDataFromEntity<RenderBounds>( false ),
83.         }.Schedule( TestEntities1.Length, 64, inputDependencies ).Complete();
84.         Profiler.EndSample();
85.  
86.  
87.         Profiler.BeginSample( "Instantiate with prefab" );      
88.         this.EntityManager.Instantiate( m_TestPrefab, TestEntities2 );
89.         Profiler.EndSample();
90.  
91.  
92.         this.EntityManager.DestroyEntity( TestEntities1 );
93.         this.EntityManager.DestroyEntity( TestEntities2 );
94.         TestEntities1.Dispose();
95.         TestEntities2.Dispose();
96.  
97.         return inputDependencies;
98.     }
99. }

if you want to test it yourself, this code should work in every ecs project...at least i hope so
and here are some screenshots of my profiler sessions:

first two thinks i think are the bottlenecks when using Instantiate(). I used the "Deep Profile" setting in the Profiler for that and had a test with 5000 entities instantiated/created in batch:


here have a look at the whole player loop with deep profile on


and here i deactivated deep profiling and changed the spawn entity count from 5k to 100k


so the question i have in mind is why is instantiate so much heavier than create. surely it should be slower because you would have only malloc's with createentity and malloc's + memcopy's with instantiate. but not 10x.

by the way: a method like "EntityManager.GetArchetype(Entity)" to get the archetype from an entity would be very nice for this. so if you have a converted gameobject-prefab you dont have to manually define a copy of the archetype and have that kind of automatically

so i hope i don't have missed something and hope this post helps in the future when you have similar problems.
and if i have made some serious error in reasoning (is this the way to say this in english?) please tell me, because it is so much overhead with this way to instantiate entities and i dont want to do that when a similar part of my project needs this in the future

 

You can get the archetype of an entity like this:

Code (csharp):

1. Entity entity = ...;
2. EntityManager.GetChunk(entity).Archetype;

This allows you to get the component types associated with the archetype using the GetComponentTypes method on the archetype.
For entities converted from prefabs you may find that the archetype contains additional components that you don't actually want to copy over (like the Prefab component, the LinkedEntityGroup etc).

 

It seems like a lot of the overhead is coming from setting and getting entity names (though deep profiling might be skewing the numbers here significantly). This only happens within the editor, here is the relevant piece of code in the instantiate logic:

Code (CSharp):

1. #if UNITY_EDITOR
2.                 for (var i = 0; i < allocatedCount; ++i)
3.                     SetName(outputEntities[i + instanceBeginIndex],
4.                         GetName(srcEntity));
5. #endif

I'd suggest remeasuring the cost in an actual build.

 

You shouldn't profile it in editor, cause it has many overheads (safety checks, job debugger, etc). SetName it's editor feature which won't be in build. Build two different versions and profile them in build for real, clear results.
Edit: a bit late, I don't update page from lunch

 

Awesome! Thanks for that

Yes, it was 100% that. In a real build both the create and instantiate are roughly the same speed. Damn i feel dumb now. I already suggested something like that but yeah, at least everything is clear now. Sorry for the desinformation, i will make an edit to my post above.

Big thanks to all of you