Most Efficient way to instantiate 200k entities in separate Threads/Jobs

i have a collection of Entities and their Positions Rotations and Scales. (around 200k)
im trying to instantiate them in different threads cause doing it in the main thread is just blocking my game flow for more than 30s which is not a considarable solution. so i tried to do it in a Multithreaded way to paralellize the work but i get errors.
cant do it inside Jobs cause it's impossible to set The SCD Region.

Code (CSharp):

1. ExclusiveEntityTransaction commands = CachingEntityManager.BeginExclusiveEntityTransaction();
2.            
3. NativeArray<Entity> enitiesContainer;
4. await Task.Run(()=> {
5.  
6. foreach (var row in entitiesMapper)
7.      {
8.      float3 scale;
9.      if (row.Value.Count > 1)
10.       {
11.         enitiesContainer = new NativeArray<Entity>(row.Value.Count, Allocator.Persistent);
12.         commands.Instantiate(row.Key, enitiesContainer);
13.         var count = enitiesContainer.Length;
14.          for (int i = 0; i < count; i++)
15.               {
16.                commands.SetComponentData(enitiesContainer[i], new Translation { Value = row.Value[i].pos });
17.                commands.SetComponentData(enitiesContainer[i], new Rotation { Value = row.Value[i].rot });
18.                scale = row.Value[i].scale;
19.                if ((math.abs(scale.x - scale.y) > 0.00f) && (math.abs(scale.x - scale.z) > 0.00f))
20.                     commands.SetComponentData(enitiesContainer[i], new Scale { Value = scale.x });
21.                else
22.                     commands.SetComponentData(enitiesContainer[i], new NonUniformScale { Value = scale });
23.                     commands.SetSharedComponentData(enitiesContainer[i], new Region { value = row.Value[i].pos.ToRegionIndex() });
24.                }
25.                commands.DestroyEntity(row.Key);
26.                enitiesContainer.Dispose();
27.         }
28.               else
29.               {
30.                     commands.SetComponentData(row.Key, new Translation { Value = row.Value[0].pos });
31.                     commands.SetComponentData(row.Key, new Rotation { Value = row.Value[0].rot });
32.                     scale = row.Value[0].scale;
33.                     if ((math.abs(scale.x - scale.y) > 0.00f) && (math.abs(scale.x - scale.z) > 0.00f))
34.                             commands.SetComponentData(row.Key, new Scale { Value = scale.x });
35.                     else
36.                             commands.SetComponentData(row.Key, new NonUniformScale { Value = scale });
37.                             commands.SetSharedComponentData(row.Key, new Region { value = row.Value[0].pos.ToRegionIndex() });
38.                     }
39.                 }
40.             });
41.             CachingEntityManager.EndExclusiveEntityTransaction()

 

The fastest way of instantiating I've found so far is by instantiating them all on the main thread with the batch operation. And then have a job that that sets all the ComponentData with Burst enabled. One way I've found with setting the SharedComponentData is by setting it on the prefab before I instantiate so everything will have the correct shared data when instantiated.

 

Use batch api to instantiate all entities

You can than use a job to initialize them.
Use a IParralelForJob and
GetComponentDataFromEntity()
GetBufferFromEntity()
to set components.

 

it's a nice workaround but it's not applicable in my case ( sorry i should provide more details)
so my map that need to be generated is just a Dictionary<int Addressable_Id, List< PosRotScale> listTrans>
the problem with setting the SCD before the Conversion workflow is i need to convert the same Asset Multiple times depending on regions his linked and i have a lot of regions (around 400). so i think setting it after the Conversion will be faster and there is no way to do it in parrallel

 

Ah I didn't mean to set it before the conversion. The prefab is just an entity like the one to be instantiated, the only difference is it has a Prefab component and queries in systems ignore entities with Prefab components on them. I'm setting the actual prefab entity's SharedComponentData just before I instantiate.

Code (CSharp):

1. NativeArray<Entity> entities = new NativeArray<Entity>(100, Allocator.Temp, NativeArrayOptions.UninitializedMemory);
2.  
3. EntityManager.SetSharedComponentData(prefab, new MapChunk() { ChunkId = new MapChunk(0, 0) });
4. EntityManager.Instantiate(prefab, entities);

 

What error message are you getting when trying to set a shared component using EET?

But as a workaround, temps12 is right. Creating a runtime prefab entity for each unique shared component on the main thread and storing them in a lookup to instantiate from in your job is an ok workaround. It's a little annoying though if you want multiple threads since with EET each thread needs its own world and you have to create your prefabs for each world.

 

Is this 200k transform rotation scale known before runtime? If so then the fastest way to instantiate things is going to be via chunk memory deserialization. (By SerializationUtility or Subscene feature) SCD is also handled by SerializationUtility, the problem is the library is not well done yet and if serialize version bumped then your previously serialized memory will not load.

 

That's awesome!
yes im collecting my transforms in edit mode at runtime im fetching them from a file.
can you give me any example or link to a documentation ?

 

It seems to not being documented much at the moment maybe because it is still unstable? But for SCD you need some help from GameObject + proxy version of that SCD. This is called hybrid serialization. Instead of SerializeUtility, you use SerializeUtilityHybrid. The proxy will allows SCD to be serialized by classic Unity object serialization because it made your SCD into a MonoBehaviour pastable onto GO, while the ECS part goes in the serialized chunk memory + only SCD indexes went with them. Assemble 2 pieces back together on deserialize which it will also ask for a game object with those SCD proxies.

Long time ago I played with this and there is a serious problem about memory architecture, I made a serialized chunk memory file from 64-bit Macbook then try to deserialize on 32-bit Android and it is not compatible since Chunk struct contains some pointer fields that weren't padded. This maybe fixed by now but just to warn you about potential problems.

 

Assuming there is no SCD involved, the fastest way I found is to use

CopyFromComponentDataArray

where you have your TRS values to set waiting in NativeArray with exact length of entity target you want to set to, which will be determined by a query. This is even faster than multithreaded job-set because

CopyFromComponentDataArray

schedules multithreaded unsafe job that copy a stride of data per chunk rather than per entity.

This is an example of jobified set. It is multithreaded per chunk, but each chunk set one by one linearly.

Code (CSharp):

1. [PerformanceTest]
2. public void InitializeValuesByJobs()
3. {
4.     var w = new World("Sample");
5.     var types = new ComponentType[]{
6.         ComponentType.ReadWrite<Translation>(),
7.         ComponentType.ReadWrite<Rotation>(),
8.         ComponentType.ReadWrite<NonUniformScale>()
9.     };
10.     var arc = w.EntityManager.CreateArchetype(types);
11.  
12.     var entityArray = new NativeArray<Entity>(200000, Allocator.TempJob);
13.     var translationArray = new NativeArray<Translation>(200000, Allocator.TempJob);
14.     var rotationArray = new NativeArray<Rotation>(200000, Allocator.TempJob);
15.     var scaleArray = new NativeArray<NonUniformScale>(200000, Allocator.TempJob);
16.     w.EntityManager.CreateEntity(arc, entityArray);
17.  
18.     Measure.Method(() =>
19.     {
20.         for (int i = 0; i < 200000; i++)
21.         {
22.             translationArray[i] = new Translation { Value = Random.insideUnitSphere };
23.             rotationArray[i] = new Rotation { Value = Random.rotation };
24.             scaleArray[i] = new NonUniformScale { Value = Random.insideUnitSphere };
25.         }
26.     }).Definition("Prepare data").WarmupCount(5).MeasurementCount(1).Run();
27.  
28.     var eq = w.EntityManager.CreateEntityQuery(types);
29.     Measure.Method(() =>
30.     {
31.         new InitializationJob
32.         {
33.             translationArray = translationArray,
34.             rotationArray = rotationArray,
35.             scaleArray = scaleArray
36.         }.Schedule(eq).Complete();
37.     }).Definition("Jobs Initialization").WarmupCount(5).MeasurementCount(1).Run();
38.  
39.     entityArray.Dispose();
40.     translationArray.Dispose();
41.     rotationArray.Dispose();
42.     scaleArray.Dispose();
43.     w.Dispose();
44. }
45.  
46. private struct InitializationJob : IJobForEachWithEntity<Translation, Rotation, NonUniformScale>
47. {
48.     [ReadOnly] public NativeArray<Translation> translationArray;
49.     [ReadOnly] public NativeArray<Rotation> rotationArray;
50.     [ReadOnly] public NativeArray<NonUniformScale> scaleArray;
51.     public void Execute(Entity entity, int index, ref Translation c0, ref Rotation c1, ref NonUniformScale c2)
52.     {
53.         c0 = translationArray[index];
54.         c1 = rotationArray[index];
55.         c2 = scaleArray[index];
56.     }
57. }

This is using

CopyFromComponentDataArray

without any jobs.

Code (CSharp):

1. [PerformanceTest]
2. public void InitializeValuesByThreadedCFCDA()
3. {
4.     var w = new World("Sample");
5.     var types = new ComponentType[]{
6.         ComponentType.ReadOnly<Translation>(),
7.         ComponentType.ReadOnly<Rotation>(),
8.         ComponentType.ReadOnly<NonUniformScale>(),
9.     };
10.     var arc = w.EntityManager.CreateArchetype(types);
11.  
12.     var entityArray = new NativeArray<Entity>(200000, Allocator.TempJob);
13.     var translationArray = new NativeArray<Translation>(200000, Allocator.TempJob);
14.     var rotationArray = new NativeArray<Rotation>(200000, Allocator.TempJob);
15.     var scaleArray = new NativeArray<NonUniformScale>(200000, Allocator.TempJob);
16.     w.EntityManager.CreateEntity(arc, entityArray);
17.  
18.     Measure.Method(() =>
19.     {
20.         for (int i = 0; i < 200000; i++)
21.         {
22.             translationArray[i] = new Translation { Value = Random.insideUnitSphere };
23.             rotationArray[i] = new Rotation { Value = Random.rotation };
24.             scaleArray[i] = new NonUniformScale { Value = Random.insideUnitSphere };
25.         }
26.     }).Definition("Prepare data").WarmupCount(5).MeasurementCount(1).Run();
27.  
28.     var eq = w.EntityManager.CreateEntityQuery(types);
29.     Measure.Method(() =>
30.     {
31.         //Array length must match entity count return from the EQ exactly.
32.         //The data lands on each entity in order of chunks returned.
33.  
34.         //Each of these is threaded per chunk, and for a chunk there is only one MemCpy to initialize
35.         //multiple entities at once.
36.         eq.CopyFromComponentDataArray(translationArray);
37.         eq.CopyFromComponentDataArray(rotationArray);
38.         eq.CopyFromComponentDataArray(scaleArray);
39.     }).Definition("CFCDA Initialization").WarmupCount(5).MeasurementCount(1).Run();
40.  
41.     entityArray.Dispose();
42.     translationArray.Dispose();
43.     rotationArray.Dispose();
44.     scaleArray.Dispose();
45.     w.Dispose();
46. }

Result comparison :

CopyFromComponentDataArray

is about 40% faster than making your own job. Those tests are added to a section of this article https://gametorrahod.com/batched-operation-on-entitymanager/#batched-set with a bit more explanation.

 

Thank you very much, i never knew about the "CopyFromComponentDataArray" that we can copy data that fast from an array directly to chunks ^_^.

a last question
is it possible to serialize and deserialize NativeContainers to a binary array directly as a managed value type without passing Unmanaged containers ?


you are awesome 5argon!
Thank you all guys for your help!

 

The ECS serialize function is meant to serialize chunk. Chunk contains mostly

IComponentData

and native containers couldn't go into those. Dynamic buffer could be serialized but that's per-entity thing so not fit for this purpose. The solution would be

BlobAssetReference

. If any entity has the ref then the real thing will be serialized somewhere. So, I think you have to find a way to use blob builder to build those needed data instead of just

NativeArray

, then use the blob ref to get the

NativeArray

on use.

(I didn't try but there is a test

SerializeEntitiesWorksWithBlobAssetReferences

in

SerializeTests.cs

that looks like it works)

But still I think the proper way is to just serialize entities that are the product of that native container prescription rather than serialize the container and you still have to instantiate entity from things in that container at runtime.

 

Yes I think the correct way to restore the map at runtime is not to recalculate everything but just deserialize a world containing the final product.
Actually I got my response from your blog .

 

Huh, I thought I demolished that old blog long time ago, had no idea they fallback to my personal account instead. Everything there is extremely outdated (before the deprecation apocalypse) and I intended to just start over because those cause misunderstandings : ( I will probably deactivate my main account to get rid of them soon.

Also that article still has no new counterpart since I feel like there are more to come to the serialization system, so I am delaying the rewrite for a bit more.

 

so the serialization process is working fine and generates my file, but the deserialization is giving me errors.

Code (CSharp):

1.  
2.  const string serializedMap = "Assets/Tests/SerializedMap.data";
3.  public void OnClickSerialize()
4.         {
5.             using (var writer = new StreamBinaryWriter(serializedMap))
6.             {
7.                 int[] sharedComponentIndices;
8.                 SerializeUtility.SerializeWorld(MainManager, writer, out sharedComponentIndices);
9.                 SerializeUtility.SerializeSharedComponents(MainManager, writer, sharedComponentIndices);
10.             }
11.         }
12.  
13.  
14.         public void OnclickDeserialize()
15.         {
16.             var transaction = CachingEntityManager.BeginExclusiveEntityTransaction();
17.             using (var reader = new StreamBinaryReader(serializedMap))
18.             {
19.                 var num = SerializeUtility.DeserializeSharedComponents(CachingEntityManager, reader);
20.                 SerializeUtility.DeserializeWorld(transaction, reader, num);
21.             }
22.             CachingEntityManager.EndExclusiveEntityTransaction();
23.         }
24.  

Errors:
OverflowException
Unity.Entities.Serialization.SerializeUtility.DeserializeSharedComponents (Unity.Entities.EntityManager entityManager, Unity.Entities.Serialization.BinaryReader reader) (at Library/PackageCache/com.unity.entities@0.0.12-preview.33/Unity.Entities/SerializeUtility.cs:466)
CWBR.Editor.Utils.TestingCode.OnclickDeserialize () (at Assets/Scripts/Test/TestingCode.cs:90) it's poiting to the line (var num = SerializeUtility.DeserializeSharedComponents(CachingEntityManager, reader)
UnityEngine.Events.InvokableCall.Invoke () (at C:/buildslave/unity/build/Runtime/Export/UnityEvent/UnityEvent.cs:166)
UnityEngine.Events.UnityEvent.Invoke () (at C:/buildslave/unity/build/Runtime/Export/UnityEvent/UnityEvent/UnityEvent_0.cs:58)
UnityEngine.UI.Button.Press () (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/UI/Core/Button.cs:66)
UnityEngine.UI.Button.OnPointerClick (UnityEngine.EventSystems.PointerEventData eventData) (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/UI/Core/Button.cs:108)
UnityEngine.EventSystems.ExecuteEvents.Execute (UnityEngine.EventSystems.IPointerClickHandler handler, UnityEngine.EventSystems.BaseEventData eventData) (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/EventSystem/ExecuteEvents.cs:50)
UnityEngine.EventSystems.ExecuteEvents.Execute[T] (UnityEngine.GameObject target, UnityEngine.EventSystems.BaseEventData eventData, UnityEngine.EventSystems.ExecuteEvents+EventFunction`1[T1] functor) (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/EventSystem/ExecuteEvents.cs:261)
UnityEngine.EventSystems.EventSystem:Update()