Chunk Iteration - How to avoid deallocation error

I've been learning about Chunk Iteration, and I've run into a problem I'm not sure how to solve. If any of you experts can suggest something, I'd really appreciate it:

I've written a system below that needs to do three things:

1. Read data about an entity from an ISharedComponentData
2. Create a new instance of that entity
3. Store a reference to the newly created instance in a DynamicBuffer

I'm using EntityManager in a simple ComponentSystem. The problem is that when I call EntityManager.Instantiate() to create the clone in step #2, it deallocates my DynamicBuffer reference.

Here's my code:

Code (CSharp):

1. public class MySystem : ComponentSystem
2. {
3.     private ComponentGroup componentGroup;
4.  
5.     private ArchetypeChunkEntityType archetypeChunkEntityType;
6.     private ArchetypeChunkSharedComponentType<MySourceData> archetypeChunkMySourceDataType;
7.     private ArchetypeChunkBufferType<MyCloneReferenceData> archetypeMyCloneReferenceDataBufferType;
8.  
9.     protected override void OnCreateManager()
10.     {
11.         EntityArchetypeQuery query = new EntityArchetypeQuery()
12.         {
13.             All = new ComponentType[] { typeof(MySourceData), typeof(MyCloneReferenceData) },
14.         };
15.  
16.         componentGroup = GetComponentGroup(query);
17.     }
18.  
19.     protected override void OnUpdate()
20.     {
21.         NativeArray<ArchetypeChunk> archetypeChunks = componentGroup.CreateArchetypeChunkArray(Allocator.TempJob);
22.  
23.         archetypeChunkEntityType = GetArchetypeChunkEntityType();
24.         archetypeChunkMySourceDataType = GetArchetypeChunkSharedComponentType<MySourceData>();
25.         archetypeMyCloneReferenceDataBufferType = GetArchetypeChunkBufferType<MyCloneReferenceData>();
26.            
27.  
28.         for (int iChunk = 0; iChunk < archetypeChunks.Length; iChunk++)
29.         {
30.             ArchetypeChunk archetypeChunk = archetypeChunks[iChunk];
31.  
32.             NativeArray<Entity> entities = archetypeChunk.GetNativeArray(archetypeChunkEntityType);
33.             int mySourceDataIndex = archetypeChunk.GetSharedComponentIndex(archetypeChunkMySourceDataType);
34.             MySourceData mySourceData = EntityManager.GetSharedComponentData<MySourceData>(mySourceDataIndex);
35.             BufferAccessor<MyCloneReferenceData> myCloneReferenceDataBufferAccessor = archetypeChunk.GetBufferAccessor(archetypeMyCloneReferenceDataBufferType);
36.  
37.             for (int iEntity = 0; iEntity < archetypeChunk.Count; iEntity++)
38.             {
39.                 Entity entity = entities[iEntity];
40.                 DynamicBuffer<MyCloneReferenceData> myReferenceDataBuffer = myCloneReferenceDataBufferAccessor[iEntity];
41.  
42.                 for (int iSourceEntity = 0; iSourceEntity < mySourceData.sourceEntities.Length; iSourceEntity++)
43.                 {
44.                     Entity sourceEntity = mySourceData.sourceEntities[iSourceEntity]; // grab the source entity
45.  
46.                     Entity cloneEntity = EntityManager.Instantiate(sourceEntity); // clone the source entity
47.  
48.                     myReferenceDataBuffer[iSourceEntity] = new MyCloneReferenceData(cloneEntity); // store a reference the clone entity in the dynamic buffer
49.                 }
50.             }
51.         }
52.  
53.         archetypeChunks.Dispose();
54.     }
55. }

If you can suggest another way to do what I'm after, I would be sincerely grateful.

 

Just a guess, you're probably invalidating the chunk you're iterating when you use EntityManager to Instantiate, try PostUpdateCommands.Instantiate instead.

 

Spot on.

Pretty much you can't use EntityManager methods that modify data (instantiate, add, remove, destroy etc) while iterating because it invalidates everything.

 

Thanks you, guys. I understand this. So What should I do instead, to achieve the desired results?

I don't believe I can use PostUpdateCommands here, since I need a reference to the Entity being created. Take a look at the steps listed in the OP. In step #2, I instantiate a new entity. In step #3. I store a reference to that entity in a DynamicBuffer.

It almost seems like the only way would be to split this simple logic up into multiple Systems that each do a tiny bit of the work. I must be missing something here.

What would you do?

 

In the latest Entities update(0.0.21) PostUpdateCommands + EntityCommandBuffer return a placeholder Entity. I'm unsure how useful it is in this case as I've tried to use it to unify two of my systems, but failed, admittedly I wasn't being too serious about it. Maybe you'll have better luck.

 

Ah, that's fantastic news! I've been hoping they would implement something like that, and I had no idea it had happened.

Say, where would I go to find patch notes, or update notes for the package updates? I feel like that's where people are finding information like this, and I'm missing out.

Thank you again!

Nevermind - found it.

https://github.com/Unity-Technologies/EntityComponentSystemSamples/blob/master/ReleaseNotes.md

 

Actually, I'm not seeing this in the release notes for 0.0.21. Maybe it's an undocumented feature?

 

It's not documented, they actually changed the "implied" behavior of creating and setting/adding components where you would create an entity and then set/add would go to the last created entity. If you use intellisense you'll see that it returns an entity now.

 

*Does a dance* This is great news. Thank you.

Looking at the source - it doesn't look like they've removed the old implicit functions. Just clarifying for anyone else who comes by.

 

@RecursiveEclipse: Unfortunately that doesn't look like the silver bullet I imagined.

I believe the temporary entities returned by these new functions can be used in further calls to EntityCommandBuffer, like this:

Code (CSharp):

1. Entity cloneEntity = PostUpdateCommands.Instantiate(sourceEntity);
2.  
3. PostUpdateCommands.AddComponent(cloneEntity, new Position());

But they can't be used to store a reference to the newly created entity, like this:

Code (CSharp):

1. Entity cloneEntity = PostUpdateCommands.Instantiate(sourceEntity);
2.  
3. // this won't work. "cloneEntity" has an index of -1, won't refer to the same entity that's eventually created by the ECB.
4. CloneReferenceData cloneReferenceData = new CloneReferenceData(cloneEntity);

...Which makes sense I suppose. Entity is a simple struct, so it couldn't be updated later with a new index and version. Should have thought of that earlier. :/

Still a cool tool to have for other cases.

 

You'd need to track the temp entities and update them in a future system after they've been created.

If you're not doing this in a job, you could just iterate your chunks and store the data required to setup in custom struct within a nativelist/queue then dispose of the chunks and just iterate the list/queue and create all your entities directly with entity manager within this iteration.

 

Thank you for your advice. However, I tried your suggestion, and I'm still getting a "NativeArray deallocated" error when I try to access the DynamicBuffer again after calling EntityManager.Instantiate(). <--(line #51).

Could you take a look and tell me what I'm doing wrong?

Code (CSharp):

1. public class MySystem : ComponentSystem
2. {
3.     private ComponentGroup componentGroup;
4.  
5.     private ArchetypeChunkEntityType archetypeChunkEntityType;
6.  
7.     protected override void OnCreateManager()
8.     {
9.         EntityArchetypeQuery query = new EntityArchetypeQuery()
10.         {
11.             All = new ComponentType[] { typeof(MyCloneReferenceData) },
12.         };
13.  
14.         componentGroup = GetComponentGroup(query);
15.     }
16.  
17.     protected override void OnUpdate()
18.     {
19.         NativeArray<ArchetypeChunk> archetypeChunks = componentGroup.CreateArchetypeChunkArray(Allocator.TempJob);
20.         archetypeChunkEntityType = GetArchetypeChunkEntityType();
21.  
22.         NativeQueue<Entity> storedEntities = new NativeQueue<Entity>(Allocator.TempJob);
23.  
24.         for (int iChunk = 0; iChunk < archetypeChunks.Length; iChunk++)
25.         {
26.             ArchetypeChunk archetypeChunk = archetypeChunks[iChunk];
27.  
28.             NativeArray<Entity> entities = archetypeChunk.GetNativeArray(archetypeChunkEntityType);
29.  
30.             for (int iEntity = 0; iEntity < archetypeChunk.Count; iEntity++)
31.             {
32.                 Entity agent = entities[iEntity];
33.  
34.                 storedEntities.Enqueue(agent);
35.             }
36.         }
37.  
38.         archetypeChunks.Dispose();
39.  
40.         for (int iEntity = 0; iEntity < storedEntities.Count; iEntity++)
41.         {
42.             Entity entity = storedEntities.Dequeue();
43.             DynamicBuffer<MyCloneReferenceData> cloneReferenceDataBuffer = EntityManager.GetBuffer<MyCloneReferenceData>(entity);
44.  
45.             for (int iData = 0; iData < cloneReferenceDataBuffer.Length; iData++)
46.             {
47.                 MyCloneReferenceData myCloneReferenceData = cloneReferenceDataBuffer[iData];
48.  
49.                 Entity clonedEntity = EntityManager.Instantiate(myCloneReferenceData.entity);
50.  
51.                 cloneReferenceDataBuffer[iData] = new MyCloneReferenceData(clonedEntity);
52.             }
53.         }
54.  
55.         storedEntities.Dispose();
56.     }
57. }

 

Yes you can't access the buffer array after you make a call to entitymanager.
You'd have to call

DynamicBuffer<MyCloneReferenceData> cloneReferenceDataBuffer = EntityManager.GetBuffer<MyCloneReferenceData>(entity);

again

 

Would you consider that a clean pattern, personally?

 

No, whole thing is a mess lol.

 

Then why did you suggest this? I spent time implementing your suggestion.

Ok, ok, what would you do?

 

Because it was the solution to your problem.

With how I design my systems, I'd do the entire thing in jobs and set the native buffer 1 frame later after the entity was created. Or just invert the referencing (instance references prefab).

I haven't really come across a situation that I would need to do something like this though so I haven't really thought about it.

 

Full job based version. Support for either same or two frame.

Not tested, just an idea.

It does unfortunately add then remove a component from an Entity. I might explore remapping entities a bit more because it seems to be a very common theme in the last few days on these forums.

Code (CSharp):

1.  
2. #define SAME_FRAME
3.  
4. using Unity.Burst;
5. using Unity.Collections;
6. using Unity.Entities;
7. using Unity.Jobs;
8.  
9. public class MySystem : JobComponentSystem
10. {
11.     private ComponentGroup componentGroup;
12.     private EndFrameBarrier barrier;
13.  
14.     /// <inheritdoc/>
15.     protected override void OnCreateManager()
16.     {
17.         this.barrier = this.World.GetOrCreateManager<EndFrameBarrier>();
18.  
19.         this.componentGroup = this.GetComponentGroup(new EntityArchetypeQuery()
20.         {
21.             All = new ComponentType[] { typeof(MyCloneReferenceData) },
22.         });
23.     }
24.  
25.     /// <inheritdoc/>
26.     protected override JobHandle OnUpdate(JobHandle handle)
27.     {
28. #if SAME_FRAME
29.         var ecb = new EntityCommandBuffer(Allocator.TempJob);
30. #endif
31.  
32.         var getEntitiesJob = new GetEntitiesJob
33.         {
34.             EntityType = this.GetArchetypeChunkEntityType(),
35.             MyCloneReferenceDataType = this.GetArchetypeChunkBufferType<MyCloneReferenceData>(true),
36. #if SAME_FRAME
37.             EntityCommandBuffer = ecb.ToConcurrent(),
38. #else
39.             EntityCommandBuffer = this.barrier.CreateCommandBuffer().ToConcurrent(),
40. #endif
41.         };
42.  
43.         handle = getEntitiesJob.Schedule(this.componentGroup, handle);
44.  
45. #if SAME_FRAME
46.         handle.Complete();
47.         ecb.Playback(this.EntityManager);
48.         ecb.Dispose();
49. #endif
50.  
51.         var fillBuffer = new FillBufferJob
52.         {
53.             EntityCommandBuffer = this.barrier.CreateCommandBuffer().ToConcurrent(),
54.             MyCloneReferenceDatas = this.GetBufferFromEntity<MyCloneReferenceData>(),
55.         };
56.  
57.         handle = fillBuffer.Schedule(this, handle);
58.  
59.         this.barrier.AddJobHandleForProducer(handle);
60.  
61.         return handle;
62.     }
63.  
64.     private struct GetEntitiesJob : IJobChunk
65.     {
66.         [ReadOnly]
67.         public ArchetypeChunkEntityType EntityType;
68.  
69.         [ReadOnly]
70.         public ArchetypeChunkBufferType<MyCloneReferenceData> MyCloneReferenceDataType;
71.  
72.         public EntityCommandBuffer.Concurrent EntityCommandBuffer;
73.  
74.         /// <inheritdoc />
75.         public void Execute(ArchetypeChunk chunk, int chunkIndex, int firstEntityIndex)
76.         {
77.             var entities = chunk.GetNativeArray(this.EntityType);
78.             var buffers = chunk.GetBufferAccessor(this.MyCloneReferenceDataType);
79.  
80.             for (int i = 0; i < chunk.Count; i++)
81.             {
82.                 var entity = entities[i];
83.  
84.                 for (var j = 0; j < buffers[i].Length; j++)
85.                 {
86.                     var e = this.EntityCommandBuffer.Instantiate(chunkIndex, entity);
87.                     this.EntityCommandBuffer.AddComponent(chunkIndex, e, new EntityReference { Parent = entity, Index = j, });
88.                 }
89.             }
90.         }
91.     }
92.  
93.     private struct FillBufferJob : IJobProcessComponentDataWithEntity<EntityReference>
94.     {
95.         public EntityCommandBuffer.Concurrent EntityCommandBuffer;
96.  
97.         [NativeDisableParallelForRestriction] // should be fine as index is unique
98.         public BufferFromEntity<MyCloneReferenceData> MyCloneReferenceDatas;
99.  
100.         /// <inheritdoc />
101.         public void Execute(Entity entity, int index, [ReadOnly] ref EntityReference er)
102.         {
103.             var buffer = this.MyCloneReferenceDatas[er.Parent];
104.             buffer[er.Index] = new MyCloneReferenceData(entity);
105.             this.EntityCommandBuffer.RemoveComponent<EntityReference>(index, entity);
106.         }
107.     }
108.  
109.     private struct EntityReference : IComponentData
110.     {
111.         public Entity Parent;
112.         public int Index;
113.     }
114. }
115.  

 

Thanks for the example. I will dig through it and learn from what you are doing.

 

Quick question: You're using EntityCommandBuffer.Concurrent in a BurstCompiled Job. I've read elsewhere in these forums that EntityCommandBuffer isn't valid for Burst. I guess that doesn't apply to EntityCommandBuffer.Concurrent?

I hadn't thought to run multiple jobs in a single system, or to playback an ECB while a system is in progress. And I didn't know about BufferFromEntity<T>. Thanks for these tips, I will use them.

 

Nah that was just my mistake. I changed what I was doing and started using ECB. Like I said I never tested it. Just realized as well I never disposed the ecb. Code updated.

 

DestroyEntity is the only function(specifically DestroyEntity/CreateEntity/Add/Set/Remove) of EntityCommandBuffer that is Burst compilable, the rest don't work right now because they use static.

 

Sorry, I am new to ECS and trying to get onboard with V 0.0.12-preview.23. I cannot find any static methods on EntityManager. That creates a problem when I try to access shared data from chunk using chunk.GetSharedComponentData(sharedType, em) inside Execute methods. Is there a simple way to access EM from BurstCompiled struct?

 

EM not allowed inside job.