[Example] IJobForEach and IBufferElementData (Unity 2020.1)

In the light of current changes, I decided to write up and post very basics example of
IJobProcessComponentDataWithEntity IJobForEach and IBufferElementData
without injection, hoping it will become useful. And maybe can be further improved.

Unity 2020.1a15+ (Entities 0.2.0-preview.18)
Last update 2019 Dec. 10
https://github.com/Antypodish/DOTS_IJobForEach_IBufferElementData

Spoiler

Code (CSharp):

1.  
2. // Unity DOTS-ECS very simple example with IJobForEachWithEntity, IJobForEach_BC and IBufferElementData.
3.  
4. // 2019.12.10
5.  
6. // Requires Unity 2020.1.0a15+
7.  
8. // Tested with
9. // Burst 1.2.0-preview.10
10. // Collections 0.2.0-Preview.13
11. // Entities 0.2.0-preview.18
12. // Jobs 0.2.1-Preview.3
13. // Mathematics 1.1.0.preview.1
14.  
15. // New repo name
16. // ECS_IJobForEach_IBufferElementData
17. // https://github.com/Antypodish/ECS_IJobForEach_IBufferElementData
18. // Old repo name
19. // ECS_IJobProcessComponentDataWithEntity_IBufferElementData
20. // https://github.com/Antypodish/ECS_IJobProcessComponentDataWithEntity_IBufferElementData
21.  
22. using Unity.Collections ;
23. using Unity.Entities ;
24. using Unity.Jobs ;
25. using Unity.Burst ; // See commented out [BurstCompile] lines, above jobs.
26.  
27. using UnityEngine ;
28.  
29. namespace ECS.Test
30. {
31.  
32.     struct Instance : IComponentData
33.     {
34.         public float f ;
35.     }
36.  
37.     // For testing in Job_BC
38.     struct SomeBufferElement : IBufferElementData
39.     {
40.         public int i ;
41.     }
42.  
43.     // For testing in JobWithEntity
44.     struct SomeFromEntityBufferElement : IBufferElementData
45.     {
46.         public int i ;
47.     }
48.  
49.     public class BufferWithJobSystem : JobComponentSystem
50.     {
51.  
52.         // protected override void OnCreateManager ( int capacity ) // Obsolete
53.         protected override void OnCreate ( )
54.         {
55.             base.OnCreate ( ) ;
56.  
57.             Debug.LogWarning ( "Burst is disabled, to use Debug.Log in jobs." ) ;
58.             Debug.LogWarning ( "Jobs are executed approx every second." ) ;
59.  
60.             Instance instance = new Instance () ;
61.  
62.             Entity entity = EntityManager.CreateEntity ( typeof (Instance) ) ;
63.  
64.             EntityManager.SetComponentData ( entity, instance ) ;
65.             EntityManager.AddBuffer <SomeBufferElement> ( entity ) ;
66.  
67.             DynamicBuffer <SomeBufferElement> someBuffer = EntityManager.GetBuffer <SomeBufferElement> ( entity ) ;
68.  
69.             // Add two elements to dynamic buffer.
70.             SomeBufferElement someBufferElement = new SomeBufferElement () ;
71.             someBufferElement.i = 100000 ;
72.             someBuffer.Add ( someBufferElement ) ;
73.             someBufferElement.i = 200000 ;
74.             someBuffer.Add ( someBufferElement ) ;
75.  
76.             EntityManager.Instantiate ( entity ) ; // Clone entity.
77.  
78.    
79.             entity = EntityManager.CreateEntity ( typeof (Instance) ) ;
80.  
81.             EntityManager.SetComponentData ( entity, instance ) ;
82.             EntityManager.AddBuffer <SomeFromEntityBufferElement> ( entity ) ;
83.  
84.             DynamicBuffer <SomeFromEntityBufferElement> someFromEntityBuffer = EntityManager.GetBuffer <SomeFromEntityBufferElement> ( entity ) ;
85.  
86.             // Add two elements to dynamic buffer.
87.             SomeFromEntityBufferElement someFromEntityBufferElement = new SomeFromEntityBufferElement () ;
88.             someFromEntityBufferElement.i = 1000 ;
89.             someFromEntityBuffer.Add ( someFromEntityBufferElement ) ;
90.             someFromEntityBufferElement.i = 10 ;
91.             someFromEntityBuffer.Add ( someFromEntityBufferElement ) ;
92.  
93.             EntityManager.Instantiate ( entity ) ; // Clone entity.
94.  
95.         }
96.  
97.         float previoudTime = 0 ;
98.  
99.         protected override JobHandle OnUpdate ( JobHandle inputDeps )
100.         {
101.    
102.             float time = Time.time ;
103.  
104.             // Execute approx every second.
105.             if ( time > previoudTime + 1 )
106.             {
107.                 previoudTime = time ;      
108.             }
109.             else
110.             {
111.                 return inputDeps ;
112.             }
113.  
114.             JobHandle job_withEntity = new Job_WithEntity ()
115.             {
116.                 time = time,
117.                 someBuffer = GetBufferFromEntity <SomeFromEntityBufferElement> ( false ) // Read and write
118.  
119.             }.ScheduleSingle ( this, inputDeps ) ; // Using instead job.Schedule ( this, inputDeps ), for single threaded test and debug.
120.             // }.Schedule ( this, inputDeps ) ; // Allow execute job in parallel, if there is enough entities.
121.    
122.             JobHandle job_BC = new Job_BC ()
123.             {
124.                 time = time,
125.        
126.             }.ScheduleSingle ( this, job_withEntity ) ; // Using instead job.Schedule ( this, inputDeps ), for single threaded test and debug.
127.             // }.Schedule ( this, jobWithEntity ) ; // Allow execute job in parallel, if there is enough entities.
128.  
129.             return job_BC ;
130.         }
131.  
132.         // [BurstCompile] // Disbaled burst, as Debug.Log is used.
133.         [RequireComponentTag ( typeof ( SomeFromEntityBufferElement ) ) ]
134.         // struct Job: IJobProcessComponentDataWithEntity <Instance> // Obsolete
135.         struct Job_WithEntity : IJobForEachWithEntity <Instance>
136.         {
137.             public float time;
138.  
139.             // Allow buffer read write in parralel jobs
140.             // Ensure, no two jobs can write to same entity, at the same time.
141.             // !! "You are somehow completely certain that there is no race condition possible here, because you are absolutely certain that you will not be writing to the same Entity ID multiple times from your parallel for job. (If you do thats a race condition and you can easily crash unity, overwrite memory etc) If you are indeed certain and ready to take the risks.
142.             // https://forum.unity.com/threads/how-can-i-improve-or-jobify-this-system-building-a-list.547324/#post-3614833
143.             [NativeDisableParallelForRestriction]
144.             public BufferFromEntity <SomeFromEntityBufferElement> someBuffer ;
145.  
146.             public void Execute( Entity entity, int index, ref Instance tester )
147.             {
148.                 tester.f = time ;
149.  
150.                 DynamicBuffer <SomeFromEntityBufferElement> dynamicBuffer = someBuffer [entity] ;
151.  
152.                 SomeFromEntityBufferElement bufferElement = dynamicBuffer [0] ;
153.                 bufferElement.i ++ ; // Increment.
154.                 dynamicBuffer [0] = bufferElement ; // Set back.
155.        
156.                 // Console will throw error when using debug and burst is enabled.
157.                 // Comment out Debug, when using burst.
158.                 Debug.Log ( "T: " + tester.f + " IJobForEachWIthEntity " + " #" + index + "; entity: " + entity + "; " + dynamicBuffer [0].i + "; " + dynamicBuffer [1].i ) ;
159.  
160.             }
161.  
162.         }
163.  
164.         // [BurstCompile] // Disbaled burst, as Debug.Log is used.
165.         struct Job_BC : IJobForEach_BC <SomeBufferElement, Instance>
166.         {
167.             public float time;
168.  
169.             // Allow buffer read write in parralel jobs
170.             // Ensure, no two jobs can write to same entity, at the same time.
171.             public void Execute( DynamicBuffer <SomeBufferElement> dynamicBuffer, ref Instance tester )
172.             {
173.                 tester.f = time ;
174.  
175.  
176.                 SomeBufferElement bufferElement = dynamicBuffer [0] ;
177.                 bufferElement.i ++ ; // Increment.
178.                 dynamicBuffer [0] = bufferElement ; // Set back.
179.        
180.                 // Console will throw error when using debug and burst is enabled.
181.                 // Comment out Debug, when using burst.
182.                 Debug.Log ( "T: " + tester.f + " IJobForEach_BC (Buffer, Component) " + "; "  + dynamicBuffer [0].i + "; " + dynamicBuffer [1].i ) ;
183.  
184.             }
185.  
186.         }
187.  
188.     }
189. }
190.  
191.  

Unity 2019 (Entities 0.0.12-preview 12)
Last update 2019 Jan 3
https://github.com/Antypodish/ECS_I...lementData/blob/master/BufferWithJobSystem.cs

Spoiler

Code (CSharp):

1. // Unity ECS very simple example with IJobProcessComponentDataWithEntity and IBufferElementData.
2.  
3. // Requires official Unity samples
4. // https://github.com/Unity-Technologies/EntityComponentSystemSamples
5. // Or own bootstrap.
6. // Just copy this script to the project
7. // Based on SimpleRotation / RotationSpeedSystem.cs
8.  
9. // 2018.11.16
10.  
11. // Tested with
12. // Entities 0.0.12-preview 12
13. // Burst 0.2.4-preview.37
14. // IncrementalCompiler 0.0.42-preview.24
15. // Jobs 0.0.7-Preview.5
16. // Mathematics 0.0.12.preview.19
17.  
18. using Unity.Collections;
19. using Unity.Entities;
20. using Unity.Jobs;
21. using Unity.Burst;
22. // using Unity.Mathematics;
23. // using Unity.Transforms;
24. using UnityEngine;
25.  
26. namespace ECS.Test
27. {
28.  
29.     struct Instance : IComponentData
30.     {
31.         public float f ;
32.         // public DynamicBuffer <int> db_a ;
33.     }
34.  
35.     struct SomeBufferElement : IBufferElementData
36.     {
37.         public int i ;
38.     }
39.  
40.     public class BufferWithJobSystem : JobComponentSystem
41.     {
42.         [BurstCompile]
43.         [RequireComponentTag ( typeof (SomeBufferElement) ) ]
44.         struct Job: IJobProcessComponentDataWithEntity <Instance>
45.         {
46.             public float dt;
47.  
48.             // Allow buffer read write in parralel jobs
49.             // Ensure, no two jobs can write to same entity, at the same time.
50.             // !! "You are somehow completely certain that there is no race condition possible here, because you are absolutely certain that you will not be writing to the same Entity ID multiple times from your parallel for job. (If you do thats a race condition and you can easily crash unity, overwrite memory etc) If you are indeed certain and ready to take the risks.
51.             // https://forum.unity.com/threads/how-can-i-improve-or-jobify-this-system-building-a-list.547324/#post-3614833
52.             [NativeDisableParallelForRestriction]
53.             public BufferFromEntity <SomeBufferElement> someBufferElement ;
54.  
55.             public void Execute( Entity entity, int index, ref Instance tester )
56.             {
57.                 tester.f = 10 * dt ;
58.  
59.                 DynamicBuffer <SomeBufferElement> someDynamicBuffer = someBufferElement [entity] ;
60.  
61.                 SomeBufferElement buffer = someDynamicBuffer [0] ;
62.  
63.                 // Uncomment as needed
64.                 // buffer.i = 99 ;
65.  
66.                 // someDynamicBuffer [0] = buffer ;
67.  
68.                 // Debug Will throw errors in Job system
69.                 // Debug.Log ( "#" + index + "; " + someDynamicBuffer [0].i + "; " + someDynamicBuffer [1].i ) ;
70.  
71.             }
72.         }
73.  
74.         protected override JobHandle OnUpdate(JobHandle inputDeps)
75.         {
76.             var job = new Job () {
77.                 dt = Time.deltaTime,
78.                 someBufferElement = GetBufferFromEntity <SomeBufferElement> (false)
79.  
80.                 } ;
81.             return job.Schedule(this, inputDeps) ;
82.         }
83.  
84.         // protected override void OnCreateManager ( ) // for Entities 0.0.12 preview 20
85.         protected override void OnCreateManager ( int capacity )
86.         {
87.             base.OnCreateManager ( capacity );
88.  
89.             Instance instance = new Instance () ;
90.  
91.             Entity entity = EntityManager.CreateEntity ( typeof (Instance) ) ;
92.  
93.             EntityManager.SetComponentData ( entity, instance ) ;
94.             EntityManager.AddBuffer <SomeBufferElement> ( entity ) ;
95.  
96.             var bufferFromEntity = EntityManager.GetBufferFromEntity <SomeBufferElement> ();
97.             var buffer = bufferFromEntity [entity];
98.  
99.             SomeBufferElement someBufferElement = new SomeBufferElement () ;
100.             someBufferElement.i = 6 ;
101.             buffer.Add ( someBufferElement ) ;
102.             someBufferElement.i = 7 ;
103.             buffer.Add ( someBufferElement ) ;
104.         }
105.     }
106. }

 

Hi Antypodish, thanks for the demo script. I have some questions.

• // Allow buffer read write in parallel jobs. In void, there is Execute DynamicBuffer is declared. A DynamicBuffer to me is an array that can be dynamically resized. I think you mean RandomAccess Buffer? But it's no a thing.

Some comments to the code. Perhaps you can explain.Thank you.

Code (CSharp):

1. protected override void OnCreateManager ( int capacity )
2.         {
3.  
4.             // ... Not sure what this does in this context.
5.             base.OnCreateManager ( capacity );
6.             Instance instance = new Instance () ;
7.  
8.             // ... Create a single entity without world?
9.             Entity entity = EntityManager.CreateEntity ( typeof (Instance) ) ;
10.          
11.             // ... Set the component value of the instance.
12.             EntityManager.SetComponentData ( entity, instance ) ;
13.  
14.             // ... AddBuffer Structure to the entity?
15.             EntityManager.AddBuffer <SomeBufferElement> ( entity ) ;
16.  
17.             // ... Get base reference from Entity Arrays?
18.             var bufferFromEntity = EntityManager.GetBufferFromEntity <SomeBufferElement> ();
19.  
20.             // ... Get offset reference from bufferFromEntity( entity * sizeof(<SomeBufferElement>))
21.             var buffer = bufferFromEntity [entity];
22.  
23.             // ... Initialize a new SomeBufferElement to handle values.
24.             SomeBufferElement someBufferElement = new SomeBufferElement () ;
25.  
26.             // ... When adding a new buffer, what happen to the struct Instance : IComponentData
27.             // ... public float f ? Where it's used or manipulated ?
28.             someBufferElement.i = 6 ;
29.             buffer.Add ( someBufferElement ) ;
30.             someBufferElement.i = 7 ;
31.             buffer.Add ( someBufferElement ) ;
32.  
33.         }

 

To be honest this is default created, when typing

Code (CSharp):

1. protected override void OnCreateManger.

I don't thing actually 'base.' is necessary in this case. It works as well without it.

Yes, that is not a problem. you can create many entities this way.

Self explanatory.

If you want dynamic buffer per entity, where you can extend size as needed, then you need assign created IBufferElementData to entity at some point.

Since we created buffer element and assigned entities to it, we can read it now.
First need get access to all entities data, which have SomeBufferElement.

And now we can access our data with entities, which hold this buffer element, using entity index.

That is irrelevant in this case.
We are not affecting IComponentData when manipulating IBufferElementData.
IComponentData is here just as an example, to better represent creation of entity.
We could equally make empty entity, or make just IComponentData as tag, without value.
I could use instance in job for example, to read / write f value, of given entity.

But instance in this case, allows me select all entities with that component.
Example assumes, entity having Instance, has also buffer data.

 

You should really add a requirecompenenttag to the job to ensure it only executes for entities with the buffer.

 

I haven't used it before.

Something like that?

Code (CSharp):

1. [RequireComponentTag ( typeof (SomeBufferElement ) ) ]
2. public class BufferWithJobSystem : JobComponentSystem
3. { ... }

 

Add it above the IJobProcessComponentData

(in bed otherwise I would have written it up properly)

 

Oh me daft, picked wrong line.
That makes more sense. If correct.

Code (CSharp):

1. [RequireComponentTag ( typeof (SomeBufferElement ) ) ]
2. struct Job: IJobProcessComponentDataWithEntity <Instance>

But don't worry, replay tomorrow, if you can not confirm.
Then I will update scripts.

 

Yep that looks good.

 

Do you think that would be possible to access all entites at once from the enitity manager by IBufferElementData and save or load them into a file or serialize? I use meshInstanceRender (mesh and material) and LocalToWorldMatrix with a fixed count of entities only.

I didnt find a way to serialize or save/load the content of an enititymanger at once.
Any idea about?

 

RotationSpeedRotation . Such a project dos not exist in EntityComponentSystemSamples
So I used SimpleRotation or GalacticConquest/Scripts/System.

The example does not compile for any reasons.
Assets\GameCode\Samples.Common\SimpleRotation\BufferWithJobSystem.cs(76,33): error CS0115: 'BufferWithJobSystem.OnCreateManager(int)': no suitable method found to override.

 

My Apology. I Haven't precised well. I have corrected ( RotationSpeedSystem.cs ) and you found right file anyway.

You are probably using later version of Entity preview, where int in OnCreatemanger is removed.
Replace line with

Code (CSharp):

1. protected override void OnCreateManager ( )

You can iterate through entities on main thread, if you want use EntityManager.
Then store appropriate data, in relevant collection.
Then I think best bet is use SharedComponent, via Hybrid ECS to read data in classic OOP. So you can use standard Unity API as you like.
But there may be better solution?

 

Updated, thx

 

Unity have special utility class for serialization/deserialization worlds.
SerializeUtilityHybrid.Serialize / Deserialize
SerializeUtility.SerializeWorld / DeserializeWorld

 

So, is this in anyway similar to ParallelFor? With your code, I check the index, and it's always 0. Am I missing something? There are two items in the buffer, so shouldn't I be seeing some batching?

 

Similar in the sense, that you multithread per entity chunk, rather per index. Also you can get entity data straight away.

You should see values in buffer. Are you sure you are looking at right entity? Asking just in case, if you got other systems active.

 

Ah, so if I just wanted to make an iteration on an "array" (i.e. buffer) on an entity parallel, this would not be the right solution for me? With ParallelFor, I was able to multi-thread on a NativeArray.

Sorry, I am trying to understand ECS a bit better.

 

It's thread per chunk, not entity.

 

Thank you for correction. I suppose was to vague.

 

To be honest, I never thought on parallel iterating, over the entity buffer array itself. You would be likely having let say 100k entities, for each having buffer array component, with 10k elements each.
Then each array is iterated inside the job, on its own thread, for currently loaded entity data. Therefore, I Process Component Data With Entity.

My understanding is, providing I am correct, normally you should not access same buffer array at the same time, on multiple threads, as I believe accessing same entity data, is illegal operation. This is to avoid race conditions, as far I am concerned.

 

That makes sense, but what about the cases in which you are 100% sure you aren't accessing the same elements? Isn't that the whole point of the index?

For example, I have an array of shorts which act as my blocks (voxelt errain). In my case, that array is ReadOnly, so there shouldn't even be an expectations of illegal operations (the data is then added to a different array).

 

The main difference with IJobParallelFor, is that, you iterate over specific array only. For example fixed size NativeArray.
And the solution for using this approach may be 100% fine.

But if you need to iterate over many arrays, lets say having 100k entities with 10 buffer elements arrays each, or even dynamic range for these buffer (buffer array advantage), you don't want IJobParralelFor for each of small arrays. That would be executed single threaded anyway. So you would go 100k entities single threaded with small array iterations each on single thread.

If you inverse situation that you have one big array of 100k elements, or have 10 entities lets say, with big buffer arrays, then IJobParallelFor may be more suitable, to iterate through arrays.

So you should pick right tool, to right job.

 

Thanks, that clears it up. Looks like I'll be going with the ParallelFor approach.

How would you recommend accessing the NativeArray from an ParallelFor job? Just pass it by reference as a parameter, or use a FixedArray, or something else?

 

FixedArray is obsolete. It has been replaced by BufferArray while ago.

Yes pretty much. And just simply iterate through it.

 

I have updated code to
Unity 2020.1 code (Entities 0.2.0-preview.18)
Please see first post.

Requires
Unity 2020.1.0a15+

Tested with
Burst 1.2.0-preview.10
Collections 0.2.0-Preview.13
Entities 0.2.0-preview.18
Jobs 0.2.1-Preview.3
Mathematics 1.1.0.preview.1

After execution, you should see in console following messages: