Resolved Access to Component Data in jobs

I moved to ECS and I am very happy with it: harder to code, but more reliable, and extremely fast!

I am asking for help, which is exceptional, as I pretty much always find answers in docs and forums (thanks by the way); but here I exhausted all options, and can’t find a satisfactory answer.

The problem is (too) simple; I would like to iterate over entities in a native container through a foreach

Adding a tag is a solution, but not efficient here, as it moves the entities to another chunk, then back once treated by the foreach

something like an array.foreach would be great!

=> not such a big deal as the good old FOR loop will do it in a job, and can be using burst.

PS: maybe (?) DOTS 0.5 will solve this, and in the interval, your help will be appreciated

 

There is likely some new stuff in this space coming in DOTS 0.50. But until then for loops are your best bet.

 

What's the problem of iterating native container in for loop?

 

Sorry for being unclear: my REAL problem is that, apart from the main thread, I can not access data from components !
- commandBuffer are only for writing
and
- use of GetComponentDataFromEntity makes no sense: I have hundreds of thousands of entities and only a few in my array / I didn’t move on to ECS to end-up iterating other the full world while data is right there!
and
- GetComponent<Component>(entity) is not possible in a job

As above, adding a tag is not really efficient as I end up loosing a lot of time for moving entities to a chunk, then back to their original ones, just to avoid a GetComponentDataFromEntity (as it would appear less expensive) but still, I have the entities (from different chunks, as the renderers are different) in an array and I can't believe that get the component of an entity is not possible in a job without iterating over the entire world...

At the beginning of my “train” of jobs, I can use the EntityManager but then, once the jobs, and their dependencies are launched, I am helpless.

I must be missing something here.

 

Yes I presume you're missing something as you can use GetComponent<T>(Entity entity) in a job. It is slower than acessing an Entities data directly in the For.Each lambda though, so should be avoided if possible. Are you using SystemBase Entities.ForEach?

 

Ok, firstly, GetComponent is the same as GetComponentFromEntity AFAIK, just syntax sugar for you to have less boilerplate code.
Secondly there is no magic way to writing logic. The most performant game is the game that have never been written. Iterating through chunks is performant, yes, because of linear access. ComponentDataFromEntity is less performant of course. But unity itself has Transform systems which use ComponentDataFromEntity a lot!
Also i see no problem to get component in a job from different chunks. You just need to construct a proper EntityQuery and implement IJobChunk / IJobEntityBatch and pass this query with ComponentDataHandles of components which you want to grab from chunks.

 

Thank you Charles. I just can't succeed in getting the GetComponent<T>(Entity entity) working in a job, getting a non-static/non-instantiated exception.

Regarding foreach, I use it constantly but, as I said in my message, unless I mark the entities in my array with a tag (which is possible, but slow given changes of chunks), I can't find any way to loop a foreach on an array of entity...

 

Thanks Tony. Indeed, if GetComponent equals GetComponentFromEntity, then I am back to square one.
As you noted, the solution remains centered on query.
Here, out of 100's of 1000's of entities all alike but for their renderer (mesh / 50 different types), I have only let say a 1000 to be treated, all in an array.
If my understand is correct, GetComponentFromEntity loop over ALL entities, thus not being efficient here (even if your point on its use by Unity for its Transform system makes me really wonder if my understanding is correct).
Alternative remains the use of tags but not efficient either.
"Casting" the array into the query is all I need at the end.

 

Tony, you are / right on ! IjobEntityBatch is exactly what I needed !!! You made my day.

The array is pass on in the scheduling / perfect : ScheduleParallel<T>(T, EntityQuery, NativeArray<Entity>, JobHandle)

Joachim Ante himself advised IjobEntityBatch in that thread https://forum.unity.com/threads/wha...es-of-a-query-over-time.1133323/#post-7281172 but for some reason, it didn’t click in my head.

Must be because the documentation https://docs.unity3d.com/Packages/com.unity.entities@0.17/api/Unity.Entities.IJobEntityBatch.html
was using GetComponentDataFromEntity so I overlooked it, sorry.

Again, thank you very much for your guidance, even more so as this processes chunks in parallel, increasing the performance!

I do need to understand GetComponentDataFromEntity better also, as my understanding on this must be wrong.

 

A couple of thoughts.

First off, that API for IJobEntityBatch doesn't work with deferred arrays, meaning you will always have a sync point on the array of entities. So that isn't a perfect solution.

Second, that API may be improved to work with ForEach in the 0.50 release. Or maybe they instead decide to drop it in favor of enabled components which would solve the same problem.

And third, ComponentDataFromEntity does not work the way you think it does. It does not assemble any array or iterate through all entities. It is much better than that. There's a whole bunch of internal dynamic lookup tables to chunks and components, and CDFE navigates these to get to the data it cares about efficiently. If your entity distribution in your array is sparse across chunks, CDFE can actually be faster than IJEB. The error you were getting suggests you weren't quite using the API correctly, so showing code and the full error messages (including corresponding line numbers) will help us troubleshoot whatever weird issue you ran into.

 

Not sure about creating a deferred array in that case, and what are deferred arrays in general: very interested to learn on that.

We are all excited to getting DOTS 0.5 (in few weeks now normally), and yes, either foreach work with arrays and/or component enabling possibility would be significant improvements, definitely solving that.

About comparing CDFE against IJEB, without obviously having your technical level (impressed by your framework by the way!), I cant agree with that IN MY CASE: an array containing pointers thus popping out the right chunks can not be slower than any indexing / dynamic look up system if the entities are pretty much all the same...

 

A good tutorial on IJEB:

Turbo Makes Games offers good videos in general.

 

On CDFE, I am confused.
1) that the point of using it if most of my world entities are alike and my entities to process (in my array) are way less numerous / indeed, CDFE is expensive (Unite talk on that topic

/ see Min 17:20 and if my array has a significant size, not sure that IJEB will be slower (as it gets the chunks, thus closer to the best possible efficiency), not to say about the initial overhead of calling it
2) worse: why not use a foreach sorted on the component (indeed, DreamingImLatios kindly answered that question => dynamic) and/or component tag => CDFE remains a random memory access, which is not fully my case
It thus appears that CDFE is the best tool, but for limited number of entities to process
3) in my case, I would not write to those entities in my job (I do it later in a commandBuffer), which is a good idea as safety can not be guaranteed if you write to CDFE, and/or structural changes to those entities would invalidate the CDFE

 

They are arrays populated in a job (typically as a NativeList) and the job hasn't completed yet but is instead a Dependency.

What in IJEB has an array containing pointers?

A lookup in CDFE does three things:
1) It looks up the archetype map, chunk, and index in chunk of the entity.
2) It looks up the component array in the chunk from the archetype map.
3) It fetches the component value.

(1) cache hits when fetching separate components from the same entity.
(2) cache hits when fetching the same component from a different entity in the same archetype.

IJEB does these:
1) It looks up the chunk and index in chunk of every entity (possibly non-Bursted?) on the main thread.
2) For each consecutive sequence of entities in the array which are also consecutive in the chunk, it creates a batch.
3) For each batch, it looks up the component array in the chunk from the archetype map.

Depending on how many batches you generate and how many components you access, CDFE can definitely win.

Admittedly I am struggling to understand what you are trying to say in your last post with the video, so I won't address those comments.

 

Thanks DreamingImLatios for this detailed explanation. I definitely agree that CDFE can win, and even more will win in most situation, and I will keep that in mind, from your input.
In my case with 300k+ entities with "only" 30 different rendermeshes, thus 10k+ items per group of chunks (with some items being much more numerous than others, here up to 50k in fact),
So lets say my array is 1000 entities, pretty much never consecutive, thus according to your post, roughly 1000 batches also. Who wins in your view? Same question for 100 or 10,000 entities (always non-consecutive, and mostly
I shall try/test both options and maybe use both depending on array's length in the first place.
Also CDFE could be great for another use in my project where I just change the value of a component, thus with no structural change, and then I could be also useful, so you may have helped me twice here!
PS: the video (section 17min 20 sec / adding video with time stamp did'nt work) was because 1) Unity treated CDFE in a video (interesting, can be useful to people reading this) and 2) they say that CDFE is key to entities interaction, as you put emphasis on, but also stating that it can be expensive therefore IJEB can be better sometimes. Again, without your input, I would have discarded it, so thanks for that.

 

If I was going to take a shot in the dark, I would say CDFE is better. But your example is missing a bunch of factors for me to provide any further insight. I did look a little more closely at the code and IJEB does seem to at least use Burst when generating batches on the main thread. So a lot of it is going to depend on your game and where you can afford latencies. And definitely profile.

 

Voila. DOTS 0.50 solved all this:

Code (CSharp):

1. Entities.WithFilter(NativeArray<Entity> entities)

https://forum.unity.com/threads/experimental-entities-0-50-is-available.1253394/
Thanks again for your help, Tony_Max & DreamingImLatios !