Mixing Entities.ForEach, WithSharedComponentFilter and WithStoreEntityQueryInField

I ran into a weird behavior when trying to mix WithSharedComponentFilter and WithStoreEntityQueryInField
The example below is taken from the documentation with some modification

Code (CSharp):

1. public class ColorCycleJob : SystemBase
2. {
3.     protected override void OnUpdate()
4.     {
5.         List<Cohort> cohorts = new List<Cohort>();
6.         EntityManager.GetAllUniqueSharedComponentData<Cohort>(cohorts);
7.         foreach (Cohort cohort in cohorts)
8.         {
9.             DisplayColor newColor = ColorTable.GetNextColor(cohort.Value);
10.             var allQueryEntities = query.ToEntityArray(Allocator.TempJob);
11.             Entities.WithSharedComponentFilter(cohort)
12.                 .WithStoreEntityQueryInField(ref query)
13.                 .ForEach((Entity entity, int entityInQueryIndex, ref DisplayColor color) => {
14.                      var e = allQueryEntities[entityInQueryIndex];
15.                      color = newColor;
16.                 })
17.                 .Schedule();
18.         }
19.     }
20. }

In the example above the entities returning from the query would belong to the previously ran ForEach, not the current. meaning that entity and e (inside the ForEach body) will never equal each other.

This also created another side effect if I use WithChangeFilter<DisplayColor>() above, no filtering will ever occur, since the shared component query is misaligned.

Is there something I'm missing here?

 

Found a workaround but it's very smelly and feels hacky! If I explicitly call SetSharedComponentFilter with the same cohort, before I call the Entities.ForEach then the query entities would match the entities in the ForEach!

Code (CSharp):

1. query.SetSharedComponentFilter(cohor);

Really need some insight to what's going on here, it feels like a bug!

 

While the EntityQuery is created during OnCreate, the filter is set at the Entities.ForEach, and query.ToEntityArray is being called before that.

 

Hmmm but since it's a "ref" to the query, it should update whenever the ForEach is called no?
for instance if I do "WithStoreEntityQueryInField" on 1 ForEach call, the query that's called before the ForEach actually matches the entities inside the foreach.
However when I have a ForEach for each cohort, the current query and the ForEach are misaligned. Meaning the current ForEach is corresponding with the previous query (first one being 0 entities)

 

The filter is not stored in the query when as @DreamingImLatios says it is created during OnCreate – I stumbled over this a while back: https://forum.unity.com/threads/resolved-issue-with-setsharedcomponentfilter.870484/#post-5729140.

 

I understand that part, however I'm not creating the query on the OnCreate, I'm getting the query from

WithStoreEntityQueryInField 

after the

WithSharedComponentFilter 

which is doing the

SetSharedComponentFilter

internally.

To clarify, I am getting different query for each cohort in the example above, they're just misaligned.
Say I have
cohorts[0] -> run ForEach with WithSharedComponentFilter I get an empty query.
cohorts[1] -> run ForEach with WithSharedComponentFilter I get the query from cohorts[0]
cohorts[2] -> run ForEach with WithSharedComponentFilter I get the query from cohorts[1]

And so on...

 

To clarify this seems to be an issue with the for loop creating some sort of a race condition, for instance if I remove the for loop from the code above, and use

WithSharedComponentFilter 

with a static value (instead of the dynamic one coming from the for loop) the produced query from

WithStoreEntityQueryInField

would match the entities inside the Entities.ForEach call.

Nevermind, seems like

WithSharedComponentFilter

is ignored on the first iteration, say I have a total of 16 entities with DisplayColor component on them, and 6 of those have the Cohort SCD with the "w/e" value set on them (see code below). On the very first OnUpdate

query.ToEntityArray(Allocator.TempJob);

will return 16 entities, however the Entites.ForEach will iterate over 6 entities. From the second OnUpdate call and onwards both will have 6 entities on them.

Code (CSharp):

1.  
2. var allQueryEntities = query.ToEntityArray(Allocator.TempJob);
3. Entities.WithSharedComponentFilter(new Cohort(){ Value = w/e})
4.             .WithStoreEntityQueryInField(ref query)
5.             .ForEach((Entity entity, int entityInQueryIndex, ref DisplayColor color) => {
6.                      var e = allQueryEntities[entityInQueryIndex];
7.                      color = newColor;
8.                 })
9.              .Schedule();

Isn't the whole concept of

WithStoreEntityQueryInField

is to return a query equal to the one inside the Entities.ForEach body, so we can use it (with confidence) with

ToComponentDataArray

or

ToEntityArray

.

 

I will try to say this again. Please spend some time trying to understand it rather than your assumptions. So an Entities.ForEach codegens two separate pieces.

The first it codegens is the creation of the EntityQuery without filters roughly during OnCreate (OnCreateForCompiler to be exact). Why without filters? Because otherwise the filters would have to be known at compile time. That would make .WithSharedComponentFilter useless for your use case of looping through multiple shared component instances.

The second part is the actual Entities.ForEach. This is codegen-ed directly where you write the Entities.ForEach. It applies the filters here, and then it runs an IJobChunk with the logic using captured variables. Therefore, your query.ToEntityArray is happening before the filters are assigned in the particular shared component iteration.

If you still don't get it, think of it this way. The EntityQuery is not 100% valid until WithStoreEntityQueryInField is called.

 

Thank you so much for detailing this, right before I saw your reply I was looking at the codegen in the DOTS compiler (had no idea that was there) what you said earlier and what you're saying now makes complete sense. Of course

WithStoreEntityQueryInField

would work on anything that's not dynamic, and filters would be ignored. I think I was hung up on the documentation more than what makes sense here.

Having said that, with Entities.ForEach using any of the below methods makes sense on it's own, but mixing any of them together makes for an outcome that's kinda unpredictable. Unless you dig deep and understand the codegen that's happening internally.

WithStoreEntityQueryInField

WithSharedComponentFilter

WithChangeFilter

 

So then, if we call SetSharedComponentFilter on the query before running the ForEach, then there is no need to use WithSharedComponentFilter, correct?

That is, the ForEach will use a filtered collection of entities of based on SetSharedComponentFilter and won't override the shared filter we set, right?

 

The codegen of ForEach will never clear the filter unless it has a filter to set. I haven't bothered to experiment with all the different combinations. If you are unsure, try it and look at the disassembly.