How to read SharedComponentData in IJobChunk?

Why can't you make the native hash map persistant?

Just create it once in OnCreateManager and dispose in OnDestroyManager.
Start of each OnUpdate call .Clear on it

 

I can take a look later, when I am home and might post an example, if your git is not along the lines, I was thinking.

But to me this remains a theoretical exercise - until we have a real example. I am very skeptical that this is an optimal solution to a real use case.

 

@Seto

I am not a big fan looking through code, but I did - my observations:

1. While this works (NativeHashMap<int, TestSharedComponentData>), I think it would be safer to use NativeHashMap<int, T> and T is the blittable value type of the SCD you want to use (i.e. the type of testSharedComponentData.test) --- i.e. ensures .test is not a managed type
2. You are not disposing the NativeHashMap correctly. Unfortunately [DeallocateOnJobCompletion] does not work with NHM. You have to dispose it when the job completes, either after you call complete in the same system, at a barrier later or (untested) in the OnUpdate, before you schedule the job with if (TSCDMap.IsCreated()) TSCDMap.Dispose() --- this should work because I think OnUpdate is not called if the job has not completed. The simplest for you to test is to call .Complete() directly after you schedule the job and then .Dispose()
3. Another option is to make the NHM persistant, but in general it's very similar -> i.e. you would have to clear it, before you schedule the job again, instead of disposing and reallocating it. I usually do it this way though (in particular if capacity can not be pre-calculated)
4. You might want to investigate how SCD index is assigned and in case it is sequential, if you can work with an array + (offset?) instead of a NHM

 

They get assigned just as they come so it might be very sparse.

So far I use a single NativeArray shared between all systems mapping global index to type index. I create multiple views on it so each system can update the SCD type it requires while threads are reading other parts. This is pretty fast. Of course you may not cache the buffer returned by GlobalToLocalBuffer anywhere except in the Jobs returned by OnUpdate as it might get recreated on structural changes.

*Use at your own risk*. I had a heavy use case for this but would probably use a NHM for most stuff as this bypasses the safety system to be able to use the buffer RW in multiple threads. But I hope there will be some API to help using compatible SCDs in jobs in the future.

Spoiler

Code (CSharp):

1. public class SharedComponentIndexSystem : ComponentSystem
2. {
3.     NativeArray<int> globalToLocalScd;
4.  
5.     public unsafe NativeArray<int> GetGlobalToLocalBuffer()
6.     {
7.         if (!globalToLocalScd.IsCreated || globalToLocalScd.Length < EntityManager.GetSharedComponentCount())
8.         {
9.             var newBuffer = new NativeArray<int>(EntityManager.GetSharedComponentCount() * 2, Allocator.Persistent);
10.             if (globalToLocalScd.IsCreated)
11.             {
12.                 newBuffer.Slice(0, globalToLocalScd.Length).CopyFrom(globalToLocalScd);
13.                 globalToLocalScd.Dispose();
14.             }
15.  
16.             globalToLocalScd = newBuffer;
17.         }
18.  
19.         var shadow = NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray<int>(globalToLocalScd.GetUnsafePtr(), globalToLocalScd.Length, Allocator.Invalid);
20. #if ENABLE_UNITY_COLLECTIONS_CHECKS
21.         NativeArrayUnsafeUtility.SetAtomicSafetyHandle(ref shadow, AtomicSafetyHandle.Create());
22. #endif
23.         return shadow;
24.     }
25.  
26.     protected override void OnDestroyManager()
27.     {
28.         if (globalToLocalScd.IsCreated)
29.             globalToLocalScd.Dispose();
30.         base.OnDestroyManager();
31.     }
32.  
33.     protected override void OnUpdate()
34.     {
35.     }
36. }

Used like this:

Code (CSharp):

1.  
2. List<SomeSCD> components;
3. List<int> localToGlobal;
4. SharedComponentIndexSystem scdIndex;
5. protected override void OnCreateManager()
6. {
7.     components = new List<SomeSCD>();
8.     localToGlobal = new List<int>();
9.     scdIndex = World.GetExistingManager<SharedComponentIndexSystem>();
10. }
11.  
12. protected override JobHandle OnUpdate(JobHandle inputDeps)
13. {
14.     var globalToLocalScd = scdIndex.GetGlobalToLocalBuffer();
15.     components.Clear();
16.     localToGlobal.Clear();
17.  
18.     EntityManager.GetAllUniqueSharedComponentData(components, localToGlobal);
19.     for (var i = 0; i < localToGlobal.Count; i++)
20.         globalToLocalScd[localToGlobal[i]] = i;
21. }

Of course you have to pass the actual components too. You could Pin the actual C# List while the jobs are running and pass it aliased by a NativeArray or just copy it, depends on the count of SCD values you have.

 

Hm, I'm pretty sure moving SCD data to a chunk component should work to make SCD data available in burst jobs as long as it's blittable. And it should be much faster than the GetAllSharedComponentValues stuff. It would only need to be done once for each new chunk. I have to try that out tomorrow.

 

I have not worked with chunk components, but yes, those might be a good solution to some of this -

Out of interest, what is your use case - I had a hard time thinking of one? OPs examples where to generic for me (i.e. changing scd frequently would always copy to new chunks & if scd is only used to save memory but not to filter data, why not just use NHM, etc. --- do the entities always stay in the same group? i.e. you could also use a 'Tag' instead of SCD and then assign a chunk component to store the shared data? I think OP had SCD instead of 'Tag' to also re-group)

 

Thank you. NHM persistent is working.
At least, now I can find an alternative. But it still gets more SCD than the filter if some are with SCD & CD, Some are with SCD only while my component system should process with SCD & CD combined. @tertle Any idea on it well? Thank you.

 

@julian-moschuering

I made a quick & dirty test with chunk component and it seems to work.
- This effectively maps chunk - > SCD.value directly, instead of separate SDC.index - > SCD.value lookup table
- You still need SCD for chunk splitting (I am not very familiar with the chunk API, i.e. lock, create, etc... maybe there are alternative ways)
- The Entity Debugger seems not to display chunk components right

 

Nice!

Yes, I don't think that is bad. The SCD is still the owner of the data. Changing the SCD moves the entity to a different chunk which is either new -> Chunk component should be added automatically by some system, or chunk did already exist -> chunk component is already there and is matching the SCD. Sounds like a pretty good match.

But SCD's and they are still linked to the entity. So that is not a problem.

 

This was used in my MaterialPropertyBlock support which stores per material overwrites in SCD's. Batching is done using the SCD index but actual rendering needs the actual component. As might be a lot of index to SCD lookups I used this system for best performance.

 

Yes, I recall, you developed a fork of the old rendering system that supported MPB. I never looked into it, as I just needed color and wrote a very basic DrawMeshInstanced system which only batched transforms (ICD) and colors (ICD) per material (SCD), in blocks of 1023 and then Draw. (no culling, no frozen, etc.)

Are you planning to extend MPB support for the new RenderMeshSystemV2 ? That would be nice, since I am still using my basic system...

 

Code (CSharp):

1.         TSCDMap.Clear();
2.         var chunkArray = m_Group.CreateArchetypeChunkArray(Allocator.Persistent);
3.         for (int i = 0; i < chunkArray.Length; i++)
4.         {
5.             var index = chunkArray[i].GetSharedComponentIndex(testSharedComponentDataType);
6.             var value = chunkArray[i].GetSharedComponentData(testSharedComponentDataType, EntityManager);
7.             TSCDMap.TryAdd(index, value);
8.         }
9.         chunkArray.Dispose();

Finally, here's the best alternative way for my case. Use ComponentGroup to get the SCD index and value. And pass it to the job. It will not contain all the SCD like GetAllUniqueSharedComponentData.

 

why don't you try chunk component data, you are almost there

Code (CSharp):

1.  var tempValue = chunkArray[i].GetSharedComponentData(testSharedComponentDataType, EntityManager);
2.  
3. var value = new TestCunkComponentData {Value = tempValue.Value}; // you need to create an IComponentData to store in the Chunk Component Data
4.  
5. chunkArray[i].SetChunkComponentData<TestCunkComponentData>(GetArchetypeChunkComponentType<TestChunkComponentData>(), value);
6.  

you can read this directly in the job + it ensures it is a blittable type because its ICD

 

So, that's not SCD anymore. You will have more memory overhead if your Value is the same for most entities. SCD is for shared data.

 

Just as my previous mentioning. 10000 entities is of Value 1, 10000 entities is of value 2. For SCD, you will have only two SCD memory allocation. But for CD, you will have 20000 CD memory allocation. That's what SCD is concerned about.

 

Yes, I understand this. I always advocate that depending on the use case you need to find the best solution (i.e. memory, speed, convenience, etc.). Just to clarify, we discussed 3 approaches so far.
1- NHM - SCD value is hashed (1 allocation per index) and reference provided to job
2- CCD - SCD value is stored per chunk (1 allocation per chunk) and directly accessible from job
3- ICD - no SCD, entities do not move chunks as value changes (1 allocation per entity)

So NHM storing SCD will have lowest memory footprint, but I think you misunderstood CCD (chunk component data) with (ICD / CD - component data) - for (2) you need to keep SCD for splitting the chunks and instead of writing it to a NHM, you write it directly to the chunk.

For 2, there is however a chance you do not allocate any additional memory and just use free space in the 16k chunks

 

Yes. Currently NHM is my best solution for now. Sorry to ask again. What's CCD, what's the short form for?

 

It’s per chunk component data. Which mean you have data linked with chunk, not with entity

 

One thing I'm not understanding is why you're so concerned with the memory overhead? This seems like a problem that's being approached from an overly optimised perspective. From what I can gather your intent is to make your SharedComponentData a struct type without any references similar to what is expected from ComponentData. If you're using ChunkComponentData that is of let's say roughly 100 bytes size that's 100 B overhead per 16KB. Why is this such a big problem? Surely if your SharedComponentData is starting to push sizes larger than that it's most likely leveraging memory spaces OUTSIDE of the struct that would push the struct size down to way less than that and that is also starting to go towards doing work in a Job that becomes much more involved to optimise.

Now keep in mind 100 bytes can accommodate 25 float values and it's much more likely that that amount of data requires the jobs to be split up and more than likely the data won't be applicable to EVERY entity that it's attached to. Ultimately I wouldn't care too much about the overhead of using ChunkComponentData. Heck I might even just use ComponentData until I have issues or it actually makes sense to update ALL related entities with the same value. Without seeing a proper use-case for this though with an actual issue you might be having I don't see any way around this other than making peace with not having a Burst capable job because it needs to access the SCD using EntityManager or using the NHM approach.