How to read SharedComponentData in IJobChunk?

I try to make a chunk iteration with IJobChunk with SharedComponentData. But I found that I can't, because I can't get access to EntityManager in a Job.

https://gist.github.com/SetoKaiba/c323ac39775023d3992132eb4072a6a7

 

You can get the index and work on it later or you can split your jobs and pass the shared data in specifically but you can't get it directly.

You should kind of only consider shared components for work that needs to be done on main thread or grouping.

 

Updated the gist.
But in the deprecated injection style. I can use IJobParallelFor.
And I can use burst compile for it.
Remove the debug string related code.

 

Is there a way to iterate entities in jobs with the same archetype in new style with burst compile?
And also, the old style should be with better performance. Because the entities can be processed parallel.
I think there should be a replacement while deprecate some API.
Or it breaks others work. Please help.

 

Anyone help?

 

By actual design it is not possible to access SCD from inside the job. Because those data are in a managed C# List<object> and job could not access reference type outside of its scope (safely). Even if you know you had stored value type, they are designed to accommodate everything.

However there is a running index in that big list that goes to each item. At most in the job you can know which chunk has which SCD index. (Still useful for filtering even though you could not get its value)

If you know you had stored value type in SCD and want to bring them to the job, then you may hack up a NativeArray of your value type mapped to SCD index also prepared outside the job, and bring them in.

 

So, how to get SCD before the job like the original style?
Could you please give me an example?
I have an example for the old style in the gist.
In the old style, I get the SharedComponentDataArray for all the entities.
But SharedComponentDataArray is deprecated as well.

 

Read docs about ChunkIteration, about GetAllUniqueSharedComponentData and indicies, about SCD indicies in chunk Or just try it, it’s simple, anyway you can’t acces data directly (safe) you only operate indicies and just “keeping in head” that “this index for this data”

 

The problem is that I actually need the data from the SCD in the iteration I'm not just working with the index.
You can read TestInjectionJobComponentSystem.cs in my gist which is actually can be achieve before injection and arrays deprecation.
I'm trying the GetAllUniqueSharedComponentData you say.
But I encounter some problem.

1. It reports this problem. It's strange. using Unity.Collections; is added. And there's no error in VS. The type or namespace name 'NativeList<>' could not be found (are you missing a using directive or an assembly reference?)
2. If it works, there're many useless data if I have Archetype(SCD,DB,CD) and Archetype(SCD only). But I only want to iterate the Archetype(SCD,DB,CD) but the GetAllUniqueSharedComponentData give me all the SCD.

The problem I claim is that the iteration with SCD, CD, DB together is working in previous version with injection. But now, there's no alternative.

 

https://gist.github.com/SetoKaiba/6accd474978e73490ee3990aa8cc8622
For problem 1, I found out the cause. I forget to add the Unity.Collections to asmdef.
But I still encounter problems. Here's the code.

And also, I can't make it persistent and [DeallocateOnJobCompletion].

 

I want to find an alternative way for the TestInjectionJobComponentSystem.cs in current version.

 

I believe that SharedComponentDataArray worked in a job previously is not intended. (It should have throw a runtime error when it see SCDA in a job's public field) As how it works inside SCDA, the chunk iterator grab the SCD master index from the current iterating chunk (safe, just a number) and use that master index to index into managed List holding all SCD in existence. This is a thread accessing reference type and likely cause crash or unexpected behaviour later (even if it looks like it is supported and you could make use of it currently).

What would happen if 2 jobs use SCDA concurrently? What if in the main thread a new SCD entry is added while you are reading from it in the thread? And remember since that List of object is just one big list for all SCD type, any SCD type added/removed would move the list and throw off your read from the job. Not sure if there is a job dependency/completion for SCD or not, in that case it might be safe. But still, pointer access from thread looks very not-ECS. (It goes against many things they said in the design)

With that said in this topic https://forum.unity.com/threads/api-deprecation-faq-0-0-23.636994 UT said you should contribute your use case that became unsupported. Maybe you can copy paste your problem there so the team see it and provide a new solution. But I think without changing the central object list design of SCD it is unlikely that job will goes well with SCD data.

 

It’s never was intended, and UT said about that many times

 

But SCD is read only in jobs. It should not be with concurrent problem. I've post my problem in that thread. But they didn't give me a reply.

 

If so, I think they should remove SCD totally. Because there will be the case SCD and CD have dependency in the component system.

 

SCD is very important in for example hybrid rendering. Because it make the system possible to

- Not having to duplicate meshes and materials stored per entity.
- Make those big data attachable to entity in the first place. You could say that Entity belongs to this or that mesh + material instead of "attach", because the data stays elsewhere.
- Automatically categorize (filter) entities, in other word they get automatic grouping for batch rendering thanks to how SCD can hash itself and know which one is equal (even with reference type data).
- With that hashing function you never have to fear about duplicate SCD. The problem of this kind of sharing is that "do I have to fetch the old data (reference or value type) to add to an another entity to be truly shared?" but with SCD, you could add integer 555 to one Entity, then add integer 555 to an another Entity and somehow both Entity are sharing that same integer.

Another good use is the whole sub scene ecosystem. Sub scene and scene section is together just 2 integer. SCD is the correct representation of this since Entity "belongs to" sub scene and scene section, not that Entity "contains" sub scene and scene section. So it is literally "shared".

Both use cases do not need to process SCD data on thread. There are use for SCD even with that limitation.

 

They shouldn’t. SCD lives perfect, for grouping, for splitting, etc.

 

Just a simple thing to claim. If you should never combine CD & SCD together. Why should it keep a SCD index in every chunk.

 

Simple sample - per chunk filtering with early out. You check and process only chunks with specific SCD index. Just read:
https://github.com/Unity-Technologi...aster/Documentation~/shared_component_data.md

 

No. You don't understand what I'm saying. One chunk share only one SCD for each type. If it should be use as you say "never use CD & SCD togehter", there's no need to keep an index to SCD. Just store it outside will be ok. They store it in the chunk. That means a system should be able to process with CD & SCD together. I know the index for every entity is sharing one SCD in a chunk. But I mean if you don't need to treat CD & SCD together, no need to place the index in the chunk. Just another storing structure for entities and SCD. No need to place them together.

 

SCD is for zero memory overhead for instances share the same SCD value. But still, it should be able to cooperate with CD in ComponentSystem.

 

You don't need to show me the doc. I read the doc for many times and understand the SCD design for what. But it still should be able to work with CD. Or it should design isolated from scratch with CD.

 

it’s wrong, you should understood what is SCD. It’s NOT per entity basis it’s only per chunk, entities not share index, only chunk “know” this index.

 

As I see you don’t understand general idea

 

And again they shouldn’t we going around circle

 

No. The entities in the same chunk shares the same SCD. So they share the same index. Just in words different.

 

No. Is there an official claim for never use CD & SCD together?
If it's not intended, they shouldn't be able to filter a ComponentGroup with SC & SCD together. If they design it, they should be able to work together.
If it's not intended, there should be a ComponentGroup and SharedComponentGroup.

 

Ohhhh...no. Index exists ONLY, and ONLY on chunk, and this not “different words”

 

https://forum.unity.com/threads/acc...-in-ijobprocesscomponentdata-possible.614488/
And here's another thread by other to have a requirement to iterate SCD in job as well. And also he's replied by you as well. You should know SCD & CD works together is a general requirement. And also, there'is a working solution in the past.

 

No. It exists only on chunk. But the entities know themselves in this chunk. So it knows what SCD is attached to entity.
I mean that's a different expression for me. Because I'm not an English native people.

 

A belongs Chunk A. There's SCD A index in Chunk A. A is with SCD A, right? I'm just expressing this. I'm not saying the SCD is in memory layout with entity.

 

Entities know their archetype and this not SCD index I stop arguing good luck in your question

 

Entities knows their archetype. But the archetype in chunk contain an index. You can just read the doc.
https://github.com/Unity-Technologi...unk_iteration.md#what-is-contained-in-a-chunk
An array of indices to specific values of SharedComponentData. A chunk cannot hold more than one specific value for a given type of SharedComponentData. (An Archetype is defined by the combination of ComponentData types and SharedComponentData values.)

 

I'm not arguing. I'm waiting the official from Unity to have an explanation on it.
I just to claim, if SCD & CD working together is forbidden.
(An Archetype is defined by the combination of ComponentData types and SharedComponentData values.)
Why should them keep them together. Just a archetype for ComponentData, another one for SharedComponentData.

 

I know docs I work with Unity ECS in production from the beginning. And your link says exact what I say. Entity knows “nothing”

 

And why they should be separated? It’s two part of one thing, every part responces for their side.

 

@Joachim_Ante can you please answer officially

 

@Seto

So something that worked in a previous version, does not work anymore. So far so clear, that happens all the time.

I had a quick look at you git and it was not immediately clear what you are actually trying to achieve- could you elaborate on that? (I mean what’s your use case, not an example of what is not working anymore)

 

For example, I have a SCD1 with property X value A shared for 100 entities. And also a SCD1 with property X value B shared for another 100 entities. So SCD using is desired here, right? And then there's CD with property Y various values for these 200 entities. And CD has property Z.
Now I have to compose a system to work with CD & SCD together. For example in the system, CD.Z = SCD1.X + CD.Y.
This is just a simple example for the use case. It's mostly simplified.

 

Thank you, I think it would be better if you described the real use case and not a simplified version. There might be trade offs (speed, memory, etc) and it would also allow a validation why you actually use SCD

Ps: to me SCD has two benefits
- references to managed objects
- tagging (I.e. SCD with int is like Tag01, Tag02, TagXX, ... XX = value of int) which allows filtering / splits storage


But in general I don’t use them much - possibly check rendering (that’s where everyone uses it)- maybe you find something you need in the source.

 

I don't see @eizenhorn say anywhere that you should never use CD and SCD together. It is just that it is physically impossible to get SCD data from thread. In the main thread it is perfectly possible, so not never.

(On thread access was possible, but possibly not intended by the strange ref tunnel SCDA provided you. Of course I don't know their true intention. It is just that it doesn't look like one when you looking at ECS design as a whole. Why should SCDA be the only one to cheat? You better wait for official response if you don't want a guess, though.)

There is a need to keep an index to SCD per chunk and also must be together. If you store SCD index elsewhere ( neither in chunk data, nor chunk's metadata), how can you determine any entity is associated with which SCD type and value?

Storing it outside is not ok. Which outside? Where is the bigger level of storage that you could still associate something with entities?

I guess you might be wondering why you get to see the "fake" part of SCD (the index) that is useless in job. "What's the point of that?" Then you can think that the number just happen to come with the chunk by default. Since in the main thread, that number is the only thing to get to the real SCD data and also possibly for some fast equality SCD check. But since chunk is linked to the worker thread, you can still see that index that became useless on thread. Something that became useless in an another situation is not strange.

Of course getting real data with the chunk instead of index is impossible since it would allow various loophole involving reference type.

Just because index is with the chunk does not imply you can always process CD and SCD together at all. Because that index need something more to make real use of it, the reference list storage. Reference type is forbidden in thread. You can do it in the main thread. You cannot do it in worker thread because the real data is in main thread. Being together does not mean everything is usable in every situation. Just think that you always got your home key with you. But that does not mean you could access things in your home at any time when you are outside.

If you wanted only official response you should have said so.. I or others can't help you in the first place in that case. Most of us just assumed that the question in forum is free to answer. (excluding something like ETA or the future of API)

I don't know if it is that general or not, but the real requirement is not SCD & CD working together. The requirement is that you want a per entity data to work with per chunk data. SCD is not designed for this.

That requirement is satisfied by "chunk component" not SCD. Chunk component is a real per-chunk data. Maybe you should try it instead. But remember that this time the data is really there, and you lose some shared characteristic of SCD, which is enabled in the most cross-thread compatible by the "holding only index not real data" thing that seems to troubling you currently, since it make you could not get the data on thread. This inconvenience is what make SCD "shared" in the first place.

Using chunk component, you now have to deal with careful chunk management (locking, etc.). You have no worry about that with SCD, since again, it is storing just the index. If you have so much data that your archetype takes multiple chunks, the index is mirrored to all of them easily. For chunk component, your data stays with the old chunk as it should be. It is not shared or linked. It's the real per chunk data. If you want real chunk data it would not be shared. If you want things shared it is got to be via index. You can't have both.

Or do you have better idea for a backend that allows sharing same data for multiple chunks, in the same archetype, than integer index storing, in a way that still not breaking all ECS safety design? The definition of "sharing data" in computer is basically, a pointer. Pointer is a memory address. But to disallow ruleless dereferencing UT probably redesigned it into SCD index. To make the index not usable by itself.

The index is like pointer address but in ECS style. Except the dereferencing is gated by shared data manager to keep jobs easy to program. If you could deref that "SCD pointer" (index) everywhere it is like all the safety feature we built for C# jobs was for nothing.

 

Oh, it’s exactly words which I searched to make analogy

 

And, what about consider the case I mention?
There's a SharedComponentData with property X. For example 10000 entities share value A. Another 10000 entities share value B. This should be the case of using SharedComponentData. Right? It prevents the memory allocation for share objects.
And then, there's a ComponentData with property Y and Z. Because the value is various in all the 20000 entities. ComponentData should be used here. Right? It keeps a sequential memory layout for performance.
And then, here you have to write JobComponentSystem to calcluate ComponentData's Z. CD.Z = SCD.X + CD.Y
I want to use a job for better parallel calculation. And here SharedComponentData is read only.
It's a common use case to calculate with large amount data shared and various. You should not use main thread here for better performance while they're independent. As ECS concerned, performance is the priority, I see no reason to force the calculation to be calculated in main thread in sequence.
What's the intended behavior to implement this?
@5argon @eizenhorn @Joachim_Ante @sngdan

 

Of course you can split the job to iterate first time for the SCD.X index for all the CD.Y. And then another iterate to calculate CD.Z = SCD.X + CD.Y. But it requires a second loop for it. It's again not performance prior.
You can just have an explanation on the case I mention both concerned with SCD to save memory and with CD to save CPU accessing performance.

 

No. I said that I'm waiting based on the answers that can't solve the case I mention.

• Why to use SCD? Use SCD for entities shared the same value to save memory. Maybe thousands of entities in my case.
• Why to use CD? Because the value is various. Sequential memory layout for better performance when accessing them.
• What to do? CD.Z = SCD.Y + CD.Z.
• Maybe a case in real is SCD is a chunk. SCD.X is chunk's position. SCD is a voxel. SCD.X is voxel's local position. SCD.Z is the global position. I'm just to express in an example easy to understand but it's not the actual case. I just mentioning you will with the need to calculate with SCD and CD together in threads other than main thread.

 

Right, this is the use of SCD. However saving memory and getting their value from thread is a completely different thing.

The reason you saved memory is because you have only index with you in the thread. Value sits somewhere else. That somewhere is not so thread friendly, since it is a central ref type list. But it worked out somehow for SCDA for you, intended or not.

The reason I guess it was not intended is because that behaviour cannot be replicated with chunk iteration, the method they say will be the lowest level possible of working with ECS. If it was intended, and I was wrong, maybe they will make SCDA replacement for you.

The next reason that they might not make a replacement for SCDA's list access is because it is not performance by default. You could be jumping to somewhere in memory mid-loop repeatedly on thread even if that works 100%.

I think at this point we all understand your valid use case (shared value to save space plus use it for calculation), there's no need to repeat. It would be very convenient if we could access any read only memory area from a job indeed, deeply we all wanted that. Last year we used to hack the job by accessing static variable from thread too. Why do you think UT patched that out? Because `static` allows exactly your use case of SCD. It allows you to jump to somewhere instantly without protection, allowing you to save space by storing only one real data. I would say the API deprecation is that protection UT tries to do.

We can't have everything. SCD is not the right tool for your case. And maybe, this ref pattern goes against ECS and will not be supported. (The most they could make ref ECS-like is what you currently seeing, just index) When something is not supported you have to change your approach.

Why? Maybe they are not sure that SCD value are truly independent (because of ref central list concern) So they are not providing you SCDA replacement yet. Imagine that let you access something like Material on thread. What would happen? Can you change colors from thread now? (You could do this even if Material itself is read only, thanks to reference type) There's no ECS mechanism to help check that read from worker thread.

ps. Also I tried your old code since I vaguely remembered I had used SCDA in a job long before and it wasn't working and wondering why yours worked. Indeed it is thowing this error.

Code (CSharp):

1. InvalidOperationException: TestJob.SharedComponentData.m_sharedComponentDataManager is not a value type. Job structs may not contain any reference types.
2. Unity.Jobs.LowLevel.Unsafe.JobsUtility.CreateJobReflectionData (System.Type type, Unity.Jobs.LowLevel.Unsafe.JobType jobType, System.Object managedJobFunction0, System.Object managedJobFunction1, System.Object managedJobFunction2) (at /Users/builduser/buildslave/unity/build/Runtime/Jobs/ScriptBindings/Jobs.bindings.cs:96)
3. Unity.Jobs.IJobParallelForExtensions+ParallelForJobStruct`1[T].Initialize () (at /Users/builduser/buildslave/unity/build/Runtime/Jobs/Managed/IJobParallelFor.cs:23)
4. Unity.Jobs.IJobParallelForExtensions.Schedule[T] (T jobData, System.Int32 arrayLength, System.Int32 innerloopBatchCount, Unity.Jobs.JobHandle dependsOn) (at /Users/builduser/buildslave/unity/build/Runtime/Jobs/Managed/IJobParallelFor.cs:50)
5. TestInjectionJobComponentSystem.OnUpdate (Unity.Jobs.JobHandle inputDeps) (at Assets/TestSystem.cs:116)
6. Unity.Entities.JobComponentSystem.InternalUpdate () (at Library/PackageCache/com.unity.entities@0.0.12-preview.24/Unity.Entities/ComponentSystem.cs:608)
7. Unity.Entities.ScriptBehaviourManager.Update () (at Library/PackageCache/com.unity.entities@0.0.12-preview.24/Unity.Entities/ScriptBehaviourManager.cs:83)
8. Unity.Entities.ScriptBehaviourUpdateOrder+DummyDelagateWrapper.TriggerUpdate () (at Library/PackageCache/com.unity.entities@0.0.12-preview.24/Unity.Entities/ScriptBehaviourUpdateOrder.cs:706)
9.  

The job analyzer detects that SCDA field on the job is a reference type and is not allowed. This is a perfectly reasonable error since as I said to use index to exchange for real value we need to access reference type container. (We are not even getting to the value container, the shared manager itself is a class type) It's a good thing that the job could detect this.

Here's my code, which removed just the buffer type. Which version you upgraded from? This is preview 24, the latest before deprecation.

Code (CSharp):

1. using Unity.Collections;
2. using Unity.Entities;
3. using Unity.Jobs;
4. using UnityEngine;
5.  
6. public struct TestComponentData: IComponentData
7. {
8.     public int test;
9. }
10.  
11. public struct TestSharedComponentData : ISharedComponentData
12. {
13.     public int test;
14. }
15.  
16. public class TestInjectionJobComponentSystem : JobComponentSystem
17. {
18.     struct TestJob : IJobParallelFor
19.     {
20.         public ComponentDataArray<TestComponentData> ComponentData;
21.  
22.         public SharedComponentDataArray<TestSharedComponentData> SharedComponentData;
23.  
24.         public void Execute(int i)
25.         {
26.             var msg = "";
27.  
28.             var testComponentData = ComponentData[i];
29.             msg += "ComponentData:" + testComponentData.test + "\n";
30.             msg += "SharedComponentData:" + SharedComponentData[i].test + "\n";
31.             Debug.Log(msg);
32.         }
33.     }
34.  
35. #pragma warning disable 649
36.     public struct Group
37.     {
38.         public ComponentDataArray<TestComponentData> testComponentData;
39.  
40.         [ReadOnly]
41.         public SharedComponentDataArray<TestSharedComponentData> testSharedComponentData;
42.  
43.         public readonly int Length;
44.     }
45.  
46.     [Inject] private Group m_Group;
47. #pragma warning restore 649
48.  
49.     protected override void OnCreateManager()
50.     {
51.         Entity e = EntityManager.CreateEntity();
52.         EntityManager.AddComponentData(e, new TestComponentData { test = 50 });
53.         EntityManager.AddSharedComponentData(e, new TestSharedComponentData
54.         {
55.             test = 555
56.         });
57.     }
58.  
59.     protected override JobHandle OnUpdate(JobHandle inputDeps)
60.     {
61.         var testJob = new TestJob
62.         {
63.             ComponentData = m_Group.testComponentData,
64.             SharedComponentData = m_Group.testSharedComponentData,
65.         };
66.         var jobHandle = testJob.Schedule(m_Group.Length, 64, inputDeps);
67.         return jobHandle;
68.     }
69. }

 

@Seto

I do not have the deep technical understanding of ECS compared to the "power" users engaged on this thread. But I usually find a good solution for problems. At this point however, I feel that @5argon has very well explained the current limitations and possible patterns to address you case. It might not meet your requirements / expectations but in that case you either adopt or hope your use case get's addressed in a future ECS release.

I still feel that the issue you describe is theoretical and not a real life use case (i.e. for a certain amount of entities, you might just replicate the data and waste memory in favor of speed - or safe memory and give up some speed).

This, I did not understand. First, the SCD values already split the data in different chunks. And for those chunks (feed SCD value to job) you can simply schedule a number of parallel jobs, running themselves in parallel. In general, you can get perfectly parallel processing by simply scheduling a few single core IJobs in parallel (parallel job is convenience in that case or if with write concurrency likely even less performant, than splitting the data up and merging it later).

 

edit: the post i quoted above this one disappeared...

This is what I was trying to say in the my previous post:

OP is effectively asking for a NativeHashmap chunk -> SCD Value (edit: or SCDindex -> SCD value, which you can do with EntityManager.GetAllUniqueSharedComponentData -> if you feed this hashmap into the job, you dont need to filter, you just lookup the value by index --- i dont know how the index is assigned, if by archetype, you might just use an array instead of NHM)

Another alternative is to store the data as CD at the cost of memory in case OP does not like to pass in the data & SCD filtering. (it might also be faster)

 

https://github.com/SetoKaiba/EntityComponentSystemSamplesTest
Check this repo, please. @5argon It's running well on my side. Thank you for your reply. And also I replace injection with ComponentGroup here. But both are working. You can have try just comment the code for ComponentGroup and uncomment the code for injection.

I found it's nice to meet you so much. The first time I met you on GitHub for Protobuf IL2CPP issue. And then I found that you're on medium to sharing stuffs and followed you as well. And later I found you on Pixiv and followed you as well.

 

Thank you for the reply. @sngdan Please check the gist I posted in the quote above. It tried to use GetAllUniqueSharedComponentData to solve the problem. But I encounter memory leak problem for TempJob or Permanent NHM. Any idea on it? Thank you.