Search Unity

Rewriting a system using chunks/queries instead of to-be-deprecated injected group structs

Discussion in 'Data Oriented Technology Stack' started by bac9-flcl, Oct 4, 2018.

  1. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    489
    How do you index the TransformAccessArray in a job together with chunks?

    Code (CSharp):
    1. private struct Job : IJobParallelFor {
    2.     ...
    3.  
    4.     [ReadOnly]
    5.     [DeallocateOnJobCompletion]
    6.     public NativeArray<ArchetypeChunk> chunks;
    7.  
    8.     public TransformAccessArray transformArray;
    9.  
    10.     public void Execute(int index) {
    11.         ArchetypeChunk chunk = this.chunks[index];
    12.         Process(ref chunk);
    13.     }
    14.  
    15.     private void Process(ref ArchetypeChunk chunk) {
    16.         for (int i = 0; i < chunk.Count; ++i) {
    17.             // How to index TransformAccessArray here
    18.         }
    19.     }
    20. }
     
  2. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,432
    Earlier ArchetypeChunk have StartIndex property, but now I can't see that...
    From documentation:
    Code (CSharp):
    1. struct CollectValues : IJobParallelFor
    2. {
    3.     [ReadOnly] public NativeArray<ArchetypeChunk> chunks;
    4.     [ReadOnly] public ArchetypeChunkComponentType<EcsTestData> ecsTestData;
    5.  
    6.     [NativeDisableParallelForRestriction] public NativeArray<int> values;
    7.  
    8.     public void Execute(int chunkIndex)
    9.     {
    10.         var chunk = chunks[chunkIndex];
    11.         var chunkStartIndex = chunk.StartIndex;
    12.         var chunkCount = chunk.Count;
    13.         var chunkEcsTestData = chunk.GetNativeSlice(ecsTestData);
    14.  
    15.         for (int i = 0; i < chunkCount; i++)
    16.         {
    17.             values[chunkStartIndex + i] = chunkEcsTestData[i].value;
    18.         }
    19.     }
    20. }
     
  3. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    489
    Yep, I don't see StartIndex property. I'm using 0.0.12-preview.17.

    I'm thinking of doing something like this, but I'm not sure if it's the right way.

    Code (CSharp):
    1. public void Execute(int index) {
    2.     ArchetypeChunk chunk = this.chunks[index];
    3.     int startIndex = ResolveStartIndex(index);
    4.     Process(ref chunk, startIndex);
    5. }
    6.  
    7. private int ResolveStartIndex(int chunkIndex) {
    8.     int startIndex = 0;
    9.     for(int i = 0; i < chunkIndex; ++i) {
    10.         startIndex += this.chunks[i].Count;
    11.     }
    12.  
    13.     return startIndex;
    14. }
     
    Last edited: Oct 16, 2018
  4. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,432
    @Joachim_Ante where is StartIndex now? I not see any changes in release notes (now I’m away from PC, thus can’t watch previous sources changes)
     
  5. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    489
    I'm also getting this error:

    InvalidOperationException: Trying to get iterator for UnityEngine.Transform but the required component type was not declared in the EntityGroup.
    Unity.Entities.ComponentGroup.GetIndexInComponentGroup (System.Int32 componentType) (at C:/Users/Marnel/AppData/Local/Unity/cache/packages/staging-packages.unity.com/com.unity.entities@0.0.12-preview.17/Unity.Entities/Iterators/ComponentGroup.cs:157)
    Unity.Entities.ComponentGroupExtensionsForComponentArray.GetComponentArray[T] (Unity.Entities.ComponentGroup group) (at C:/Users/Marnel/AppData/Local/Unity/cache/packages/staging-packages.unity.com/com.unity.entities@0.0.12-preview.17/Unity.Entities.Hybrid/Iterators/ComponentArray.cs:17)
    Unity.Entities.ComponentGroupExtensionsForTransformAccessArray.GetTransformAccessArray (Unity.Entities.ComponentGroup group) (at C:/Users/Marnel/AppData/Local/Unity/cache/packages/staging-packages.unity.com/com.unity.entities@0.0.12-preview.17/Unity.Entities.Hybrid/Iterators/TransformAccessArrayIterator.cs:37)

    I think my query is wrong:

    Code (CSharp):
    1. protected override void OnCreateManager() {
    2.     this.group = GetComponentGroup(this.ConstructQuery(null, new ComponentType[] {
    3.         typeof(Static)
    4.     }, new ComponentType[] {
    5.         typeof(Sprite), typeof(UseYAsSortOrder), typeof(Transform)
    6.     }));
    7. }
    8.  
    9. // ConstructQuery() is just a utility method that I created to simplify query creation
    10. public static EntityArchetypeQuery ConstructQuery(this ComponentSystemBase system, ComponentType[] any, ComponentType[] none, ComponentType[] all) {
    11.     return new EntityArchetypeQuery() {
    12.         Any = any ?? Array.Empty<ComponentType>(),
    13.         None = none ?? Array.Empty<ComponentType>(),
    14.         All = all ?? Array.Empty<ComponentType>()
    15.     };
    16. }
     
  6. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,432
    Your component group must have Transform component.
     
  7. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    489
    How though? It's already added in the current query:

    Code (CSharp):
    1. protected override void OnCreateManager() {
    2.     this.group = GetComponentGroup(this.ConstructQuery(null, new ComponentType[] {
    3.         typeof(Static)
    4.     }, new ComponentType[] {
    5.         typeof(Sprite), typeof(UseYAsSortOrder), typeof(Transform)
    6.     }));
    7. }
     
  8. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,432
    Why you do it in OnCreateManager? You must get component group in OnOpdate every time when you want to use this:
    Code (CSharp):
    1. protected override void OnUpdate()
    2. {
    3.     //Get Group
    4.     // Do stuff
    5.     // Do Other stuff
    6.     // Do one more stuff
    7. }
     
  9. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    489
    I don't think so. In TransformSystem, groups are created only in OnCreateManager().

    Anyways, I used this query:

    Code (CSharp):
    1. this.group = GetComponentGroup(typeof(Transform), typeof(Sprite), typeof(UseYAsSortOrder), ComponentType.Subtractive<Static>());
    but I'm getting another error:

    InvalidOperationException: Job.transformArray.m_TransformArray uses unsafe Pointers which is not allowed. Unsafe Pointers can lead to crashes and no safety against race conditions can be provided.
    If you really need to use unsafe pointers, you can disable this check using [NativeDisableUnsafePtrRestriction].
    Unity.Jobs.LowLevel.Unsafe.JobsUtility.CreateJobReflectionData (System.Type type, Unity.Jobs.LowLevel.Unsafe.JobType jobType, System.Object managedJobFunction0, System.Object managedJobFunction1, System.Object managedJobFunction2) (at C:/buildslave/unity/build/Runtime/Jobs/ScriptBindings/Jobs.bindings.cs:96)
    Unity.Jobs.IJobParallelForExtensions+ParallelForJobStruct`1[T].Initialize () (at C:/buildslave/unity/build/Runtime/Jobs/Managed/IJobParallelFor.cs:23)
    Unity.Jobs.IJobParallelForExtensions.Schedule[T] (T jobData, System.Int32 arrayLength, System.Int32 innerloopBatchCount, Unity.Jobs.JobHandle dependsOn) (at C:/buildslave/unity/build/Runtime/Jobs/Managed/IJobParallelFor.cs:50)
    CommonEcs.UseYAsSortOrderGameObjectSystem.OnUpdate (Unity.Jobs.JobHandle inputDeps) (at Assets/CommonEcs/Scripts/Rendering/UseYAsSortOrderGameObjectSystem.cs:127)
     
  10. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,432
    [NativeDisableUnsafePtrRestriction] attribute on TAA field in job
     
  11. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,432
    I'm from mobile sry, In my head I talk about ArchetypeChunkArray :D
     
  12. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    489
    Tried it. Same error.

    I must be missing something. I think I need some kind of ArchetypeChunk.GetTransformAccessArray(). I just couldn't find it. Here's my current job code which feels very wrong:

    Code (CSharp):
    1. private struct Job : IJobParallelFor {
    2.     public ArchetypeChunkComponentType<Sprite> spriteType;
    3.  
    4.     [ReadOnly]
    5.     public ArchetypeChunkComponentType<UseYAsSortOrder> useYType;
    6.  
    7.     [ReadOnly]
    8.     [DeallocateOnJobCompletion]
    9.     public NativeArray<ArchetypeChunk> chunks;
    10.  
    11.     [NativeDisableUnsafePtrRestriction]
    12.     public TransformAccessArray transformArray;
    13.  
    14.     public void Execute(int chunkIndex) {
    15.         ArchetypeChunk chunk = this.chunks[chunkIndex];
    16.         Process(ref chunk, ResolveStartIndex(chunkIndex));
    17.     }
    18.  
    19.     private void Process(ref ArchetypeChunk chunk, int startIndex) {
    20.         NativeArray<Sprite> sprites = chunk.GetNativeArray(this.spriteType);
    21.         NativeArray<UseYAsSortOrder> useYArray = chunk.GetNativeArray(this.useYType);
    22.  
    23.         for (int i = 0; i < chunk.Count; ++i) {
    24.             // Is this even allowed?
    25.             Transform transform = this.transformArray[startIndex + i];
    26.             Vector3 position = transform.position;
    27.             UseYAsSortOrder useY = useYArray[i];
    28.             position.z = position.y + useY.offset;
    29.             transform.position = position; // Modify
    30.  
    31.             Sprite sprite = sprites[i];
    32.             sprite.RenderOrder = -position.z;
    33.             sprites[i] = sprite; // Modify
    34.         }
    35.     }
    36.  
    37.     private int ResolveStartIndex(int chunkIndex) {
    38.         int startIndex = 0;
    39.         for (int i = 0; i < chunkIndex; ++i) {
    40.             startIndex += this.chunks[i].Count;
    41.         }
    42.  
    43.         return startIndex;
    44.     }
    45. }
     
  13. psuong

    psuong

    Joined:
    Jun 11, 2014
    Posts:
    55
    So I was reading this thread and I was wondering about ISharedComponentData if that is also going way. Since this thread mainly focuses on accessing IComponentData using chunks, would getting values for ISharedComponent type still come from the EntityManager using the following function as the data doesn't live in chunks according to the Github Docs?
    Code (csharp):
    1. EntityManager.GetAllUniqueSharedComponentData<T>(List<T> values)
    Or would getting ISharedComponentData change in a different way as well?
     
  14. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,666
    You can get shared components of a chunk

    Code (CSharp):
    1. private readonly List<SharedComponent> _sharedComponents = new List<SharedComponent>();
    2. private readonly List<int> _sharedComponentIndices = new List<int>();
    3.  
    4. // ...
    5.  
    6. _sharedComponentRO = GetArchetypeChunkSharedComponentType<SharedComponent>();
    7. EntityManager.GetAllUniqueSharedComponentData(_sharedComponents, _sharedComponentIndices);
    8.  
    9. var chunks = EntityManager.CreateArchetypeChunkArray(_query, Allocator.TempJob);
    10.  
    11. foreach (var chunk in chunks)
    12. {
    13.         var cvSharedIndex = chunk.GetSharedComponentIndex(_sharedComponentRO);
    14.         var cvIndex = _sharedComponentIndices.IndexOf(cvSharedIndex);
    15.         var sharedComponent = _sharedComponents[cvIndex];
    16. }
    It is a bit of a roundabout though, maybe there's a better way?

    -edit- there is, this is outdated look below
     
    Last edited: Oct 16, 2018
    psuong likes this.
  15. elcionap

    elcionap

    Joined:
    Jan 11, 2016
    Posts:
    87
    Code (CSharp):
    1.  
    2. _sharedComponentRO = GetArchetypeChunkSharedComponentType<SharedComponent>();
    3.  
    4. var chunks = EntityManager.CreateArchetypeChunkArray(_query, Allocator.TempJob);
    5.  
    6. foreach (var chunk in chunks)
    7. {
    8.         var sharedIndex = chunk.GetSharedComponentIndex(_sharedComponentRO);
    9.         var sharedComponent = EntityManager.GetSharedComponentData<SharedComponent>(sharedIndex);
    10. }
    11.  
    []'s
     
    wobes, Sylmerria and psuong like this.
  16. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,666
    Ah there we go. Thanks for that. I suspected I was using an older method.
     
  17. psuong

    psuong

    Joined:
    Jun 11, 2014
    Posts:
    55
    Ah cool, I probably should've looked at the EntityManager a bit more. Thanks!

     
  18. elcionap

    elcionap

    Joined:
    Jan 11, 2016
    Posts:
    87
    Looking at the source, I couldn't dig anything that let you access the hybrid part of a chunk. Every route uses ArchetypeManager, an internal class.
    So, I think you have no other way than calculating the start index by yourself:

    Code (CSharp):
    1.  
    2. struct MoveJob : IJobParallelFor {
    3.     [NativeDisableParallelForRestriction]
    4.     public TransformArrayAccess Transform;
    5.  
    6.     [DeallocateOnJobCompletion]
    7.     public NativeArray<int> StartIndex;
    8.  
    9.     [DeallocateOnJobCompletion]
    10.     public NativeArray<ArchetypeChunk> Chunk;
    11.  
    12.     public ArchetypeChunkComponentType<Velocity> VelocityType;
    13.  
    14.     public float DeltaTime;
    15.  
    16.     public void Execute(int index) {
    17.         var velocities = Chunk[index].GetNativeArray(VelocityType);
    18.    
    19.         for (var entityIndex = 0; entityIndex < chunk.Count; entityIndex++) {
    20.             var transformIndex = StartIndex[index] + entityIndex;
    21.        
    22.             Transform[transformIndex].position += velocities[entityIndex].Value * DeltaTime;
    23.         }
    24.     }
    25. }
    26.  
    27. protected override JobHandle OnUpdate(JobHandle inputDeps) {
    28.     var chunks = m_Group.CreateArchetypeChunkArray(Allocator.TempJob);
    29.     var indices = new NativeArray<int>(chunks.Length, Allocator.TempJob);
    30.     var startIndex = 0;
    31.  
    32.     for (var chunkIndex = 0; chunkIndex < chunks.Length; chunkIndex++) {
    33.         indices[chunkIndex] = startIndex;
    34.    
    35.         startIndex += chunks[chunkIndex].Count;
    36.     }
    37.  
    38.     var moveJob = new MobJob();
    39.  
    40.     moveJob.DeltaTime = Time.deltaTime;
    41.     moveJob.VelocityType = GetArchetypeChunkComponentType<Velocity>(true);
    42.     moveJob.Chunk = chunks;
    43.     moveJob.StartIndex = indices;
    44.     // TODO: Dispose?
    45.     moveJob.Transform = m_Group.GetTransformAccessArray();
    46.  
    47.     return moveJob.Schedule(chunks.Length, 64, inputDeps);
    48. }
    49.  
    I haven't test the code above, but I think should work properly.
     
    Last edited: Oct 17, 2018
    davenirline likes this.
  19. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    489
    I can't get rid of this error:

    InvalidOperationException: Job.transformAccess.m_TransformArray uses unsafe Pointers which is not allowed. Unsafe Pointers can lead to crashes and no safety against race conditions can be provided.
    If you really need to use unsafe pointers, you can disable this check using [NativeDisableUnsafePtrRestriction].

    I already have both NativeDisableParallelForRestriction and NativeDisableUnsafePtrRestriction on the TransformAccessArray field.
     
  20. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,432
    Oh I’m totally forgot - you use IJobParallelForTransform?
     
  21. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    489
    Seems like this page of the documentation has been removed. https://github.com/Unity-Technologi...ster/Documentation/content/chunk_iteration.md

    I'm still remaking my systems into using chunk iteration. I have a system that uses an ISystemStateComponentData to tag a certain shared component that it was already processed by the system. I wanted to select only the entities with shared components that wasn't "tagged" yet. Kind of like using ISystemStateComponentData as an added event handler. However, I fail to find a similar ArchetypeChunk.GetNativeArray() method for a ArchetypeChunkSharedComponentType.

    How do I filter shared components by tag components and still be able to loop through them? I'm tempted to use EntityManager.GetSharedComponentData() to the resolved entities instead but is there a better way? I'm using this component group:

    Code (CSharp):
    1. this.group = GetComponentGroup(this.ConstructQuery(null, new ComponentType[] {
    2.         // None
    3.         typeof(Processed)
    4.     }, new ComponentType[] {
    5.         // All
    6.         typeof(SomeSharedComponent)
    7.     }));
    This is the injection query that I want to replicate:
    Code (CSharp):
    1. private struct Data {
    2.     public readonly int Length;
    3.  
    4.     [ReadOnly]
    5.     public EntityArray Entity;
    6.  
    7.     [ReadOnly]
    8.     public SharedComponentDataArray<SomeSharedComponent> SharedComponent;
    9.  
    10.     [ReadOnly]
    11.     public SubtractiveComponent<Processed> Processed;
    12. }
     
    Last edited: Oct 19, 2018
  22. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,432
    Read release notes. Documentation moved, your page in capsicum: https://github.com/Unity-Technologi...temSamples/blob/master/Documentation/index.md (chuk iteration here - https://github.com/Unity-Technologi.../blob/master/Documentation/reference/index.md)
     
  23. TZ-

    TZ-

    Joined:
    Oct 23, 2017
    Posts:
    26
  24. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    489
  25. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    489
    I have given up on using chunk iteration with TransformAccessArray. I was still able to remove injection, though. I have opted to use ComponentGroup.GetComponentDataArray() instead so that my transform jobs could still work.

    However, ComponentDataArray will be deprecated soon. So my code is not safe.
     
  26. snowcave

    snowcave

    Joined:
    Nov 15, 2018
    Posts:
    3
    Array.Empty isn't valid code for me, am I missing some namespace I have to import or something?

    ETA: Nevermind, found it, System.Array.Empty.
     
    Last edited: Nov 15, 2018