Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

GetBufferFromEntity length check.

Discussion in 'Entity Component System' started by Deleted User, Feb 24, 2019.

  1. Deleted User

    Deleted User

    Guest

    Good evening everyone.

    Was following @Antypodish example of DynamicBuffer processing in job.
    https://forum.unity.com/threads/exa...datawithentity-and-ibufferelementdata.584548/

    My question is, is it possible somehow to schedule a job for the entities that actually has an element in their DynamicBuffer. My scenario is a networking, when we have an IComponentData NetworkConnection with a DynamicBuffer of NetworkPackets.

    So the archetype is [NetworkConnection][NetworkPacketsBuffer]

    At the beginning of the frame receiving system adds the packets to the buffer so the other systems can process the data in the buffer. By the end, Collector system clears the entire DynamicBuffer.

    But as I can see in between if we schedule a IJobProcessComponentData<NetworkConnection> it will schedule a job for every connections and the
    GetBufferFromEntity
    even though it is empty and we have no packets in it. which is quite costly.

    So the question is, what would be a good and compact approach to process the NetworkConnections with Buffers that actually have the packets to process?

    Cheers.
     
    Last edited by a moderator: Feb 24, 2019
  2. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,752
    A faster way to poll to check if your buffer has elements would be to have a second command, struct HasElement : IComponentData { Bool value; }
    And set that to true when an element is added and false when you clear (don't add remove component).

    You can pass that in as a param in a job and it is extremely fast to poll elements in burst jobs. You can do 6 figures of entities a fraction of time.

    How many entities do you have that you find GetBufferFromEntity becomes noticeably slow out of interest? It surprises me how quick it usually is and I'm interested at what point I may need to refactor.
     
    Deleted User likes this.
  3. Deleted User

    Deleted User

    Guest

    Hi tertle.

    Thank you for the quick reply.
    That is the good suggestion though. So I will have some sort of IJobProcessComponentDataWithEntity<NetworkConnection, HasElement> and then just do early checkout in the Execute method. However it would be awesome if I could schedule the job and call GetBufferFromEntity only for the NetworkConnections that has that HasElement.

    Or I misunderstood the approach completely?

    Because passing it as a parameter will require getting HasElement Component from a Archetype Query at least.
     
    Last edited by a moderator: Feb 24, 2019
  4. Deleted User

    Deleted User

    Guest

    4000 Entities of connections, each have their Buffer. Capacity of a buffer 256, however there are only up to 2-3 elements in one frame. The part that is slow is actual calling GetBufferFromEntity for all of the 4000 connections and the fact that I schedule a job for all of 4000 entities rather than for the one's that have something in their buffers, in a Non-Burst job it would take more than 1ms. And even though I enable Burst, the performance is pretty much stable, however the more processing systems I will have - the more overhead I will accumulate. So I have to make sure that I schedule a job only for those who have elements in their buffers.
     
    Last edited by a moderator: Feb 24, 2019
  5. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,683
    Maybe better is use SCD for split data to chunks and process only required chunks, it’s much faster than per entity basis. And with burst is huge performance.
     
    Deleted User likes this.
  6. Deleted User

    Deleted User

    Guest

    I think I will revise my architecture and avoid use of DynamicBuffer. Each NetworkPacket could be an individual entity with a reference to the NetworkConnection entity.
    So I can work with IJobProcessComponentData<NetworkPacket> in burst jobs and add/remove/set appropriate components on NetworkConnection.

    Thank you so much for your answers gentlemens.
     
  7. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,752
    It's actually just the fastest way if you don't have ridiculously large amounts of entities. Burst and the optimized memory layout of chunks just seems to eat through this.

    However, I'm going to throw you an idea of something you should probably disregard. It's not something that should be done unless A) you know what you're doing B) you really need performance C) you know what you're doing.

    I'm only posting this because network libraries should be optimized, so if you want absolute performance to be able to filter without GetBufferFromEntity, instead of a dynamic buffer, just store a pointer to a chunk of data in a IComponentData...

    Code (CSharp):
    1.     public unsafe struct NetworkData : IComponentData
    2.     {
    3.         public NetworkConnection Connection;
    4.         public void* Buffer;
    5.         public int Length;
    6.     }
    7.  
    8.     [BurstCompile]
    9.     public struct ProcessConnection : IJobProcessComponentData<NetworkData>
    10.     {
    11.         public void Execute(ref NetworkData network)
    12.         {
    13.             if (network.Length == 0) // filter
    14.             {
    15.                 return;
    16.             }
    17.      
    18.             var buffer = NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray<NetworkPacketsBuffer>(
    19.                 network.Buffer, network.Length, Allocator.Invalid);
    20.  
    21. #if ENABLE_UNITY_COLLECTIONS_CHECKS
    22.             NativeSliceUnsafeUtility.SetAtomicSafetyHandle(ref buffer, AtomicSafetyHandle.Create());
    23. #endif
    24.      
    25.             // do work
    26.         }
    27.     }
    -edit-

    I actually use this method to store mesh data of static meshes so I can access them in a job.

    Code (CSharp):
    1.     /// <summary>
    2.     /// Pointers to mesh components so they can be used in a job.
    3.     /// </summary>
    4.     public unsafe struct MeshPointer : IComponentData
    5.     {
    6.         public void* Vertices;
    7.         public void* Normals;
    8.         public void* Triangles;
    9.  
    10.         public int VerticesLength;
    11.         public int TriangleLength;
    12.     }
    Again you should probably disregard this suggest. I try to avoid posting hacks here.
     
    Last edited: Feb 24, 2019
    Deleted User likes this.
  8. Micz84

    Micz84

    Joined:
    Jul 21, 2012
    Posts:
    447
    Sorry for of topic question. Is it possible to modify mesh data in job this way?
     
  9. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,752
    I don't. I just convert RenderMeshes to MeshPointer (cached of course) which I then use to perform raycasts. I just use dynamic buffers for editable meshes.

    The original system I wrote for mesh manipulation before DynamicBuffers existed kind of did that though. It stored a pointer to a managed array which I could manipulate in a job (actually i was the internal array of a List<T> but that's a different story.) It still requires you to sync the arrays back to the mesh on the main thread.

    However, there is actually a
    Mesh.GetNativeIndexBufferPtr
    https://docs.unity3d.com/ScriptReference/Mesh.GetNativeIndexBufferPtr.html
    but it's a bit of a (huge) pain to use as it varies per platform, the graphics API etc.

    Anyway off-topic, if you want more info just PM me or create another thread.
     
  10. Deleted User

    Deleted User

    Guest

    Yes, I have some sort of accessing data through pointers in a job.

    Code (CSharp):
    1. /*
    2. -------------------------------------------------------
    3. Developer:  Alexander - twitter.com/wobes_1
    4. Date:       23.12.2018 22:39
    5. -------------------------------------------------------
    6. */
    7.  
    8. using ENet;
    9. using System;
    10. using Unity.Entities;
    11.  
    12. namespace ECSNet
    13. {
    14.     public struct NetworkPacket : IComponentData
    15.     {
    16.         public Opcode Opcode;
    17.         public IntPtr Data;
    18.         public Peer Peer;
    19.         public Channel Channel;
    20.  
    21.         public Entity ConnectionEntity;
    22.     }
    23. }

    So I can call later on UnsafeUtility.CopyPtrToStructure(native.ToPointer(), out T output);
    I think the most performant way will be have NetworkPackets as separate entities that will be destroyed by the end of frame.


    Code (CSharp):
    1. /*
    2. -------------------------------------------------------
    3. Developer:  Alexander - twitter.com/wobes_1
    4. Date:       24.02.2019 09:13
    5. -------------------------------------------------------
    6. */
    7.  
    8. using ECSNet;
    9. using System;
    10. using Unity.Jobs;
    11. using UnityEngine;
    12. using Unity.Entities;
    13. using Unity.Burst;
    14. using Unity.Collections;
    15.  
    16. [BurstCompile]
    17. public struct TestPacketProcessJob : IJobProcessComponentData<NetworkPacket>
    18. {
    19.     public void Execute([ReadOnly] ref NetworkPacket networkPacket)
    20.     {
    21.         switch (networkPacket.Opcode)
    22.         {
    23.             case Opcode.TestMessage:
    24.                 {
    25.                     var packet = Memory.ReadPacketData<TestMessage>(networkPacket.Data);
    26.                     break;
    27.                 }
    28.         }
    29.     }
    30. }
    31.  
    32. public class ServerTestReceiveSystem : NetworkJobComponentSystem, IServerSystem, INetworkPacketListener
    33. {
    34.     protected override JobHandle OnUpdate(JobHandle inputDeps)
    35.     {
    36.         Log.Info(string.Format("Process {0} packets on frame {1}", networkPacketGroup.CalculateLength(), Time.frameCount));
    37.  
    38.         return new TestPacketProcessJob().Schedule(this, inputDeps);
    39.     }
    40. }
    Here is some results 4095 connections, 64 tickrate:

    BurstJob



    NonBurst

     
    Last edited by a moderator: Feb 24, 2019
  11. Deleted User

    Deleted User

    Guest

    @tertle , I think I found my problem. My buffer [InternalBufferCapacity] was set up to a high value which made the data to fit only in (N) connections chunks. So I had 4095 chunks (4095 connections) which of course are harder to proceed on the worker threads. Reducing Capacity to 16 packets per frame instead of 128 made my data to fit into much smaller amount of chunks which are of course got faster to proceed on the worker threads.
     
    Last edited by a moderator: Mar 28, 2019
  12. Deleted User

    Deleted User

    Guest

    Hi there. Do you have any idea how am I able to dispose the original NativeArray based on its obtained pointer?

    after I do NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray
    and call Dispose after I done with the converted Array it throws an error about the array being allocated with not a proper allocator.
     
  13. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,683
    You should dispose initial container
     
  14. Deleted User

    Deleted User

    Guest

    How would I? I meant, if the original container was allocated in a method scope and its pointer was obtained. Even when I am deallocating the obtained pointer memory an exception occurs. I should be able to control the container lifetime on other end, not the initial.