Question Alternatives to transfer cyclic received data arrays from monobehaviour to ecs?

I'm struggling with the question of how to transfer cyclic received data of a 3rd party c# dll, which I'm controlling through a Monobehaviour, into the ecs context.

Lets assume the following situation: I have 100 robots, each of them is described by a dozen of parameters, like geometry, joint positions, assigned tool, etc, which I receive over network.
In my Monobehaviour class I receive a new set of data every 16ms. This dataset consists of arrays of floats and arrays of strings.
Then I have multiple ECS systems which are using different parts of this dataset to control entities.

My idea is to have a static DataContainer class with static public fields. The Monobehaviour then writes to the DataContainer class everytime it receives a new dataset. The ECS systems directly read the DataContainer fields and update the corresponding entities.
One disadvantage is, that none of the systems reading from the DataContainer class can be Burst compiled (Unity gives an error that the static fields are not readonly).

Code (CSharp):

1. public static class DataContainer
2. {
3.   public static uint RobotId;
4.   public static NativeHashMap<uint, float3> PositionByRobotId = new NativeHashMap<uint, float3>();
5.   public static NativeHashMap<uint, float3> RotationByRobotId = new NativeHashMap<uint, float3>();
6. }

And inside a system:

Code (CSharp):

1. foreach (RefRW<RobotComponent> robot in SystemAPI.Query<RefRW<RobotComponent>>())
2. {
3.   robot.ValueRW.Id = DataContainer.RobotId;
4. }

I assume, their has to be a better, cleaner, more elegant - and most important - more scalable solution.

 

A potentially really fast way to do this would be to use UnsafeUtility.MemCpy (which has native performance benefits) to copy your raw received data into a DynamicBuffer<SomeByteSizedBufferElementData> when you receive it in a MonoBehavior. And then have a system with Burst jobs do the actual parsing of these dynamic buffers.

 

Thanks, this sounds good. That said, I have a lot to learn on this topic and have trouble to fill the dynamic buffer.

This class is inside the 3rd party library and defines my data.

Code (CSharp):

1. public class DataValue
2. {
3.   public object Value { get; protected set; }
4.   public uint? StatusCode { get; protected set; }
5.   public DateTime? SourceTimestamp { get; protected set; }
6. }
7.  

My code so far:

Code (CSharp):

1. public struct DataValueBufferComponent : IBufferElementData
2. {
3.   // public DataValue Value; // compiler gives error that I can't use generic AddBuffer<T> method
4.   public byte Value;
5. }
6.  
7. public class MyClient : Library.Client // 3rd Party DLL
8. {
9.   private Entity m_brokerEntity;
10.  
11.   public MyClient(/*...*/)
12.     : base(/*...*/)
13.   {
14.      // Create Entity with buffer - works
15.      var entityManager = World.DefaultGameObjectInjectionWorld.EntityManager;
16.      m_brokerEntity = entityManager.CreateEntity();
17.      entityManager.AddBuffer<DataValueBufferComponent>(m_brokerEntity);
18.   }
19.  
20.   public override void NotifyDataChangeNotifications(DataValue[] notifications)
21.   {
22.      var entityManager = World.DefaultGameObjectInjectionWorld.EntityManager;
23.      var dataBuffer = entityManager.GetBuffer<DataValueBufferComponent>(m_brokerEntity);
24.  
25.      //dataBuffer.Add(new DataValueBufferComponent { Value = 1}); // This works
26.            
27.      unsafe
28.      {
29.          // I assume I have to somehow convert the DataValue[] to raw bytes?
30.          fixed (void* sourcePtr = &notifications[0])
31.          {
32.             void* destinationPtr = dataBuffer.GetUnsafePtr();
33.             long sizeInBytes = 0 // how to get size in bytes? notifications.length * sizeof(DataValue) would not work because DataValue has not a fixed size
34.             UnsafeUtility.MemCpy(destinationPtr, sourcePtr, sizeInBytes);
35.           }
36.        }
37.     }
38. }
39.  
40.            
41.  

 

Oh. If you are using a third-party library that already deserializes everything into managed objects, you'll need to come up with some scheme to re-encode that into unmanaged bytestreams again if you want to use the technique I proposed.

As someone fully immersed in this DOD stuff, that interface provided by that third party looks pretty disgusting to work with.

 

Yeah it is because it's an OPC-UA Library. It is opensource and has an Apache License, so in theory I can grab the bytestream and move all the conversion of UAVariants and such things into ECS context. Well, in theory....

Maybe I can do something like this for now (notifications.Length will be <= 100 for the prototype):

The notification.Value can only be a blittable type or a string - maybe I can change the specification so that it has only blittable types. In addition with the DataValue array I get an Id, which can be mapped to the type of the value with the help of a dictionary. I will have one dynamic buffer for each type. Then I can iterate through the notification entries and use the dictionary to write it to the corresponding dynamic buffer with floatBuffer.Add(new FloatBufferComponent....). Or I create an according NativeArray<float> and copy the nativearray with MemCpy to the DynamicBuffer...I have no clue which one would be better.

 

I faced some thread race conditions the last couple of days and have now a working solution (hopefully).
Nevertheless I think there might be much better implementations and I would love to here the input of you guys.

The goal: Assign data of a separate networking thread to my entity components.
My approach: Create a broker entity which collects all data and then distribute this data to individual entities via a job.

My Monobehaviour connecting to a server, using a 3rd party dll:

Code (CSharp):

1. public class OpcUaManager : MonoBehaviour
2. {
3.     private OpcUaClient m_client;
4.     private Entity m_brokerEntity;
5.  
6.     public void Connect()
7.     {
8.         // Create Singleton Entity to function as a broker for the network data
9.         var entityManager = World.DefaultGameObjectInjectionWorld.EntityManager;
10.         m_brokerEntity = entityManager.CreateEntity();
11.  
12.         m_client = new OpcUaClient(/*...*/);
13.         m_client.Connect();
14.     }
15.  
16.     public void Update()
17.     {
18.         // Copy data from m_client's thread to the dynamic buffer here on the main thread to access the data thread safe in jobs
19.         if (m_client != null && m_client.IsConnected)
20.         {
21.             var entityManager = World.DefaultGameObjectInjectionWorld.EntityManager;
22.             var buffer = entityManager.GetBuffer<CyclicConveyorBufferComponent>(m_brokerEntity); // only valid on main thread
23.             buffer.CopyFrom(m_client.CyclicConveyorBufferList.AsArray());
24.         }
25.     }
26. }
27.  
28. public class OpcUaClient : LibUA.Client
29. {
30.     public NativeList<CyclicConveyorBufferComponent> CyclicConveyorBufferList = new NativeList<CyclicConveyorBufferComponent>(Allocator.Persistent);
31.  
32.     // This notification method gets called from another thread
33.     public override void NotifyDataChangeNotifications(uint[] clientHandles, DataValue[] notifications)
34.     {
35.         for (int i = 0; i < clientHandles.Length; i++)
36.         {
37.             var cyclicBuffer = new CyclicConveyorBufferComponent
38.             {
39.                 BeltPosition = notifications[i].Value.ConvertTo<float>(),
40.             };
41.            
42.             if (CyclicConveyorBufferList.Length <= i)
43.             {
44.                 CyclicConveyorBufferList.Add(cyclicBuffer);
45.             }
46.             else
47.             {
48.                 CyclicConveyorBufferList[i] = cyclicBuffer;
49.             }
50.         }
51.     }
52. }

And here is my System with a job to assign the data to each entities component:

Code (CSharp):

1. [BurstCompile]
2. public partial struct ConveyorSystem : ISystem
3. {
4.     [BurstCompile]
5.     public void OnUpdate(ref SystemState state)
6.     {
7.         var buffer = SystemAPI.GetSingletonBuffer<CyclicConveyorBufferComponent>(true);          
8.         var jobHandle = new UpdateConveyorBeltDataJob
9.         {
10.             CyclicConveyorBuffer = buffer
11.         }.Schedule(state.Dependency);
12.     }
13. }
14. /// <summary>
15. /// Assigns the databroker content to the entity component.
16. /// </summary>
17. [BurstCompile]
18. public partial struct UpdateConveyorBeltDataJob : IJobEntity
19. {
20.     [ReadOnly] public DynamicBuffer<CyclicConveyorBufferComponent> CyclicConveyorBuffer;
21.  
22.     public void Execute(ref ConveyorBeltComponent conveyorbelt)
23.     {
24.         if (CyclicConveyorBuffer.Length > conveyorbelt.BufferIndex)
25.         {
26.             conveyorbelt.ActualBeltPosition = CyclicConveyorBuffer[conveyorbelt.BufferIndex].BeltPosition;
27.         }
28.     }
29. }