Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Join us on Dec 8, 2022, between 7 am & 7 pm EST, in the DOTS Dev Blitz Day 2022 - Q&A forum, Discord, and Unity3D Subreddit to learn more about DOTS directly from the Unity Developers.
    Dismiss Notice
  3. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

Feature Request Output data from PopEventForConnection to native array/slice instead of a DataStreamReader

Discussion in 'Unity Transport' started by PolarTron, Apr 4, 2022.

  1. PolarTron

    PolarTron

    Joined:
    Jun 21, 2013
    Posts:
    56
    I want to pop an event in a job and store a pointer to the internal sub array of that data so that I can later parse the data in other jobs.

    I cannot use the DataStreamReader with ReadBytes in this case because I am inside a job and cannot allocate temp memory (with ReadBytes) and return that as it's not allowed.
    Passing in a pre-allocated block of memory to the job increases code complexity and I really don't want to do that.

    I've made a tiny modification to NetworkDriver to work around this. It's probably unsafe.

    My ParsePacketsJob
    See how PopEventForConnection now outputs a NativeArray instead of DataStreamReader

    Code (CSharp):
    1. while ((cmd = Driver.PopEventForConnection(Connections[index], out NativeArray<byte> bytes)) != NetworkEvent.Type.Empty)
    2.                 {
    3.                     if (cmd == NetworkEvent.Type.Data)
    4.                     {
    5.                         var reader = new DataStreamReader(bytes);
    6.                         PacketType type = Packets.ReadPacketType(ref reader);
    7.                         reader.SeekSet(0);
    8.                      
    9.                         ReceivedMessages.Add((int) type, new PacketArrayWrapper()
    10.                         {
    11.                             Pointer = bytes.GetUnsafeReadOnlyPtr(),
    12.                             Length = bytes.Length,
    13.                             InternalId = Connections[index].InternalId
    14.                         });
    15.                     }
    16.                     else if (cmd == NetworkEvent.Type.Disconnect)
    17.                     {
    18.                         Debug.Log("Client disconnected from server");
    19.                         Connections[index] = default(NetworkConnection);
    20.                     }
    21.                 }
    Modification of NetworkDriver.cs
    Added


    Code (CSharp):
    1.             public NetworkEvent.Type PopEventForConnection(NetworkConnection connectionId, out NativeArray<byte> bytes)
    2.             {
    3.                 return PopEventForConnection(connectionId, out bytes, out var _);
    4.             }
    5.  
    6.             public NetworkEvent.Type PopEventForConnection(NetworkConnection connectionId, out NativeArray<byte> bytes, out NetworkPipeline pipeline)
    7.             {
    8.                 pipeline = default;
    9.  
    10.                 bytes = default;
    11.                 if (connectionId.m_NetworkId < 0 || connectionId.m_NetworkId >= m_ConnectionList.Length ||
    12.                     m_ConnectionList[connectionId.m_NetworkId].Version != connectionId.m_NetworkVersion)
    13.                     return (int)NetworkEvent.Type.Empty;
    14.  
    15.                 var type = m_EventQueue.PopEventForConnection(connectionId.m_NetworkId, out var offset, out var size, out var pipelineId);
    16.                 pipeline = new NetworkPipeline { Id = pipelineId };
    17.  
    18.                 if (type == NetworkEvent.Type.Disconnect && offset < 0)
    19.                     bytes = m_DisconnectReasons.GetSubArray(math.abs(offset), 1);
    20.                 else if (size > 0)
    21.                     bytes = ((NativeArray<byte>)m_DataStream).GetSubArray(offset, size);
    22.  
    23.                 return type;
    24.             }
     
    Last edited: Apr 4, 2022
  2. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    150
    I understand your use case and will bring it up to the team, but unfortunately I'm not sure this is a change we'll be willing to make. The data stream API serves an important role for us in mediating access to our internal data structures.

    To process data in a separate job, I'd recommend just passing the
    DataStreamReader
    itself to the other job. In your example code, that would mean modifying
    PacketArrayWrapper
    to store a
    DataStreamReader
    instead of a pointer and length. All readers obtained when popping data events are valid until the next update job is scheduled.

    (Also small aside regarding the example code; I would advise against using
    NetworkConnection.InternalId
    . It's not a unique connection identifier, since the internal ID of a connection can be reused after it's closed. I'm not sure why it's part of our public API, honestly. It would be preferable to store
    NetworkConnection
    structures directly. They're meant to be small and usable as value types.)
     
    PolarTron likes this.
  3. PolarTron

    PolarTron

    Joined:
    Jun 21, 2013
    Posts:
    56
    Aha. I now see where things went wrong. I assumed that DataStreamReader was only meant to be used in the current scope in a temporary way. Storing and passing DataStreamReader around feels weird but I now see that it's a more elegant and proper solution to what I'm trying to achieve. Thank you for the response :)
     
  4. PolarTron

    PolarTron

    Joined:
    Jun 21, 2013
    Posts:
    56
    One problem with passing in DataStreamReader to a job is that DataStreamReader.m_bufferPtr produces errors because I am passing an unsafe pointer to the job.

    I cannot use [NativeDisableUnsafePtrRestriction] because it has to be applied on the m_bufferPtr field itself inside of DataStream.cs

    This attribute will not do anything and it will still produce errors about using an unsafe pointer.

    Code (CSharp):
    1.     [BurstCompile]
    2.     public unsafe struct ParsePublicSnapshotJob : IJob
    3.     {
    4.         [NativeDisableUnsafePtrRestriction] public DataStreamReader Reader;
    Using Entities 0.50
     
  5. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    150
    Oh, good catch! I'll file a task on our end to fix this. Thanks for raising this!
     
    PolarTron likes this.