Search Unity

Question Transport: How to use Steamworks.Net

Discussion in 'NetCode for ECS' started by Deathwing, Dec 19, 2022.

  1. Deathwing

    Deathwing

    Joined:
    May 8, 2013
    Posts:
    25
    Hey everyone!

    I was looking at using Steam Networking (with the Steamworks.NET DLL). (or could also be the native part, doesn't really matter about the library, its more about the topic itself)
    With the GameObject approach it seems quite easy to add it:
    https://github.com/Unity-Technologi...etworking/Runtime/SteamNetworkingTransport.cs

    I was looking at the NetCode for ECS code, but I can't find a way (or a start) to create a custom transport on it. Do I have to make a new INetworkInterface based on the IPC transport type? Really a bit clueless yet :D

    Cheers

    Edit: From what I see, I don't have to create a new pipeline, so I thought about adding a SteamLayer, but this seems impossible (not considering if it makes sense xD) , as all the Layers are internal only. So I guess my way to go is to create a new SteamNetworkInterface, and I basically need to map from the above-mentioned SteamNetworkinigTransport.cs the following things:
    Code (CSharp):
    1. INetworkInterface                                                               vs NetworkTransport
    2. int Bind(NetworkEndpoint endpoint);                                             -> public abstract bool StartClient();
    3. int Initialize(ref NetworkSettings settings, ref int packetPadding);            -> public abstract void Initialize(NetworkManager networkManager = null);
    4. int Listen();                                                                   -> public abstract bool StartServer();
    5. JobHandle ScheduleReceive(ref ReceiveJobArguments arguments, JobHandle dep);    -> public abstract NetworkEvent PollEvent(out ulong clientId, out ArraySegment<byte> payload, out float receiveTime);
    6. JobHandle ScheduleSend(ref SendJobArguments arguments, JobHandle dep);          -> public abstract void Send(ulong clientId, ArraySegment<byte> payload, NetworkDelivery networkDelivery);
    Or do I miss something?

    Edit 2:
    So I tried my luck with a custom NetworkInterface to get it done...
    but I encounter a few problems :)
    Like on the ReceiveJob site, I basically can't add the events to the NetworkEventQueue, as it is internal, but I guess that would be the correct approach...
    iterating over the incoming p2p packages, and converting them into the associated NetworkEvents (disconnect, connect, date, empty)

    On the SendJob (but maybe the problem, for now, exists only because the reading is not clear),
    I iterate over the SendQueue and try to convert them into byte[] to send them as p2p packets, but now comes the problem:
    I can't get the clientId, as the ConnectionRef inside the PacketProcessor is internal, so I can basically not find the correct client to which the data should go.

    Secondly, NetCode for GameObjects does have a NetworkDelivery type on the sending, how can I know in the network interface, what kind of sending type it is? Like Reliable, Unreliable, etc.
     
    Last edited: Dec 19, 2022
    Deathwing, Dec 19, 2022
    #1
    PolarTron likes this.

  2. Deathwing

    Deathwing

    Joined:
    May 8, 2013
    Posts:
    25
    So I spent a couple of (more) hours, and I think it would be achievable, once some of the internal restrictions are lifted :)
    I mean I can receive the pipeline id during the sending (as it is part of the packet payload), but I can't retrieve the pipeline from it due to accessibility restrictions.
    Also, I could get the connectionId (clientId) from the packet, but unfortunately, that part is also internal :) (if the connectionId would be a long it would also be great :D as it could fit in the steamUserId :D )
    Also, I could add the received P2P steam events to the NativeEventQueue, if again, it would be public inside the ReceiveJobArguments :)
     
    Deathwing, Dec 21, 2022
    #2
    Kichang-Kim likes this.

  3. Deathwing

    Deathwing

    Joined:
    May 8, 2013
    Posts:
    25
    So after more research, my old answer is still the way to go, without further help from Unity, these seems not possible :)
     
    Deathwing, Jan 2, 2023
    #3
    PolarTron likes this.

  4. miniwolf_unity

    miniwolf_unity

    Unity Technologies

    Joined:
    Apr 10, 2018
    Posts:
    136
    To solve your problem, which you mention requires changing the internal methods, you can internalize the package and do those changes yourself.

    You can find the package inside: root/Library/PackageCache. Copy the package folder to root/Packages and you can do development on it and change what you may need.
     
    miniwolf_unity, Jan 5, 2023
    #4

  5. Kichang-Kim

    Kichang-Kim

    Joined:
    Oct 19, 2010
    Posts:
    1,011
    Kichang-Kim, Jan 6, 2023
    #5
    te_headfirst likes this.

  6. Deathwing

    Deathwing

    Joined:
    May 8, 2013
    Posts:
    25
    A modified package isn't a pretty good solution, as we want to keep it easy to grab updates of the Netcode package itself, etc. I would rather like a Unity-provided solution, like exposing stuff publicly :D Is that something that can be achieved in the foreseeable future? Or is there a reason why almost everything is internal, and not planned to be adopted by users, even so, the regular Netcode package is not that restrictive on the access level.

    Cheers
     
    Deathwing, Jan 6, 2023
    #6

  7. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    441
    Hi!

    A network interface would indeed be the way to support Steam Networking in the transport. But I think trying to implement it the same way as it is implemented for Netcode for GameObjects (NGO) is the wrong approach.

    Compared to the 
    NetworkTransport
    interface in NGO, 
    INetworkInterface
    is meant for very low-level operations. It was designed as a replacement for socket-type of constructs, not entire transports. (You are correct that perhaps a 
    INetworkLayer
    would be preferable for those cases, but making this public comes with API stability guarantees that we're not willing to provide at this point.)

    So to implement Steam Networking using 
    INetworkInterface
    , you should basically use it in a socket-like manner. This would mean, for example:
    • Don't try to implement any of the network delivery mechanisms in the interface, since that's already taken care of by the pipelines that live in the upper layers. Just send everything unreliably at the Steam level.
    • Don't try to map connection events from Steam Networking into connection events from the transport. Let the upper layers of the transport handle that. The Steam Networking events would just become internal implementation details (for example you might need to use them to buffer traffic until the Steam connection is established).
    • Use 
      NetworkEndpoint
      to track connections at the interface level, instead of trying to access the internal connection IDs. While the API of 
      NetworkEndpoint
      may look like it only handles IP addresses and ports, you can actually store anything you want in there (as long as it fits). You could put a Steam ID or anything you want. We do that ourselves when using Unity Relay, where the allocation ID is shoe-horned into the endpoint (look at 
      RelayAllocationId.ToNetworkEndpoint
      ).
    I understand that not all of this is always very practical. We might offer more ways to extend the transport package in the future (e.g. maybe by making 
    INetworkLayer
    public), but we want to do this in a very considerate manner. It might seem simple to "just make it public", but it comes with an important maintenance cost to us (pipelines are already a maintenance burden because they are so open). There are API stability guarantees that come with something being public, and it also means the code needs to handle all sorts of situations we might not have thought about. For example, I'm pretty sure having a network interface generate connection events would break assumptions in other parts of the code.

    Let me know if you have further questions!
     
    simon-lemay-unity, Jan 6, 2023
    #7
    Kmsxkuse likes this.

  8. Deathwing

    Deathwing

    Joined:
    May 8, 2013
    Posts:
    25
    Hey Simon!

    Thanks for the post.

    So I got it already "working" that I can bind/listen all using netcode with the nice endpoint trick you mentioned of wrapping the steam id inside an endpoint :) The initial connect bytes are sent via the SendJob -> SteamNetworking.SendP2PPacket. They are also received by the server in ReceiveJob -> ReadP2PPacket.

    Now I have the problem the client stays in a 'connecting' state, (sending the bytes all the time) and the server is still having 0 connections, as the underlying ConnectionList is still in the connecting state. And unfortunately, I can't do it the way the TCPNetworkInterface / WebSocketNetworkInterface are doing it, as both the ConnectionList class is internal, but also I can't pass the ConnectionList from my interface to the network stack like the other interfaces are doing it.

    I guess I miss something here :D
     
    Deathwing, Jan 9, 2023
    #8

  9. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    441
    If the connections are still in the connecting state but your interface's receive job is correctly receiving the data, my first guess would be that perhaps the receive job is not passing the data up to the next layer (or perhaps not passing it up correctly).

    Basically, you'd want your receive job to do something like this:
    Code (CSharp):
    1. byte[] receivedMsg = ReceiveFromSteam();
    2.  
    3. // Remove any Steam-specific headers from the message, if any.
    4.  
    5. if (!receiveArguments.ReceiveQueue.EnqueuePacket(out var packetProcessor))
    6. {
    7.     receiveArguments.ErrorCode = (int)Error.StatusCode.NetworkReceiveQueueFull;
    8.     return;
    9. }
    10.  
    11. fixed (byte* ptr = receivedMsg)
    12. {
    13.     packetProcessor.AppendToPayload(ptr, receivedMsg.Length);
    14. }
    You should do that for any message received by your interface. This will allow the message to be processed by upper layers like pipelines. In particular this will pass the connection handshake messages to the connection management layer, which will take care of marking the connection as connected and of creating a new connection on the server end.
     
    simon-lemay-unity, Jan 9, 2023
    #9

  10. Deathwing

    Deathwing

    Joined:
    May 8, 2013
    Posts:
    25
    Hey Simon!

    Thanks for the update!

    So I got it partially working :) It's just odd that not ALWAYS the clients are going into the game state after connecting. (Even the host on itself), still trying to figure out why so, but at least Steam packages are sent and received now between them :)
     
    Last edited: Jan 10, 2023
    Deathwing, Jan 9, 2023
    #10
    cristianm_unity and simon-lemay-unity like this.

  11. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    441
    What do you mean by clients not always going into the game state?

    Otherwise it's great that you got it (mostly) working! We were wondering if the API would allow creating a custom network interface for services like Steam. I'm glad to see that it's possible. If you feel like it, I'd really like to read your impressions on the process and what could be improved. (I already know that our documentation is lacking on that front. That's something we wish to improve soon.)
     
    simon-lemay-unity, Jan 10, 2023
    #11