Networking: feedback and questions

[Edited September 2020]

In the Road to 2021 blog, we committed to a fully-supported solution for networking for Game Objects. This forum will continue the games networking discussion related to Game Objects, and if you are already using the DOTS networking solution - please share your needs and challenges with the team in the DOTS Forum.

Here are a couple of links that should provide relevant information and guidance as you move forward with networking for your game.

• “Choosing the right netcode for your game” - a blog post and research report to help make the best existing solution for your game’s needs - Sept 2020.
• "Connected Games: Real-time Multiplayer" - a Networking 101 talk - Sept 2018.
  

- Brandi House on behalf of the Unity Networking team

 

So I'm not sure if this is the right place for this question but I hope it's a relevant place. Please redirect me if not.

I'm new to multiplayer networking and starting my researching on how to get things setup for my particular use case.

There is a bewildering amount of information out there and most of it directed at the older multiplayer systems.

Starting with a clean slate and the new developments where would you point me to solve this scenario

1. I have a multi device already existing "game" that is used primarily on desktop in WebGL and on mobile iOS and Android in app form. It also exists in standalone exe format and legacy Webplayer - but I don't need to worry about those. I need to add multi-player into this existing ecosystem.

2. In terms of multi-player I need to only setup initially to account for one user as host and many (but only small numbers say max. about 30) of clients that the host needs to communicate with.

3. A specific invitation can be sent by the host to the clients to join a multi-player session. It's not designed to be accessed by anyone anywhere, in general all the players will in fact be in the same physical room.

So this is effectively a closed small group of networked players with no complicated match making required and no real need to worry about latency or distance.

Are the initial new tools working in a peer to peer manner suitable for this situation?
And if so will there be built-in methods to handle the grouping of players using either an "invite" scheme or a join if on the same IP setup scenario.

Where do I start to figure this out?

 

I wonder how suitable will new networking stack be for AR games? Most of them are played on LAN where one of the players hosts the game and others just join it. There are some examples of this done with UNet, such as:

https://github.com/Unity-Technologies/SharedSpheres

Will I be able to recreate something similar with new multiplayer solution?

 

I'd like to comment on the current state of the documentation and getting developers involved with alpha testing. Yes the introductory PingPong example is pretty basic, but I feel we need a little more, such as some of the UNet examples where it demonstrated creating two character controllers and getting them to sync.

The FPS example, in my opinion, right now is overkill for just trying to wrap your mind around the new Networking API. I'm hoping some more examples come soon, because right now I feel like I can't do anything beyond tweaking the PingPong example. For example, what's the best way to serialize my data and associate it with a GameObject? Will there be an API for that or do I need to rely upon some other serialization library?

And a final comment as it relates to the overall coding style in some of the Unity examples I've seen. If Unity really wants to use its samples and code as a means to teach developers how to utilize a library, I would personally find it helpful if the development team adopted more of the normalized C# coding standards. A couple of things on that note that I've seen:

• Things such as nested classes within the same file cause the code to seem bloated and very hard to follow, especially when you're just trying to understand what a class/file is responsible for.
• Class member names with prefixes are redundant and very uncommon in C#. I've only seen them used in Unity. https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/names-of-type-members

 

The best set-up for XR installations is usually a dedicated server onsite, and the new systems already support this for faster local iteration. For now, you may still need to manually enter the server's IP address (or hard-code it in), and in the future we are considering more options for local discovery / server-list type systems. Feedback about specific needs will be helpful as we prioritize what's next.

 

This is good feedback, and we definitely have a lot of room to grow here. We are initially working on figuring out how we convert the FPS sample to a simpler starter kit, and beyond that, we'll keep thinking about how best to balance complexity with supporting "real" game scenarios.

 

I would echo the sentiment of the FPS example being too much. That approach while still valid for some very limited use cases, has gone out of favor over the years as a general approach. Latency is not what it used to be, and responsiveness is more important in most games then accuracy. You handed developers a hammer that will likely be used on a lot of flies and very few nails.

 

I was looking at the Ping example. 800 LOC to send a simple ping and pong message seems insane to me.

Do you expect indie developers to manually serialize data into streams, keeping track of sequence numbers, connections, jobs, etc... just to send a few messages?

It looks like "Performance by default" really means "Usability as an afterthought".

 

Overall my thoughts so far.

BIggest concern is lack of structure. My idea of a good networking system is it supports some type of plugin system. In another thread I suggested pipelines as made popular in Netty, as that allows layering plugins as well on a per channel basis. But regardless of the specific approach, without an intentional and somewhat opinionated structure there will be very little reuse as people just make up their own approaches to extending the system.

Along the same thought patterns, why put the implementation of the snapshot networking into the FPS demo and not in the multiplayer project. There is a lot of code in the FPS demo that IMO belongs in the multiplayer project. Parts that would be reusable in other approaches as well and can and should be generalized.

What it looks like to me is that there was pressure to get something that worked out the door. There are just too many holes in what was released for anything else to make sense really. The red flag to me is that you seldom recover from that when you do it at the system design stage.

 

@timjohansson and @BHouse
Anyone care to comment on my original question?

 

That sounds fine to me, I'm only looking at proof of concept at the moment.
Is there a sample of this using UNet that you can point me to?

 

Thanks for adding some clarity to this. I suppose it wasn't immediately clear to me that this was just an initial feedback release of the low level transport, with more to come for assisting with the API. It's something that confused me and caused a little hesitation about being willing to adopt the new framework.

So this totally makes sense now. Thanks! Do we expect that "utility code" to eventually be incorporated into the FPS example? And will that be an iterative process, as the netcode samples are updated the FPS sample will likewise be changed?

 

Some questions:

• Seems that the plans is to directly use the public structs (BasicNetworkDriver<T>, NetworkConnection, etc) and in the future different INetworkInterface's can be used to provide stuff such as WebSockets or a TCP fallback?
• There's any way to use the DataStreamWriter to measure or clamp the size after it has been written? I need to fill up the packets as closest to MTU as possible
• There are plans to make a INetworkInterface to simulate latency, packet loss, etc?
• Will the multiplayer package at the FPSSample be updated to the latest version?
• Seems that FPSSample had some handy utilities (PacketFragmenter and a empty PacketNotifier), why they were dropped? I've ended up making a PacketNotifier on my own.
• How's the ETA to IPv6? it may be a blocker for Apple AppConnect review.

Thanks!

 

Will the new libraries support peer to peer networking or will they require a separate server?

 

The transport is agnostic (i.e. can be used P2P or DGS), however, beyond that we are not currently focused on a new Relay service or NAT punchthrough, so you will need to find alternatives to this. Also our future higher-level simulation systems (i.e. delta compression, forward prediction, etc) will be focused first on a server-authoritative client-prediction model (i.e. typical FPS / fast-paced game model), which will assume network traffic is centralized (typically a game server).

 

What is "Server Query Protocol" in multiplayer sample project? Is there any documentation about it?

 

I am looking through a code and have a few questions about quality of service. For this UDP transport layer, are there currently built in options for reliable or reliable sequences data? Or is it currently meant only for high passed action games that can afford small amounts of packet loss?

Thanks!

 

We are currently working on features for reliability and sequence numbers that can be included when you need them, so they'll be the next on the list to reach alpha customers.

 

SQP is required to run in the "Game Server Hosting" environment - it provides information to the server management services (like player count and some perf metrics), so it can determine the health and activity of a server. The Hosting service is in alpha, and we are gradually adding users as we continue to stabilize.

 

Thank you for the quick reply that sounds excellent! I am spending the next month of so doing an extensive deep dive into networking technologies. I currently have a prototype game under the HLAPI and want to move to the new system with as few hiccups as possible, so I am very interested in the underlying techniques used to obtain reliability with UDP.

Do you have any technical insight into how this is achieve or methods your team is looking at to achieve this? For the example of a turn based game, from my limited knowledge I believe some messages would to be reliably transmitted such as turn changing and unit movement where 1 move is made each turn (I would envision a packet failing to deliver when a unit is moved on a player's turn would be a substantial problem in this type of game).

Thanks again

 

Excellent! That is exactly what I need

 

Really looking forward to this.

Are there any plans for creating a broadcasting API? Right now it's the only thing I still need from the old HLAPI and I'm already tired of seeing the warnings about them being deprecated.

 

There were some talks about networking on Unite, which for some reason are not published yet:
https://forum.unity.com/threads/unite-la-networking-talks.580375/

When they will be published?

 

I checked Unity 2018.3b and it's all LLAPI are marked as "deprecated". But it seems that new networking API does not have full-feature compared with old LLAPI, like packet fragmentation, reliability, packet order and so on. Will new API be updated to same feature-level to LLAPI in 2018.3 cycle?

 

They've said no, and that they deprecated it just to tell people that its going away in the future, not because theres a viable alternative.

 

Hello, I've been taking a look at the Networking Transport alpha package that is published on github. I was wondering if webGL support for clients is a priority in the roadmap or not yet, since I understand that a different network driver will be needed. Thanks

 

Does new networking stack support custom header for raw UDP packet? It is need for custom packet relaying. Also, is there any information for top-level header structure for new networking stack? Thanks.

 

These actually just got posted yesterday to youtube, and I just updated the initial sticky post with the relevant links!

At Unite LA, we had a heavy focus on networking and sharing more about our current direction. Here are the 3 most relevant to networking:

• "Connected Games: Real-time Multiplayer" - A review of topologies, protocols, and services related to networking, and thoughts about our goals/priorities as a result.
• "Connected Games: New Networking Foundation" - TimJ (our networking lead) gives an overview of the new transport, its goals, and some initial thoughts about how we foresee future networking packages will leverage ECS, Job System, Burst etc to provide the scale and performance required for today's games.
• "FPS Sample Game: Networking Deep Dive" - Peter (our sample game lead) reviews the "reference implementation" of a client-prediction / server-authoritative implementation shown in the FPS sample. This code will inform future generalized systems, and can be used today as an example for devs eager to get started with similar types of games.

As ever - feedback welcome!

 

I know it's still being explored, but in the meanwhile, @MichalBUnity would you be willing to share the simulation socket you talked about in the overview video a couple months ago? It'd be a great thing for new learners to pick apart while playing with the new transport layer.

 

Awesome, thanks!

 

Thanks for the excellent samples. I have successfully separated the ECS code into two separate projects; server project and a client project. I would be helpful going forward if the examples also do this for those of us running dedicated headless Linux servers.

Please consider these in future releases:

• How to authenticate with the server? Stub code
• How to validate each packet is from a valid client? In web world we would receive a token from the authentication process and resend with each future packet. Though this token most likely would be too heavy for real-time networking.
• Elaborate more on the concept of what the DriverList is for and when would we ever have more than one item in the list?
• How to detect inactivity of a client, then drop their connection and remove from them from the list of connections?
• How to detect robot clients, then drop their connection? For example, a user tapes their forward arrow key down while they are at work in order to stay in the server.
• Pure ECS sample for every non-ECS sample
• I also agree that splitting the files into smaller code is easier to understand. For example breaking out the jobs from component code. I understand now, but trying to learn ECS at the same time as learning the new library...
• Don't include multiple topics in the same code base for those who are learning. Work up to the next feature. For example, removing the matchmaking feature from ping. Add it in another sample related to matchmaking. Not everyone needs it.
• Add more comments in the code
• Maybe consider creating community open source projects for each game type so we can help out? Then these can server as starter packs for others. Sure we can create the project in GitHub, but I feel it would be best sponsored Unity's watchful eye. We would love to help!

Thanks for the considerations. We look forward to the next code drop

VR Architect

 

Hi, I am looking at the example and I only found UDP protocol, does the package include TCP as well, or will be available in the near future?

Code (CSharp):

1. using UdpCNetworkDriver = Unity.Networking.Transport.BasicNetworkDriver<Unity.Networking.Transport.IPv4UDPSocket>;

 

Hello, also I found out there is a limitation on maximum size 1400 bytes per send, what is the suggested way to send a message larger than 1400 bytes?

Error code:
SocketException: A message sent on a datagram socket was larger than the internal message buffer or some other network limit, or the buffer used to receive a datagram into was smaller than the datagram itself.

 

TCP would be rather helpful for prexisting connections that are not necessarily game bound. I too would like to know if an example were to be made for handling TCP websockets to a secure connection (wss).

 

Unreliable messaging can never send big messages. Suppose you want to send 1MB of data, the hardware cannot send packages that large, so the transport needs to break it up into many small packages and then reassemble the entire message on the other end. This is called "fragmentation". If you are using an unreliable transport, and one of those fragments gets lost, then the other end would not be able to reassemble the entire message. In practice it would actually be highly unlikely that your 1MB will make it across this way, the odds of all fragments making it intact are very low, especially over wifi.

For sending larger packages, you need to have a reliable, fragmented transport. TCP does that automatically. You can send any size message in TCP. The OS does the fragmentation for you, and send the fragments reliably. To do it in UDP, you need to first implement RUDP before you can implement fragmentation, which Unity has not done yet.

 

This ^^^

The big advantages of using RUDP over TCP is you can choose to send data reliably or unreliably on a message by message basis, and have more control over message buffering and what happens when a reliable message is dropped.

As far as sending large messages in practice I've seen that larger sized UDP packets are not reliable to send to some clients due to the routing hardware in use, dropping 100% of UDP packets above certain sizes while allowing smaller packets through. Setting the outgoing packet size so it doesn't get much higher than 500 bytes I've found is the most reliable. The problem is there is a lot of times you will want to send a message containing more than 500 bytes, so fragmentation is important even for relatively small amounts of data.

Spoiler

Most people won't care about this, because they would expect Unity to just make something like this available, but this is basically how I've implemented reliable fragmentation in my own UDP networking solution.

1) First I create a byte array from 1 byte to almost max int bytes containing whatever data I want to send.
2) Based on the max size of a packet I want to use and the headers I will be bundling into the packet, I chop up that message into fragments (usually a little under 500 bytes each), and assign each fragment a fragment ID so I can put it back together in the correct order on the other end.
3) I assign low level message ID numbers to the fragments and a few other bytes of header data and send those out the wire as individual UDP packets.
4) On the receiving end, for reliable messages I send acknowledgement messages back to the sender for each low level message received
5) If the sender didn't receive an acknowledgement for one or more low level messages sent, it resends those packets after a timeout period.
6) The receiver sorts all the messages received by low level message ID (because they can be received out of order), removes any messages with duplicate low level ID's, and holds them until it receives all the message fragments to reconstruct the higher level message.
7) When all the messages have been received I reconstruct the higher level message and send it off to do its thing to whatever script cares about that type of message

 

How could we port existing UNET LLAPI game to the new Multiplayer?

For example, if we plan to release later this year, we might want to migrate our UNET code to the new system to future proof our game

I see 2 problems:

1. The new system is Client/Server and UNET is Peer to Peer via relay
I guess the solution here is for us to write the relay code as the server code? Might we nice if there was one off the shelf for everyone to clone

2. UNET Matchmaking has a Create/List/Join model, the new matchmaker does not. Our game relies on this.
For example, we want one peer to create the match and act as both a client and server, and then offer other clients to select it and join.
I cannot see how this can be implemented with a system which creates/joins automatically on demand. Perhaps with a some custom rules filling in user roles

I appreciate these features may be implemented sometime this year, but you can understand our concern about when this service offers this in production servers if we also plan on a late 2019 release.

Also, on the topic of big messages, would be neat if you included a "Reliable/Sequenced/Fragmented" protocol type (all other transports we use do)