Networking: feedback and questions

@BHouse Would you be able to provide any insight into the current development efforts of the new DOTS Netcode? And by that I mean, are they piloting the project based on any input at all from current game developers, or is it solely an in-house project at this point and developers will get their first look in the fall?

I for one would advocate at least for the former approach, getting developers involved as early in the process as possible. Whether that means a lot of input or limiting input, I think it's the way to ensure that you won't turn away a good majority of those looking forward to using a networking solution coming from Unity.

One thing that I've appreciated from open source projects like Mirror is that the community is strong, and discussions with the developers on Discord has (at least for me) been pleasant and very helpful. Having the same sort of community involvement at the Unity level I think would be a major step forward.

 

Hello,

We are developing a multiplayer application based on the new networking system (https://github.com/Unity-Technologies/multiplayer).

We already have a working server conected with multiple clients, but we are having some problems with package loss.
Is there any example of any implementation of package reliability/acknowledgement?

Just to clarify: What I would like to find is an example of how to get ride of package loss, so for example when the server send a command to all clients and some of them do not receive it correctly, the server resend the command when necessary.

 

Thank you for your fast answer!

We are already using the reliability pipeline, but I'm not sure if we are doing it correctly.

This is our server code:

Code (CSharp):

1.         public void StartServer(int port, int maxConnections)
2.         {
3.             Debug.Log("Starting server, port: " + port);
4.             _serverDriver = new UdpNetworkDriver(new ReliableUtility.Parameters { WindowSize = 32 });
5.             _pipeline = _serverDriver.CreatePipeline(typeof(ReliableSequencedPipelineStage));
6.             NetworkEndPoint endpoint = new NetworkEndPoint();
7.             endpoint = NetworkEndPoint.Parse(IPAddress.Any.ToString(), (ushort)port);
8.  
9.             if (_serverDriver.Bind(endpoint) != 0)
10.                 Debug.Log("Failed to bind to port " + port);
11.             else
12.             {
13.                 _serverDriver.Listen();
14.                 Debug.Log("Server started successfully");
15.  
16.                 ServerStarted?.Invoke();
17.             }
18.  
19.             _clientConnections = new NativeList<NetworkConnection>(maxConnections, Allocator.Persistent);
20.             _driverStarted = true;
21.         }

And client code:

Code (CSharp):

1.         public void StartClient()
2.         {
3.             if (_driverStarted)
4.             {
5.                 Debug.Log("Client driver already started");
6.                 return;
7.             }
8.  
9.             Debug.Log("Starting client");
10.             _clientDriver = new UdpNetworkDriver(new ReliableUtility.Parameters { WindowSize = 32 });
11.             _pipeline = _clientDriver.CreatePipeline(typeof(ReliableSequencedPipelineStage));
12.             _serverConnection = default;
13.             _driverStarted = true;
14.         }

Is it OK?

We are testing our product and what I want to know is if we are doing it right (programming-side).

Another interesting thing would be to know if we are losing packages for debuging purposes. Is it possible track somehow?

Regards,
Andrés Y.

 

Does the new networking system located here: https://github.com/Unity-Technologies/multiplayer work with the newest Unity 2019.3.0b1 that was just released ?

 

Hi, Is there any ETA for packet pre-fragmentation for large chunk of data? (equivalent to LLAPI's QosType.XXXFragmented)

 

hello, will the fps sample receive updates with multiplayer's new reliable udp and client prediction feature?

 

Will the networking system be ready by October of this year? I only see one branch for that repo.

 

Hello,

I'm still having a lot of problems with package loss. I'm working on a project where we have 1 server and multiple clients over a WiFi local network and I'm experiencing so many problems with package never reaching the endpoints.

I'm already working with the reliability pipeline (as I shown in my previous message), but I think it is not enough.

Is there any other way to ensure packages will reach to server/clients?

And another (related) question, is there any way to work with TCP instead of UDP?

Regards,
Andrés Y.

 

@larus If Unity does not support pre-fragmentation for Unity Transport soon, can you provide documentation about pipelines to Unity users? It seems that the only way for implementing fragmentation is using pipelines, but there is no useful documentation.

https://github.com/Unity-Technologi...y.transport/Documentation~/pipelines-usage.md

This is only introduction about Untiy's builtin pipeline, not for creating custom pipelines. Of course, the source code can be resources, but official documentation can boost up developer's understanding I think.

Thanks.

 

After reading again your message, I realised that I was using the reliability pipeline for every communication but I have to use it only for specifics events.

Now I'm tring to implement two different pipelines (reliable and unreliable), but since then the app no longer runs OK and I suspect that it's caused because I'm doing something wrong when using both pipelines from the same driver.

This is my code now for server start:

Code (CSharp):

1.         public void StartServer(int port, int maxConnections)
2.         {
3.             Debug.Log("Starting server, port: " + port);
4.             _serverDriver = new UdpNetworkDriver(new ReliableUtility.Parameters { WindowSize = 32 });
5.             _reliablePipeline = _serverDriver.CreatePipeline(typeof(ReliableSequencedPipelineStage));
6.             _pipeline = _serverDriver.CreatePipeline(typeof(UnreliableSequencedPipelineStage));
7.             NetworkEndPoint endpoint = new NetworkEndPoint();
8.             endpoint = NetworkEndPoint.Parse(IPAddress.Any.ToString(), (ushort)port);
9.  
10.             if (_serverDriver.Bind(endpoint) != 0)
11.                 Debug.Log("Failed to bind to port " + port);
12.             else
13.             {
14.                 _serverDriver.Listen();
15.                 Debug.Log("Server started successfully");
16.  
17.                 ServerStarted?.Invoke();
18.             }
19.  
20.             _clientConnections = new NativeList<NetworkConnection>(maxConnections, Allocator.Persistent);
21.             _driverStarted = true;
22.         }

And this is what I do to send with or without the reliability pipeline:

Code (CSharp):

1.                         if (clientWriter.isReliable)
2.                             _serverDriver.Send(_reliablePipeline, _clientConnections[i], clientWriter.writer);
3.                         else
4.                             _serverDriver.Send(_pipeline, _clientConnections[i], clientWriter.writer);

What I'm doing wrong?

Is there any example of using more than one pipeline at the same time?

Thank you.

 

will pricing be based on concurrent players ? or server resources used? (cpu ram etc), will it be scalable?Will I be able to host client-servers on player machines (with the LLAPI)?

 

Hi all, is it possible to hit the Server Query Protocol with an external service? We are working on our own intelligent server scaling system which needs to know information about the player count inside a game server.

 

Hey guys, just tried out the sample scenes from GitHub. All works very well on PC.

But running NetCube on android, the moving cube from another device (PC) appears choppy.

Is there a setting I have to change?

 

Hi everyone. When will the new complete network system be available?
Is this the link for the alpha version or final version:
https://github.com/Unity-Technologies/multiplayer

Will applications based on old networks continue to work? For example, those that have been developed using unity 2018.1

Is photon 2 a recommended alternative?

Thanks!

 

I too would like to know the status of the new multiplayer package https://github.com/Unity-Technologies/multiplayer

@larus I don't see it listed on the Unity roadmap. I recall reading somewhere that it'd be "release ready" by end of Q1 2020 is that still true?

 

@timjohansson @larus @Joachim_Ante What happend to the NetCode project is it abandoned or postponed or you guys still work on it? No news from the devs for quite a while.

 

@timjohansson Thanks for reply - amazing job so far

 

lookforward to the new features..

 

@timjohansson Thank for the news, what about a dedicated subforum with the next release ? (requestesd by many)

Question remains when is the next entities release

Edit : did not realize I was in the connected game forum wich seem to contains lot of netcode related post. Still connected game is more than just netcode if I'm not mistaken, so a subforom if possible referenced both by the connected game and dots forum would be great IMO.

 

Pls take the survey and ask among other things you are interested in for special NetCode subforum, discord server and more tutorials video especially - thx in advance

 

Would be interesting to learn how you guys rated scalability/performance.
Mirror is the only one tested with ~500 CCU, yet we got 3/5

Edit: just looked at the study:

"Unfortunately, the solution’s core architecture is based on UNet, which had fundamental challenges with achieving high performance and scalability. Primarily for this reason, this solution may not be sufficient for games with high scale needs."

This is a false assumption. UNET being slow is the whole point why we forked it. UNET performance does not imply Mirror performance.

"Unfortunately, HLAPI struggled with a monolithic design that made it hard to optimize performance. Mirror struggles with the same core design, and it uses LINQ libraries heavily to reduce codebase size. In our profiling, the memory allocations and garbage collection spikes could get pretty rough as a result, leading to spikes in performance issues."

Untrue. We do not have hot path allocations if you use Apathy. Telepathy has some, but it still scales to ~500 CCU nonetheless.

"Like HLAPI, there are inherent scale challenges in Mirror. We know of a few games that heavily modified HLAPI to reach 64 players per session, but without significant changes to HLAPI, we haven’t identified many shipped games have succeeded with more than 32, and the vast majority are closer to ~4 players per session."

Your rated Mirror based on projects that use UNET. You should rate it based on benchmarks or on projects that use Mirror, e.g. Population: ONE.

"However, the core design of the code is not ideal for modification and extension (i.e., it’s monolithic and has heavy LINQ library usage)."
Untrue. We don't use LINQ in hot path at all. We use Linq for one-time operations that are not in hotpath, i.e. Weaver.

"Mirror on its own does not support consoles. However, it supports a wide variety of transports that may make it feasible to support console platforms."
Untrue. Mirror runs on all platforms where Unity runs. By your definition, Unity does not support consoles.

Additionally, you also rated all solutions by Stability. Yet there is no mention of test coverage whatsoever, which is arguable the best metric for software stability.

 

Why was none of this relevant information provided in the post? I can't imagine why there is absolutely no evidence to support the findings that were conveyed in the post. Could we see, for instance, the baseline project that was used to test these systems with? Or some profiler results of where and when each system failed benchmarks?

I suppose the blogpost itself is nice if you're a complete beginner looking for the right buzzwords.

- EDIT //
Ah, I missed the link to the 'full report' which led me to a survey form (that no one wants to fill out), then a off-site hosted link which my browser and multiple addons aggressively warned me about visiting.

Maybe host the data somewhere secure and reputable? Like your own domain?

 

If possible add support for big worlds(for example earth size planets etc) since many especially space arcade/exploring etc games support it - also not need to say other game engines support it too. This applies both to dots and that future promised classic unity version of unity network frameworks.

 

I'm disappointed.
Or maybe this is the wrong word. I am Shocked. I'm angry.

Unity, my dear friends, I love you guys, I really do, My company and it's games are made on Unity, and I have done plenty of consultant work for companies making real production multiplayer games. and I can tell you, this blog, and this report is just.. Wrong, I mean.. completely wrong.. outright. misinformation.

Who wrote this blog? who made this research?
Is it anybody that actually knows how to make multiplayer games?
Doesn't seem like it.

I am just perplexed. This document reads like a high school report. I don't want to offend somebody but really. it does.
Somebody has to be harsh and say the truth. as example you use a unknown made up term "DGS", again, no offence to the author, it's just a very entry level paper, which isn't expected. Neither is networking entry level stuff

Sorry for the longread but here are the facts.

TLDR:
- You are comparing completely different things, more like apples and oranges, on arbitrary undefined meaningless statistics.
- The market research doesn't include half of the solutions.
- Some solutions are outright a wrong way how to network your game and will hurt anyone trying to build a game on them.
- The products are not tested on actual real world production games successes.
- Winner is not a winner
- Unity has no viable solution.

Wrong compare
Each solution presented is very different, some don't do much for you more than syncing some variables/RPC's you present it, some take the whole task of networking out of your hands so you can actually just make a fun game.
This is a giant difference. As example your "winner" MLAPI doesn't do much of what you want a library to do for you, Concepts like RPCs are fire and forget, you have no idea which order they come in. Every server auth movement example have to make up for this, by implementing reconciliation... this library doesn't take care about a basic tick-engine, correction, reconciliation, prediction, etc etc etc.
Whilst on the other side of the spectrum Quantum, is a fully deterministic framework where you can literally make a game almost like a single player game, and it's completely networked out of the box, you don't have to think about anything, it just works... and it's amazing too. How is MLAPI better?
The statistics you give stars for in your compare are just silly, it is not defined how you get those stars, what they mean and they are on points that are not even helping your customers choose a library. You have not provided any actual data, just your conclusion with no way to back it up (which basically means its arbitrary).
Also I seen some straight up miss-information and/or lies about some solutions (Quantum, Mirror, etc.)

Leaving out solutions
I understand you have to make a scope for this compare, but from my research and knowledge in the production game field, MLAPI is barely used, whilst Forge was left out, as example. (not that those are good solutions but just saying)

Wrong way to do networked games
I see the options a developer has to network their game in Unity as a trap into a minefield.
Once a developer chose a certain solution, and tries to make his game in it, he usually is stuck in it's (usually wrong) way of doing things. As a industry professional with experience, and in contact with many other industry professionals in the field of networked games I have seen too many projects fail.
Most networking solutions are focused on one thing:
How to pack data and send it and receive it, whilst this is just not how you network a game.
A networked game is all bout the simulation. Let that sink in, SIMULATION.
Everything revolves around that, How do you simulate a game, across devices. so for a player it is fun and smooth.
The way how to pack data, and take care about packet loss or delays, it is things way less important than the simulation.
Exactly in this field, most solutions fall flat, they don't take care about it, you have to build it yourself, so what is the point of the library then at that point?
Networking is way more than having some data on one end and having it on the other end.
Even worse is advising people to use a low level library that uses TCP for it's reliability, ever network engineer knows that TCP data will push UDP data out of the pipe, which is why reliable UDP libraries were designed in the first place.

Real world production games
This is the big/main one for me, What actually should matter is what games have been made on each solution, and what these developers had to do/workaround or make themselves to make their games in a released and working state.

Winner is not a winner
You recommend your users to MLAPI as the best overall, I want to say that this is biased.
The lead developer recently got hired by Unity (this is not mentioned)
There are no launched games or proven to actually work in real world scenarios (both MLAPI itself and the relay and punch and its udp library) Compared to other solutions there is really no community surrounding it.
The library itself is basically a re-hashed version of UNET, which by most is deemed not a good library or the right way to do networked games.

The real winner, is a solution that solves the problems your users have to make a networked game.
hint; it is not MLAPI or Mirror.

So this is what you recommend... a fresh re-hash of your old UNET-architecture with no launched games, not proven to work in real world scenario and no community almost?
Where the author works for you since this month..

Unity has not it's own viable solution
The real reason we are all talking about these things, and we are having libraries to compare is because Unity has not reacted to the market and presented us with a proper networking solution developer in-house, it has failed on this job for 10 years already.
Every solution presented was a wrong way to do networking, from Raknet to UNet and now to the new dots networking, this is not how you network a game.
You guys have to get together, possibly to be harsh, fire the networking team, fire the networking lead, and start over.
Find real industry professionals that have multiplayer production games under their belt, hire them, give them a salary they cannot refuse steal them if you have to.. Let them make an actual multiplayer solution that we can use.
Recently with all the IPO build-up you bought companies that didn't help your customers, like Multiplay, or Vivox you paid millions!, why don't you guys use such money to hire good developers with multiplayer experience.. or as an alternative buy up Exitgames/Photon, integrate it into the engine, they seem to have multiplayer down. That would have been an actual useful acquisition.

Whatever you do, the clock is ticking, Your competition, as example Unreal has a good multiplayer solution build into the engine for years, and developers are not blind, if your choice is to walk this minefield, or roll your own solution, or just go for the competing engine.. think about it.

With love and hope,

Maxim.

 

What you imply here is that we sacrifice performance for code size by using LINQ. This is not the case. I just searched for every use of LINQ in our codebase:

• We use it a lot in tests, the weaver, and a few editor scripts. These are only used during build time and do not affect game performance at all.
• Once in awake in NetworkAnimator
• Once in NetworkManager during initialization, while searching for start positions
• Once in NetworkRoomManager, which is for making a lobby while counting which players are ready to begin playing
• Once in WebSocket transport while querying protocols. This transport is not used by default. It is only invoked once per connection when it is established.

And that is it. Our runtime only uses LINQ in 4 insignificant places. We don't use it anywhere else.

They are either compile-time only or are invoked just once when creating/initializing the object. If we got rid of them, it would not make any difference to any game at all.

We used to have more LINQ, which is perhaps how we got that reputation, but we systematically removed them as they showed during our profiling exercises. Our reliance on LINQ has been greatly exaggerated for a long time.

 

Not sure what your criticism of DOTS Netcode is. Doesn't it focus on handling the "simulation" that you mentioned earlier? It tells you exactly where to put your simulation systems in the update loop in order for prediction or interpolation to work, so users don't have to worry about that

Now, I'm not at all an expert so you guys correct me if I'm wrong, but: while it may not use a networking model that is as automagical as full rollback networking (which Quantum is, I think?), it does have 2 advantages over it:

• better fit for large-scale games like 100-player battle royales due to not having the *extreme* CPU cost on clients and servers of rollback networking. Rollback networking pretty much multiplies the cost of your player simulation by a lot
• you don't have to abandon all Unity APIs (physics, math, navmesh, etc....) and cut yourself off from all third party assets like you have to do in Quantum. Specialized deterministic APIs for everything is a requirement for Quantum to work the way it works, but if Unity can pull off their floating point determinism in DOTS, then this networking model could become possible with the standard DOTS APIs

I do feel like DOTS netcode has a good design for a first netcode solution of several. It seems like if I had to choose one architecture to start with, that would fit the largest amount of use cases, that would probably be it

 

I want to know about one thing,i watched some Tutorials of how to do a multiplayer Game,And they used two Components,Network Identity and Network Transform,but i don't have this two,the truth i have nothing of Network,what i need to do to have this two Components?

 

Those 2 components are part of HLAPI. HLAPI was deprecated and removed from Unity, but it is available as a UPM package.

You can also get those components from Mirror, which is a fork of HLAPI and maintained by the community.

 

I’d love to hear something about normcore in this matter, perhaps compared to the other ones and also considering Vincenzo’s point of view.

 

Normcore is just fluff, it has no good systems at all for synchronization of network state, no rollbacks, no client side prediction server reconcilliation etc. just garbage.

If you want proper networking in your game today, use Photon Fusion. Their new library that is fully released soon.

It has everything you ever wanted to have in a networking library for Unity and more.

 

Any reactions on Microsoft mesh?

 

As the creator of Normcore, I am pretty confident this person has never used Normcore fwiw.

 

thanks for this