BenchmarkNet (Stress test for ENet, UNet, LiteNetLib, Lidgren, MiniUDP, Hazel, Photon and others)

@Rexima I don't use libraries in the project that are compiled by other people or automated build services for many reasons (especially for security). Everything should be done in my environment. UNet is an exception for obvious reasons.

 

@toreau Thank you for your support. It was an unexpected surprise.

 

Forge has a nightly build on the GitHub page, there is no need to compile source. Also the source has not been tested against the Roslyn compiler as there were reported issues with building against .NET 3.5 a few months back. To build Forge you can just use the standard VS toolchain on windows or the Mono build chain on other platforms.

 

Thanks for the information, Brent and welcome to the party. I don't use VS for a very long time (something around 5 years)... I'll take a look what I can do, one more time.

 

@nxrighthere add traffic usage to benchmark. This one of the most useful information.

 

I was thinking about it, but it's better to just monitor the loopback (filtered by process ID) with any available protocol analyzer, because I can't measure packet overhead in the application.

 

I finished working on an integration with Photon and Forge. I'm just waiting until Forge developers investigate this issue. Perhaps, I have time to add support for Netcode.io.

@RevenantX I'll implement in the next version the payload flow measurement in Mbps for the server. Something like this:

Code (CSharp):

1. (clientsChannelsCount * (messageLength * sendRate * 2) * 8 / (1000 * 1000)) * 2

I hope it will be enough.

 

I think better will be calculate total sent/received bytes after bench.

 

BenchmarkNet 1.03 has been released.

• Added support for Photon
• Added payload flow measurement for the server
• Fixed sending data after the client drop
• Fixed minor bugs

First post and wiki page updated with the new results of Photon.

Release with support for Forge postponed due to this issue.
Thanks to guys in Forge's Discord for the warm welcome (very friendly people are there ).

As for Photon, it's a quite different networking solution. You need to follow this guide if you want to test the on-premise by yourself. With 1000 clients, Photon was performed not that fast as ENet, but still a good result.

At the moment, I have some problems with Netcode.io and I just need some rest...

 

Poor Unet :'(

 

btw, paypal.me page doesnt work, says
"This recipient accepts PayPal payments only through their website.
To complete this payment, please go to their website and follow their instructions, or email the recipient for instructions."

 

@Driiades Please, give some time to Alex... I believe that with this tool in his arsenal he can improve the UNet, but it requires many changes to the code.

@mgear Yes, PayPal has some restrictions for my country. It looks like I can't do anything with this... Thanks for the information.

 

I finished working on an integration with Netcode.io, but some additional checks are needed. Also, I got some very good ideas for new features. I'm going to implement them in the next release.

 

Nice Photon test, thanks! Seems it does hold up well enough...

 

Yeah, Photon also has a very flexible configuration. I played with the settings a little bit and I really like that I have an advanced control over the server. Other networking solutions do not provide such control and you need to write some of these things manually from scratch while Photon provides them out of the box.

 

Well your hard work helped convince us that Photon is probably the best solution for now (if you agree). I'm doing a moderate load action title with around 8 players in MP for now. I don't have much experience in networking so it's probably for the best?

UNet will probably take a fair amount of time before it can match Photon for reliability and features and performance, what do you think?

Thanks for your work!

 

Photon is a reliable choice, but personally, I prefer ENet. Both solutions are battleforged. One paid and with the closed source code, another absolutely free and you can always change something in its code. ENet has an incredible congestion control, you don't need to do anything if the server becomes overloaded. ENet will be easier to deal with if you don't have much experience in networking. Photon is a more complex but has some examples that may help you to understand how to write the server-side logic of the game and load-balancing. ENet can be used directly inside Unity as the server which gives you unlimited possibilities, Photon is a standalone application which makes things harder.

As for UNet, I think it has a great potential, but if I had just started developing a network game right now, I would prefer ENet. Not only because it has an impressive performance, but also because it's technically a well-designed networking library. That's just only my opinion.

 

@nxrighthere

Netcode.io is not "realiable" library, you mean: "https://github.com/KillaMaaki/ReliableNetcode.NET" ??

I am very corious about result!! ;-)

 

Netcode.io with C# bindings and a reliability layer.

 

Great benchmarks! I hope some optimization will allow uNet to handle few hundreds connection in the future release. One important advantage of uNet is multiple platforms support. Can ENet clients be compiled for android/ios?

 

I'm not familiar with mobile platforms, I've never worked with them. It's better to test it yourself. ENet supports Unix-like operating systems.

 

@aabramychev Don't forget to update to the latest version, I fixed some bugs.

 

@snacktime External tools slows down the performance when I monitor the time spent in GC. This is particularly noticeable under high loads. So, I found a very simple solution:

Code (csharp):

1. private static void GCLatency() {
2.    Stopwatch latency = new Stopwatch();
3.  
4.    while (processActive) {
5.        latency.Restart();
6.        Thread.Sleep(1);
7.        latency.Stop();
8.  
9.        if (latency.ElapsedMilliseconds > 2)
10.            Interlocked.Exchange(ref gcLatency, latency.ElapsedMilliseconds);
11.    }
12. }

After some tests, it looks like it works fine, but I'm not sure if this is a good way to measure GC pauses. The article is here. What do you think about it?

 

...I highly doubt that loop will properly record time spent in GC. Thread.Sleep(1) isn't accurate to 1 ms at all, often sleeping for 5-10 ms on various machines

Edit: obviously depends on how much threads are active in the background, the less the more precise the sleep will be

 

Yeah, you are right, but I can't find another method... And even if it works I don't like it.

 

It's worth noting you can start and stop the GC (ish) in critical regions using GC.TryStartNoGCRegion and GC.EndNoGCRegion. You can also invoke it manually using GC.Collect.

Also, this article seems to suggeset you can get notifications on when it runs which might aid the accuracy of your timing there.

 

Thank you for the great work. Thanks to you, you created a sort of competition that incites the creators, including unity' team to do better. I just had a little question: where did you get the ENet.dll file? I didn't found it in the c# wrapper repository

 

Yea, I played with GC API a few days ago, but it didn't give any positive results. Only the Sustained Low Latency has some interesting impact, but the application becomes a little bit unstable so I reverted changes.

At the moment, I'm trying to find a solution to measure GC latency without affecting performance of the application.

Thanks, I'll take a look.

Thank you. You need to compile the library, follow the instructions here.

 

Ya I found that article too while looking for a way to measure pauses at runtime, and I didn't find anything better. I think it's better then nothing. On a real app I like just measuring a couple of places on the hot path that I know should be deterministic.

GC has been a huge pain point for me in .NET on the server side compared to java. You basically get two modes that are at the far ends of the spectrum while what you really want is something in between for games.

The goal is you trade memory for pause times in a way that best fits the app. Combined with making the GC more or less aggressive in specific generations. On the JVM you have enough control to really define that pattern exactly how you want. For example I tuned it for frequent 1ms-2ms pauses with one 5ms max pause every couple of minutes. And a 20ms max pause around once per hour and manually do a full collect during non peak times. This required a lot of RAM FYI.

That level of control is impossible in .NET right now. You just have to look at other ways to solve the problem. Go horizontal faster, split stuff out into services, etc..

 

Thank you Chris, very informative. I'll postpone this feature for a while.

I need to read a couple more things about GC in .NET. This is also directly related to my future projects and I can't ignore the impact of GC on performance and memory consumption.

 

BenchmarkNet 1.04 has been released.

• Added optional Sustained Low Latency mode
• Optimized Photon client logic
  
• Improved performance
• Fixed minor bugs

I've updated the results for all libraries and added a new tier with 128 clients. Also, you can read about the new option here.

I mentioned earlier that I wanted to integrate the original Netcode.io written by Glenn Fiedler (using C# bindings and a reliability layer). But after the tests, I found that it doesn't support more than 256 connections. As far as I understand, it's unrecommended to change the preprocessor directive to bypass this limitation... Well, at least Netcode.io works absolutely flawless! If someone wants to test it, I can send you a source code.

As for .NET implementation of Netcode.io, it's obviously slower than pure C library. But beyond that, it looks like the library has some problems with connections. As soon as clients start connecting one after another, the server performance degrades earlier than the number of connections reached ten. Thus, I don't see the point of adding support for it.

By the way, I see that people still want to support the project, and I appreciate it. Thank you for your support.

 

So on a related note to optimizing network stuff, if it would be helpful I have a collection of techniques I've put together over the last few years for conserving bandwidth. Some are known but others not so much. Used together you can do things like 500 characters moving/rotating at under 80kbps bandwidth usage, without using tricks that say rely on low tick rates or other things that make it unusable outside of a test case.

I could write it up and hand it off to @nxrighthere to add to his stuff.

 

It would be great!

By the way, I'm reading your threads right now and wonder how I missed all this stuff before...

 

@nxrighthere about bandwith. Can you add something like this: https://stackoverflow.com/questions...ork-bandwidth-usage-of-a-specific-application ? There already usefull methods to measure bandwith usage in .NET. Total sent/received traffic.

 

Sure. I'll take a look what I can do.

 

your pm is closed, but can you consider some crypto-currencies?
(its easy to create deposit address in exchanges, and use them for trading or just keep if value goes up..)

 

How could this happen.

Feel free to contact me via email, I'm always responding.

Thanks for the idea! I completely forgot about it...

 

Well, I tried but it's not working (this code is terrible by the way). I tried to use this API properly, but still nothing. PerformanceCounterCategory just doesn't contain this category.

 

I'm experimenting with RakNet and the old version of Lidgren right now, and I got the results that confuse me... While RakNet is completely useless (I'm glad that Unity deprecated it in favor of UNet), the old Lidgren a bit shocked me.



CPU usage 60%, memory consumption 360 megabytes.
Not perfect, but it's passed the test with 1000 clients while generation 3 can not...

 

Did you add a custom (i.e. plain) RakNet or did you use Unity's integrated RakNet? Must admit that we only target a range of 16 - 30 players per server with our game but Unity's RakNet has been incredibly stable for us (can't say a lot about performance though). Unlike UNET which still can't properly handle big amounts of data and loses data all the time

 

I used the latest version from the master branch with C# bindings generated with Swig.

 

Would be cool to see comparison values for Unity's own implementation vs the others that you already have

 

Sure, but where I can get it?

 

It's in Unity, you don't have to get anything else
Manual: https://docs.unity3d.com/Manual/NetworkReferenceGuide.html
Scripting ref: https://docs.unity3d.com/ScriptReference/Network.html

Easiest way: Add a NetworkView component to a behaviour and in the script and then add a method to a script on that same gameobject that has the RPC attribute set, like this:

Code (CSharp):

1. [RPC]
2. protected void RPC_RawData(byte[] _data, NetworkMessageInfo _sender) {
3. // _data is your own stuff, _sender contains information from the networking layer
4. }

Starting and closing a server:

Code (CSharp):

1. NetworkConnectionError res = Network.InitializeServer (_maxConnections, _serverPort, false);
2. Network.Disconnect (200);

Connecting to a server and disconnecting:

Code (CSharp):

1. Network.Connect (ip, port, password);
2. Network.Disconnect (200);

Code (CSharp):

1. yourGameObject.GetComponent<NetworkView>().RPC ("RPC_RawData", _networkPlayer, _data);

from server to a client identified by _networkPlayer and

Code (CSharp):

1. yourGameObject.GetComponent<NetworkView>().RPC ("RPC_RawData", RPCMode.Server, _data);

from client to server.

That should be most of what you need

Regards,
Chris

 

It's Legacy RakNet, which is not supported anymore by Unity Team.

 

Oh, my... So, it's still in the engine? What a surprise... I thought that it was removed a long time ago. I can't use it unfortunately, BenchmarkNet is a standalone application.

 

Ah, ok, didn't look at the code just thought you were doing this stuff as a Unity application

 

Hm. Why there not all messages was sent?

 

One client was dropped in the process.