BenchmarkNet (Stress test for ENet, UNet, LiteNetLib, Lidgren, MiniUDP, Hazel, Photon and others)

@nxrighthere please add traffic usage) Maybe use some external application that can capture that information. And add to table.

 

All the tools I tried affect the performance, unfortunately... I'm also wanted to monitor the bandwidth, but I still can't find a solution for this.

 

Is there a chance of adding steam SDK's networking capabilities?

It's pretty inconvenient though, it wants a steamID for every client & host. So you'd have to load the game server api on every client to get an anonymous steamID.

 

Nah, too much headaches with Steamworks API. And I can tell you without any benchmarks that Steam networking can't handle more than 64 clients per server with similar setup. It's not suited for this. The server performance will degrade before 48 connections will be reached.

 

BenchmarkNet 1.05 has been released.

• Added support for Neutrino
  
• Increased the server thread priority
  
• Optimized IL code
• Updated LiteNetLib to the latest version
  
• Fixed minor bugs

I've added Neutrino to the results and updated them for all other libraries that are affected by latest changes. If someone doesn't know, you can compare the current version with the previous like this.

Neutrino is an interesting networking library. It's tightly integrated with MessagePack and using it in almost all transmissions. It also reuse instances of messages to minimize GC pressure which is quite effective. This works absolutely wonderful until no more than 256 clients are connected to the server. But after this amount of connections, the performance is highly degraded.

The application is now built using the Roslyn's optimization logic for IL code. You can see the difference here in the part of UNet. I've never used this feature before, but now I think that it's one of the most useful things in Roslyn. For some reasons, I didn't recompile the libraries with this feature, but maybe in the near future it'll happen. Also, after some experiments, I increased the server thread priority by one level to above normal. This change has a beneficial effect.

By the way, PayPal is now working fine and I made a Supporters section in the first post and on GitHub page (some nice guys are already there). Currently, I'm building a server machine that I'm going to use in the development. If you like this project, you can help me with the next one... I'm grateful for any support.

In the last two weeks, I've tested up to 15 different networking libraries. As you can see, not all of them supported due to some issues, bugs and limitations. But I keep looking for more.

 

I found the best way to measure the total bandwidth, but it still takes 10% of CPU power. Well, it seems like I have no choice and I'll run the tests twice for each networking library. One for the performance and one for the bandwidth usage. You know, to avoid any spikes I don't even move my mouse while the benchmark is performing...

So, here is the updated tables.

 

It might be failing tests but at least Hazel's winning on bandwidth right?

 

Yep, it looks like it has the lowest packet overhead: 51 bytes for reliable (3 bytes overhead) and 49 bytes for an unreliable message (1 byte overhead). But the other libraries are using sequencing, packet merging and so on.

ENet


Hazel

 

Sorry,
what is the program that you use in these images? Wireshark?

Thank you for your answer...

 

Nope, I don't like Wireshark.

This is a simple yet powerful protocol analyzer SNPA. To capture the loopback packets, use Tools -> Loopback Monitor, select the process and start capturing.

 

Any news on the test outcome?

 

They are still working on it. I'm in touch with Alex. Perhaps the update will be this week.

 

nxrightthere how did you do your original uNet tests? with HLAPI or just LLAPI? Oh it looks like Low level api then. I found your .cs file it looks like low level api.

 

This is a separate product which is called the UNet Server. This library is very similar to LLAPI, but it works outside of the Unity environment.

 

maybe you should join kcp
https://github.com/libinzhangyuan/reliable_udp_bench_mark/blob/master/bench_mark.md

 

KCP is just a protocol, not a networking library.

 

@nxrighthere so as i expected there is users that doesn't understand what is benchmarked here

So maybe you add some notice? Because I just get a bad reputation...
500 clients + server with 500 connections != just server with 500 connections

 

Yea, sure. I'll add a note about this.

 

Done. I hope that this is enough.

By the way, if anyone is interested, here is the LiteNetLib fork which shows that everything is in your hands.



CPU usage 46%, memory consumption 198 megabytes, bandwidth 313 megabytes.

 

@nxrighthere do you think this is really important for now - to optimize library for synthetic benchmark?)
I have many other things to do. And separately server can handle more connections. In real application 99% of CPU usage will be by game logic and bad code

 

You have your own vision in which direction to improve the project. For me, performance and stability have the highest priority in development. Benchmarking, stress-testing, analyzing and so on... We often talk with Danis. For him, it's a challenge to make the library competitive to lower level solutions, and he achieved good results. In general, as Chris mentioned here, it's extremely difficult to reach 1000 clients on the server in a real scenario. To make this possible, we need something more than just a simple server-side logic written on top of a networking library. It should be a system that will work and withstand high loads. This is what I'm currently working on.

Here's a simplified diagram of the infrastructure of my current project.

 

Now I can confirm that ENet works on mobile platforms. I slightly improved my fork, but the original version should work fine.

 

BenchmarkNet 1.06 has been released.

• Added support for DarkRift
• Added optional Instant mode
• Fixed detection of process failure
• Fixed minor bugs

You can find the results of DarkRift as usual on the wiki page.

I finally got the latest beta version of DarkRift 2, and it actually works quite well! There's still work to be done, and Jamie knows about it. I'm glad that this tool was useful for him, and as a result, he improved several places in the library. Looking forward to the DarkRift release!

 

Hi there, still no Forge support?

 

Nope. No progress on this issue (and others as well) for a month.

 

This is great, thank you for sharing this.

 

TL;DR The test should separate clients and server in different process

I tried to optimize a net library I am working on to use less resources and have a better score with this benchmark, but after thinking about it (I have spent more than a week on this, not working fulltime), I reached the conclusion that the test only makes sense about bandwidth, but not resources (cpu and memory).

Right now it appears to test an executable with lots of interconnected peers with its sockets and buffers, when in a real application you should only have one socket as a client, or one as a server. Even server 2 server communication can be done using that same socket, and in the "worst" case, a secondary socket could be made.Right now this is testing lots of sockets, so instead of for example testing a server with 1000 clients, it tests a 1000 clients server plus 1000 clients.

A solution to this would be to allow the process to be launched as a server, single client or multiple clients, and measure them independently. Given that multiple instances of the process would be needed, and that the benchmark will not be accurate if there is more than one process being measured (it would be the same as before), using a second machine would be a good idea.

Thank you for the library, the problem made me think a lot about how to solve it. I was able to use a single buffer with multiple sockets and it did run fast with Windows, but was painfully slow with Linux. After using one thread per socket with its own buffer (one of many solutions) the problem was no more.

 

Thank you for the review. Which library are you working on? I remember you worked on LiteNetLib and Lidgren, so I guess you are using one of them?

As for the separate processes, yes, I was thinking about it a month ago. I also had a discussion with @aabramychev about the sockets and yes, this is not good at all. But don't forget it's a localhost and for purposes the developers using this tool it more than enough. Using a single process made it possible to determine and eliminate a bunch of bugs, GC pressure, deadlocks, and many other problems in the libraries (there are a lot of them). Yes, it doesn't give accurate information about the resources usage, but all libraries are in almost equal conditions. And I can name a few forks that are already close to ENet in the context of this benchmark.

I'm glad that this tool was useful for you, and I'll think about an option of separating processes.

 

I did some small changes on those libraries, they are good and helped me a lot

I am working on a net library that is still green, so green that it only has unreliable QoS, and it does not have a name. The core concept is lightweight "channels" with QoS with ids that are equivalent to "state queues". Instead of sending several "states" with an id (and maybe length) through a channel, have a small channel per component that needs to serialize states, and have the channel id to be the packed id. Client-Server, and channel management would be done by the server accepting requests from clients. I am still working on some details, like ACKs, and maybe the performance ends up not being good.

To be able to have lots of peers with low memory usage and not so bad cpu what I did is:
-Store the Sockets inside an array, wrapped inside an object that also contains a timestamp in milliseconds of when was the last time Socket.Available was called on it (it is not free) and the delay in milliseconds that we want to wait until the next check
-In one or more threads iterate over the array of sockets (without overlaping the checks between threads). In each iteration check for each socket if the timestamp + its delay is less than the current timestamp. If it is, check socket.Available and increase the timestamp. If Available is greater than one save the ammount in a tmp var and repeat socket.ReceiveFrom substracting its return from tmp, until tmp reaches 0. It after iterating over all the sockets there was one that called ReceiveFrom, inmediately iterate over them again, otherwise use the smalles ammount of milliseconds needed to wait by any socket and use that to sleep the thread.

By doing this I was able to use only one buffer per receiver thread, keeping the ram usage super low (about 10MB).
With my cpu I was able to get 60,000 pps (packets per second) in Windows, but only 160 pps in Linux, so this is not usable in practice.
Increasing the ammount of receiving threads did not increase the ammount of pps in my tests.

 

I knew UNet was a bit bad, but I'm surprised just how bad it performed here.

Hopefully the devs can sort things out.

 

I do not have any experience with c++ and Unity plugins, so I am sorry If a don't get something obvious. I can't get ENet working from Unity.

I am on Windows.

Built a native enet dll.
Built a dll for c# wrapper.
Created small c# project (server/client depending on command line args), added reference to wrapper dll.
Built project.
Copied native dll to the folder with exe.
Ran as server and then as client.
Everything works great.

But when I try to connect from client in Unity to standalone server, nothing happens. Like nothing at all. Server does not receive any connection attemps.

I added both native and wrapper DLLs to Assets/Plugins. Wrote small script, which just initializes library, creates a host and connects to the server. No compilation errors.

When I hit Play, nothing happens. host.Connect(...) is executed, but server does not receive anything. Address is correct. Almost the same code works well from standalone client.

 

Did you enable port forwarding? And be sure to check firewall settings.

 

Server and client are both on localhost, there isn't any real internet connection. And it works outside Unity, so problem is with Unity setup. I do something wrong, but not sure exactly what.

 

You don't need to manually copy the native ENet library to the root directory of your standalone build. Instead, you just need to move the DLL to a special subfolder into the Plugins folder in your project. The native library will be automatically copied to the proper directory in the build process. In your case, it'll be Standalone\Data\Plugins\ENet.dll.

Also, apply this and this changes to the wrapper.

 

Yeah, I made the first change earlier, because Unity complained that it can't find dll. Thanks for your response, I'll try it tomorrow.

 

Is it an open source Project with Unity using enet and a server in c#?

Regards.

 

You can use ENet as the server/client in Unity or in your standalone .NET application. All that you need is just compile the native ENet library, and drag & drop it with the C# wrapper into your project.

 

Great work @nxrighthere , this is very valuable! Someone should really pin this here. (@hippocoder ?)

Small suggestion: it would be great to see a 'uses TCP or UDP' column on the Wiki page as performance is expected to be slightly slower for TCP libraries, while providing other benefits etc.

Did you get any different results for LLAPI with the latest Unity versions?
Or any news from @aabramychev ?

 

If I understand you correctly, as far as I know only Hazel and DarkRift support TCP. All the libraries use only UDP for communications.

Well, I've tested only the UNet Server, and I'm still waiting for an updated version. All I know is that they fixed some bugs, and an improved version of LLAPI must be already shipped with latest patches for the Unity.

 

Just found this thread and think it's really exciting! Thanks for doing this work.
If you don't mind, we would check if we can use this for more tests.

Very late input and maybe nitpicking but: Comparing Photon and ENet might be a bit misleading. ENet is more or less a low level protocol and not meant to be a middleware for higher level functionality. Our UDP protocol is actually based on an earlier version of ENet.

As feedback: It would be cool if you could include some version info in the test-results. That's not unimportant (and might help show the evolution of the network solutions, too).

 

Yea, I saw a month ago that the Photon's binary protocol is based on ENet and the service function is very similar. Yes, comparing them is not entirely correct, it's more about how I see the difference between them and my personal preference in general.

Thank you for the feedback, I'll include the version info in the results very soon.

 

In case if someone gets stuck with setting up ENet to work from Unity on Windows, as I did, here are the steps that helped me:
- Build one native enet dll for x32 and the other for x64. On the enet website there are instructions for what you have to do before building. In short, there are preparation 3 steps: (1) make sure that you build a dll, not a lib. (2) add links to ws2_32.lib and winmm.lib. (3) define ENET_DLL preprocessor directive. All these steps can be done via project properties.
- Make changes to ENetCS library that nxrighthere showed in #134. Build library for Any CPU.
- Copy enet x32 dll and ENetCS.dll to Assets/Plugins/x86.
- Copy enet x64 dll to Assets/Plugins/x86_64.
- Make sure that in inspector enet x32 dll has x86 chosen in settings for Editor and Standalone.
- Similarly make sure that enet x64 dll has x86_64 chosen.
- And finally make sure that ENetCS.dll has everything chosen.

 

Done. By the way...

 

Haha, that's why I don't use it. for LAN purpose you have to buy a license... SUPER !

Did Unet work on memory leak ? And when it will be fixed ?
oh, and I don't see a memory leak on my project running for 4-5 hours, so I don't really know if the issue is important or not =/

 

I'd like to know too

Also @nxrighthere did you try modifying the UNET test so that the SentMessagePoolSize and ReceivedMessagePoolSize aren't initialized with ushort.MaxValue? I wouldn't be surprised if that was why UNET's memory usage in this test is so preposterous

From what I understand, the SentMessagePoolSize can remain low because you can always send multiple times per frame using SendQueuedMessages and a sendDelay of 0. However, I don't know what to do about the ReceivedMessagePoolSize. With 1000 clients sending 1500 messages per tick, your server could be receiving 1.5 million messages per tick on average, which is much higher than ushort.MaxValue. It seems ridiculous to initialize a pool of 1.5 million messages for this, but from what I understand, with the way things are set up currently, you'd be getting a whole lot of messages that are dropped because your receive pool would be full. Actually even at 64 clients your received messages pool would be full because you'd be getting 96k messages per tick on average. I may not understand this well, though. Maybe @aabramychev can help us figure this out?

 

Unfortunately, guys, I don't have any news about it. At the moment, all I know is in this post.

Well, it doesn't change anything except without these tweaks the server's message pool will be filled very quickly and as a result, the server will throw debug info about it. The problem is not there actually, I think it sits at the client initialization stage, and it's not related to this options.

The biggest thing that makes me sad is that UNet's source code is closed and I can't take a look at what exactly is happening under the hood...

 

BenchmarkNet 1.07 has been released.

• Switched GC mode to concurrent Server GC
• Updated DarkRift from beta to release version
• Improved performance
• Fixed minor bugs

I've updated the results for all libraries and added information about how to switch GC modes here.

When I started working on this tool, I couldn't enable the concurrent Server GC mode because I never did that before for a console application with .NET Framework. It's pretty easy to do with .NET Core, but I've been wondering why the almost similar method doesn't work with .NET Framework and now I finally solved it.

So, what has changed? Everything...

I knew that one of the biggest problems of .NET networking libraries is GC because while monitoring CLR memory counters I saw how unstable was generations of objects on the heap. And only now I did enough investigation why the concurrent Server GC doesn't turn on with .NET Framework... It was my fault as a developer who knows, how the Microsoft's Workstation/Server GC modes are working and where they should be used.

But better late than never. Now GC trades memory in favor of throughput. It's crucial for tests with more than 256 simulated clients. You can find more information in this article about how different GC modes are working.

Also, very soon I'll write some stuff here, which I found while working with developers of .NET networking libraries. Perhaps this information could be useful for some guys here.

 

Very interesting, but I have a great problem! I don't know how to build a NetworkManager. I think that is must been a Singleton with connect, disconnect and Receive Data event. But EnetCS doesn't have real event, it use while(true) infinite cycle to intercept type data (too bad for Unity class..)

Someone has a very good example or a stub to start with?

Thank you for your help! I would create a complete example in MIT license usafull to learn for everything ;-)
(Sorry for my bad English)

 

As a starting point, you can just initialize the library in Awake() method and call the service function asynchronously on the thread/task which will process network events.

 

If you don't buy software to make your life easier, that's up to you. With that attitude, I hope you don't try to earn your money with Unity games.

@nxrighthere: Please PM me the email of your account. I'll try to get you a license.