BenchmarkNet (Stress test for ENet, UNet, LiteNetLib, Lidgren, MiniUDP, Hazel, Photon and others)

It's not calculated at all, that you see on the results page is the overall data captured at the IP level using an external application.

 

That sounds awesome.

Looking forward to this.
Thank you for your hard work.

 

@Vincenzo It's released. A bit later I'm going to write an API reference with some tips and explanations. A lot of stuff has changed, so drop me a message if you run into any issues.

 

I'll get to try to switch from LLAPI to your Enet wrapper sometime next week, i'll keep you posted.

Thanks for the hard work nxrighthere!

 

@Zuntatos Here's new information regarding the Valve's networking library.

 

You think there is a bug in the valve library? or is it simply to do with scrambling from supplying managed data?

 

I'm currently trying to find a source of the problem, but I don't think that something wrong with the wrapper (I've double checked everything). It seems that something is broken in the native library (deadlock probably). I'll try to perform simulation using a native program, and I want to hear what Fletcher thinks about this issue.

 

I'll post all the information here so you can keep track stuff related to this issue there.

 

@nxrighthere will you ever publish the Telepathy results? A benchmark should always show all the things that were tested

 

When I finish working on BenchmarkNet 2.0, I'll publish the results.

 

The C# wrapper for Valve sockets is available here. Please, read this before using it.

 

Sorry, I can not read all the previous posts, since my English is bad, so I'll just ask here. What is the best solution for creating a multiplayer game in which large online are planned, up to 200-400 people on 1 server? (for example as in the game Rust)

 

It depends on many factors:

1. Your experience in networking, concurrency, and data optimization.
   
2. Your knowledge of memory management in .NET.
3. Per client resources budget.
   
4. Server software platform.
   
5. How good the target server hardware.

 

I did not think about it deeply, but I want to understand in what direction I should move. For example, to create a game like Counter Strike, it's enough for me to be a Photon, I've already done different prototypes on it and they worked. I understand there is no tool that will allow me to do the same simply, but only for online 200-400 people on the server (without rooms)? As in the games of Rust or San Andreas Multiplayer (SAMP).

 

Unity 5 vs UE4 vs Photon vs DIY for MMO
Development & Deployment of Multiplayer Online Games Vol. I
MMOs and modern scaling techniques
Actor model

 

ok, I so understood from the above that there is no simply way (like Photon) to create a game with a large on-line in one room (on 1 server).

 

yeah no there really isn't

 

Mirror & Telepathy can handle 200-400 people on a server easily if you have decent hardware. I tested my uMMORPG Asset with 200+ players on a small scene, everyone broadcasting to everyone else and it works fine.

 

@nxrighthere hi! Can you update benchmark results? I added recycle method to NetPacketReader (NetDataReader extension) and some receive speed improvements.

 

Sure, I'll update it soon.

 

Hmm, thanks, I'll look at this. And what was your hardware on the server?

 

BenchmarkNet 1.10 has been released.

• Added multi-process instances
• Added self-monitoring and execution of instances
• Added memory-mapped data exchange between instances
• Updated LiteNetLib to the latest version
• Updated DarkRift to the latest version
  
• Reworked layout of output text
• Reworked user-specified configuration
• Reworked Photon server logic
  
• Replaced unsafe ENet with ENet-CSharp
• Changed Sustained Low Latency mode to work with the server instance only
• Minor improvements

The results are updated for all networking libraries. Please, keep in mind: everything that you see there (except the screenshots from the monitoring instance) is now related only to the server process which is isolated from the clients simulation. More information is below.

This is the last major update before the upcoming 2.0 version, which will introduce the GUI with modern visualization of the simulation (classic CLI version will still be an option), detailed statistics with graphs (like minimalist netdata), TCP libraries support (regardless of my opinion and experience in general), and other stuff.

But for now, let's talk about the latest 1.10 version. Developers requested several times to separate the server and clients processes for easier debugging, tracing, and resources monitoring. So, here it is: now there are three fully automated instances: monitoring instance CLI/GUI (master), server instance (child) and clients simulation instance (child). From the task manager perspective it looks like this:



The child processes communicate with the master via memory-mapped files (the data serialized using well-known binary formatter). The named pipes used to check the heartbeat of the master and self-termination if something goes wrong. When a benchmark is complete, the supervisor task will terminate the child processes automatically.

One of the biggest advantages of isolated processes is that I got perfectly linear changes in resources usage (except memory consumption) which means that it's precise and very predictable data. This is most likely what you will see in a real-world application on target hardware. Look at the tiers with 500 and 1000 clients, compare them, it's awesome how linear this data is.

You can use something like the famous Process Explorer or Process Hacker to monitor CPU usage, memory consumption, network I/O (loopback), GC generations, heap size, time spent in Jit/GC, and tons of other useful information for each process. For example, here are graphs with captured resources usage of ENet server process.

*Takes a deep breath*

In the 1.10 version, I've changed the layout of output text into a more readable form, and user-specified configuration now works a bit differently. Now at the runtime, the user can select only the networking library and a number of the simulated clients. All other parameters can be set in the config file before launching the application.

The original ENet with an unsafe C# wrapper was replaced with our fork. It's sad that Lee is not working on the library anymore, but the source code is open, so we took the project on our shoulders. I mentioned several times across the forum that we improved it a lot, and now you can see the results.

@RevenantX is doing a great job (as always). LiteNetLib with the latest changes now works faster and consumes fewer resources.

Same thing with DarkRift. (Keep it going @Jamster!)

As for the Valve sockets. As soon as Fletcher fix some stuff, I'll integrate it into the application since I've finished working on the C# wrapper, so stay tuned.

@tobiass After reworking the Photon server logic I instantly found two bugs (or I missed something?): 1. OnReceive() returns an incorrect size of the payload, take a look. 2. Channels with ID > 1 are not working for some reason.

 

@nxrighthere: I will bring in the server guys for the payload-size in OnReceive(). Channels: The client needs to set the client-count before connect, so it's setup for the connection. Else, they won't be accessible. Do you do that?

Edit: @nxrighthere: Would you mind mailing us the relevant code for the OnReceive() problem? My colleague didn't check the code yet but seems positive that we just provide the byte[] data and no separate length parameter. So the data.Length should be correct. Is the value larger than what you expect? It could include type info and or something wrapping your raw data? Mail to: developer@photonengine.com please.

 

Great work there @nxrighthere. Our switch from LLAPI to your Enet wrapper is 50%, it had some delays because of work i had to do on other parts of our project. Your new benchmark shows very epic results for Enet to be honest, it is the lightest library out there today. Can't wait to test it in our project.

Thank you for all the hard work here!

If your interested for an alpha testing with our game just hit me a Private message.

 

Ah, you are right I forgot about it.

Yea, it always returns more length than expected - 50 bytes instead of 48 bytes per message in this case. The source code of server logic is here.

Done.

Thank you. Actually, I'm very rarely playing the games (Fortnite/World of Warcraft sometimes), but would be interesting to see how things are going with your project.

 

@tobiass Photonian told me that OnReceive returns a buffer with headers, and deserialized data can be retrieved via OnMessage callback. Everything works fine now, thanks.

 

Only just seen this somehow!

Keep up the great work @nxrighthere, I'm not sure you realize quite how much seeing these results drives me to put DR at the top!

You'll be glad to know that the next update will allow DR to use other networking layers instead of it's bi-channel default. Would be good fun to colab and give it a test run with DR on ENet or Neutrino! After all, you do seem to hate TCP

 

@nxrighthere can you share more about this

 

Here are a few posts (1, 2) about this networking system. It utilizes the ENet for low-level communications over UDP and the Hydrogen (optional module) for secure per channel encryption. The primary goal is high-performance networking backend that works right inside Unity but scalable without any interruptions of a game loop. The second goal is modular extensibility to kickstart any multiplayer project within a few hours.

This networking system is the successor of a complex project. It will be released as soon as I finish working on some high-level stuff, and when it will pass all the tests.

 

Our MMO currently utilizes UNET and we've been eyeing ENet as a replacement but your solution is sounding very appealing. Would you use the analogy that LLAP<-->ENet and HLAPI<-->ScionNet? Looking forward to your release!

 

The first one is correct. The ENet is used as a transport module and shuttles packets across the network at low-level. But the programmer can replace it with any other networking library in less than an hour.

The second is very different. Instead of trying to solve everything for the programmer via attributes, codegen, and components, the ScionNet offers modules with an intuitive, clean API. Using these modules, the programmer can create a logic for the multiplayer game in a few hours and automatically benefit from the concurrency, data management, and other stuff under the hood that leads to high-performance and low-latency.

I'm planning to cover in a series of video tutorials how and where the programmer can involve each module and how to make them work in harmony with specific game mechanics. This system is user-friendly, and it's shaped into a form of how I see modern networking backend right in the Unity environment where 90% of your code sitting in the main thread.

 

Ah I see sounds good. I'm not afraid of getting my hands dirty if it gives us more refined control and is less error prone!

 

Great stuff @nxrighthere.

What does the interface look like for sending messages to an existing framework that has it's own concurrency model? Is it just here is the message in whatever async context your system has, and you do what you want from there? (which would be fine with me).

 

Basically, it's just a method that we call after obtaining a reference to the core class and initialization. For example, let's say that we want to send a message to a peer and we have an array with serialized data of a typical MessagePack/Protobuf class. We call the send method in the core that expects destination peer, message ID, an array with serialized data, a flag with delivery type, and channel ID (everything is almost like with the ENet). As soon as the send method is called, the life-cycle of the message looks like this:

Synchronously (the call of the send function comes from the main or any other thread/task/job):

1. Acquire segment in a concurrent pool for the send function (essentially this the segment of a self-stabilizing circular buffer that's used as a storage for class instances with data of function calls).
2. Clone an array with serialized data using rented buffer from a buffers pool and put the reference to it with other function-related stuff into the acquired segment.
3. Enqueue function call into a concurrent circular queue (this is the disruptor-like Ring Buffer, we have one for events and one for functions).

Asynchronously:

1. Check for any functions in the circular queue.
2. If we have something there, let's try to process as many functions as possible depending on time window between calls to the queue functions by producer and consumer.
3. If the consumer successfully dequeued the send function call, create a packet for the message, send it to the designated peer, return the rented buffer back to the pool, and release the acquired segment.

 

@nxrighthere your solution sound very appealling your videos shows how stable the system is, no lag, no ridiculous memory consumption and memory doesn't increase over the time. I do wish you to finish ASAP .
Are you using Enet-Csharp or it's just based on native Enet library.

 

Thank you. Yes, the transport module is the ENet-CSharp, but it was slightly changed for this system.

 

Great!

 

Hi there. Do you plan to do a benchmark of your C# wrapper of Valve's networking?
https://github.com/nxrighthere/ValveSockets-CSharp

It would be nice to see the results on your machine.

 

Yes, I mentioned this some time ago:

I've described the problems here with the library. It's still not fixed.

 

Oh. Sorry about that.
Thank you.

 

Hi I'm analyzing the best approach to implement a mobile MMO with simple physics on a 2d world for around 100 players together and this post is helping me a great deal.

I read through (most of) this, and I'm not sure if there's any reason not to include SpatialOs in the benchmark. I thought it seems a good option, but maybe a I’m wrong.


At this very moment I was thinking about using ENet. If possible I’m searching for a solution that would require just quick adjustments on server side, to let us focus on client side. Any suggestion is highly appreciated.

 

I'm not sure about the mobile platforms it's not my area, unfortunately. But on the desktop, with my concurrent networking system using ENet, I'm running high-load simulations with low resource consumption and everything works without any problems.

I've recently made simulation tool for it using the same technique that I use in the BenchmarkNet where's each client is an asynchronous task and here's a demo with 128 clients, 20 state updates per second each, right in the Unity environment on AMD FX-4300 (4-core 4GHz): https://a.doko.moe/mtybzt.mp4

The biggest consumer of CPU power on this video is Unity's built-in physics. In this demo, I'm using a single-producer single-consumer pattern, but the system has other several options that allow you to have multiple producers and consumers with any number of threads you want.

Here you can read what I'm using to minimize GC influence.

@Jack-Mariani and thank you for your contribution on Bountysource, Jack, I appreciate that.

 

If you are going to write high-level stuff for the game by yourself, you can use any networking library from the list on the first page. With any of those libraries, you can prototype client-server logic and make some adjustments in no time. All those libraries can handle 100 clients per server instance with a reasonable amount of data for transmission, but GC is the biggest problem of the managed libraries, especially in Unity.

If you are looking for something like HLAPI, I don't recommend anything for many reasons (unnecessarily overcomplicated inefficient design and so on).

 

Hey @nxrighthere , we noticed that you included TCP on your Wiki page now. Will you ever publish the Telepathy results?

You are the only person who took the time to benchmark networking libraries for Unity and people got immense value out of it. We are all counting on you, and a scientifically correct benchmark shouldn't omit results based on personal preference, etc. Let the people draw their own conclusions

 

BenchmarkNet 2.0 is not finished yet, and in the current state, I can't add any TCP library. Nanomsg, NetMQ, Telepathy, it doesn't matter. If you add UDP support like how DarkRift does, I'll integrate the library into the application on the next day, since the unreliable protocol will be available and we can compare things relative to the other libraries.

 

Hello nxrighthere

Can you say something about TNet?
I am surprised that there is not benchmark result for TNet, this is a hidden gem in my opinion. Do you have a plans to test it?
TNet have some very unique and powerful features, so I would like to see an opinion of professionals in network technologies about TNet.

 

Here's an answer regarding TNet integration.

@mgear Thank you for the contribution, Mika.

 

@nxrighthere got an ETA on the release of ScionNet?

Are you accepting beta testers?

 

Hopefully two more weeks from now (fingers crossed). I finished working on compression module for floats, vectors, and quaternions + I've integrated LZ4 at packet-level into ENet, you can read here about it. Now I'm working on AoI logic for network culling which should work in .NET Core and Unity as other modules. When I've done with the culling module, I'm planning to release ScionNet, and then start working on vectorized buffers cloning, server instances orchestrator, and other stuff.

As for beta testing, I already have a few guys they are work with me.

By the way, it seems that I found a very critical bug in Unity where's memory barriers fail in some cases, even if you are using built-in concurrent collections (I don't use them). I'm creating a test case for this issue right now.