Search Unity

BenchmarkNet (Stress test for ENet, UNet, LiteNetLib, Lidgren, MiniUDP, Hazel, Photon and others)

Discussion in 'Connected Games' started by nxrighthere, Jan 13, 2018.

Thread Status:
Not open for further replies.
  1. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    It's not calculated at all, that you see on the results page is the overall data captured at the IP level using an external application.
     
    Last edited: Sep 22, 2018
  2. Vincenzo

    Vincenzo

    Joined:
    Feb 29, 2012
    Posts:
    46
    That sounds awesome.

    Looking forward to this.
    Thank you for your hard work.
     
    nxrighthere likes this.
  3. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    @Vincenzo It's released. A bit later I'm going to write an API reference with some tips and explanations. A lot of stuff has changed, so drop me a message if you run into any issues.
     
    Last edited: Aug 19, 2018
    wobes likes this.
  4. Vincenzo

    Vincenzo

    Joined:
    Feb 29, 2012
    Posts:
    46
    I'll get to try to switch from LLAPI to your Enet wrapper sometime next week, i'll keep you posted.

    Thanks for the hard work nxrighthere!
     
    nxrighthere likes this.
  5. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    @Zuntatos Here's new information regarding the Valve's networking library.
     
    Zuntatos likes this.
  6. Vincenzo

    Vincenzo

    Joined:
    Feb 29, 2012
    Posts:
    46
    You think there is a bug in the valve library? or is it simply to do with scrambling from supplying managed data?
     
  7. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    I'm currently trying to find a source of the problem, but I don't think that something wrong with the wrapper (I've double checked everything). It seems that something is broken in the native library (deadlock probably). I'll try to perform simulation using a native program, and I want to hear what Fletcher thinks about this issue.
     
    Last edited: Sep 13, 2018
  8. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    I'll post all the information here so you can keep track stuff related to this issue there.
     
    Last edited: Sep 13, 2018
  9. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    3,301
    @nxrighthere will you ever publish the Telepathy results? A benchmark should always show all the things that were tested :)
     
  10. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    vis2k likes this.
  11. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    The C# wrapper for Valve sockets is available here. Please, read this before using it.
     
    Kirsche likes this.
  12. scorp2007

    scorp2007

    Joined:
    Aug 17, 2014
    Posts:
    34
    Sorry, I can not read all the previous posts, since my English is bad, so I'll just ask here. What is the best solution for creating a multiplayer game in which large online are planned, up to 200-400 people on 1 server? (for example as in the game Rust)
     
  13. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    It depends on many factors:
    1. Your experience in networking, concurrency, and data optimization.
    2. Your knowledge of memory management in .NET.
    3. Per client resources budget.
    4. Server software platform.
    5. How good the target server hardware.
     
  14. scorp2007

    scorp2007

    Joined:
    Aug 17, 2014
    Posts:
    34
    I did not think about it deeply, but I want to understand in what direction I should move. For example, to create a game like Counter Strike, it's enough for me to be a Photon, I've already done different prototypes on it and they worked. I understand there is no tool that will allow me to do the same simply, but only for online 200-400 people on the server (without rooms)? As in the games of Rust or San Andreas Multiplayer (SAMP).
     
  15. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
  16. scorp2007

    scorp2007

    Joined:
    Aug 17, 2014
    Posts:
    34
  17. rasto61

    rasto61

    Joined:
    Nov 1, 2015
    Posts:
    311
    yeah no there really isn't
     
  18. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    3,301
    Mirror & Telepathy can handle 200-400 people on a server easily if you have decent hardware. I tested my uMMORPG Asset with 200+ players on a small scene, everyone broadcasting to everyone else and it works fine.
     
  19. RevenantX

    RevenantX

    Joined:
    Jul 17, 2012
    Posts:
    127
    @nxrighthere hi! Can you update benchmark results? I added recycle method to NetPacketReader (NetDataReader extension) and some receive speed improvements.
     
  20. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Sure, I'll update it soon.
     
  21. scorp2007

    scorp2007

    Joined:
    Aug 17, 2014
    Posts:
    34
    Hmm, thanks, I'll look at this. And what was your hardware on the server?
     
  22. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    3,301
    upload_2018-9-17_10-16-54.png
     
  23. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    BenchmarkNet 1.10 has been released.
    • Added multi-process instances
    • Added self-monitoring and execution of instances
    • Added memory-mapped data exchange between instances
    • Updated LiteNetLib to the latest version
    • Updated DarkRift to the latest version
    • Reworked layout of output text
    • Reworked user-specified configuration
    • Reworked Photon server logic
    • Replaced unsafe ENet with ENet-CSharp
    • Changed Sustained Low Latency mode to work with the server instance only
    • Minor improvements

    The results are updated for all networking libraries. Please, keep in mind: everything that you see there (except the screenshots from the monitoring instance) is now related only to the server process which is isolated from the clients simulation. More information is below.

    This is the last major update before the upcoming 2.0 version, which will introduce the GUI with modern visualization of the simulation (classic CLI version will still be an option), detailed statistics with graphs (like minimalist netdata), TCP libraries support (regardless of my opinion and experience in general), and other stuff.

    But for now, let's talk about the latest 1.10 version. Developers requested several times to separate the server and clients processes for easier debugging, tracing, and resources monitoring. So, here it is: now there are three fully automated instances: monitoring instance CLI/GUI (master), server instance (child) and clients simulation instance (child). From the task manager perspective it looks like this:

    ProcessInstances.PNG

    The child processes communicate with the master via memory-mapped files (the data serialized using well-known binary formatter). The named pipes used to check the heartbeat of the master and self-termination if something goes wrong. When a benchmark is complete, the supervisor task will terminate the child processes automatically.

    One of the biggest advantages of isolated processes is that I got perfectly linear changes in resources usage (except memory consumption) which means that it's precise and very predictable data. This is most likely what you will see in a real-world application on target hardware. Look at the tiers with 500 and 1000 clients, compare them, it's awesome how linear this data is.

    You can use something like the famous Process Explorer or Process Hacker to monitor CPU usage, memory consumption, network I/O (loopback), GC generations, heap size, time spent in Jit/GC, and tons of other useful information for each process. For example, here are graphs with captured resources usage of ENet server process.

    *Takes a deep breath*

    In the 1.10 version, I've changed the layout of output text into a more readable form, and user-specified configuration now works a bit differently. Now at the runtime, the user can select only the networking library and a number of the simulated clients. All other parameters can be set in the config file before launching the application.

    The original ENet with an unsafe C# wrapper was replaced with our fork. It's sad that Lee is not working on the library anymore, but the source code is open, so we took the project on our shoulders. I mentioned several times across the forum that we improved it a lot, and now you can see the results.

    @RevenantX is doing a great job (as always). LiteNetLib with the latest changes now works faster and consumes fewer resources.

    Same thing with DarkRift. (Keep it going @Jamster!)

    As for the Valve sockets. As soon as Fletcher fix some stuff, I'll integrate it into the application since I've finished working on the C# wrapper, so stay tuned.

    @tobiass After reworking the Photon server logic I instantly found two bugs (or I missed something?): 1. OnReceive() returns an incorrect size of the payload, take a look. 2. Channels with ID > 1 are not working for some reason.
     
    Last edited: Sep 22, 2018
    Jamster and tobiass like this.
  24. tobiass

    tobiass

    Joined:
    Apr 7, 2009
    Posts:
    2,241
    @nxrighthere: I will bring in the server guys for the payload-size in OnReceive(). Channels: The client needs to set the client-count before connect, so it's setup for the connection. Else, they won't be accessible. Do you do that?

    Edit: @nxrighthere: Would you mind mailing us the relevant code for the OnReceive() problem? My colleague didn't check the code yet but seems positive that we just provide the byte[] data and no separate length parameter. So the data.Length should be correct. Is the value larger than what you expect? It could include type info and or something wrapping your raw data? Mail to: developer@photonengine.com please.
     
    Last edited: Sep 19, 2018
    nxrighthere likes this.
  25. Vincenzo

    Vincenzo

    Joined:
    Feb 29, 2012
    Posts:
    46
    Great work there @nxrighthere. Our switch from LLAPI to your Enet wrapper is 50%, it had some delays because of work i had to do on other parts of our project. Your new benchmark shows very epic results for Enet to be honest, it is the lightest library out there today. Can't wait to test it in our project.

    Thank you for all the hard work here!

    If your interested for an alpha testing with our game just hit me a Private message.
     
    nxrighthere likes this.
  26. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Ah, you are right I forgot about it.

    Yea, it always returns more length than expected - 50 bytes instead of 48 bytes per message in this case. The source code of server logic is here.

    Done.

    Thank you. Actually, I'm very rarely playing the games (Fortnite/World of Warcraft sometimes), but would be interesting to see how things are going with your project.
     
    Last edited: Sep 19, 2018
    tobiass likes this.
  27. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    @tobiass Photonian told me that OnReceive returns a buffer with headers, and deserialized data can be retrieved via OnMessage callback. Everything works fine now, thanks.
     
    tobiass likes this.
  28. Jamster

    Jamster

    Joined:
    Apr 28, 2012
    Posts:
    1,026
    Only just seen this somehow!

    Keep up the great work @nxrighthere, I'm not sure you realize quite how much seeing these results drives me to put DR at the top!

    You'll be glad to know that the next update will allow DR to use other networking layers instead of it's bi-channel default. Would be good fun to colab and give it a test run with DR on ENet or Neutrino! After all, you do seem to hate TCP ;)
     
    TwoTen and nxrighthere like this.
  29. buFFalo94

    buFFalo94

    Joined:
    Sep 14, 2015
    Posts:
    204
  30. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Here are a few posts (1, 2) about this networking system. It utilizes the ENet for low-level communications over UDP and the Hydrogen (optional module) for secure per channel encryption. The primary goal is high-performance networking backend that works right inside Unity but scalable without any interruptions of a game loop. The second goal is modular extensibility to kickstart any multiplayer project within a few hours.

    This networking system is the successor of a complex project. It will be released as soon as I finish working on some high-level stuff, and when it will pass all the tests.
     
    Last edited: Oct 13, 2018
    buFFalo94 and philwinkel like this.
  31. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    233
    Our MMO currently utilizes UNET and we've been eyeing ENet as a replacement but your solution is sounding very appealing. Would you use the analogy that LLAP<-->ENet and HLAPI<-->ScionNet? Looking forward to your release!
     
  32. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    The first one is correct. The ENet is used as a transport module and shuttles packets across the network at low-level. But the programmer can replace it with any other networking library in less than an hour.

    The second is very different. Instead of trying to solve everything for the programmer via attributes, codegen, and components, the ScionNet offers modules with an intuitive, clean API. Using these modules, the programmer can create a logic for the multiplayer game in a few hours and automatically benefit from the concurrency, data management, and other stuff under the hood that leads to high-performance and low-latency.

    I'm planning to cover in a series of video tutorials how and where the programmer can involve each module and how to make them work in harmony with specific game mechanics. This system is user-friendly, and it's shaped into a form of how I see modern networking backend right in the Unity environment where 90% of your code sitting in the main thread.
     
    Last edited: Oct 4, 2018
    mons00n likes this.
  33. mons00n

    mons00n

    Joined:
    Sep 18, 2013
    Posts:
    233
    Ah I see sounds good. I'm not afraid of getting my hands dirty if it gives us more refined control and is less error prone!
     
    nxrighthere likes this.
  34. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,478
    Great stuff @nxrighthere.

    What does the interface look like for sending messages to an existing framework that has it's own concurrency model? Is it just here is the message in whatever async context your system has, and you do what you want from there? (which would be fine with me).
     
  35. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Basically, it's just a method that we call after obtaining a reference to the core class and initialization. For example, let's say that we want to send a message to a peer and we have an array with serialized data of a typical MessagePack/Protobuf class. We call the send method in the core that expects destination peer, message ID, an array with serialized data, a flag with delivery type, and channel ID (everything is almost like with the ENet). As soon as the send method is called, the life-cycle of the message looks like this:

    Synchronously (the call of the send function comes from the main or any other thread/task/job):
    1. Acquire segment in a concurrent pool for the send function (essentially this the segment of a self-stabilizing circular buffer that's used as a storage for class instances with data of function calls).
    2. Clone an array with serialized data using rented buffer from a buffers pool and put the reference to it with other function-related stuff into the acquired segment.
    3. Enqueue function call into a concurrent circular queue (this is the disruptor-like Ring Buffer, we have one for events and one for functions).
    Asynchronously:
    1. Check for any functions in the circular queue.
    2. If we have something there, let's try to process as many functions as possible depending on time window between calls to the queue functions by producer and consumer.
    3. If the consumer successfully dequeued the send function call, create a packet for the message, send it to the designated peer, return the rented buffer back to the pool, and release the acquired segment.
     
    Last edited: Jan 5, 2019
    wobes likes this.
  36. buFFalo94

    buFFalo94

    Joined:
    Sep 14, 2015
    Posts:
    204
    @nxrighthere your solution sound very appealling your videos shows how stable the system is, no lag, no ridiculous memory consumption and memory doesn't increase over the time. I do wish you to finish ASAP :):):).
    Are you using Enet-Csharp or it's just based on native Enet library.
     
  37. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Thank you. Yes, the transport module is the ENet-CSharp, but it was slightly changed for this system.
     
  38. buFFalo94

    buFFalo94

    Joined:
    Sep 14, 2015
    Posts:
    204
  39. wobes

    wobes

    Joined:
    Mar 9, 2013
    Posts:
    709
  40. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Yes, I mentioned this some time ago:
    I've described the problems here with the library. It's still not fixed.
     
  41. wobes

    wobes

    Joined:
    Mar 9, 2013
    Posts:
    709
    Oh. Sorry about that.
    Thank you.
     
  42. Jack-Mariani

    Jack-Mariani

    Joined:
    Jul 12, 2013
    Posts:
    10
    Hi I'm analyzing the best approach to implement a mobile MMO with simple physics on a 2d world for around 100 players together and this post is helping me a great deal.

    I read through (most of) this, and I'm not sure if there's any reason not to include SpatialOs in the benchmark. I thought it seems a good option, but maybe a I’m wrong.


    At this very moment I was thinking about using ENet. If possible I’m searching for a solution that would require just quick adjustments on server side, to let us focus on client side. Any suggestion is highly appreciated.
     
  43. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    I'm not sure about the mobile platforms it's not my area, unfortunately. But on the desktop, with my concurrent networking system using ENet, I'm running high-load simulations with low resource consumption and everything works without any problems.

    I've recently made simulation tool for it using the same technique that I use in the BenchmarkNet where's each client is an asynchronous task and here's a demo with 128 clients, 20 state updates per second each, right in the Unity environment on AMD FX-4300 (4-core 4GHz): https://a.doko.moe/mtybzt.mp4

    The biggest consumer of CPU power on this video is Unity's built-in physics. In this demo, I'm using a single-producer single-consumer pattern, but the system has other several options that allow you to have multiple producers and consumers with any number of threads you want.

    Here you can read what I'm using to minimize GC influence.

    @Jack-Mariani and thank you for your contribution on Bountysource, Jack, I appreciate that. ;)
     
    Last edited: Nov 6, 2018
    wobes likes this.
  44. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    If you are going to write high-level stuff for the game by yourself, you can use any networking library from the list on the first page. With any of those libraries, you can prototype client-server logic and make some adjustments in no time. All those libraries can handle 100 clients per server instance with a reasonable amount of data for transmission, but GC is the biggest problem of the managed libraries, especially in Unity.

    If you are looking for something like HLAPI, I don't recommend anything for many reasons (unnecessarily overcomplicated inefficient design and so on).
     
  45. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    3,301
    Hey @nxrighthere , we noticed that you included TCP on your Wiki page now. Will you ever publish the Telepathy results?

    You are the only person who took the time to benchmark networking libraries for Unity and people got immense value out of it. We are all counting on you, and a scientifically correct benchmark shouldn't omit results based on personal preference, etc. Let the people draw their own conclusions :)
     
  46. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    BenchmarkNet 2.0 is not finished yet, and in the current state, I can't add any TCP library. Nanomsg, NetMQ, Telepathy, it doesn't matter. If you add UDP support like how DarkRift does, I'll integrate the library into the application on the next day, since the unreliable protocol will be available and we can compare things relative to the other libraries.
     
  47. Artaani

    Artaani

    Joined:
    Aug 5, 2012
    Posts:
    411
    Hello nxrighthere

    Can you say something about TNet?
    I am surprised that there is not benchmark result for TNet, this is a hidden gem in my opinion. Do you have a plans to test it?
    TNet have some very unique and powerful features, so I would like to see an opinion of professionals in network technologies about TNet.
     
  48. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Here's an answer regarding TNet integration.

    @mgear Thank you for the contribution, Mika. ;)
     
    Last edited: Oct 29, 2018
    Artaani likes this.
  49. logixworx

    logixworx

    Joined:
    Jun 29, 2017
    Posts:
    72
    @nxrighthere got an ETA on the release of ScionNet? :)

    Are you accepting beta testers?
     
    mons00n likes this.
  50. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Hopefully two more weeks from now (fingers crossed). I finished working on compression module for floats, vectors, and quaternions + I've integrated LZ4 at packet-level into ENet, you can read here about it. Now I'm working on AoI logic for network culling which should work in .NET Core and Unity as other modules. When I've done with the culling module, I'm planning to release ScionNet, and then start working on vectorized buffers cloning, server instances orchestrator, and other stuff.

    As for beta testing, I already have a few guys they are work with me. ;)

    By the way, it seems that I found a very critical bug in Unity where's memory barriers fail in some cases, even if you are using built-in concurrent collections (I don't use them). I'm creating a test case for this issue right now.
     
    PixelMind, wobes, mons00n and 2 others like this.
Thread Status:
Not open for further replies.