Search Unity

  1. We want you to join us at GDC this year! Take a peek at all of the events we will be hosting during the week of GDC.
    Dismiss Notice
  2. Tell us about your experience here and you’ll get early access to the 2018 Game Studios report + more goodies.
    Dismiss Notice
  3. Unity 2017.3 has arrived! Read about it here.
    Dismiss Notice
  4. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice
  5. We've closed the job boards. If you're looking for work, or looking to hire check out Unity Connect. You can see more information here.
    Dismiss Notice

BenchmarkNet (Stress test for ENet, UNet, LiteNetLib, Lidgren, MiniUDP, Hazel, Photon and others)

Discussion in 'Multiplayer Networking' started by nxrighthere, Jan 13, 2018.

  1. RevenantX

    RevenantX

    Joined:
    Jul 17, 2012
    Posts:
    70
    @nxrighthere please add traffic usage) Maybe use some external application that can capture that information. And add to table.
     
  2. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    All the tools I tried affect the performance, unfortunately... I'm also wanted to monitor the bandwidth, but I still can't find a solution for this.
     
  3. Zuntatos

    Zuntatos

    Joined:
    Nov 18, 2012
    Posts:
    421
    Is there a chance of adding steam SDK's networking capabilities?

    It's pretty inconvenient though, it wants a steamID for every client & host. So you'd have to load the game server api on every client to get an anonymous steamID.
     
  4. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    Nah, too much headaches with Steamworks API. And I can tell you without any benchmarks that Steam networking can't handle more than 64 clients per server with similar setup. It's not suited for this. The server performance will degrade before 48 connections will be reached.
     
    Last edited: Jan 26, 2018
  5. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    BenchmarkNet 1.05 has been released.
    • Added support for Neutrino
    • Increased the server thread priority
    • Optimized IL code
    • Updated LiteNetLib to the latest version
    • Fixed minor bugs

    I added Neutrino to the results and updated them for all other libraries that are affected by latest changes. If someone doesn't know, you can compare the current version with the previous like this.

    Neutrino is an interesting networking library. It's tightly integrated with MessagePack and using it in almost all transmissions. It also reuse instances of messages to minimize GC pressure which is quite effective. This works absolutely wonderful until no more than 256 clients are connected to the server. But after this amount of connections, the performance is highly degraded.

    The application is now built using the Roslyn's optimization logic for IL code. You can see the difference here in the part of UNet. I've never used this feature before, but now I think that it's one of the most useful things in Roslyn. For some reasons, I didn't recompile the libraries with this feature, but maybe in the near future it'll happen. Also, after some experiments, I increased the server thread priority by one level to above normal. This change has a beneficial effect.

    By the way, PayPal is now working fine and I made a Supporters section in the first post and on GitHub page (some nice guys are already there). Currently, I'm building a server machine that I'm going to use in the development. If you like this project, you can help me with the next one... I'm grateful for any support.

    In the last two weeks, I've tested up to 15 different networking libraries. As you can see, not all of them supported due to some issues, bugs and limitations. But I keep looking for more.
     
    Last edited: Jan 30, 2018
    Vytek, alexr1221 and bddckr like this.
  6. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    I found the best way to measure the total bandwidth, but it still takes 10% of CPU power. Well, it seems like I have no choice and I'll run the tests twice for each networking library. One for the performance and one for the bandwidth usage. You know, to avoid any spikes I don't even move my mouse while the benchmark is performing...

    So, here is the updated tables.
     
    Last edited: Jan 30, 2018
    -chris and RevenantX like this.
  7. Jamster

    Jamster

    Joined:
    Apr 28, 2012
    Posts:
    864
    It might be failing tests but at least Hazel's winning on bandwidth right? :p
     
  8. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    Yep, it looks like it has the lowest packet overhead: 51 bytes for reliable (3 bytes overhead) and 49 bytes for an unreliable message (1 byte overhead). But the other libraries are using sequencing, message combining and so on.

    ENet
    ENetPacket.PNG

    Hazel
    HazelPacket.PNG
     
    Last edited: Feb 8, 2018
    Vytek likes this.
  9. Vytek

    Vytek

    Joined:
    Apr 29, 2016
    Posts:
    41
    Sorry,
    what is the program that you use in these images? Wireshark?

    Thank you for your answer...

     
  10. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    Nope, I don't like Wireshark.

    This is a simple yet powerful protocol analyzer SNPA. To capture the loopback packets, use Tools -> Loopback Monitor, select the process and start capturing.
     

    Attached Files:

    Last edited: Jan 30, 2018
    forestrf, Vytek and wobes like this.
  11. JesseLord

    JesseLord

    Joined:
    Jan 5, 2015
    Posts:
    2
    Any news on the test outcome?
     
  12. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    They are still working on it. I'm in touch with Alex. Perhaps the update will be this week.
     
  13. JesseLord

    JesseLord

    Joined:
    Jan 5, 2015
    Posts:
    2
    nxrightthere how did you do your original uNet tests? with HLAPI or just LLAPI? Oh it looks like Low level api then. I found your .cs file it looks like low level api.
     
  14. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    This is a separate product which is called the UNet Server. This library is very similar to LLAPI, but it works outside of the Unity environment.
     
    Last edited: Feb 5, 2018
  15. orangetech

    orangetech

    Joined:
    Sep 30, 2017
    Posts:
    1
  16. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    KCP is just a protocol, not a networking library.
     
  17. RevenantX

    RevenantX

    Joined:
    Jul 17, 2012
    Posts:
    70
    @nxrighthere so as i expected there is users that doesn't understand what is benchmarked here
    So maybe you add some notice? Because I just get a bad reputation...
    500 clients + server with 500 connections != just server with 500 connections
     
    wobes likes this.
  18. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    Yea, sure. I'll add a note about this.
     
  19. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    Done. I hope that this is enough.

    By the way, if anyone is interested, here is the LiteNetLib fork which shows that everything is in your hands.

    LiteNetLib-Fork.PNG

    CPU usage 46%, memory consumption 198 megabytes, bandwidth 313 megabytes.
     
    Last edited: Feb 8, 2018
  20. RevenantX

    RevenantX

    Joined:
    Jul 17, 2012
    Posts:
    70
    @nxrighthere do you think this is really important for now - to optimize library for synthetic benchmark?)
    I have many other things to do. And separately server can handle more connections. In real application 99% of CPU usage will be by game logic and bad code ;)
     
    wobes likes this.
  21. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    You have your own vision in which direction to improve the project. For me, performance and stability have the highest priority in development. Benchmarking, stress-testing, analyzing and so on... We often talk with Danis. For him, it's a challenge to make the library competitive to lower level solutions, and he achieved good results. In general, as Chris mentioned here, it's extremely difficult to reach 1000 clients on the server in a real scenario. To make this possible, we need something more than just a simple server-side logic written on top of a networking library. It should be a system that will work and withstand high loads. This is what I'm currently working on. :rolleyes:

    Here's a simplified diagram of the infrastructure of my current project.

    SwarmNet.png
     
    Last edited: Feb 19, 2018 at 8:15 PM
    david55527 and alexr1221 like this.
  22. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    Now I can confirm that ENet works on mobile platforms. I slightly improved my fork, but the original version should work fine.
     
    dnnkeeper likes this.
  23. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    BenchmarkNet 1.06 has been released.
    • Added support for DarkRift
    • Added optional Instant mode
    • Fixed detection of process failure
    • Fixed minor bugs

    You can find the results of DarkRift as usual on the wiki page.

    I finally got the latest beta version of DarkRift 2, and it actually works quite well! There's still work to be done, and Jamie knows about it. I'm glad that this tool was useful for him, and as a result, he improved several places in the library. Looking forward to the DarkRift release!
     
    Last edited: Feb 10, 2018
    wobes and Jamster like this.
  24. wobes

    wobes

    Joined:
    Mar 9, 2013
    Posts:
    327
    Hi there, still no Forge support?
     
  25. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    193
    Nope. No progress on this issue (and others as well) for a month.
     
  26. Kirsche

    Kirsche

    Joined:
    Apr 14, 2015
    Posts:
    14
    This is great, thank you for sharing this.
     
    nxrighthere likes this.