Search Unity

  1. Calling all beginners! Join the FPS Beginners Mods Challenge until December 13.
    Dismiss Notice
  2. It's Cyber Week at the Asset Store!
    Dismiss Notice

BenchmarkNet (Stress test for ENet, UNet, LiteNetLib, Lidgren, MiniUDP, Hazel, Photon and others)

Discussion in 'Connected Games' started by nxrighthere, Jan 13, 2018.

Thread Status:
Not open for further replies.
  1. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    @Rexima I don't use libraries in the project that are compiled by other people or automated build services for many reasons (especially for security). Everything should be done in my environment. UNet is an exception for obvious reasons.
     
    Last edited: Jan 17, 2018
  2. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    @toreau Thank you for your support. It was an unexpected surprise. :oops:
     
  3. Brent_Farris

    Brent_Farris

    Joined:
    Jul 13, 2012
    Posts:
    881
    Forge has a nightly build on the GitHub page, there is no need to compile source. Also the source has not been tested against the Roslyn compiler as there were reported issues with building against .NET 3.5 a few months back. To build Forge you can just use the standard VS toolchain on windows or the Mono build chain on other platforms.
     
    rogueknight likes this.
  4. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Thanks for the information, Brent and welcome to the party. I don't use VS for a very long time (something around 5 years)... I'll take a look what I can do, one more time.
     
    Last edited: Jan 20, 2018
    NFMynster likes this.
  5. RevenantX

    RevenantX

    Joined:
    Jul 17, 2012
    Posts:
    125
    @nxrighthere add traffic usage to benchmark. This one of the most useful information.
     
  6. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    I was thinking about it, but it's better to just monitor the loopback (filtered by process ID) with any available protocol analyzer, because I can't measure packet overhead in the application.
     
    Last edited: Jan 18, 2018
    wobes likes this.
  7. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    I finished working on an integration with Photon and Forge. I'm just waiting until Forge developers investigate this issue. Perhaps, I have time to add support for Netcode.io.

    @RevenantX I'll implement in the next version the payload flow measurement in Mbps for the server. Something like this:
    Code (CSharp):
    1. (clientsChannelsCount * (messageLength * sendRate * 2) * 8 / (1000 * 1000)) * 2
    I hope it will be enough.
     
    Last edited: Jan 21, 2018
    Munchy2007 likes this.
  8. RevenantX

    RevenantX

    Joined:
    Jul 17, 2012
    Posts:
    125
    I think better will be calculate total sent/received bytes after bench.
     
  9. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    BenchmarkNet 1.03 has been released.
    • Added support for Photon
    • Added payload flow measurement for the server
    • Fixed sending data after the client drop
    • Fixed minor bugs

    First post and wiki page updated with the new results of Photon.

    Release with support for Forge postponed due to this issue.
    Thanks to guys in Forge's Discord for the warm welcome (very friendly people are there ;)).

    As for Photon, it's a quite different networking solution. You need to follow this guide if you want to test the on-premise by yourself. With 1000 clients, Photon was performed not that fast as ENet, but still a good result.

    At the moment, I have some problems with Netcode.io and I just need some rest...
     
    Last edited: Jan 26, 2018
  10. Driiades

    Driiades

    Joined:
    Oct 27, 2015
    Posts:
    151
    Poor Unet :'(
     
  11. mgear

    mgear

    Joined:
    Aug 3, 2010
    Posts:
    5,598
    btw, paypal.me page doesnt work, says
    "This recipient accepts PayPal payments only through their website.
    To complete this payment, please go to their website and follow their instructions, or email the recipient for instructions."
     
  12. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    @Driiades Please, give some time to Alex... I believe that with this tool in his arsenal he can improve the UNet, but it requires many changes to the code.

    @mgear Yes, PayPal has some restrictions for my country. It looks like I can't do anything with this... Thanks for the information.
     
    Last edited: Jan 26, 2018
  13. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    I finished working on an integration with Netcode.io, but some additional checks are needed. Also, I got some very good ideas for new features. I'm going to implement them in the next release.
     
    Last edited: Jan 23, 2018
  14. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,919
    Nice Photon test, thanks! Seems it does hold up well enough...
     
    nxrighthere likes this.
  15. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Yeah, Photon also has a very flexible configuration. I played with the settings a little bit and I really like that I have an advanced control over the server. Other networking solutions do not provide such control and you need to write some of these things manually from scratch while Photon provides them out of the box.
     
    hippocoder likes this.
  16. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,919
    Well your hard work helped convince us that Photon is probably the best solution for now (if you agree). I'm doing a moderate load action title with around 8 players in MP for now. I don't have much experience in networking so it's probably for the best?

    UNet will probably take a fair amount of time before it can match Photon for reliability and features and performance, what do you think?

    Thanks for your work!
     
    nxrighthere likes this.
  17. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Photon is a reliable choice, but personally, I prefer ENet. Both solutions are battleforged. One paid and with the closed source code, another absolutely free and you can always change something in its code. ENet has an incredible congestion control, you don't need to do anything if the server becomes overloaded. ENet will be easier to deal with if you don't have much experience in networking. Photon is a more complex but has some examples that may help you to understand how to write the server-side logic of the game and load-balancing. ENet can be used directly inside Unity as the server which gives you unlimited possibilities, Photon is a standalone application which makes things harder.

    As for UNet, I think it has a great potential, but if I had just started developing a network game right now, I would prefer ENet. Not only because it has an impressive performance, but also because it's technically a well-designed networking library. That's just only my opinion.
     
    Last edited: Jan 23, 2018
    Brodus and alexr1221 like this.
  18. Vytek

    Vytek

    Joined:
    Apr 29, 2016
    Posts:
    50
  19. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Netcode.io with C# bindings and a reliability layer.
     
    Last edited: Jan 24, 2018
  20. dnnkeeper

    dnnkeeper

    Joined:
    Jul 7, 2013
    Posts:
    66
    Great benchmarks! I hope some optimization will allow uNet to handle few hundreds connection in the future release. One important advantage of uNet is multiple platforms support. Can ENet clients be compiled for android/ios?
     
    dzmitry-lahoda likes this.
  21. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    I'm not familiar with mobile platforms, I've never worked with them. It's better to test it yourself. ENet supports Unix-like operating systems.
     
    Last edited: Jan 23, 2018
  22. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    Just for info: So far no news :( I'm still working with stress testing (including @nxrighthere code) will put my update here immediately after finish.
     
    nxrighthere likes this.
  23. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
  24. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    @snacktime External tools slows down the performance when I monitor the time spent in GC. This is particularly noticeable under high loads. So, I found a very simple solution:
    Code (csharp):
    1. private static void GCLatency() {
    2.    Stopwatch latency = new Stopwatch();
    3.  
    4.    while (processActive) {
    5.        latency.Restart();
    6.        Thread.Sleep(1);
    7.        latency.Stop();
    8.  
    9.        if (latency.ElapsedMilliseconds > 2)
    10.            Interlocked.Exchange(ref gcLatency, latency.ElapsedMilliseconds);
    11.    }
    12. }
    After some tests, it looks like it works fine, but I'm not sure if this is a good way to measure GC pauses. The article is here. What do you think about it?
     
    Last edited: Jan 25, 2018
  25. Zuntatos

    Zuntatos

    Joined:
    Nov 18, 2012
    Posts:
    523
    ...I highly doubt that loop will properly record time spent in GC. Thread.Sleep(1) isn't accurate to 1 ms at all, often sleeping for 5-10 ms on various machines

    Edit: obviously depends on how much threads are active in the background, the less the more precise the sleep will be
     
    Last edited: Jan 22, 2018
    Jamster likes this.
  26. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Yeah, you are right, but I can't find another method... And even if it works I don't like it.
     
  27. Jamster

    Jamster

    Joined:
    Apr 28, 2012
    Posts:
    1,025
    It's worth noting you can start and stop the GC (ish) in critical regions using GC.TryStartNoGCRegion and GC.EndNoGCRegion. You can also invoke it manually using GC.Collect.

    Also, this article seems to suggeset you can get notifications on when it runs which might aid the accuracy of your timing there.
     
  28. alexr1221

    alexr1221

    Joined:
    Mar 5, 2016
    Posts:
    43
    Thank you for the great work. Thanks to you, you created a sort of competition that incites the creators, including unity' team to do better. I just had a little question: where did you get the ENet.dll file? I didn't found it in the c# wrapper repository
     
  29. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Yea, I played with GC API a few days ago, but it didn't give any positive results. Only the Sustained Low Latency has some interesting impact, but the application becomes a little bit unstable so I reverted changes.

    At the moment, I'm trying to find a solution to measure GC latency without affecting performance of the application.

    Thanks, I'll take a look.

    Thank you. You need to compile the library, follow the instructions here.
     
    Last edited: Jan 24, 2018
  30. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,473
    Ya I found that article too while looking for a way to measure pauses at runtime, and I didn't find anything better. I think it's better then nothing. On a real app I like just measuring a couple of places on the hot path that I know should be deterministic.

    GC has been a huge pain point for me in .NET on the server side compared to java. You basically get two modes that are at the far ends of the spectrum while what you really want is something in between for games.

    The goal is you trade memory for pause times in a way that best fits the app. Combined with making the GC more or less aggressive in specific generations. On the JVM you have enough control to really define that pattern exactly how you want. For example I tuned it for frequent 1ms-2ms pauses with one 5ms max pause every couple of minutes. And a 20ms max pause around once per hour and manually do a full collect during non peak times. This required a lot of RAM FYI.

    That level of control is impossible in .NET right now. You just have to look at other ways to solve the problem. Go horizontal faster, split stuff out into services, etc..
     
    nxrighthere likes this.
  31. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Thank you Chris, very informative. I'll postpone this feature for a while.

    I need to read a couple more things about GC in .NET. This is also directly related to my future projects and I can't ignore the impact of GC on performance and memory consumption.
     
    Last edited: Jan 23, 2018
  32. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    BenchmarkNet 1.04 has been released.
    • Added optional Sustained Low Latency mode
    • Optimized Photon client logic
    • Improved performance
    • Fixed minor bugs

    I've updated the results for all libraries and added a new tier with 128 clients. Also, you can read about the new option here.

    I mentioned earlier that I wanted to integrate the original Netcode.io written by Glenn Fiedler (using C# bindings and a reliability layer). But after the tests, I found that it doesn't support more than 256 connections. As far as I understand, it's unrecommended to change the preprocessor directive to bypass this limitation... Well, at least Netcode.io works absolutely flawless! If someone wants to test it, I can send you a source code.

    As for .NET implementation of Netcode.io, it's obviously slower than pure C library. But beyond that, it looks like the library has some problems with connections. As soon as clients start connecting one after another, the server performance degrades earlier than the number of connections reached ten. Thus, I don't see the point of adding support for it.

    By the way, I see that people still want to support the project, and I appreciate it. Thank you for your support.
     
    Last edited: Jul 15, 2018
    hippocoder likes this.
  33. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,473
    So on a related note to optimizing network stuff, if it would be helpful I have a collection of techniques I've put together over the last few years for conserving bandwidth. Some are known but others not so much. Used together you can do things like 500 characters moving/rotating at under 80kbps bandwidth usage, without using tricks that say rely on low tick rates or other things that make it unusable outside of a test case.

    I could write it up and hand it off to @nxrighthere to add to his stuff.
     
    dnnkeeper and nxrighthere like this.
  34. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    It would be great!

    By the way, I'm reading your threads right now and wonder how I missed all this stuff before...
     
  35. RevenantX

    RevenantX

    Joined:
    Jul 17, 2012
    Posts:
    125
  36. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Sure. I'll take a look what I can do.
     
  37. mgear

    mgear

    Joined:
    Aug 3, 2010
    Posts:
    5,598
    your pm is closed, but can you consider some crypto-currencies?
    (its easy to create deposit address in exchanges, and use them for trading or just keep if value goes up..)
     
  38. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    How could this happen. :eek:

    Feel free to contact me via email, I'm always responding.

    Thanks for the idea! I completely forgot about it...
     
    Last edited: Jan 25, 2018
  39. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Last edited: Jan 25, 2018
  40. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    I'm experimenting with RakNet and the old version of Lidgren right now, and I got the results that confuse me... While RakNet is completely useless (I'm glad that Unity deprecated it in favor of UNet), the old Lidgren a bit shocked me.

    Lidgren-Old.PNG

    CPU usage 60%, memory consumption 360 megabytes.
    Not perfect, but it's passed the test with 1000 clients while generation 3 can not...
     
    Last edited: Jan 26, 2018
    Jamster likes this.
  41. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    230
    Did you add a custom (i.e. plain) RakNet or did you use Unity's integrated RakNet? Must admit that we only target a range of 16 - 30 players per server with our game but Unity's RakNet has been incredibly stable for us (can't say a lot about performance though). Unlike UNET which still can't properly handle big amounts of data and loses data all the time :(
     
  42. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    I used the latest version from the master branch with C# bindings generated with Swig.
     
  43. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    230
    Would be cool to see comparison values for Unity's own implementation vs the others that you already have :)
     
  44. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Sure, but where I can get it?
     
  45. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    230
    It's in Unity, you don't have to get anything else ;)
    Manual: https://docs.unity3d.com/Manual/NetworkReferenceGuide.html
    Scripting ref: https://docs.unity3d.com/ScriptReference/Network.html

    Easiest way: Add a NetworkView component to a behaviour and in the script and then add a method to a script on that same gameobject that has the RPC attribute set, like this:
    Code (CSharp):
    1. [RPC]
    2. protected void RPC_RawData(byte[] _data, NetworkMessageInfo _sender) {
    3. // _data is your own stuff, _sender contains information from the networking layer
    4. }
    Starting and closing a server:
    Code (CSharp):
    1. NetworkConnectionError res = Network.InitializeServer (_maxConnections, _serverPort, false);
    2. Network.Disconnect (200);
    Connecting to a server and disconnecting:
    Code (CSharp):
    1. Network.Connect (ip, port, password);
    2. Network.Disconnect (200);
    Code (CSharp):
    1. yourGameObject.GetComponent<NetworkView>().RPC ("RPC_RawData", _networkPlayer, _data);
    from server to a client identified by _networkPlayer and
    Code (CSharp):
    1. yourGameObject.GetComponent<NetworkView>().RPC ("RPC_RawData", RPCMode.Server, _data);
    from client to server.

    That should be most of what you need :)

    Regards,
    Chris
     
  46. wobes

    wobes

    Joined:
    Mar 9, 2013
    Posts:
    705
    It's Legacy RakNet, which is not supported anymore by Unity Team.
     
  47. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    Oh, my... So, it's still in the engine? What a surprise... I thought that it was removed a long time ago. I can't use it unfortunately, BenchmarkNet is a standalone application.
     
    wobes likes this.
  48. Alloc

    Alloc

    Joined:
    Jun 5, 2013
    Posts:
    230
    Ah, ok, didn't look at the code just thought you were doing this stuff as a Unity application :)
     
  49. RevenantX

    RevenantX

    Joined:
    Jul 17, 2012
    Posts:
    125
    Hm. Why there not all messages was sent?
     
  50. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    546
    One client was dropped in the process.
     
Thread Status:
Not open for further replies.