Search Unity

  1. Improved Prefab workflow (includes Nested Prefabs!), 2D isometric Tilemap and more! Get the 2018.3 Beta now.
    Dismiss Notice
  2. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  3. Let us know a bit about your interests, and if you'd like to become more directly involved. Take our survey!
    Dismiss Notice
  4. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice
  5. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

BenchmarkNet (Stress test for ENet, UNet, LiteNetLib, Lidgren, MiniUDP, Hazel, Photon and others)

Discussion in 'Connected Games' started by nxrighthere, Jan 13, 2018.

  1. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341


    BenchmarkNet is a console application for testing the reliable UDP networking solutions.

    Features:
    • Asynchronous simulation of a large number of clients
    • Stable under high-loads
    • Simple and flexible simulation setup
    • Detailed session information
    • Multi-process instances
    Supported networking libraries:
    More information and source code on GitHub.
    You can find the latest benchmark results on the wiki page.

    Special thanks to @aabramychev for providing UNet libraries, help with integration and blessing the development.
    Also, thanks to @RevenantX for the quick response and implementation of some missing features in LiteNetLib.

    This project has already had an impact and helped developers in an improvement of the networking libraries. If you like this project, you can support me on PayPal or Bountysource. Any support is much appreciated.

    Supporters

     
    Last edited: Sep 18, 2018
    philwinkel, LostPanda, vis2k and 10 others like this.
  2. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    23,350
    Looks like for now, UNet is not doing very well. Didn't expect it to be that bad though. Was the C# wrapper for ENet used in the test?
     
    tequyla likes this.
  3. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    Yep, this wrapper was used in the test.

    UNet devs already know about this problem, and they got all technical details from me.
     
    Last edited: Jan 23, 2018
    hippocoder likes this.
  4. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    23,350
    Thanks a lot :)
     
  5. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    904
    Sad to see Unet's poor performance, but thanks a lot for this.

    Have you heard any news back from the devs after you've told them this?
     
  6. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    Yeah, we talked with @aabramychev about this and they are working on it. It took 2 days to determine why it's happening. More information will be later (after this weekend I think).
     
    wobes, NoxCaos and PhilSA like this.
  7. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    904
    I was really curious so I tried the same sort of test - but directly inside Unity

    What the test does:
    • Start a server and 64 clients connected to it (I can't test for 500 and 1000, because the max nb of hosts started on a single machine is 128)
    • 64 times per second, each client writes a 500 reliable messages and 1000 unreliable messages, and queue them for sending
    • both reliable and unreliable messages are 48 bytes each
    • 15 times per second, or whenever our queued msgs reached the queue limit, each client sends its queued messages
    • When recieving a reliable message from a client, the server sends back a 48 byte reliable message to that client. Same goes for unreliable messages
    • Done in unity 2018.1b2, .Net 4.6, Windows x64 architecture, in a release (non-dev) build, all logging turned off in Player settings
    Results:
    • Memory usage stays perfectly constant over an hour. No signs of memory leaks
    • 40% CPU usage and 1.8 GB memory usage
    To try it yourselves, just put the attached script on an empty object in an empty scene and press play. You can keep all default settings in the inspector

    Now... I'm really not sure how similar both our tests are, so I'm not gonna worry too much about that 1.8GB memory usage for 64 players (versus 800 mb in your test). But what I do think is important is that I don't see any memory leakage in this test. Could it be that there is a difference between the server dll and the unity NetworkTransport? Or maybe there is a big dumb flaw somewhere in my test that makes it not do what I think it does, but nothing seems suspicious after doing a bit of logging sent/received counts
     

    Attached Files:

    Last edited: Jan 13, 2018
    moco2k likes this.
  8. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    They are not much different as Alex told me, but this problem specifically addressed to the UNet Server.

    If you want to try to reproduce it in Unity, you need to initialize each client as an individual process and send the data independently of the others (each in its own update cycle).
     
    Last edited: Jan 13, 2018
  9. Whippets

    Whippets

    Joined:
    Feb 28, 2013
    Posts:
    1,725
    Be interesting to see how uNet does once the memory leak is fixed. I hope someone's on the case with that.
     
    wobes likes this.
  10. wobes

    wobes

    Joined:
    Mar 9, 2013
    Posts:
    440
  11. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    904
    something I just noticed:

    The fact that we're setting topology.SentMessagePoolSize and topology.ReceivedMessagePoolSize to ushort.MaxValue has a tremendous effect on memory consumption, and it might explain why UNET seems to be taking so much more memory than everything else. Maybe more sensible values are needed in topology, golbalConfig, ConnectionConfig, etc... but my understanding of all this is somewhat vague at the moment

    In a more realistic networked game scenario, I think there would be way less messages sent than in those tests. Especially if the world state is sent as one big fragmented message, and all the rest of the messages are just occasional damage/shoot/playerInput messages. In such cases, huge message pool/queue sizes wouldn't be necessary

    But still, none of that explains the memory leaks you are getting
     
    Last edited: Jan 13, 2018
  12. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    545
    First of all thanks for work done. Excellent. Few words.
    Test itself is slightly different from what we are using in the game, take a look
    For 1000 clients test will:
    Create 1001 instances of NetLibraryManager,
    Each netLibraryManager will
    create minimum 3 threads (3001 in common)
    create 4 different memory pools and preallocate memory for them
    create libwebsocket instance, and prealocate memory for tcp sockets
    create 1001 queue and populate them by prealocated events...
    When we created library we did not imagine this use case :) sorry.

    Memory leaks, I didn't find them with running 200 clients. memory growth but then stop growing when system is stabilized. 1000 clients - yes it is problem, they try to get a lot of memory, hence unet implementation itself is different and depends of platform for "server" os like Win, Mac, Unix, tread pooling is used, for mobile only io thread in use.

    Hence, I should note, thanks again, looks like configuration is not so flexible as we expected. And I expected better results :( so it is good opportunity temporary stop to think about new features and return back to stability and stressing.

    I will send my update in a week or two after this work will be finished. Many thanks to Stas who provide this work and you guys for discussion
     
  13. TwoTen

    TwoTen

    Joined:
    May 25, 2016
    Posts:
    1,004
    I experianced these issues with UNET in July. This was what my game looked like after being idle with one client exchanging messages after 12 hours
     
    PhilSA and nxrighthere like this.
  14. Jamster

    Jamster

    Joined:
    Apr 28, 2012
    Posts:
    949
    I'd be interested to see how Hazel does in this test since it's used by a few Unity projects now. It's not had much stress testing (it's been a while since I did anything on it to be honest) but I'd like to think I write well performing code!

    If not it would certainly provide some excellent feedback...!

    I'll try and make a PR at some point if you like but I'm not sure when I'll get around to it.
     
    nxrighthere likes this.
  15. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    Oh, I was going to integrate it, but I thought that you no longer work on Hazel in favor of DarkRift and therefore I didn't. I'll do it in the next release if everything goes smoothly!
     
    Jamster likes this.
  16. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    545
    @TwoTen Do you see this issue now? If yes I want to repro this in my side.
     
  17. TwoTen

    TwoTen

    Joined:
    May 25, 2016
    Posts:
    1,004
    I faced this issue in july with the HLAPI. We have since moved to the LLAPI with our own abstraction layer ontop. I will check if these issues are still present later today. I can also checkout the commit at the date i uploaded the video to see if the problem is with present then.

    Ill get back with results.
     
    aabramychev likes this.
  18. Rexima

    Rexima

    Joined:
    Jan 28, 2014
    Posts:
    25
  19. Jamster

    Jamster

    Joined:
    Apr 28, 2012
    Posts:
    949
    It's true I'm definitely more focused on DR2 (which has a different networking layer) but I don't think I'll be abaondoning Hazel anytime soon. It's more because people are using it, it would be good information for them :)
     
    nxrighthere likes this.
  20. RevenantX

    RevenantX

    Joined:
    Jul 17, 2012
    Posts:
    112
    Found some places that i can optimize in LiteNetLib after this benchs)
     
    nxrighthere likes this.
  21. RevenantX

    RevenantX

    Joined:
    Jul 17, 2012
    Posts:
    112
    @nxrighthere i see some difference between tests) In Lidgren you sending just byte array. And in LiteNetLib you creating NetDataWriter all time) You can send just byte[] too. This can reduce some memory and CPU usage.
     
    nxrighthere likes this.
  22. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    @Rexima I had some problems with Forge, but I'll try again after the next release.

    @RevenantX You are right... It will be fixed soon! I just forgot to change it after you had implemented the feature for determining the channel type.
     
    Last edited: Jan 23, 2018
  23. Whippets

    Whippets

    Joined:
    Feb 28, 2013
    Posts:
    1,725
    One of the most important threads in Unity right now.
     
    letscodestuff, Vytek and wobes like this.
  24. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    1,755
    Some things to keep in mind here.

    These tests would be more useful with some GC stats. PerfView works pretty well for that.

    I think what you will find is that unacceptable levels of GC pauses are happening long before you hit a few hundred connections in most cases. That is with any kind of realistic workload that an average realtime game will have.

    In the context of Unity I would think somewhere around 100 is going to be a realistic max. If you are not running unity you can push that up a ways but it's extremely difficult to hit 1000.

    The reason is GC doesn't really scale. You hit hard diminishing returns fairly quickly for low latency apps, it's just not worth the time to try and optimize beyond a certain point.

    These tests are useful for basic smoke testing, but don't really give details that matter the most. Like what is the average latency and standard deviation at a specific payload size/throughput/concurrency.

    The single biggest impact you can make is manual allocation and management of buffers. Like I would expect DotNetty to trounce everyone in these tests primarily because of that. While you can't use DotNetty in Unity (maybe 2018 haven't tried yet) you can probably borrow/steal some of the work they have done. Like just ripping out IByteBuffer and friends would provide a really good base for buffer managment.
     
    Whippets and nxrighthere like this.
  25. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    @snacktime I was thinking about the same things (especially the DotNetty part). Quite useful information, thanks!

    @Whippets Let's wish good luck to Alex. :D
     
    Whippets likes this.
  26. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    BenchmarkNet 1.02 has been released.
    • Added support for Hazel
    • Updated LiteNetLib to the latest version
    • Fixed sending of byte arrays for LiteNetLib
    • Fixed integer overflow in calculations
    • Fixed minor bugs

    First post updated with the new results of Hazel and LiteNetLib.

    Thanks to @Jamster for the help and rapid improvements to Hazel (changes are already in the master branch).
     
    Last edited: Jan 16, 2018
    Whippets and Jamster like this.
  27. Jamster

    Jamster

    Joined:
    Apr 28, 2012
    Posts:
    949
    I still have work to do! I'm not happy with those results! :)
     
    Vytek, Whippets and nxrighthere like this.
  28. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    Developers from different teams/companies bombard my email with results of their forks and suggestions. I didn't expect such an effect... :eek:
     
    Last edited: Jan 16, 2018
    letscodestuff, tobiass and Whippets like this.
  29. Whippets

    Whippets

    Joined:
    Feb 28, 2013
    Posts:
    1,725
    What you've started is massively important.
     
    letscodestuff and nxrighthere like this.
  30. Vytek

    Vytek

    Joined:
    Apr 29, 2016
    Posts:
    44
    Please @Jamster, make the miracle! ;-) I need Hazel and I like library very much!!!

     
    Jamster likes this.
  31. Vytek

    Vytek

    Joined:
    Apr 29, 2016
    Posts:
    44
    nxrighthere likes this.
  32. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    Yeah, I know about all of these networking solutions. I'll integrate them into the application, but a little bit later.
     
    Vytek likes this.
  33. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    23,350
    Is it relevant at all to include Photon results (it has a free tier) ?
     
  34. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    Sure, why not. I'll take a look what I can do.
     
    Munchy2007 and hippocoder like this.
  35. Lisk

    Lisk

    Joined:
    Oct 23, 2013
    Posts:
    91
    This looks like an awesome project and may singlehandedly improve the quality of Unity networking solutions across the board.
     
    Whippets and nxrighthere like this.
  36. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    ENet (3000 clients) on 6-core Intel Xeon...

    ENet (3000 clients).png

    CPU usage 34%, memory consumption 200 megabytes.
     
    Vytek, Whippets, Lisk and 1 other person like this.
  37. Vytek

    Vytek

    Joined:
    Apr 29, 2016
    Posts:
    44
  38. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    23,350
    Irrelevant stats as the same testing is not performed. I don't think we can perform the test either.
     
    Whippets and nxrighthere like this.
  39. TwoTen

    TwoTen

    Joined:
    May 25, 2016
    Posts:
    1,004
    That's not a real benchmark. That's player counts? For all we know they could be 10 players per server while having the game world sharded across many servers.
     
    GameDevCouple_I and nxrighthere like this.
  40. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    The second attempt to integrate Forge was again unsuccessful. The server is working fine, but the clients are not connecting. Moving forward to Photon.
     
    Jamster, Whippets and Munchy2007 like this.
  41. Whippets

    Whippets

    Joined:
    Feb 28, 2013
    Posts:
    1,725
    @nxrighthere This must be taking up a ton of your time. Have you got a patreon or donation page set up? It's quality information that we're all benefiting from XD
     
    Jamster likes this.
  42. NFMynster

    NFMynster

    Joined:
    Jul 1, 2013
    Posts:
    41
    Have you tried using the binary frame for sending the messages with Forge?
    Link to docs
    Very interested in Forge results
     
  43. Vytek

    Vytek

    Joined:
    Apr 29, 2016
    Posts:
    44
    It is only a joke ;-)

     
  44. toreau

    toreau

    Joined:
    Feb 8, 2014
    Posts:
    203
    This is some amazing work, but you _have to_ to post these benchmarks on the github page. If not, people will google and end up here looking at outdated benchmarks in the future. :(
     
  45. NFMynster

    NFMynster

    Joined:
    Jul 1, 2013
    Posts:
    41
    He did?
     
  46. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    @NFMynster I tried everything using the latest version from the master branch, compiled with Roslyn. Forge is just not working and I have no idea why.

    @toreau A note about the results is mentioned in readme on GitHub page.

    @Whippets I have a PayPal and Bountysource, but I don't think anyone is interested in this.
     
    Last edited: Jan 26, 2018
  47. toreau

    toreau

    Joined:
    Feb 8, 2014
    Posts:
    203
    No.
     
  48. toreau

    toreau

    Joined:
    Feb 8, 2014
    Posts:
    203
    Yes, but it would be much better (IMO) if you posted results over time on your github wiki page. Progress would be nice to see, to get a grasp of a project's "livelyness", plus you can always refer to an updated benchmark instead of linking to this already outdated benchmark in this thread.
     
    nxrighthere likes this.
  49. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    341
    Well, it makes sense. Thank you for the suggestion.
     
    Last edited: Jan 17, 2018
  50. Rexima

    Rexima

    Joined:
    Jan 28, 2014
    Posts:
    25