BenchmarkNet (Stress test for ENet, UNet, LiteNetLib, Lidgren, MiniUDP, Hazel, Photon and others)

​

BenchmarkNet is a console application for testing the reliable UDP networking solutions.

Features:

• Asynchronous simulation of a large number of clients
• Stable under high-loads
• Simple and flexible simulation setup
• Detailed session information
• Multi-process instances

Supported networking libraries:

• ENet
• UNet
• LiteNetLib
• Lidgren
• MiniUDP
• Hazel
• Photon
• Neutrino
• DarkRift

More information and source code on GitHub.
You can find the latest benchmark results on the wiki page.

Special thanks to @aabramychev for providing UNet libraries, help with integration and blessing the development.
Also, thanks to @RevenantX for the quick response and implementation of some missing features in LiteNetLib.

This project has already had an impact and helped developers in an improvement of the networking libraries. If you like this project, you can support me on PayPal, Bountysource or Coinbase. Any support is much appreciated.

These wonderful people make open-source better:

 

Looks like for now, UNet is not doing very well. Didn't expect it to be that bad though. Was the C# wrapper for ENet used in the test?

 

Yep, this wrapper was used in the test.

UNet devs already know about this problem, and they got all technical details from me.

 

Thanks a lot

 

Sad to see Unet's poor performance, but thanks a lot for this.

Have you heard any news back from the devs after you've told them this?

 

Yeah, we talked with @aabramychev about this and they are working on it. It took 2 days to determine why it's happening. More information will be later (after this weekend I think).

 

I was really curious so I tried the same sort of test - but directly inside Unity

What the test does:

• Start a server and 64 clients connected to it (I can't test for 500 and 1000, because the max nb of hosts started on a single machine is 128)
• 64 times per second, each client writes a 500 reliable messages and 1000 unreliable messages, and queue them for sending
• both reliable and unreliable messages are 48 bytes each
• 15 times per second, or whenever our queued msgs reached the queue limit, each client sends its queued messages
• When recieving a reliable message from a client, the server sends back a 48 byte reliable message to that client. Same goes for unreliable messages
• Done in unity 2018.1b2, .Net 4.6, Windows x64 architecture, in a release (non-dev) build, all logging turned off in Player settings

Results:

• Memory usage stays perfectly constant over an hour. No signs of memory leaks
• 40% CPU usage and 1.8 GB memory usage

To try it yourselves, just put the attached script on an empty object in an empty scene and press play. You can keep all default settings in the inspector

Now... I'm really not sure how similar both our tests are, so I'm not gonna worry too much about that 1.8GB memory usage for 64 players (versus 800 mb in your test). But what I do think is important is that I don't see any memory leakage in this test. Could it be that there is a difference between the server dll and the unity NetworkTransport? Or maybe there is a big dumb flaw somewhere in my test that makes it not do what I think it does, but nothing seems suspicious after doing a bit of logging sent/received counts

 

They are not much different as Alex told me, but this problem specifically addressed to the UNet Server.

If you want to try to reproduce it in Unity, you need to initialize each client as an individual process and send the data independently of the others (each in its own update cycle).

 

Be interesting to see how uNet does once the memory leak is fixed. I hope someone's on the case with that.

 

@nxrighthere thank you for this benchmarking tool.

 

something I just noticed:

The fact that we're setting topology.SentMessagePoolSize and topology.ReceivedMessagePoolSize to ushort.MaxValue has a tremendous effect on memory consumption, and it might explain why UNET seems to be taking so much more memory than everything else. Maybe more sensible values are needed in topology, golbalConfig, ConnectionConfig, etc... but my understanding of all this is somewhat vague at the moment

In a more realistic networked game scenario, I think there would be way less messages sent than in those tests. Especially if the world state is sent as one big fragmented message, and all the rest of the messages are just occasional damage/shoot/playerInput messages. In such cases, huge message pool/queue sizes wouldn't be necessary

But still, none of that explains the memory leaks you are getting

 

I experianced these issues with UNET in July. This was what my game looked like after being idle with one client exchanging messages after 12 hours

 

I'd be interested to see how Hazel does in this test since it's used by a few Unity projects now. It's not had much stress testing (it's been a while since I did anything on it to be honest) but I'd like to think I write well performing code!

If not it would certainly provide some excellent feedback...!

I'll try and make a PR at some point if you like but I'm not sure when I'll get around to it.

 

Oh, I was going to integrate it, but I thought that you no longer work on Hazel in favor of DarkRift and therefore I didn't. I'll do it in the next release if everything goes smoothly!

 

I faced this issue in july with the HLAPI. We have since moved to the LLAPI with our own abstraction layer ontop. I will check if these issues are still present later today. I can also checkout the commit at the date i uploaded the video to see if the problem is with present then.

Ill get back with results.

 

@nxrighthere Could you make a benchmark with forge?

 

It's true I'm definitely more focused on DR2 (which has a different networking layer) but I don't think I'll be abaondoning Hazel anytime soon. It's more because people are using it, it would be good information for them

 

Found some places that i can optimize in LiteNetLib after this benchs)

 

@nxrighthere i see some difference between tests) In Lidgren you sending just byte array. And in LiteNetLib you creating NetDataWriter all time) You can send just byte[] too. This can reduce some memory and CPU usage.

 

@Rexima I had some problems with Forge, but I'll try again after the next release.

@RevenantX You are right... It will be fixed soon! I just forgot to change it after you had implemented the feature for determining the channel type.

 

One of the most important threads in Unity right now.

 

Some things to keep in mind here.

These tests would be more useful with some GC stats. PerfView works pretty well for that.

I think what you will find is that unacceptable levels of GC pauses are happening long before you hit a few hundred connections in most cases. That is with any kind of realistic workload that an average realtime game will have.

In the context of Unity I would think somewhere around 100 is going to be a realistic max. If you are not running unity you can push that up a ways but it's extremely difficult to hit 1000.

The reason is GC doesn't really scale. You hit hard diminishing returns fairly quickly for low latency apps, it's just not worth the time to try and optimize beyond a certain point.

These tests are useful for basic smoke testing, but don't really give details that matter the most. Like what is the average latency and standard deviation at a specific payload size/throughput/concurrency.

The single biggest impact you can make is manual allocation and management of buffers. Like I would expect DotNetty to trounce everyone in these tests primarily because of that. While you can't use DotNetty in Unity (maybe 2018 haven't tried yet) you can probably borrow/steal some of the work they have done. Like just ripping out IByteBuffer and friends would provide a really good base for buffer managment.

 

@snacktime I was thinking about the same things (especially the DotNetty part). Quite useful information, thanks!

@Whippets Let's wish good luck to Alex.

 

BenchmarkNet 1.02 has been released.

• Added support for Hazel
• Updated LiteNetLib to the latest version
  
• Fixed sending of byte arrays for LiteNetLib
• Fixed integer overflow in calculations
• Fixed minor bugs

First post updated with the new results of Hazel and LiteNetLib.

Thanks to @Jamster for the help and rapid improvements to Hazel (changes are already in the master branch).

 

I still have work to do! I'm not happy with those results!

 

Developers from different teams/companies bombard my email with results of their forks and suggestions. I didn't expect such an effect...

 

What you've started is massively important.

 

Please @Jamster, make the miracle! ;-) I need Hazel and I like library very much!!!

 

@nxrighthere

I think that is very important to add to your very good benchmark also:
https://github.com/KillaMaaki/ReliableNetcode.NET (Please also: https://github.com/KillaMaaki/Netcode.IO.NET)
It is the "C# port of Glenn Fiedler's secure UDP communication protocol inspired by reliable.io and yojimbo" (See: https://github.com/networkprotocol/yojimbo)

And:
https://github.com/MarkioE/Networker
https://github.com/BeardedManStudios/ForgeNetworkingRemastered

Your project is very interesting!!

 

Yeah, I know about all of these networking solutions. I'll integrate them into the application, but a little bit later.

 

Is it relevant at all to include Photon results (it has a free tier) ?

 

Sure, why not. I'll take a look what I can do.

 

This looks like an awesome project and may singlehandedly improve the quality of Unity networking solutions across the board.

 

ENet (3000 clients) on 6-core Intel Xeon...



CPU usage 34%, memory consumption 200 megabytes.

 

Photon has real benchmark ;-)

https://steamdb.info/app/438100/graphs/

 

Irrelevant stats as the same testing is not performed. I don't think we can perform the test either.

 

That's not a real benchmark. That's player counts? For all we know they could be 10 players per server while having the game world sharded across many servers.

 

The second attempt to integrate Forge was again unsuccessful. The server is working fine, but the clients are not connecting. Moving forward to Photon.

 

@nxrighthere This must be taking up a ton of your time. Have you got a patreon or donation page set up? It's quality information that we're all benefiting from XD

 

Have you tried using the binary frame for sending the messages with Forge?
Link to docs
Very interested in Forge results

 

It is only a joke ;-)

 

This is some amazing work, but you _have to_ to post these benchmarks on the github page. If not, people will google and end up here looking at outdated benchmarks in the future.

 

He did?

 

@NFMynster I tried everything using the latest version from the master branch, compiled with Roslyn. Forge is just not working and I have no idea why.

@toreau A note about the results is mentioned in readme on GitHub page.

@Whippets I have a PayPal and Bountysource, but I don't think anyone is interested in this.

 

No.

 

Yes, but it would be much better (IMO) if you posted results over time on your github wiki page. Progress would be nice to see, to get a grasp of a project's "livelyness", plus you can always refer to an updated benchmark instead of linking to this already outdated benchmark in this thread.

 

Well, it makes sense. Thank you for the suggestion.

 

@nxrighthere You can find the latest Forge builds on this site.