Unity Multiplayer Mirror - Open Source Networking for Unity

i don't get what you're saying

 

Can I use it as a message forwarding service separate from Mirror?

 

I tried to use Mirror for my local network project. The project was originally built with UNET but since how unstable UNET was (disconnecting all the time), Mirror was used to replace it. The project was divided into two parts. One is an Android app only for client and another is a PC app that is only for server. They are built with two different Unity projects. The server mainly controls how the client goes, changing scenes or performing actions by sending commands to client. And clients basically just send back their current status and progress data for server side to monitor. Because of that, the two apps only use NetworkClient.Send and NetworkConnection.Send to communicate. They both have RegisterHandler for receiving the message of same NetworkMessage struct. However, while testing, messages don't seem to be received. It seems like the handlers are not set but it was totally set. After some debugging, it seems like they have different message IDs even when the struct name and structures are the same. My question is, is it because they are two different builds from two different projects, the apps couldn't recognize the exact data type?

 

Help needed

Became sponsor, but cannot use Discord due to some really sad reasons.
Is there any other way to get in touch with the Mirror team and get the profiler?

 

This forum thread works
Profiler is outdated though.
It was made by Paul, and we don't have the rights to modify it so we can't update it.
Sorry about that.

 

Thanks for the reply Got most of the work done without using the profiler during the weekend

I think this should be mentioned in the sponsor page or the profiler documentation page

 

Where did you still see a mention of profiler?
Should be gone pretty much anywhere by now.

 

Snapshot Interpolation V2 preview:
https://github.com/vis2k/Mirror/tree/snapint2_preview

I recommend to try this:
- open the snapshot interpolation demo
- select client cube, set interpolate=false
- press play, see default behaviour with 5% jitter, drops, etc.
- set interpolate=true to see it become smooth

stop

- set jitter = 0.20
- interpolate=false to see how it looks without interpolation
- set interpolate=true, notice **blue** color while slowing down (still jittery)
- notice **bufferTimeMultiplier** increasing in inspector dynamically
- after a few seconds, when it's not blue anymore, it'll be stable

you can increase the dynamic tolerance for even more stability, with the extra cost of latency.

 

https://mirror-networking.gitbook.io/docs/components/network-profiler

 

Error: Connection was closed by peer, Timed out attempting to connect

When I used mirror+fizzy to make steam multiplayer online games, I encountered a problem that I couldn't connect to the host. I haven't figured out why for several days. Can the official mirror staff answer it for me?

Is there anything wrong with my settings?

Looking forward to your answer.Tanks！

 

**Snapshot Interpolation V2** is live on Github.
You can also test it easily with the example.
More news soon...

 

Hello. We have been looking into and testing mirror networking for a large scale PVP using telepathy transport. We have a problem though. During stress tests it seems the server is bottle necking at around 350 mega bits output in a local network. The server is running on a higher end gaming PC with an 8 core 16 thread AMD CPU, 32 gigs ram. The motherboard ethernet connection is capable of 2.5 giga bits, so I do not believe there is any bottle neck there.

Here are a few conditions we are simulating, to emulate a massive PVP battle.

-We are using RPCs on the server to sum up all player data and send as one large message to each connection(Very optimized, each player is an average of 2-3 bytes per send, 10 times per second).

-The processing side in these test is extremely light weight, each client can run upwards of 40 exe instances to simulate high connection count at good framerate.

-To simulate even higher player counts in the thousands, we increase multipliers appropriately. A player multiplier of 10x, sends all player positions 10x per send to eachclient, as well as player byte data being multiplied 10x. (We reach 1200-1500 players before the server seems to bottle neck, and cannot keep up with messages, but bandwidth is only 350 megabits per second).

Our question here is.. Is this a server CPU bottle neck? Or is there something we are missing here? The performance on the server is a flawless 60FPS with no hiccups. Just seems a little strange that a high end gaming PC seems unable to get more than 350 megabits of messages out in a local network. To clarify, when 1500 players are simulated, roughly 15,000 messages are being sent per second, each message being 3 kilobytes per message (15,000 X 3kb = 45 megabytes).

 

Unity & Mono are the bottleneck.

For example, let's say you want to make a TCP transport like Telepathy.
The best solutions with the most throughput are generally SocketAsyncEventArgs or async/await.
The worst solution is to use 2 threads per connection.

Now unfortunately, SocketAsyncEventArgs, async/await, or generally anything with 'async' in the name is horribly slow only in Unity (we tried).

Okay, another approach might be to use nonblocking TCP sockets.
That's actually a thing, even though not very common.
Where you set the sockets to non-blocking, and you won't need threads as long as the buffers don't get full.
Which can be fine, if you set a reasonable buffer size.

Alright, that also works fine outside of Unity.
If you try to do it inside of Unity (which is what we did), you'll notice that your nonblocking sockets are actually still blocking sometimes, which is not great to say the least.

Aparrently this is known. If you look at the mono source, there's actually a class in there that attempts to detect and unblock non-blocking sockets which sometimes occasionally block.
Besides the fact that this class shouldn't exist in the first place, the unblocking of blocking non-blocking sockets is actually broken too, it does not always work.

.... as result, that's why we use the super old '2 threads per connection' model for our TCP transport in Unity.
Which in general is the worst way to do, but in Uniity is unfortunately still the best scaling way to do it for TCP.

----------

For more practical advice, I recommend trying our kcp transport.
It's essentially reliability (like TCP) but on top of UDP sockets which don't block, and don't require threads.
Additionally, it also allows for lower latency.
That's why kcp exists, it's basically 'let's do tcp but tweak the parameters for lower latency for games'.

Now there are also a few issues (and open bugs) with kcp.
Fortunately, we fixed those for you in our kcp transport

It's really working quite well at this point.

----------

Note that we are going to do a V2.0 version for that transport in the near future
TLDR: try kcp transport; hope that Unity switches from mono to netcore asap.

----------

What game are you working on by the way? Any videos?

 

Thanks for the advice. However, with more testing last night, I actually concluded that the bottle neck seemed to be on the client side. The reason the client was bottlenecking bandwidth, was because I was running 20 Unity exe instances to simulate 20 clients connected. This was causing the CPU usage to be too high to keep up with the very high amount of messages coming through. Basically, because of the way I was simulating high player counts, the client was doing around 20X more work than the server had to do.

With a single client connected with my player count simulation multiplier ((playercount x playermultiplier) x (playercount x playermultiplier) x (playerbytes) and sending all data to a single client, I was able to simulate near 4000 players with over 800 megabits per second output.



So. I am pretty certain I should be able to get near this output in a normal scenario. Since with the simulation I made, a single client ends up having to do too much work when he is acting as 20, which causes the server to back up.

Is there any way to send an RPC as 'unreliable' so that the server does not keep sending the message? Also how much memory is used in a single empty RPC? I'm guessing some sort of index is attached to know what RPC function to call(4-8 bytes?)?

"What game are you working on by the way? Any videos?" This is very early, and is not public so I cannot show anything here. However, you can take a look at our latest title, Ultimate Epic Battle Simulator 2(Hundreds of videos on Youtube). We specialize in crowd tech, and heavy optimization.

 

Looks good, keep us updated!

For the RPC:


Check our RpcMessage struct to see the total size

 

Thanks so much, I never saw this command in documentation(Might have missed it?). Another question, why is it that unreliable messages have a 1200 byte limit in kpc transport? PS, I was still having issues getting all the bytes out when using TPC while connected over the internet. However with KPC I was able to max out my internet connection(750 megabits), but at the cost of poor performance on the server. The frame rate goes down to around 4-8FPS when simulating 1800 players over internet connection, which limits our ambitions of being able to simulate near 4000 players with a 2.5 gigabit internet connection which we are upgrading to soon. Is there any way I can modify the server side to use threading with KPC? I have extensive experience in multithreaded operations so I should be able to make the modifications myself if it's possible.

 

@vis2k
Hello my friend, lately I'm having a hard time finding solutions whenever I run into some issue with Mirror.

Most of the troubleshoot is happening in Discord, which is a nightmare to search.
Also it is inefficient. In discord you give a solution to a single person asking, but in a forum a solution is used by multiple people.

Can we get some kind of Mirror forum? It is a much better format to find and offer help.
I can even help with that if you want.

 

The unreliable channel does not fragment message, so the limit will be MTU.

The reason it does not fragment is because the more packets you send the higher chance the whole message will be dropped. If any of the fragments are lost the whole message would need to be counted as lost. If you have 2% packet loss, and you spit up a 5000 byte message into 5 packets you now have a 10% change that message will be dropped.

 

We have enough on our plate, but if the community would like to help then sure.

We asked the Unity guys for a Mirror subforum here before, since we are supporting UNET users anyway.
But that didn't happen obviously

We still own r/MirrorNetworking on reddit.
Looks like people are actually using it now, too.

Would that work for you?

 

4000 players is a lot(!) for Unity.
Common wisdom is to target 300-500 CCU if you run the server in Unity.
Maybe you can give more details about your game so we can help you better.
Outside of Unity that would be fine though.

I wanted to announce this later, but since you asked.
We are working on Mirror V2.0, where in order to scale, the critical part of Broadcast() + Transport are spread across worker threads.
I am not saying you should wait for Mirror V2, but perhaps we can backpart the thread safe transport to V1.

If you like to help with the transport, please tag me in Discord (@mischa) so we can discuss some more details.

 

soon.

 

EXCITING

That'd be better, but still very volatile. Non trendy topics get buried away, even if they have 1000s of upvotes. We'd have google indexing which would be nice.

If people are interested, I could setup up a Forum on my VPS and give full admin rights to your people.

Also looks like Discord is testing a forum feature. Maybe that's worth a look.
It seems to be accessible if you have Discord Partner status:

https://support.discord.com/hc/en-us/articles/6208479917079-Forum-Channels-FAQ

 

We will try discord forums when they are available to us.
For now, I think reddit should do.
Community members disappear sometimes, so we shouldn't take the risk of 'someone else' hosting again

 

Which has lower bandwidth, tcp or kcp? Hope to get help thanks.

 

kcp is basically a tcp implementation on top of UDP sockets, where a little more bandwidth is traded for a little lower latency.

 

Yes 4000 is a lot. With the heavy optimizations we have done for bandwidth, it should be possible, and that's without any network culling or LOD. With network culling and Lod, 10,000 players should theoretically be possible. I think processing is going to be our bottle neck here. I tried to find the RPC struct you mentioned, and could not find anything. However I did a test sending a bunch of empty unreliable RPC messages and checked the network output. It appears RPCs take somewhere between 12-16 bytes per message, which seems pretty high. Are you guys sending an entire string of the function name for the RPC calls? Cause if you used indexing, you could use a single uint16(2 bytes) to transmit the function that should be called. All indexing in Mirror should be using uint16, since you would never need more than 65,000 functions and variable types. (Apologies if you are already doing this)

 

this is the RPC message that mirror has https://github.com/vis2k/Mirror/blob/master/Assets/Mirror/Runtime/Messages.cs#L37-L45

Code (CSharp):

1.     public struct RpcMessage : NetworkMessage
2.     {
3.         public uint netId;
4.         public byte componentIndex;
5.         public ushort functionHash;
6.         // the parameters for the Cmd function
7.         // -> ArraySegment to avoid unnecessary allocations
8.         public ArraySegment<byte> payload;
9.     }

the size breaks down like this:
- 2 bytes for NetworkMessage (hash of the struct name)
- 4 bytes for netid
- 1 byte for componentIndex
- 2 bytes for functionHash (hash of the rpc function name)
- 4 bytes for payload length
= 13 bytes total

I dont think mirror currently uses var-int compression, which would effect he size of netid and payload length. If you were to add var-int it would save 4-6 bytes from each rpc.

Best case you are looking at something like this:
- 2 bytes for NetworkMessage
- 1-3 bytes for netid (1 byte if under 240, 2 bytes if under 2287)
- 1 byte for componentIndex
- 1 bytes for functionIndex (index of rpc inside each NB)
- 1 bytes for payload length
= 6-8 bytes total

If you dont need to target Networkbehaviours, then you can optimize by avoiding RPC and just send custom NetworkMessages

 

When I try to connect to SteamLobby. Mirror said“We don't have cert, and self-signed certs not allowed.”
What does this mean?

Code (CSharp):

1. networkManager.networkAddress = param.m_ulSteamIDLobby.ToString();// SteamMatchmaking.GetLobbyData(new CSteamID(param.m_ulSteamIDLobby), HostAddressKey);
2. networkManager.StartClient();

Is the networkaddress not the SteamID now?
FizzySteamworks v5.0.1
Mirror Version 66.0.9

 

Hello,Can you focus on this issue? It seems to be very tricky.
https://forum.unity.com/threads/fiz...t-attempting-to-negotiate-rendezvous.1199293/

 

Steam transport was made by a community member.
Please try to join discord, find that member and ask him

 

@vis2k Is there anything you can tell us about Mirror V2?

1. What made you decide to have it V2, is it being rewritten completely?
2. Is there any special focus you want to have on the new version, like performance, usability, more built-in features (e.g. lag compensation)?
3. Have you learned some mysterious data locality secrets from DOTSNET that you wanna apply there?

 

Thanks. However, I've tried switching over to generic network messages but I had an issue. It causes clients joining to sometimes fail and disconnect. Even when I send the message to 'ready clients only'. Any idea why this might be happening? I can't seem to get it to fail in editor as client, so it's hard to debug. So I just went back to RPCs.

 

Sure.
Will link a longer text here soon.

 

Made with Mirror.
https://github.com/vis2k/Mirror#made-with-mirror

 

https://mirror-networking.com
all links, one place.

 

Hi, I was working on KCP server-side networking, not directly on mirror. But I've read Mirror's kcp transport implementation, it's differ approach from my project. Can you tell me more detail about which is the root cause for low fps on server side of your server? Any helpful logs for now?

I'm curious about your project too and was wondering how your server stress test is going? AFAIK, 4000++ ccu for mmorpg is an impressive high number, even with greatly optimized servers.

 

@vis2k
Just thought I would throw an update your way. We decided to op out of Mirror and make our own networking system. We went with UDP and we are getting 250 megabytes per second(2000 megabits) output over the internet using 10 threads on medium end Ryzen. Thanks for all the help though, mirror is a great tool, just not quite at the bandwidth level we were looking for. PS, we tried TPC and for some reason no matter what we did, we couldn't get above 1.6 megabytes per second on a single thread, is this normal? I basically tried everything and then moved over to UDP, and got far better results(Yes I am aware that UDP is unreliable).

 

You built your own UDP transport with fragmentation & reliability?
Would you be interested in open sourcing that?

 

Mirror II - coming soon


After 3 months of development, Mirror II syncs the Benchmark demo for the first time today.
There will be a detailed blog post with all the improvements near the end of this month!

 

It's still pretty young at the moment. Just got RPCs functioning yesterday, and we need to ensure the threading part is completely safe still(Atomic counters, ect..). Also, we haven't done reliable messages yet, though I haven't seen any packet loss yet, even when we did VPM tests on the other side of the world(Reliable message are our next task). Our RPC messages are really low overhead, just 4 bytes.

It's possible we might be able to share some code with you if we have the time, and it doesn't compromise our project.

 

Got it.
Well good luck, reliability & fragmentation on top of UDP are difficult to get right.
That's why we use KCP - it's battle tested in a lot of games, and independent of UDP / sockets.
You could use that for reliability & fragmentation, and still put it on threads FYI

 

When you say fragmentation, are you referring to large packets? Or hackers messing with the server? We already know how we are going to do reliable messaging though, so we aren't super worried about that one.

 

Yes, for packets > 1200 ish bytes MTU you would need fragmentation.

 

Code (CSharp):

1. > Configure project :launcher
2. WARNING: The option setting 'android.enableR8=false' is deprecated.
3. It will be removed in version 5.0 of the Android Gradle plugin.
4. You will no longer be able to disable R8

Code (CSharp):

1. * What went wrong:
2. Execution failed for task ':launcher:mergeReleaseNativeLibs'.
3. > A failure occurred while executing com.android.build.gradle.internal.tasks.Workers$ActionFacade
4.    > More than one file was found with OS independent path 'lib/armeabi-v7a/libEOSSDK.so'. If you are using jniLibs and CMake IMPORTED targets, see https://developer.android.com/studio/preview/features#automatic_packaging_of_prebuilt_dependencies_used_by_cmake

I'm running into this issue when trying to build to android. I've tried these solutions: How to remove android.enableR8' from the project - Unity Answers but unable to solve it. The issue might be related to the EOS transport. Any advice is appreciated.

Have also set minimum android api level to marshmellow as indicated in the EOS Transport of Mirror's Android guide.

EDIT:
Could this be caused by the fact that I didn't download Mirror and the Epic Transport via the Asset Store and Package Manager? I pulled directly from github repo.

 

Asset store is always the safest.
But likely not related.

 

Made with Mirror.

https://store.steampowered.com/app/1062520/Dinkum/

 

In order to scale, Mirror II will be multithreaded.
Dedicated servers with 8/16/32/64/96 cores are easily available.
It's time for us to utilize them in order to develop larger worlds.

Mirror II has interchangeable Broadcaster components.
-> SequentialBroadcaster: original Mirror main thread.
-> ThreadedBroadcaster: multiple worker threads.


Worker threads handle a configurable amount of connections each.
In the future, there could also be a Burst/Jobs Broadcaster.

For now, the goal is to use fully managed C# with minimum dependencies.
As of today, the Threaded Broadcaster is working.
While there will be more improvements over classic Mirror, this feature is the most significant advancement.

As mentioned, a blog post with more answers will follow soon.

 

Hello,
I like to use mirror in my projects.
But I'm having trouble using mirror in unity 2021.3.7 version.
This error occurs when I drag an object with NetworkIdentity component to the hierarchy panel.
How can i solve this problem?

 

Please show the full stack trace that you see when clicking the error

 

Very good progress on Mirror II.
Both Benchmark & Tanks demo are fully working now.