Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Unity 2022.2 is now available as the latest Tech release.
    Dismiss Notice
  3. We are making some changes to the DOTS forums.
    Dismiss Notice
  4. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

Unity Multiplayer Mirror - Open Source Networking for Unity

Discussion in 'Multiplayer' started by vis2k, Aug 11, 2016.

  1. yoonitee

    yoonitee

    Joined:
    Jun 27, 2013
    Posts:
    2,337
    i don't get what you're saying :(
     
  2. dhhdumi

    dhhdumi

    Joined:
    Aug 3, 2019
    Posts:
    3
    Can I use it as a message forwarding service separate from Mirror?
     
  3. ysleungrockman

    ysleungrockman

    Joined:
    Mar 16, 2014
    Posts:
    31
    I tried to use Mirror for my local network project. The project was originally built with UNET but since how unstable UNET was (disconnecting all the time), Mirror was used to replace it. The project was divided into two parts. One is an Android app only for client and another is a PC app that is only for server. They are built with two different Unity projects. The server mainly controls how the client goes, changing scenes or performing actions by sending commands to client. And clients basically just send back their current status and progress data for server side to monitor. Because of that, the two apps only use NetworkClient.Send and NetworkConnection.Send to communicate. They both have RegisterHandler for receiving the message of same NetworkMessage struct. However, while testing, messages don't seem to be received. It seems like the handlers are not set but it was totally set. After some debugging, it seems like they have different message IDs even when the struct name and structures are the same. My question is, is it because they are two different builds from two different projects, the apps couldn't recognize the exact data type?
     
  4. Lion-Chen

    Lion-Chen

    Joined:
    Oct 13, 2021
    Posts:
    4
    Help needed :(

    Became sponsor, but cannot use Discord due to some really sad reasons.
    Is there any other way to get in touch with the Mirror team and get the profiler?
     
    Last edited: Jun 10, 2022
  5. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    This forum thread works :)
    Profiler is outdated though.
    It was made by Paul, and we don't have the rights to modify it so we can't update it.
    Sorry about that.
     
  6. Lion-Chen

    Lion-Chen

    Joined:
    Oct 13, 2021
    Posts:
    4
    Thanks for the reply :D Got most of the work done without using the profiler during the weekend :D

    I think this should be mentioned in the sponsor page or the profiler documentation page :oops:
     
  7. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Where did you still see a mention of profiler?
    Should be gone pretty much anywhere by now.
     
  8. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Snapshot Interpolation V2 preview:
    https://github.com/vis2k/Mirror/tree/snapint2_preview

    I recommend to try this:
    - open the snapshot interpolation demo
    - select client cube, set interpolate=false
    - press play, see default behaviour with 5% jitter, drops, etc.
    - set interpolate=true to see it become smooth

    stop

    - set jitter = 0.20
    - interpolate=false to see how it looks without interpolation
    - set interpolate=true, notice **blue** color while slowing down (still jittery)
    - notice **bufferTimeMultiplier** increasing in inspector dynamically
    - after a few seconds, when it's not blue anymore, it'll be stable

    you can increase the dynamic tolerance for even more stability, with the extra cost of latency.
     
  9. Lion-Chen

    Lion-Chen

    Joined:
    Oct 13, 2021
    Posts:
    4
  10. MisterZhou

    MisterZhou

    Joined:
    Nov 9, 2013
    Posts:
    13
    Error: Connection was closed by peer, Timed out attempting to connect

    When I used mirror+fizzy to make steam multiplayer online games, I encountered a problem that I couldn't connect to the host. I haven't figured out why for several days. Can the official mirror staff answer it for me?

    Is there anything wrong with my settings?

    Looking forward to your answer.Tanks!
    upload_2022-6-24_17-31-41.png upload_2022-6-24_17-32-6.png
     
  11. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    **Snapshot Interpolation V2** is live on Github.
    You can also test it easily with the example.
    More news soon... :)
     
  12. brilliantgames

    brilliantgames

    Joined:
    Jan 7, 2012
    Posts:
    1,937
    Hello. We have been looking into and testing mirror networking for a large scale PVP using telepathy transport. We have a problem though. During stress tests it seems the server is bottle necking at around 350 mega bits output in a local network. The server is running on a higher end gaming PC with an 8 core 16 thread AMD CPU, 32 gigs ram. The motherboard ethernet connection is capable of 2.5 giga bits, so I do not believe there is any bottle neck there.

    Here are a few conditions we are simulating, to emulate a massive PVP battle.

    -We are using RPCs on the server to sum up all player data and send as one large message to each connection(Very optimized, each player is an average of 2-3 bytes per send, 10 times per second).

    -The processing side in these test is extremely light weight, each client can run upwards of 40 exe instances to simulate high connection count at good framerate.

    -To simulate even higher player counts in the thousands, we increase multipliers appropriately. A player multiplier of 10x, sends all player positions 10x per send to eachclient, as well as player byte data being multiplied 10x. (We reach 1200-1500 players before the server seems to bottle neck, and cannot keep up with messages, but bandwidth is only 350 megabits per second).

    Our question here is.. Is this a server CPU bottle neck? Or is there something we are missing here? The performance on the server is a flawless 60FPS with no hiccups. Just seems a little strange that a high end gaming PC seems unable to get more than 350 megabits of messages out in a local network. To clarify, when 1500 players are simulated, roughly 15,000 messages are being sent per second, each message being 3 kilobytes per message (15,000 X 3kb = 45 megabytes).
     
    Last edited: Jun 28, 2022
  13. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Unity & Mono are the bottleneck.

    For example, let's say you want to make a TCP transport like Telepathy.
    The best solutions with the most throughput are generally SocketAsyncEventArgs or async/await.
    The worst solution is to use 2 threads per connection.

    Now unfortunately, SocketAsyncEventArgs, async/await, or generally anything with 'async' in the name is horribly slow only in Unity (we tried).

    Okay, another approach might be to use nonblocking TCP sockets.
    That's actually a thing, even though not very common.
    Where you set the sockets to non-blocking, and you won't need threads as long as the buffers don't get full.
    Which can be fine, if you set a reasonable buffer size.

    Alright, that also works fine outside of Unity.
    If you try to do it inside of Unity (which is what we did), you'll notice that your nonblocking sockets are actually still blocking sometimes, which is not great to say the least.

    Aparrently this is known. If you look at the mono source, there's actually a class in there that attempts to detect and unblock non-blocking sockets which sometimes occasionally block.
    Besides the fact that this class shouldn't exist in the first place, the unblocking of blocking non-blocking sockets is actually broken too, it does not always work.

    .... as result, that's why we use the super old '2 threads per connection' model for our TCP transport in Unity.
    Which in general is the worst way to do, but in Uniity is unfortunately still the best scaling way to do it for TCP.

    ----------

    For more practical advice, I recommend trying our kcp transport.
    It's essentially reliability (like TCP) but on top of UDP sockets which don't block, and don't require threads.
    Additionally, it also allows for lower latency.
    That's why kcp exists, it's basically 'let's do tcp but tweak the parameters for lower latency for games'.

    Now there are also a few issues (and open bugs) with kcp.
    Fortunately, we fixed those for you in our kcp transport :)

    It's really working quite well at this point.

    ----------

    Note that we are going to do a V2.0 version for that transport in the near future :)
    TLDR: try kcp transport; hope that Unity switches from mono to netcore asap.

    ----------

    What game are you working on by the way? Any videos?
     
  14. brilliantgames

    brilliantgames

    Joined:
    Jan 7, 2012
    Posts:
    1,937

    Thanks for the advice. However, with more testing last night, I actually concluded that the bottle neck seemed to be on the client side. The reason the client was bottlenecking bandwidth, was because I was running 20 Unity exe instances to simulate 20 clients connected. This was causing the CPU usage to be too high to keep up with the very high amount of messages coming through. Basically, because of the way I was simulating high player counts, the client was doing around 20X more work than the server had to do.

    With a single client connected with my player count simulation multiplier ((playercount x playermultiplier) x (playercount x playermultiplier) x (playerbytes) and sending all data to a single client, I was able to simulate near 4000 players with over 800 megabits per second output.

    traffic.png

    So. I am pretty certain I should be able to get near this output in a normal scenario. Since with the simulation I made, a single client ends up having to do too much work when he is acting as 20, which causes the server to back up.

    Is there any way to send an RPC as 'unreliable' so that the server does not keep sending the message? Also how much memory is used in a single empty RPC? I'm guessing some sort of index is attached to know what RPC function to call(4-8 bytes?)?

    "What game are you working on by the way? Any videos?" This is very early, and is not public so I cannot show anything here. However, you can take a look at our latest title, Ultimate Epic Battle Simulator 2(Hundreds of videos on Youtube). We specialize in crowd tech, and heavy optimization.
     
    Last edited: Jun 29, 2022
    vis2k likes this.
  15. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Looks good, keep us updated!

    For the RPC:
    2022-06-30 - 13-02-40@2x.png

    Check our RpcMessage struct to see the total size :)
     
  16. brilliantgames

    brilliantgames

    Joined:
    Jan 7, 2012
    Posts:
    1,937
    Thanks so much, I never saw this command in documentation(Might have missed it?). Another question, why is it that unreliable messages have a 1200 byte limit in kpc transport? PS, I was still having issues getting all the bytes out when using TPC while connected over the internet. However with KPC I was able to max out my internet connection(750 megabits), but at the cost of poor performance on the server. The frame rate goes down to around 4-8FPS when simulating 1800 players over internet connection, which limits our ambitions of being able to simulate near 4000 players with a 2.5 gigabit internet connection which we are upgrading to soon. Is there any way I can modify the server side to use threading with KPC? I have extensive experience in multithreaded operations so I should be able to make the modifications myself if it's possible.
     
  17. akuno

    akuno

    Joined:
    Dec 14, 2015
    Posts:
    54
    @vis2k
    Hello my friend, lately I'm having a hard time finding solutions whenever I run into some issue with Mirror.

    Most of the troubleshoot is happening in Discord, which is a nightmare to search.
    Also it is inefficient. In discord you give a solution to a single person asking, but in a forum a solution is used by multiple people.

    Can we get some kind of Mirror forum? It is a much better format to find and offer help.
    I can even help with that if you want.
     
  18. JamesFrowenDev

    JamesFrowenDev

    Joined:
    Oct 10, 2015
    Posts:
    18
    The unreliable channel does not fragment message, so the limit will be MTU.

    The reason it does not fragment is because the more packets you send the higher chance the whole message will be dropped. If any of the fragments are lost the whole message would need to be counted as lost. If you have 2% packet loss, and you spit up a 5000 byte message into 5 packets you now have a 10% change that message will be dropped.
     
    vis2k likes this.
  19. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    We have enough on our plate, but if the community would like to help then sure.

    We asked the Unity guys for a Mirror subforum here before, since we are supporting UNET users anyway.
    But that didn't happen obviously :)

    We still own r/MirrorNetworking on reddit.
    Looks like people are actually using it now, too.

    Would that work for you?
     
    akuno likes this.
  20. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    4000 players is a lot(!) for Unity.
    Common wisdom is to target 300-500 CCU if you run the server in Unity.
    Maybe you can give more details about your game so we can help you better.
    Outside of Unity that would be fine though.

    I wanted to announce this later, but since you asked.
    We are working on Mirror V2.0, where in order to scale, the critical part of Broadcast() + Transport are spread across worker threads.
    I am not saying you should wait for Mirror V2, but perhaps we can backpart the thread safe transport to V1.

    If you like to help with the transport, please tag me in Discord (@mischa) so we can discuss some more details.
     
    Last edited: Jul 1, 2022
  21. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    horizontal 2500 x 1000.png
    soon.
     
    akuno likes this.
  22. akuno

    akuno

    Joined:
    Dec 14, 2015
    Posts:
    54
    EXCITING

    That'd be better, but still very volatile. Non trendy topics get buried away, even if they have 1000s of upvotes. We'd have google indexing which would be nice.

    If people are interested, I could setup up a Forum on my VPS and give full admin rights to your people.

    Also looks like Discord is testing a forum feature. Maybe that's worth a look.
    It seems to be accessible if you have Discord Partner status:

    https://support.discord.com/hc/en-us/articles/6208479917079-Forum-Channels-FAQ

     
  23. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    We will try discord forums when they are available to us.
    For now, I think reddit should do.
    Community members disappear sometimes, so we shouldn't take the risk of 'someone else' hosting again :)
     
    akuno likes this.
  24. LiLyDarcy

    LiLyDarcy

    Joined:
    Oct 10, 2018
    Posts:
    15
    Which has lower bandwidth, tcp or kcp? Hope to get help thanks.
     
  25. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    kcp is basically a tcp implementation on top of UDP sockets, where a little more bandwidth is traded for a little lower latency.
     
  26. brilliantgames

    brilliantgames

    Joined:
    Jan 7, 2012
    Posts:
    1,937
    Yes 4000 is a lot. With the heavy optimizations we have done for bandwidth, it should be possible, and that's without any network culling or LOD. With network culling and Lod, 10,000 players should theoretically be possible. I think processing is going to be our bottle neck here. I tried to find the RPC struct you mentioned, and could not find anything. However I did a test sending a bunch of empty unreliable RPC messages and checked the network output. It appears RPCs take somewhere between 12-16 bytes per message, which seems pretty high. Are you guys sending an entire string of the function name for the RPC calls? Cause if you used indexing, you could use a single uint16(2 bytes) to transmit the function that should be called. All indexing in Mirror should be using uint16, since you would never need more than 65,000 functions and variable types. (Apologies if you are already doing this)
     
  27. JamesFrowenDev

    JamesFrowenDev

    Joined:
    Oct 10, 2015
    Posts:
    18
    this is the RPC message that mirror has https://github.com/vis2k/Mirror/blob/master/Assets/Mirror/Runtime/Messages.cs#L37-L45
    Code (CSharp):
    1.     public struct RpcMessage : NetworkMessage
    2.     {
    3.         public uint netId;
    4.         public byte componentIndex;
    5.         public ushort functionHash;
    6.         // the parameters for the Cmd function
    7.         // -> ArraySegment to avoid unnecessary allocations
    8.         public ArraySegment<byte> payload;
    9.     }
    the size breaks down like this:
    - 2 bytes for NetworkMessage (hash of the struct name)
    - 4 bytes for netid
    - 1 byte for componentIndex
    - 2 bytes for functionHash (hash of the rpc function name)
    - 4 bytes for payload length
    = 13 bytes total

    I dont think mirror currently uses var-int compression, which would effect he size of netid and payload length. If you were to add var-int it would save 4-6 bytes from each rpc.

    Best case you are looking at something like this:
    - 2 bytes for NetworkMessage
    - 1-3 bytes for netid (1 byte if under 240, 2 bytes if under 2287)
    - 1 byte for componentIndex
    - 1 bytes for functionIndex (index of rpc inside each NB)
    - 1 bytes for payload length
    = 6-8 bytes total

    If you dont need to target Networkbehaviours, then you can optimize by avoiding RPC and just send custom NetworkMessages
     
    vis2k and akuno like this.
  28. MisterZhou

    MisterZhou

    Joined:
    Nov 9, 2013
    Posts:
    13
    When I try to connect to SteamLobby. Mirror said“We don't have cert, and self-signed certs not allowed.
    What does this mean?

    Code (CSharp):
    1. networkManager.networkAddress = param.m_ulSteamIDLobby.ToString();// SteamMatchmaking.GetLobbyData(new CSteamID(param.m_ulSteamIDLobby), HostAddressKey);
    2. networkManager.StartClient();
    Is the networkaddress not the SteamID now?
    FizzySteamworks v5.0.1
    Mirror Version 66.0.9
     
    Last edited: Jul 6, 2022
  29. MisterZhou

    MisterZhou

    Joined:
    Nov 9, 2013
    Posts:
    13
  30. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Steam transport was made by a community member.
    Please try to join discord, find that member and ask him :)
     
    MisterZhou likes this.
  31. akuno

    akuno

    Joined:
    Dec 14, 2015
    Posts:
    54
    @vis2k Is there anything you can tell us about Mirror V2?

    1. What made you decide to have it V2, is it being rewritten completely?
    2. Is there any special focus you want to have on the new version, like performance, usability, more built-in features (e.g. lag compensation)?
    3. Have you learned some mysterious data locality secrets from DOTSNET that you wanna apply there?
     
  32. brilliantgames

    brilliantgames

    Joined:
    Jan 7, 2012
    Posts:
    1,937

    Thanks. However, I've tried switching over to generic network messages but I had an issue. It causes clients joining to sometimes fail and disconnect. Even when I send the message to 'ready clients only'. Any idea why this might be happening? I can't seem to get it to fail in editor as client, so it's hard to debug. So I just went back to RPCs.
     
  33. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Sure.
    Will link a longer text here soon.
     
    akuno likes this.
  34. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Last edited: Jul 10, 2022
    akuno and Willbkool_FPCS like this.
  35. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
  36. nolaggg

    nolaggg

    Joined:
    Jul 17, 2022
    Posts:
    1
    Hi, I was working on KCP server-side networking, not directly on mirror. But I've read Mirror's kcp transport implementation, it's differ approach from my project. Can you tell me more detail about which is the root cause for low fps on server side of your server? Any helpful logs for now?

    I'm curious about your project too and was wondering how your server stress test is going? AFAIK, 4000++ ccu for mmorpg is an impressive high number, even with greatly optimized servers.
     
    Last edited: Jul 17, 2022
  37. brilliantgames

    brilliantgames

    Joined:
    Jan 7, 2012
    Posts:
    1,937
    @vis2k
    Just thought I would throw an update your way. We decided to op out of Mirror and make our own networking system. We went with UDP and we are getting 250 megabytes per second(2000 megabits) output over the internet using 10 threads on medium end Ryzen. Thanks for all the help though, mirror is a great tool, just not quite at the bandwidth level we were looking for. PS, we tried TPC and for some reason no matter what we did, we couldn't get above 1.6 megabytes per second on a single thread, is this normal? I basically tried everything and then moved over to UDP, and got far better results(Yes I am aware that UDP is unreliable).
     
    Last edited: Jul 19, 2022
  38. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    You built your own UDP transport with fragmentation & reliability?
    Would you be interested in open sourcing that?
     
  39. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Mirror II - coming soon :eek:
    2022-07-02 - 01-31-12@2x.png

    After 3 months of development, Mirror II syncs the Benchmark demo for the first time today.
    There will be a detailed blog post with all the improvements near the end of this month!
    2022-07-19 - first benchmark demo.png
     
    akuno and brilliantgames like this.
  40. brilliantgames

    brilliantgames

    Joined:
    Jan 7, 2012
    Posts:
    1,937
    It's still pretty young at the moment. Just got RPCs functioning yesterday, and we need to ensure the threading part is completely safe still(Atomic counters, ect..). Also, we haven't done reliable messages yet, though I haven't seen any packet loss yet, even when we did VPM tests on the other side of the world(Reliable message are our next task). Our RPC messages are really low overhead, just 4 bytes.

    It's possible we might be able to share some code with you if we have the time, and it doesn't compromise our project.
     
  41. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Got it.
    Well good luck, reliability & fragmentation on top of UDP are difficult to get right.
    That's why we use KCP - it's battle tested in a lot of games, and independent of UDP / sockets.
    You could use that for reliability & fragmentation, and still put it on threads FYI :)
     
  42. brilliantgames

    brilliantgames

    Joined:
    Jan 7, 2012
    Posts:
    1,937
    When you say fragmentation, are you referring to large packets? Or hackers messing with the server? We already know how we are going to do reliable messaging though, so we aren't super worried about that one.
     
  43. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Yes, for packets > 1200 ish bytes MTU you would need fragmentation.
     
  44. neoangelique

    neoangelique

    Joined:
    Nov 19, 2014
    Posts:
    25
    Code (CSharp):
    1. > Configure project :launcher
    2. WARNING: The option setting 'android.enableR8=false' is deprecated.
    3. It will be removed in version 5.0 of the Android Gradle plugin.
    4. You will no longer be able to disable R8
    Code (CSharp):
    1. * What went wrong:
    2. Execution failed for task ':launcher:mergeReleaseNativeLibs'.
    3. > A failure occurred while executing com.android.build.gradle.internal.tasks.Workers$ActionFacade
    4.    > More than one file was found with OS independent path 'lib/armeabi-v7a/libEOSSDK.so'. If you are using jniLibs and CMake IMPORTED targets, see https://developer.android.com/studio/preview/features#automatic_packaging_of_prebuilt_dependencies_used_by_cmake
    I'm running into this issue when trying to build to android. I've tried these solutions: How to remove android.enableR8' from the project - Unity Answers but unable to solve it. The issue might be related to the EOS transport. Any advice is appreciated.

    Have also set minimum android api level to marshmellow as indicated in the EOS Transport of Mirror's Android guide.

    EDIT:
    Could this be caused by the fact that I didn't download Mirror and the Epic Transport via the Asset Store and Package Manager? I pulled directly from github repo.
     
    Last edited: Jul 20, 2022
  45. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Asset store is always the safest.
    But likely not related.
     
  46. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
  47. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    In order to scale, Mirror II will be multithreaded.
    Dedicated servers with 8/16/32/64/96 cores are easily available.
    It's time for us to utilize them in order to develop larger worlds.

    Mirror II has interchangeable Broadcaster components.
    -> SequentialBroadcaster: original Mirror main thread.
    -> ThreadedBroadcaster: multiple worker threads.
    2022-07-22 - broadcasters.png

    Worker threads handle a configurable amount of connections each.
    In the future, there could also be a Burst/Jobs Broadcaster.

    For now, the goal is to use fully managed C# with minimum dependencies.
    As of today, the Threaded Broadcaster is working.
    While there will be more improvements over classic Mirror, this feature is the most significant advancement.

    As mentioned, a blog post with more answers will follow soon.
     
    brilliantgames and akuno like this.
  48. dvkfurkanbey

    dvkfurkanbey

    Joined:
    Apr 15, 2017
    Posts:
    2
    Hello,
    I like to use mirror in my projects.
    But I'm having trouble using mirror in unity 2021.3.7 version.
    This error occurs when I drag an object with NetworkIdentity component to the hierarchy panel.
    How can i solve this problem?

    1.PNG

    2.PNG
     
  49. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Please show the full stack trace that you see when clicking the error :)
     
  50. vis2k

    vis2k

    Joined:
    Sep 4, 2015
    Posts:
    4,260
    Very good progress on Mirror II.
    Both Benchmark & Tanks demo are fully working now.
    2022-08-01 - tanks.png