Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Messeges seem to stop, with no error or warning.

Discussion in 'Multiplayer' started by donnysobonny, Mar 22, 2017.

  1. donnysobonny

    donnysobonny

    Joined:
    Jan 24, 2013
    Posts:
    220
    So I'm nearing the launch of a game that I've been working on for some time, and recent testing has seemed to bring to light a bug that I cannot seem to get to the bottom of.

    I am using a fairly custom setup, where I avoid using HLAPI components such as the NetworkTransform and NetworkManager. Instead, i'm using a simple request/response/event setup, where the client sends requests, and the server can send responses and events. As far as I can tell, this all works fine, but I do wonder if there's maybe something that i've missed. Frustratingly, it's all built and reliant on unet, which makes it hard to know if it's an issue with something i'm doing...

    Ultimately, sending just stops, without any error on the client or server side. Send requests invoked after sending stops don't fail (no errors are generated) and seem to continue to pile up until eventually the sending-side (server) disconnects the other side (client) due to timeout, although this doesn't happen every time. On some occasions also, the sending-side (server) will report that the channel buffer has maxed out (16) which I assume is just because the messages have piled up too much.

    Pretty much everything that I send over the network is sent on a reliable channel, because I have been able to avoid the need to send things like positional/rotational updates where I would need to send them frequently and can account for package loss. In stead, I am sending the position/rotation heading reliably, when it changes, so that the client can continue to move/rotate objects at the speed that the object should move. Because of the way it's all set up, everything is kept in sync and very snappy.

    So could it be something to do with this? Is there something that could cause the sending of messages to just "hang"? I first thought that if a reliable message didn't get through (on a sequenced channel), this could cause other messages to queue up, but surely if the acknowledgement isn't received on the sending-side, it just re-sends the message until it gets through?

    Any help here would be greatly appreciated.
     
  2. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    Unfortunately You do not give enough information for answer.
    -What version of Unity do you use?
    -What is your configuration?
    -Is it lan, wifi, global internet (wan)?
    -What is your average message size?
    -How ofted do you send messages?
    What does it mean: "Ultimately, sending just stops, without any error on the client or server side." ? In our API we do not have any "Send requests invoked after sending stops don't fail", what is it? :)

    What does it mean: "Is there something that could cause the sending of messages to just "hang"?" What function hang? What's happened?

    Please clarify your question: )
    Sorry, so far I'm not able to help :(
    Alex
     
  3. donnysobonny

    donnysobonny

    Joined:
    Jan 24, 2013
    Posts:
    220
    5.5.2f1

    If you are referring to the connection config/global config/host topology etc, pretty much entirely as-is. The only thing i've changed is the ConnectionConfig.packetSize to 1470 from the default 1500 as per the documentation.

    I test the game over lan, building my server-side to my local computer and then connecting to that to test the client-side. When testing like this, I never see the issue mentioned above. It only seems to happen when I run my server-side on a dedicated server and connect to that instead. It seems to happen more, the more unstable the internet connection is (for example, it's happens more on mobile devices over wifi than it does in the web build connected directly to the router). So it's definitely more problematic the more unstable the connection is.

    I doubt that it's got anything to do with this. I've spent a lot of time ensuring that the size and count of my messages are as small as possible using things like delta compression, and implementing my own combining of small messages on top of unet's combining of small messages. We're probably talking about an average of around 200 bytes per second per player (so the client side will see that, while the server side will see that multiplied by how many players there are). It's worth noting that the issue mentioned in my original post happens with only one player on the server.

    The connection config's send rate is at the default 10ms, while I have implemented my own combining/delta compression where it will combine/compress the data to be sent and send it every 0.1 seconds during quiet times, and 0.15 seconds during peak times.

    It's probably easier to explain visually. On the client-side, what happens is the player connects, gets all of the data of networked objects on connect, and objects start moving around as per normal. However, normally after a few seconds (although this interval varies), the objects just stop moving, although they are still moving on the server-side. I did a good bit of debugging and found that the server was still sending updates about the movement of objects (via NetworkConnection.Send), but the client was not receiving the messages. Neither the server or the client side were reporting any errors while this was happening so it was as if the server thought it was successfully sending the messages but the client wasn't receiving them... hence why I though maybe the messages were being queued up successfully on the server but for some reason not being processed out of the queue?

    So as mentioned above, the NetworkConnection.Send method seems to work fine (returning true), even when messages appear to stop reaching the client, suggesting the messages themselves are maybe hanging in the queue? Not sure...


    UPDATE

    Since posting the original issue, i've managed to stop it from happening. I'm not sure exactly what I did to stop it from happening but i've been doing some heavy testing over the past few days and the issue hasn't happened once. The things that I have done though:
    • ditched the use of fragmented and sequenced channels. Now I only use a reliable/unreliable channel (although I rarely use the unreliable channel)
    • when the player initially connects, I "fragment" the sending of all of the game data myself now. I basically split all of the data up into smaller arrays (which are serialized into byte arrays) and send each array on a fixed interval of around 0.2 seconds
    UPDATE 2

    The delay of 0.2 between sending my custom fragmented room data was proving problematic, so I set out to try and lower it, and the issue started happening again. So I can confidently say that the issue has something to do with the large amount of data that is being sent when you initially connect, which due to the number of objects on the server can be anything from 10k to 40k bytes. Regardless of how I try and send the data (using my own fragmentation of the data, sending it in about 1k byte messages on a fixed interval until it is all sent; or sending it all in one go on a reliable fragmented channel), the result is always the same: sending stops without any error on either the client or server side, although sometimes the server does end up reporting that the channel buffers have maxed out after some time and in some cases it even disconnects the client without notifying the client (probably because it can't).

    So it almost seems like there is some sort of limit being reached, most likely due to how many bytes I need to send out to the client when they connect. If this is what is causing the issue then that's fine, I can work on that, but unet doesn't seem to be reporting on this to notify me that there is an error.

    Let me know if there is any other information that I can provide.
     
    Last edited: Mar 24, 2017
  4. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    Two cents, please check with the last 5.6 beta (problems should gone) and pleace set you disconnecttimeout to 3 or 4 sec - connectionconfig.disconnecttimeout = 4000 (wifi could have very large jitter) if it wont help ping me by mail we will try to manage this together
     
  5. donnysobonny

    donnysobonny

    Joined:
    Jan 24, 2013
    Posts:
    220
    I had a look at the patch notes for 5.6 and was tempted to give it a try, I was going to wait until it was released though since we're so close to releasing the game and I didn't want to cause more problems but this issue is causing concern. So yeah I'll give 5.6 a shot and PM you if I have any issues.

    I haven't ever been able to get the sending of the 10-40k bytes (when connecting) to work with unet alone. How should I be sending that? Should I be able to (in theory) send that amount of data in one single Send request, as long as it's on a reliable fragmented channel?

    Thanks a lot for the help. Much appreciated.
     
  6. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    I haven't ever been able to get the sending of the 10-40k bytes (when connecting) to work with unet alone. How should I be sending that? Should I be able to (in theory) send that amount of data in one single Send request, as long as it's on a reliable fragmented channel?

    Yes you are correct, reliable fragmented should work.. ok, anyway, lets see what's next
     
    donnysobonny likes this.
  7. donnysobonny

    donnysobonny

    Joined:
    Jan 24, 2013
    Posts:
    220
    I had a few teething problems in getting 5.6 up and running, although mainly related to visual issues, but now that I've got it set up... wow. Awesome work to the improvements on the networking transport layer man! In some places I'm seeing around 10% better performance, and everything is literally 100% more stable. The issue described in my initial post here is also completely sorted.

    Again, awesome work! Thanks again for the help.
     
  8. aabramychev

    aabramychev

    Unity Technologies

    Joined:
    Jul 17, 2012
    Posts:
    574
    :( 10% is too bad. I did mistake in two algorithms both cause cache line loosing... Working right now on performance stuff. should be available in 5.6p1