Search Unity

Question Reliability can't be maintained/too many pipeline updates/metrics went over configured limits

Discussion in 'Unity Transport' started by CosmoM, Jul 13, 2022.

  1. CosmoM

    CosmoM

    Joined:
    Oct 31, 2015
    Posts:
    204
    I am using Unity Transport in combination with Netcode for GameObjects. Recently, Unity Transport gives me a load of warnings whenever I connect even a single client to my server. Initially, the warning I was spammed with was:

    Some metrics in the collection went over the configured values limit. Some values were ignored.


    When I set the max payload size to what I think are extremely high values (e.g. 200,000), these went away, but they were replaced by:

    A lot of pipeline updates have been queued, possibly too many being scheduled in pipeline logic, queue count: 68


    I get these messages constantly, and often, at some point the connection is closed with the following error:

    Couldn't add payload of size 391 to reliable send queue. Closing connection 1 as reliability guarantees can't be maintained. Perhaps 'Max Send Queue Size' (98304) is too small for workload.


    What really gets me is that increasing the payload size helped for a little bit, but even though I'm not spawning any more network objects than I did before, the warnings and disconnections happen more and more frequently. What can I do to prevent this?
     
  2. CosmoM

    CosmoM

    Joined:
    Oct 31, 2015
    Posts:
    204
    To add to this: with every warning the delay between client and server is going up. When the round-trip-time reaches 3 seconds (this takes less than 20 seconds), the client is disconnected. So is the ping going up because every tick there are too many messages to send, causing a buildup...?
     
  3. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    441
    As the last error message mentions, it seems the 'Max Send Queue Size' setting might be too small for the workload you have. Since the send queue is where messages are stored if they can't immediately be sent to the network, its value should be larger than the maximum payload size, since it possibly needs to accommodate a maximally-sized payload. I'd suggest making it a multiple of the maximum payload size (the default value is 16 times the default maximum payload size, although for a maximum size of 200,000, 16 times is probably a bit overkill).

    Regarding the warning about metrics values being ignored, I believe this is caused by the multiplayer tools package. I know a fix was made, but I'm uncertain whether it's been released yet. It might be worth trying to update the package to its latest version.

    For the warning about the pipeline updates, are you perhaps using the simulator to add delay or drop packets? That's one way the counter triggering this warning can go up (although I'm perplexed about it reaching such high values, we might have a bug here).
     
    CosmoM likes this.
  4. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    441
    And yes, increasing round-trip times would tend to indicate a buildup in the queues, which will eventually lead to a disconnection when the send queue is filled. Increasing the send queue size would help, but if round-trip times are increasing for seconds on end, it's likely there's something else going on. Perhaps the server is not being updated frequently enough to process all the incoming data (which will cause reliable packets to back up waiting for ACKs)?

    Another possibility is having too much delay or packet loss on the network, and the flow of outgoing traffic outpaces the bandwidth of what can be reliably delivered. Normal networks shouldn't have that much delay or packet loss (unless we're talking about massive amounts of outgoing traffic), but it'd be easy to add that much delay or packet loss artificially. One way would be the simulator that's bundled with Unity Transport. In newer versions of Netcode for GameObjects, the parameters for the simulator are under the Unity Transport component. In older versions, they're in a window accessible under the Netcode menu item in the editor. I'd double-check that these values are all at 0.
     
    CosmoM likes this.
  5. CosmoM

    CosmoM

    Joined:
    Oct 31, 2015
    Posts:
    204
    Thanks for the quick reply! I tried setting the max send queue size to several times the max payload size, but that didn't help. However, I just tried setting the packet delay which I always have set to 100 ms (with jitter and drop rate to 0) to 0 as well, and that made all the issues go away!

    I've been working with a 100 ms delay for months though, so why is this suddenly a problem? It could well be since the last update.

    EDIT: After some more testing the "some metrics..." warnings do come back occasionally, any telling which limits I should be adapting? They are now accompanied by warnings like:

    [Netcode] Deferred messages were received for a trigger of type OnSpawn with key 17, but that trigger was not received within within 1 second(s).
     
    Last edited: Jul 13, 2022
  6. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    441
    Not sure why the delay would suddenly start causing more problems. I'm not aware of any changes on the Unity Transport side that would cause this change in behavior. I suppose there could have been changes on the Netcode side that cause more traffic to go out (perhaps unintentionally), which might explain it.

    I'm not familiar enough with Netcode for GameObjects to explain the deferred messages warning, but it seems to me like it would be caused by a high transmission time for some messages. This would also point to something changing in recent versions of Netcode that would cause extra delay (or more reliable traffic to be sent, which would cause delay too). If you're willing to go through the hassle, you could try measuring the traffic used by the previous version of Netcode and compare it with the current version (for the same scene and gameplay). If you notice significantly more traffic, I'd consider this worthy of filing a bug.

    Regarding the metrics warning, there is active work on improving its behavior. Unfortunately I don't know when this will land or be released. In the meantime, I think it should be fine to ignore them (although I understand that this is annoying). The warnings are only indicating that some metrics events could not be processed, which should have no impact on gameplay.
     
    CosmoM likes this.
  7. CaptainCrouton89

    CaptainCrouton89

    Joined:
    Apr 18, 2019
    Posts:
    2
    My team is having all of the problems except the pipeline issue that the original poster listed, and we are using all of the latest updates. Just wanted to throw it out there :)