Search Unity

Question MaxPacketQueueSize ?

Discussion in 'Unity Transport' started by itisMarcii_, Aug 2, 2022.

  1. itisMarcii_

    itisMarcii_

    Joined:
    Apr 5, 2022
    Posts:
    111
    Is it safe to say that the MaxPacketQueueSize is the worst case of the Network?
    So basically if I got N-Clients with M-PossibleMessages. That the formular would look something like:

    N * NM
    for 100 Clients and 1 Message: 100 * 100 = 10.000

    Meaning 10.000 MaxPacketQueueSize?
     
    itisMarcii_, Aug 2, 2022
    #1

  2. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    441
    MaxPacketQueueSize
    is the number of packets that can be sent/received in a single update, across all connections. If you have 100 clients that are each expected to be sending 100 messages per update, then yes 10000 would be an appropriate value. But I don't see how that's a realistic scenario (with the smallest packets possible, that'd be roughly 150 Mbps of bandwidth assuming 30 updates per second). I'd only expect clients to send at most a handful of messages per update, if that. So for 100 clients a value of ~1000 should be plenty enough.
     
    simon-lemay-unity, Aug 2, 2022
    #2

  3. itisMarcii_

    itisMarcii_

    Joined:
    Apr 5, 2022
    Posts:
    111
    We are doing a stress test on the Netcode component to see how Netcode in a larg environment (MMO) would behave. As test we spawn a bunch of Clients that each hold a NetworkVariable and change that variable each tick for each individuel Client. In that environment everyone could decide to launch an attack at the same time and would probably cause heavy issues on the network.
     
    itisMarcii_, Aug 2, 2022
    #3

  4. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    441
    My comment about the scenario being unrealistic was perhaps a bit premature. I should have inquired about your use case first. If you believe your game has a realistic chance of having 100 clients each send 100 message in a single update, then as I mentioned above 10000 would indeed be the correct value to use.

    And while I may not necessarily recommend setting the value that high, it should still work fine. So reading in your other thread that you are observing a stack overflow when doing so is concerning for us. Could you provide more details on the issue? Does it overflow on startup or later during the execution (e.g. on the first update)? Is there a stack trace associated with the exception?
     
    simon-lemay-unity, Aug 2, 2022
    #4

  5. itisMarcii_

    itisMarcii_

    Joined:
    Apr 5, 2022
    Posts:
    111

    It is immediatly on the Start. But an interesting behaviour is that on a dedictated server build there wont be the same issue, or at least there is no error. On windows the Editor and the Build both encounter the same issue at the start. However, it seems like increasing the value on runtime doesnt cause the same issue (I am not sure about that right now, i would try it tomorrow again).

    On the other hand, like i mentioned previously about the environment (N-Clients with each NetworkVariable update per tick), right now i hit the treshhold of around 120 Clients before i run into issues. I change the three values MaxPacketQueueSize, MaxPayloadSize and MaxPayloadQueueSize on runtime expecting that:

    MaxPacketQueueSize is (Clients * Clients + 10) * 2 (10 for a small buffer, 2 for the reliable message system)
    MaxPayloadSize is: ((Clients + 10) * (sizeof(float) * 3 * 3) * 10 (10 again buffer, the flaot times three shall represent the Transform at the Client spawn, and times 10 just as safeguard not being to low)

    Right now i am not sure if the limit of 120 Clients is because of the RAM issue on the devel machine (since i came close to my currently 1.8GB) or the RAM on my pc itself (since it also was at about 97% usage). But tomorrow i got more RAM on the devel machine and should be able to check on that.

    But are the PayloadSize calculations in a reasonable size or am I forgetting something?
     
    itisMarcii_, Aug 2, 2022
    #5

  6. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    441
    Is the dedicated server build made for Windows too?

    Increasing the value at runtime wouldn't have any effect, since the buffers are all allocated at initialization.

    Regarding the stack overflow issue, I've had a look at our code and identified an instance where we 
    stackalloc
    an amount of memory that scales with the packet queue size. I'll file an issue on our end to address this.

    Regarding RAM usage, the likeliest largest contributors are 
    MaxSendQueueSize
    and 
    MaxPayloadSize
    , since their contribution to the total memory usage scales with the number of clients. In comparison, MaxPacketQueueSize only contributes a fixed amount of memory (a value of 10000 would contribute about 40 MB to memory usage).

    Maximum payload size is typically more related to the number of network objects in the scene than to the number of clients. By far the biggest message that gets sent is the world state at the beginning of a game, and its size mostly depends on the total number of network objects/variables. With recent changes, getting this value right will be less of a concern (at least for reliable traffic).
     
    simon-lemay-unity, Aug 2, 2022
    #6

  7. itisMarcii_

    itisMarcii_

    Joined:
    Apr 5, 2022
    Posts:
    111
    No it is only made for Linux.

    So if I read that correctly, in the future updates the MaxPayloadSize will be ignored for the reliable message while the unreliable is bound to the MaxPayloadSize, is that right?

    So the MaxPayload is like:
    the number of NetworkObjects times Transform information plus
    the number of NetworkObjects with 1 NetworkVariable times the amount of NetworkVariables (1) plus
    the number of NetworkObjects with 2 NetworkVariables times the amount of NetworkVariables (2) plus
    the number of NetworkObjects with 3 NetworkVariables times the amount of NetworkVariables (3) plus
    ...

    Does that seems about right?

    PS: Thank you very much for the insight, its kinda hard to find information to those values at all :)
     
    itisMarcii_, Aug 3, 2022
    #7

  8. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    441
    That would explain why you don't see the stack overflow, then. Linux usually has higher stack size limits than Windows.

    That is correct.

    Your calculation of the maximum payload size also appears to be about right. If you go that route, I'd still add a generous extra overhead to account for other things that might need to be synchronized (e.g. network lists). Also note that this world state synchronization message is sent reliably, so with the change mentioned above, there shouldn't be any need to make these kinds of calculations to determine the best maximum payload size. If you wish to try these changes now, you can install the package directly from its Git repository.
     
    simon-lemay-unity, Aug 3, 2022
    #8

  9. itisMarcii_

    itisMarcii_

    Joined:
    Apr 5, 2022
    Posts:
    111
    Another question would be, how does MaxPacketQueueSize behave if I receive more Packages than the MaxPacketQueueSize allows. What happens with the extra packages that are already send through the network and how will the Server behave in regard of the reliably?

    Thank you very much, I will try it out as soon as possible (tomorrow). Is there a place for feedback or using the github issue page is enough?

    Again, I cant thank you enough for your time to answer all those questions! :)
     
    itisMarcii_, Aug 3, 2022
    #9

  10. simon-lemay-unity

    simon-lemay-unity

    Unity Technologies

    Joined:
    Jul 19, 2021
    Posts:
    441
    On the receive end, packets will be stored in OS buffers until they are read. So say you have 
    MaxPacketQueueSize
    set to 10, and 12 packets are received during a tick, then the first 10 packets are going to be processed immediately, while the last 2 will sit in OS buffers until the next update, where they'll be read.

    Now of course, there's a limit to how many packets the OS can store, so if packets keep piling up there, eventually some will be dropped. If that happens, the effect is exactly the same as if the packet was lost elsewhere on the network. So if the packet was sent reliably, it will be resent when the sender realizes there's a missing acknowledgment.

    Yes, the GitHub issues page would be the best place for bugs and even feature requests. Questions or support requests would be best asked on these forums or on Discord.
     
    simon-lemay-unity, Aug 3, 2022
    #10

  11. itisMarcii_

    itisMarcii_

    Joined:
    Apr 5, 2022
    Posts:
    111
    Back to MaxSendQueueSize:
    So I did a few tests and the threshold of the ~1200 MaxSendQueueSize deliver a cap of around 110 Clients for Windows. (Expecting that each tick a reliable message will received/be send). Since the Server cant provide an reliable connection, Clients will be disconnected over time. I am sure that is to expect for now till the issue got fixed.

    On the other hand unreliable messages will have no real impact on the Client number, but that again was to be expected.
    (Atleast for the server. Trying to connect 285 Clients will not result in 285 Clients connected, just around 270 or less)

    To the new MaxPayloadSize on the develop branch:
    It seems to work just fine, i was sending an ulong array of 500 through a max payload of 1500 and no issues. After increasing the array to 2800 and limiting the MaxPayloadQueueSize to 20000, meaning the payload is 2400 more than the MaxPayloadQueueSize, the connection was closed. So i figure the extra payload regarding the MaxPayloadSize get stored in the MaxPayloadQueueSize. I expect that to be the intended case. So no known issues about that change.
     
    itisMarcii_, Aug 4, 2022
    #11

  12. felipemuniznet

    felipemuniznet

    Joined:
    Jun 5, 2018
    Posts:
    6
    Did you get any better results after that? I also intend to create a massive mmorpg, ecs is the best option for that, but the netcode does not provide much security to support this.
     
    felipemuniznet, Jan 24, 2024
    #12
    te_headfirst likes this.