Search Unity

DOTS Multiplayer discussion

Discussion in 'Data Oriented Technology Stack' started by PhilSA, Jun 14, 2019.

  1. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    131
    Any news on that topic? I'm looking for networking solution for my early access fps game - already released but for now singleplayer only.

    2 main requirements:
    - low latency (highly competitive gameplay)
    - not too difficult to implement - single person would be able to do it in max 6 months

    What I found so far:
    Photon (Bolt) requires additional services like GameLift to run on dedicated servers.
    SpatialOS looks good but it's pricey + meh latency + impossible to have player's hosted servers. At least first server (normal cost ~180$/month) is free.
    DOTS Multiplayer + PlayFab Multiplayer Servers - I'll probably go this path as I'm using PlayFab for LiveOps. Probably most difficult route but in the end (in 1-2 years) will be better than others.

    I've seen Unity also acquired ChilliConnect. They have few advantages over PlayFab (better leaderboards, C# cloud scripting) but lack multiplayer servers. Would be great if in the future DOTS Multiplayer + ChilliConnect + Multiplay could work together.

    2nd question: how Multiplay operates compared to PlayFab Servers (docs)? What service is easier to implement?

    TLDR
    What is (or will be in few months) the easiest way to run DOTS Multiplayer servers including orchestration and matchmaking?
     
    dzamani likes this.
  2. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,834
    DOTS Multiplayer + ChilliConnect + Multiplay (Hosting & Matchmaking) is what we are focused on making sure works exceptionally well together, is easy to setup and our samples are configured with it by default.

    Here is several presentation sbout hosting & match making from Unite:
    https://multiplay.com/2019/10/02/multiplay-at-unite-copenhagen-2019-highlights/


    We are proving all of it out ourselves using DOTS shooter to make sure it works well for both development & deployment. Internally we are already using multiplay in playsessions for DOTS shooter, but the self serve functionality of multiplay is still being worked on before it can be released.
     
    optimise and Kamyker like this.
  3. Kamyker

    Kamyker

    Joined:
    May 14, 2013
    Posts:
    131
    Good to know, Ill start making multiplayer and once it's ready reconsider switching to Chilli. I'll have few options:
    - Playfab + Playfab Servers (Azure)
    - Playfab + Multiplay (GCloud or AWS)
    - ChilliConnect + Multiplay (GCloud or AWS)

    That's the DOTSSample, right?
     
  4. SebLazyWizard

    SebLazyWizard

    Joined:
    Jun 15, 2018
    Posts:
    42
    I'm not sure if this has been discussed before, but I'm curious what techniques you're going to utilize to save bandwidth.

    I'm planning to use DOTS Multiplayer for large scale battles with several hundred players per server, in order to archive this there must be some optimization going on, like;

    Only send data that has changed since the last tick, e.g. when a player presses a button send that info to the server once and once if he releases the button. So only state changes should be synced instead of a constant stream sending the same data over and over again. This way you can get rid of a lot redundant data.

    Distance related tick-rate and data compression. Players that are far away from you don't necessarily need to be updated that often and accurate as those who are close up to you. So it's basically a LOD system to save bandwidth.


    I haven't really touched the NetCode package yet, so I'm asking if those systems can be easily implemented, or are ideally on the ToDo list.
     
  5. timjohansson

    timjohansson

    Unity Technologies

    Joined:
    Jul 13, 2016
    Posts:
    138
    All the snapshots sent from the server are prioritized and sent with a cap to the bandwidth used. We prioritize and send the most important state until we reach the cap, the age is taken into account so anything not sent will have higher priority and be more likely to be sent next frame.
    We also have delta compression in place which compresses well enough that we can fit ~80 players per packet in DotsSample.

    The commands/inputs from the client to the server needs to be sent repeatedly, only sending changes would require reliable messages which would significantly increase the input latency. We also need to send acks of received snapshots frequently for delta compression of snapshots to work well - and commands compress really well so it only has a marginal effect on bandwidth.
    For snapshots we are planning to skip sending ghosts which did not change and/or are not visible. It will save some bandwidth, but our delta compression is really good at handling static objects so I don't expect it to make a huge difference for that.

    It is already possible to use distance as a factor for prioritization so things far away are sent less frequently - see https://docs.unity3d.com/Packages/c...host-snapshots.html#distance-based-importance
     
    pal_trefall likes this.
  6. Brendon_Smuts

    Brendon_Smuts

    Joined:
    Jun 12, 2017
    Posts:
    50
    I’ve noticed when playing with the multiplayer cube example and my own implementation that there are some simulation inaccuracies between the client and server due to the server not receiving client input packets in time. This happens when running the example with even a moderate latency 60+. Does anyone else get this issue or am I doing something wrong? I would think the example should accurately out the box. Are there still impromemts to be made to command buffering/transmission to improve this? At the moment you can notice the client predicted cube jittering back and forth when changing input directions even at low latency simulations. At higher latency/packet loss I could understand this but at this level it seems an unacceptable level of misprediction?
     
  7. Flipps

    Flipps

    Joined:
    Jul 30, 2019
    Posts:
    38
    I also recognized this behaviour. Already wondered why nobody else mentioned this.

    Logging the Ticks on the Server when the CommandData is received shows that the CommandData is sometimes too late.
    Here is some log at 100ms recv/send delay.

    upload_2020-1-11_15-59-10.png
    The "Applied tick" is the tick which was last received and should match to the server tick for appropriate simulation.

    The NetDbg delivered by Unity shows the CommandAge (oranage Line). The CommandAge appears to go up and down like a wave form. This can be explained by the continous adjustment of the estimated ServerTick on the client.

    White line = No delay of command data
    Orange line below white line = CommandData received earlier than needed
    Orange line above white line = CommandData received too late

    At 0 ms send/recv delay; commandData always on time:

    upload_2020-1-11_15-50-6.png

    At 100 ms send/recv delay; commandData sometimes to Late
    upload_2020-1-11_15-51-5.png

    At 300 ms send/recv delay; gets worse
    upload_2020-1-11_15-52-12.png


    Which CommandData is applied on which serverTick is calculated in the client. I had no closer look at the calculation but it seems wrong for larger latency.
     

    Attached Files:

  8. Kender

    Kender

    Joined:
    Nov 16, 2012
    Posts:
    178
    Any ETA on this feature?
    Actually, this could save bandwidth for cases where something changes extremely rare. Like a door -- it won't get it's open/close state switch multiple times per second. Why would we need delta compression for something that doesn't really change? Does this also mean it'll make the ghosts possible to despawn/spawn based on distance? It would've been awesome to have not just LOD for ghosts updates, but also for their presence.
     
  9. Enzi

    Enzi

    Joined:
    Jan 28, 2013
    Posts:
    258
    Hm, this sounds like the bug I mentioned: https://forum.unity.com/threads/dots-multiplayer-synchronization-lags.757415/#post-5048408

    That was with the old multiplayer asteroids sample, I haven't seen this in the new one but I haven't done any accurate testing. Could still be the same issue, just moved to a different place.
    Sadly, I don't have the time right now to investigate.

    @Flipps Debug.Logs are really bad for network testing. The delay it takes for logging distorts every measure.
    Write the log data you need to a List struct and save it to disk when quitting.
     
  10. Flipps

    Flipps

    Joined:
    Jul 30, 2019
    Posts:
    38
    @Enzi I will do that, but i think the results won't be different, because the CommandAge log in the NetDb shows the same results. Or maybe i am completly wrong.

    The sluggery movement is really easy to reproduce. Either with the NetCube Sample or the Tutorial on the NetCode Manual page (with a send recv delay of 200+ms)
     
  11. timjohansson

    timjohansson

    Unity Technologies

    Joined:
    Jul 13, 2016
    Posts:
    138
    The problem is that the calculation is based on command age reported from the server to the client. When your ping is high it takes too long to get this updated result and the client ends up over compensating since the calculation is based on several 100ms old data.
    Thanks for investigating this, the graphs really helped. I've filed a bug and we will improve this going forward.
     
  12. timjohansson

    timjohansson

    Unity Technologies

    Joined:
    Jul 13, 2016
    Posts:
    138
    Delta compression needs to handle properties which does not change since ghosts usually contain a mix of frequently changed and almost never changed properties. Having only almost never changing properties is a special case of that which happens to compress really well, even if it is not specifically written for that case.

    That is a separate thing, but we are also planning to support a PVS (potentially visible set) for ghosts so you can hide things which are not visible. The visibility calculations will be controlled by game code.
    I don't have an ETA on either of these features yet.
     
  13. Flipps

    Flipps

    Joined:
    Jul 30, 2019
    Posts:
    38
    I had some strange behaviour where the animation of an predicated ability got fired twice.

    After a little bit of research i found out that this was caused by the Clients calculated ServerTick.
    At 200ms+ send/recv it could happen that the Clients ServerTick could jump back in Ticks (see Screenshot).
    So the predicted Entities jump back in time because the prediction always runs until the ServerTick.

    I know that the ServerTick is variable adjusted, but is it intended to be less than the time before?
    I think this could unintentionally trigger Systems which run after the GhostPredictionSystemGroup twice.

    upload_2020-2-9_1-11-16.png
     

    Attached Files:

  14. timjohansson

    timjohansson

    Unity Technologies

    Joined:
    Jul 13, 2016
    Posts:
    138
    The NetworkTimeSystem tries to adjust a time scale with +/- 10% to keep the client time in sync with the server. If for some reason the time drifts too much it will however do a hard reset and just set its current tick to what it thinks the server has. So it can happen - but it is expected to be extremely rare. The limits and logic for exactly when and how this happens has not been tweaked yet though so it could trigger too often in some cases.

    In the prediction group the time will be reset to the last received tick and simulate forward to what the NetworkTimeSystem calculated - so if you are printing the tick there it is expected that it jumps back every frame.
     
    Sarkahn and pal_trefall like this.
  15. Flipps

    Flipps

    Joined:
    Jul 30, 2019
    Posts:
    38
    Thanks for the really fast and helpful answer - as always :)
    This behaviours seems to be triggered alot when above 150+ Send/recv delay or when the client has a low framerate.

    I am aware of this and printed the log outside of the GhostPredictionSystemGroup of the client.
     
  16. Jawsarn

    Jawsarn

    Joined:
    Jan 12, 2017
    Posts:
    96
    In the DOTSSample we can see that some of the data of Ghosts are grouped as "PredictedData" "InterpolatedData" "ReplicatedData" which doesn't seem to have cohesive data at my first glance of it. Is this approach more performant than having it separated in many smaller components? Which is the recommended approach?
     
  17. timjohansson

    timjohansson

    Unity Technologies

    Joined:
    Jul 13, 2016
    Posts:
    138
    That setup is mostly because that's how it worked in the fps sample rather than a deliberate design specifically for the new netcode. I did not measure which performs better but I would be surprised if there is much of a difference.

    The netcode does have a feature where owner predicted ghosts can send different data to predicted and interpolated clients though, and that setting is per component. So if you want to use that you should make sure that predicted only, interpolated only and shared data are in different components - but it does not have to be one component per mode and it does not have to be named anything special.
     
    Jawsarn and pal_trefall like this.
unityunity