Search Unity

DOTS Multiplayer discussion

Discussion in 'Data Oriented Technology Stack' started by PhilSA, Jun 14, 2019.

  1. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,257
    Blog post
    Github
    Github - NetCode

    I'd like to start a discussion on the DOTS networking framework currently being developed (not just the Transport layer, but also the "NetCode" framework), and hopefully get some devs to share thoughts on where they want to go with this.

    Here's a few questions:

    ---

    1- I'd be curious to hear what are the fundamental principles behind the network architecture planned in the "NetCode" project:
    • Is it going towards a Quake3/Overwatch-style architecture with automatic world rollback, prediction & re-simulation for lag-compensation?
    • Or is it more of a Tribes2/HaloReach-style architecture where lag-compensation is left to handled manually at the benefit of not having the CPU overhead of auto-resimulation?
    • Are there plans for congestion control, priority systems and network culling solutions?
    • Will the codegen remain explicit, or will it be converted to a Cecil approach (honestly that's not a big concern to me, but I'm still curious)
    • etc...
    ---

    2- How heavily in-development is the NetCode project? I think people are concerned about this due to the unity+networking situation of the last few years, and it would be great to be reassured. Is there a "frontman/frontwoman" to this NetCode project that we can follow in the forums or on twitter? Is the public "multiplayer" repo the same as the one devs actually work on, or is it just used for curated releases?

    ---

    3- the other major concern people have, because of the way Unity's networking announcements suspiciously tend to focus on talking about dedicated servers (something nobody is worried about) instead of actual netcode (something everybody is worried about), is that there will be some artificial restrictions that make us obligated to use Unity's server hosting. Can you confirm that both the transport and NetCode framework will remain usable with any server hosting solution, as well as self-hosted / player-hosted if we want to? I totally get it if things like matchmaking are unity-exclusive, but our hosting options have to be open. I've had to work on a bunch of connected "interactive installation"-like projects that require a LAN setup, as well as confidential game projects that we want to be able to test in LAN before moving to DGS, and I'd like to know if NetCode will support those use cases
     
    Last edited: Jun 14, 2019
    Mikael-H, James3421, Kender and 16 others like this.
  2. psuong

    psuong

    Joined:
    Jun 11, 2014
    Posts:
    58
    I'd also would love to know since I've been trying to see where the NetCode is going and (also learn how it works, especially the RPC Layer) so question #2 is something I'm heavily interested/vested in.

    For my purposes, the network architecture I'm following on their Github works quite well for me. I'm doing more of LAN with a Server Client Architecture (demo vid below):

     
  3. timjohansson

    timjohansson

    Unity Technologies

    Joined:
    Jul 13, 2016
    Posts:
    138
    This is a broad question and I can't really cover all the details in a forum post, but I can try to outline the basics.

    The basic architecture we are implementing is server authoratitive with client prediction and lag compensation on the server - like most modern FPS games.
    For interpolated ghosts (server controlled networked object) which are just displayed at the last position received from the server (minus interpolation time) we want to make it as automatic and easy to work with as possible since there are usually a lot of them and they all tend to have exactly the same logic.
    For predicted ghosts which the player have control over - that needs to be displayed at the tick the server will be at when it receives the input from the client - we are heading in the direction of automatic prediction/re-simulation, but it is still TBD exactly how automatic it will be. Right now it is only semi automatic, it rolls back for you and there are some utilities for re-simulating but you need to handle much of it and make sure it matches the server simulation yourself. This is something we are planning to improve, at the very least we will make the utilities easier to use and make it easier to share code between server simulation and prediction.
    The lag compensation on the server is mostly about knowing which tick the client saw for interpolated ghosts and raycast against that when firing hitscan weapons so you don't have to lead your targets, it is not implemented yet.
    We also have support for predictive spawning, where you can spawn entities on the client you think the server is going to spawn. The server will take ownership of the entitiy when the snapshot arrives and it will be automatically destroyed if it does not arrive. The matching function from incomming server snapshots to predictivly spawned entities is something you need to implement yourself.

    The core of the NetCode where we have spent the most time is an efficient way of processing large amount of entities, prioritizing them per ECS chunk and ghosting them to the clients with delta compression.
    So the base of the priority system is already there. Each ghost type calculates an importance factor for a chunk of entities (if you want to prioritize on distance you need to partition your entities spatially in chunks with shared components and calculate bounds per chunk right now, per entity prioritization is something we will investigate if we find a use-case where it is required).
    We send snapshots with a fixed bandwidth, once we know all the chunks we sort them based on importance, fill up a packet with the most important entities and send it. Importance values are always scaled by time since last send, so anything which did not fit will have higher importance next frame. We will make this more configurable going forward, so you can reduce package size or send rate as a response to congestion, and also provide the data to make those decisions.

    We are not planning to move the codegen to cecil, we plan to keep generating cs files. The main reason is debuggability. Once you generate assemblies rather than code it is much harder to debug since you can't just set a breakpoint or add a Debug.Log somewhere. Another reason is that we want to make sure it is possible to override the generated code if you want to do something custom for a specific type, so you can generate the code as a starting point and then hand-edit it.

    Another important pillar of the NetCode which is often forgotten is tooling and workflows. We have experimental workflows for playing multiplayer games with multiple clients and in-proc server in the editor and tools for visualizing the data flow and composition of snapshots. We think these kind of things are critical for the new NetCode to succeed and which we will continue to improve them.


    I can quantify the development investment in a meaningful way. We have a dedicated DOTS Multiplayer team working on it which I am leading. It is staffed with developers who have been working on DOTS since long before it was called DOTS and with substantial experience from big AAA multiplayer games.

    The public github is not the main repo yet. Since we pushed the NetCode there we try to push to it as soon as we integrate significant feature branches to master on our internal repo. We are also going to do some development in real games since we want to drive the development by real use-cases, but we will try to keep pushing the the public repo frequently regardless of where we are doing the active development.

    Server hosting, matchmaking and other services are developed by a different team, which tend to do more announcements since they have a more mature product. We have no plans to require any of those services for the transport or NetCode.
    Being able to run the game with a server and client in the same process is IMO required to make good development workflows which - as mentioned earlier - is something we consider imporant. And as you point out, for many projects it is a strict requirement to run with a lan server during development (I would say it is required for practically all projects since you need to be able to debug and profile your server locally)
     
    GliderGuy, Mikael-H, Sarkahn and 19 others like this.
  4. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,257
    This is incredible news and I can safely say that it clears all of my doubts! Really happy about your decisions and your progress so far

    Also,
    This is gonna make a bunch of people fall off their chairs and go through the floor
     
    Last edited: Jun 14, 2019
    Mikael-H, shodude, Shinyclef and 2 others like this.
  5. psuong

    psuong

    Joined:
    Jun 11, 2014
    Posts:
    58
    Thanks for the info @timjohansson ! :)
     
  6. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,555
    I think productive flows are really the hard problem here. The rest is fairly well known, even if it's time consuming to get the details right.

    One key thing most everyone new to this gets wrong is that server flows are necessarily and correctly different from the client. You can still get code reuse, but it needs to happen at the right abstraction layer. So while you might very well share the same data structures on client and server, the ECS systems would correctly look quite different on a per feature basis.

    Currently my favored approach is abstract core logic in a way that makes it easy to plug that into the client as a form of mocking server functionality. Although in this case the mocks are actually the real logic shared by client and server. This is primarily for the 90% of features that aren't on the hot path per say, but where most of your dev time ends up going in a larger complex multiplayer game.

    With core logic abstracted towards this goal, then an approach of mocking server functionality is very simple and can be done with very few lines of code per feature. In our game which is a rather large complex mmo, outgoing messages are redirected to a mock server class. It's a single file. 90% of features are mocked with a single call into shared logic and the creation of the response message, so 2-4 LOC on average to mock out an entire feature. Response is then injected back into the incoming message routing

    Having to run a client and server for doing most of your iteration is a less then productive flow. With the above approach I can work for days on a feature without using the server, with a high degree of confidence that it will mostly just work when I do the client/server testing. Of course this assumes some degree of unit testing, but even that is faster when it's outside Unity.

    Client and server in the same process has a lot of the problems that mocking/stubbing were designed to fix. It's better then having two processes but I don't think it's the best solution we have available.
     
  7. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,257
    One use case that I think is interesting to think about is the problem Glenn Fiedler describes here (but disregard the solution he opted for, because it would not be suitable for a game where cheat-prevention is important). In short: how to make a responsive online game where players are all controlling dynamic rigidbodies that can interact with eachother

    For a game like that, I'd imagine the "ideal" setup would look like this:
    • Start with the assumption that our physics simulation is "deterministic enough"
    • Whenever the server receives inputs from clients, revert world to the tick of the earliest of those received inputs and resimulate to present using all the inputs we know of, or predicted inputs otherwise
    • Whenever a client receives world snapshot from the server, revert world to that snapshot's tick, apply it, and resimulate to present using our own inputs as well as predicted inputs for other players
    Basically; whenever either the client or the server receives any data from eachother, revert world to the tick of that data and resimulate absolutely everything to present. It would have a much heavier CPU cost than other network architectures for sure, but this is a case where I'm hoping DOTS performance might help make this a viable option for some games.

    I'd totally understand if NetCode decides to not support such use cases, but I still think it might be worth thinking about if there are ways to structure NetCode in a way that would make this strategy easily implementable by users if they need it. I think it would also be very helpful for lag-compensation of non-hitscan projectiles

    And by the way, don't hesitate to tell me if I'm wrong about this stuff. My online game programming experience is limited
     
    Last edited: Jun 16, 2019
  8. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,834
    Our focus is on the FPS netcode architecture Tim describes since it is the most commonly used in games and the most broadly applicable. Is fair in competitive play, safe against cheating etc.

    We do believe that flexibility with netcode architectures is important. Different games sometimes need different netcode architectures, for different tradeoffs. There is no one size fits all for netcode. So we are building all simulation systems to enable that flexibility.

    A large amount of the work for good netcode is ensuring that all subsystems are built with the right principles. Performance by default, simulation determinsm by default, complete & performant access to all state / data is obviously the primary pillars that make it so that you can build your own custom netcode architectures on top of ECS yourself without fighting the system...


    Over time we want to provide examples of different architectures like LockStep deterministic for RTS or GGPO for fighting games. For now the focus is on providing a great FPS netcode architecture & examples.
     
    Last edited: Jun 16, 2019
    dzamani, psuong, thelebaron and 3 others like this.
  9. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,834
    We use multiple worlds for server & client. This allows for easy debugging in the same process. The asteroids demo even lets you switch between multiple client visualizations quickly to ease debugging of multiple players all in the same process.

    This seperation is a best practice used in all Major AAA FPS games. Even the single player missions in FPS shooters usually run through the normal client / server model, simply using memcpy instead of sockets to copy the data.

    This is to ensure that all game play programmers always test the same systems, to avoid the issue that there is a single network dude on a team who gets to fix everyones code because everyone else can just "happily ignore the annoying details of making game code work with the netcode".
     
  10. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,182
    This is all what I expected however for a large number of Unity customers/users there will still be monobehaviour projects, so is there a plan for a bridge? I can't really use the word "hybrid" if I'm honest as it doesn't really describe what functional purpose there is.
     
    shodude likes this.
  11. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,257
    My guess is that getting data from the DOTS world in order to represent it in the GameObjects world will be easily doable, but implementing simulation logic in monobehaviour is another story. You'd at least need to use a custom update that is called from the main DOTS simulation world and does the following:
    - get the data from the right entities in the DOTS world
    - do some logic
    - set back the data to the right entities in the DOTS world

    I think the real requirement is this: there is a specific point in the DOTS simulation loop where all of your Entities have to be properly updated by your gameplay logic, so that the DOTS snapshot/serialization systems can then do their job

    I gotta admit, though; part of me hopes that things like NetCode, VisualScripting, Subscene Streaming, etc... just remain 100% focused on DOTS in order to accelerate the community's transition to DOTS as soon as possible
     
    Last edited: Jun 16, 2019
    Kender likes this.
  12. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,182
    I think DOTS has a stigma, a fear attached to it that will prevent any migrations en-masse for quite some time. Firstly, it's still in development and secondly it's a new way of thinking, so that kind of situation can't IMHO happen until at least visual scripting is fully there too - people will generally adopt without too much friction at that point...
    Sorry for slightly off topic.
     
    psuong and PhilSA like this.
  13. Jawsarn

    Jawsarn

    Joined:
    Jan 12, 2017
    Posts:
    96
    I'm trying to combine DOTS stuff into one project, multipalyer, phyiscs mainly. So just adding the two caused physics not to be run on any worlds. So I modified the Physics packet code to add the SimulationGroup attribute, but quickly realized the worlds will need the Transform systems as well. How is this all planned out to work in the end with packets that might or might not depend on Client/Server Simulation player loop structure? Are/will the developers be required to manually create bootstraps to filter all systems from all packets in this sense if they should be added to simulation or not?
     
  14. timjohansson

    timjohansson

    Unity Technologies

    Joined:
    Jul 13, 2016
    Posts:
    138
    We noticed the same thing recently, it is not how we want it to work and are currently investigating how to best deal with it. We are not there yet - but in the end you should not have to write a custom bootstrap nor modify packages in order to run the default or package systems in the client or server world.
     
  15. doctorpangloss

    doctorpangloss

    Joined:
    Feb 20, 2013
    Posts:
    239
    I looked through the alpha code, and I see things aren't really settled in terms of synchronizing general data. For example, diffing isn't really a thing here. Am I mistaken?
     
  16. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,834
    Delta compression is part of the net code and works very well already.
     
    PhilSA likes this.
  17. Jawsarn

    Jawsarn

    Joined:
    Jan 12, 2017
    Posts:
    96
    Updated my project with the new things. Neat to see that systems are by default added to both server and client now. What I still dont like is the Archetype usage to create ghosts entities, a player usually have children etc, and from what I understand it can't be represented in an archetype? (maybe I'm wrong here) And that we have to manually write the Archetype for the server instead of something generated from/using the ghost prefab

    Edit: I guess you could create the avatar of the ghost object delayed as a solution as well.
     
    Last edited: Jul 18, 2019
  18. timjohansson

    timjohansson

    Unity Technologies

    Joined:
    Jul 13, 2016
    Posts:
    138
    We do want to support ghosts consisting of multiple entities, but we have not implemented that yet.

    Which archetype usage is it you do not like?
    The archetype usage to instantiate ghosts on the client is optional - if you have a reference to the prefab in the client world (which you can get by creating a GameObject with GhostPrefabAuthoringComponent and ConvertToClientServerEntity set to Client only) it will instantiate the ghost prefab instead of using the archetype.
    You can also use the ghost prefab to instantiate entities on the server.

    We do not support finding ghost types with multiple entities on the server, nor serializing / deserializing ghosts with multiple entities yet though. We also have not tried creating ghost prefabs with multiple entities yet, so it might not solve your root issue today.
     
    shodude and Jawsarn like this.
  19. Jawsarn

    Jawsarn

    Joined:
    Jan 12, 2017
    Posts:
    96
    Sweet, didn't see GhostPrefabAuthoringComponent before. Saw that you added a quickguide as well now with some bits. I don't require any serialization of data on the child entities for now, so I think it will solve it. : )
     
  20. doctorpangloss

    doctorpangloss

    Joined:
    Feb 20, 2013
    Posts:
    239
    I guess the code is at a level of abstraction where I really have no idea what's going on or what I'm supposed to do :)

    I look forward to seeing a tutorial / documentation!
     
  21. Enzi

    Enzi

    Joined:
    Jan 28, 2013
    Posts:
    258
    I'm using the CharacterController from @PhilSA, so on that occasion, hi! and thanks man, fantastic work!

    Any help or tips to make GhostAuthoring work with hybrid ECS or give an example? I'm using the Asteroid sample as a template which is pure ECS. Got most working and I have the server hybrid gameobject initializing correctly and the client can move it.
    The thing that is missing now is the ClientGhost spawning and on that I have some questions:

    - How does the client know which prefab to spawn? I'm not finding the connections. The server spawns an archetype with GhostComponent and on the client side the first element of the arrary of GhostClientPrefabComponent is taken, otherwise it falls back to archetypes? I don't get it really. Is this prototype code or am I missing something here?
    - How can I, instead of Convert and Destroy, Convert and Inject in a GhostPrefabAuthoringComponent? At first I thought this would be easy but then I was unable to find a way to call ConvertAndInjectOriginal as it's missing in GameObjectConversionUtility. - I found ot that ConverToEntity.ConvertAndInjectOriginal exists but it has no entity as return value.
     
    PhilSA and TZ- like this.
  22. Jawsarn

    Jawsarn

    Joined:
    Jan 12, 2017
    Posts:
    96
    @Enzi
    I started off hybrid as well but went with as much pure ECS as possible with the ease of use the new systems etc. The feeling I've had is that they are trying to make the hybrid stage as small as possible and focus on pure ECS stuff, the network guide even tell you to look at FPS sample for hybrid if I'm not mistaken.

    But ideas how to go hybrid, (I did this not for hybrid before but with my own prefab for client). You can mod the "DefaultGhostSpawnSystem" with some abstract methods like "Spawn/SpawnPredicted" that you can manually add prefabs you want to spawn for your specific spawn systems(generated ones). In the "OnUpdate" in the "DefaultGhostSpawnSystem" you can swap out/add code to spawn your own prefabs from these abstract methods instead :)

    From the query "m_PrefabGroup" of the generic system "DefaultGhostSpawnSystem" it has the Snapshot component of the specific ghost. So if you added the GhostPrefabAuthoring in the scene, the snapshot component and GhostClientPrefabComponent is added to that an created entity. So if there is no entities with GhostClientPrefabComponent and the Snapshot, it will use the abstract "GetGhostArchetype()/predicted".

    No idea. I would guess you can't do it like that since it's converted as an entityPrefab for later use, so there is no gameObject to bind it to? But I might be wrong here.
     
    shodude and Enzi like this.
  23. Enzi

    Enzi

    Joined:
    Jan 28, 2013
    Posts:
    258
    Thanks, I missed the GetGhostArchetype implemented in the partial class. Ghosts are linked via the SnapShot type.

    That's basically what I did, that is replacing Archetype instantiates with Hybrid GOs in the DefaultGhostSpawnSystem.
    For whatever reason the convert workflow fails for me in this instance and a GO is created without any components at all, I had to manually add them but other than that it's working now with KCM.

    Is the StateHistory window available for the new Multiplayer package? That was pretty nice in the FPSSample but it's not compatible with the new package and I've not found any replacement.

    Now that I know how this works I wonder why not more use this package already. It's awesome and much better structured then working with the FPSSample.
     
    shodude likes this.
  24. Enzi

    Enzi

    Joined:
    Jan 28, 2013
    Posts:
    258
    I spent around 4 days now on the multiplayer package. There are some things I should have done before hand because one of the biggest issues I see now is the really high bandwidth the netcode needs.
    What's the best way to set the send rate?
    How much overhead is in editor mode in terms of bandwidth? There are lots of ENABLE_UNITY_COLLECTIONS_CHECKS
    Why are vectors/floats turned into packedInts? For determinism? I spent the longest time on choppy movement until I realized that a quanitization value of 10 is too low and I turned it to 1000 to have more decimal precisions.

    The prediction weirds me out and I'm not able to find a state where no corrections are needed and I tried a lot of different methods. What gets me most is that snapshot.GetDataAtTick(currentTick, out snapshotData); ends in an interpolation of a tick that is different to the tick that was set in the GhostUpdateSystem before, so the place where new snapshots are applied. I get the "best" results when I set the position the the received snapshot position but even the "best" result is still choppy and wrong in prediction.

    If you take the Asteroids sample and change it so the velocity is 0 when no thrust is applied and you turn the asteroids to 0 to test better you'll see that the ship movement at higher speeds start to get very choppy.
    As far as I can tell the client side prediction is not working correctly in Asteroids and obscured because of the interpolation and low speed.

    For the net devs I would suggest to use a twitch based character controller like the FPSSample. (the character controller doesn't have to be complicated or good, twitchy movement is just more demanding for anything prediction related)

    Another question, how does more snapshots and entities like the asteroids scale? Are Asteroid snapshots also sent every time, 60 times per second? The default is 200 and they take roughly around 30kB/sec which doesn't seem to bad, all things considered but it's still a huge factor for something that predictable, so, I wonder if it's possible that these snapshots are hashed, compared and only sent when there are errors. Further testing then has it that with 2000 asteroids and the size of the level at 20480 doesn't increase the bandwidth usage much. Is compression kicking in more and just the initial costs higher?
     
    Last edited: Jul 28, 2019
    daschatten likes this.
  25. timjohansson

    timjohansson

    Unity Technologies

    Joined:
    Jul 13, 2016
    Posts:
    138
    If you want to do it without modifying the NetCode you could probably manually spawn a GameObject after the entitiy is created from the archetype by overriding and having your spawn system do custom things in
    Code (CSharp):
    1. protected virtual JobHandle UpdateNewInterpolatedEntities(NativeArray<Entity> entities, JobHandle inputDeps)
    and
    Code (CSharp):
    1. protected virtual JobHandle UpdateNewPredictedEntities(NativeArray<Entity> entities, JobHandle inputDeps)
    The only tool we have for inspection at the moment is the web based snapshot debugger, open NetCode/Stats/netdbg.html and connect when the game is in play mode. We will add more tools and integrate them better going forward.

    Send rate is tied to the simulation rate - which is not the normal simulation rate but hardcoded in NetCode/ClientServerWorld.cs - FixedTimeLoop.fixedTimeStep .
    We will make it possible to control the send rate independently of simulation rate, but have not implemented it yet.

    The collection checks does not change the data transfered, it only performs more validation, so it does not affect the bandwidth at all.

    They are quantized and sent as ints to make the compression work better. We send small ints with as few bits as possible, by converting values to quantized ints and delta compressing them each value will only use a few bits on the wire in the common case.

    There are two different paths in there. If a ghost is marked as predicted (by overriding
    Code (CSharp):
    1. protected virtual JobHandle MarkPredictedGhosts(NativeArray<T> snapshots, NativeArray<int> predictionMask, NativeList<PredictSpawnGhost> predictSpawnGhosts, JobHandle inputDeps)
    in the spawner) it will apply the last snapshot it received and relies on user code running the simulation from that point in time up until where it is supposed to be.
    If a ghost is not predicted it will apply an interpolated snapshot which is older than the latest in order to make sure time progresses smoothly even if there is jitter.
    The prediction will rarely be perfect though, we will be using different character controllers to investigate how to make it feel better going forward and add more documentation - and if required utilities - for implementing good prediction and correction.

    The snapshots are partial, we fill one network package with the serialized entities having the highest importance. Whatever does not fit will get higher importance next frame and will be more likely to get sent then. Because of that it will cap the bandwidth at ~ 1200 bytes * 60 Hz per connection. So if your game state does not fit in that bandwidth you will not get new state for all entities every frame. We are planning to look at ways to compress the data further by not sending things which are not visible and by sending less / no data for things which did not move at all or only moved in predictable ways.
     
    pal_trefall likes this.
  26. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    507
    Hi @timjohansson. Is that currently any features still missing to develop MOBA multiplayer mobile/pc game? Currently I'm considering whether it's the good choice to start using it or should I wait for a little bit longer?
     
    Limnage and Opeth001 like this.
  27. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,182
    Question: what happens if you have a mono project that's hard to port to DOTS say something involving XR, but still want to integrate DOTS mp?
     
  28. Enzi

    Enzi

    Joined:
    Jan 28, 2013
    Posts:
    258
    Hippo, that's actually quite straight-forward if you use hybrid gameobjects. I've it running that way and I don't see an issue porting from more complex projects.
    What needs a lot of work though is porting game logic to server/client based gamecode. If you already have the groundwork because the project was MP from the beginning, not much issues. When it was SP, well, I hope you have a good architecture, otherwise it will be the same pain as any other bringing MP to SP.
     
    hippocoder and shodude like this.
  29. shodude

    shodude

    Joined:
    Feb 22, 2016
    Posts:
    4
    This is a great read for anyone thinking about Multiplayer for their Unity game, thank you all for the wise advice and analysis

    Is DOTS considered 'a step up' or a 'step sideways' in terms of multiplayer code
    ie. is it an order of magnitude faster or just different

    I'm struggling to find much documentation of e.g. Photon vs Forge vs Dots pros/cons for
    1. Many entities vs fewer entities and more accuracy
    2. synchronizing between players (p2p?)
    3. packet size vs latency vs frequency
    hmm this does cover it very nicely(https://blogs.unity3d.com/2019/06/13/navigating-unitys-multiplayer-netcode-transition/)

    Does DOTS mean a transferal of a conversion of the Hierarchy in top left Unity Editor main view?
     
  30. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,834
    DOTS netcode is a step up obviously. A massive one at that.
    Otherwise why would we do it? I think we have given enough examples by now how everything we do in DOTS is a step change as opposed to something thats slightly different.


    1. It comes with lag compensation, interpolation, prediction, performance & time control builtin. All of this is enabled by DOTS by building DOTS from the group up to support netcode properly from the start. If you look back 2 years ago at presentations from me, I talked about that early on as our principles we took into account while building the core. Now we are reaping the benefits and soon releasing the first version. Still in preview obviously, there are obviously still many things missing from a complete solution, but the base features we ship with are on a solid foundation.

    2. Other than FPS netcode architectures are easy to build on top of DOTS too because our principles on top of which we built it are solid. But currently our focus is on building an end to end solution + sample for FPS netcode specifically.

    3. The focus on performance in DOTS enables us to make servers that use low memory consumption & high performance. Cost is a critical component of any server. I don't think there is anything quite like DOTS out there in the market. Not even close... The only thing that can compete is 100% custom specialized server components, but even that is exceptionally difficult to get to the level of performance of DOTS. DOTS is incredible at scaling simulation code.

    4. Full control over the DOTS deployed main loop & multiple world support, allows exciting opportunities with sharing blob assets, running async simulations all in one process tightly controlled. Being able to write your game code once while being able to seperate the server component out easily and only deploy that is a big deal. More to come on the full .NET Core based DOTS runtime later this year, which should make it clearer how all of this is meant to fit together when making super efficient server runtimes.

    5. On the scaling / orchestration & hosting side we have a few years ago acquired multiplay
    https://multiplay.com/
    It is quite impressive. Multiplay has been used in the largest player base scaling game in the world apex legends. And the orchestration layer performed flawlessly in that. It's the only orchestration tech that can claim that.

    Multiplay can be used from any game engine, but of course we are making sure that the integration into DOTS is very good. We do not yet have a self serve model as part of the upcoming netcode release but are working on it for early next year.
     
    PhilSA, GliderGuy, filod and 13 others like this.
  31. Kender

    Kender

    Joined:
    Nov 16, 2012
    Posts:
    178
    At the core, NetCode sends the messages over all the connections. Is it planned for any foreseeable future to have manually managed areas of interest, so that I could only share some area's data to only the players that are around? Perhaps, some sort of channels for ghosts, commands, RPCs, to subscribe to / unsubscribe from?
     
    MrCool92 likes this.
  32. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,834
    Yes. We are working on prioritisation but thats not part of the first package drop.
     
    pal_trefall and Kender like this.
  33. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    507
    I would like to know how much has been improved internally starting from lowest level Unity Transport package?

     
  34. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    552
    They are not touching it, the most experienced employee in that area left the company.
     
  35. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,257


    @Joachim_Ante @timjohansson
    I think a bunch of us got a bit worried when we saw this during the Unite presentation. We know it was running on older hardware, and that this is 4 ticks of simulation made in a single frame, but still. For only 4 players in a bare environment, these numbers are worrying, even if we look at the times for a single tick. I would guess that this machine could probably handle hosting a game like UT2004 with 32 players without any problems. (I'm making the assumption that rendering doesn't matter here since the big CPU time chunk in this pic is all Simulation-related)

    Can you reassure us about the conditions in which this test happened and about whether there were any known performance problems with NetCode (that won't be there in the future) at that time? Is this just the price of playing in-editor with simulated clients?

    Have you made stress tests with the NetCode sample with a high amount of players (60+) on "average-good" gaming machines?
     
    Last edited: Nov 18, 2019
    GliderGuy, pal_trefall and dzamani like this.
  36. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,182
    I'm interested in intense simulation but more data, for example I have 6 players (with more if it can support) but each player is essentially 3 players in VR (head, left hand, right hand) - all have independent and spatially important data to be tracked. That 6 players becomes 18 effective players worth of data and must work on mobile and also console, without hard connection, and otherwise, at very high framerates.

    I'm wondering how DOTS will handle fewer connections but higher rate of data or bigger data, and how to mitigate that, especially as it's fine-grained. If I reach over and physically disarm someone in MP VR, that's huge complexity vs a single player action like shooting in part due to just how high frequency a VR simulation would run at.

    Before DOTS, I would laugh anyone out of the room. Now, I am hoping it might be possible.
     
    Mikael-H and GliderGuy like this.
  37. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,257
    If you're trying to accomplish something like this:
    https://developer.oculus.com/blog/n...orking-a-stack-of-cubes-with-unity-and-physx/

    ...where players can move physics objects online in VR, then you'll probably need a network architecture that is different from the "FPS architecture" that NetCode will start out with. What you'll need is something similar to a "Fighting game architecture" (aka GGPO, but a server-authoritative variant) where every client and server resimulates absolutely everything whenever they receive any inputs/snapshot message from someone else. In the article however, they end up deciding that this would be too heavy for simulation and they end up going with an architecture where clients have authority over objects and can therefore cheat very easily

    But otherwise, you can still probably get away with sending data at 30hz or 15hz even if your game runs at 90hz and interpolates

    _________

    Side note: I kinda wish we wouldn't use terms like "FPS architecture" and "Fighting Game architecture". Both of these can be used outside of these genres and I'd prefer using names that are more descriptive of what they're actually doing. Even DOOM3 and Overwatch essentially use the "Fighting Game architecture" in a way. I'm just nitpicking here
     
    Last edited: Nov 18, 2019
    GliderGuy and Evil-Otaku like this.
  38. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    FPS Architecture = Client Side Prediction + Snapshots With Delta Compression
    Fighting Game Architecture = Deterministic Predict/Rollback
     
    GliderGuy and PhilSA like this.
  39. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,834
    Yes we have successfully played with ~50 players at 60 FPS tickrate.
    It's hard to coordinate that many players, internally when we haven't released it publicly yet, so we haven't done more yet. Wasn't a focus yet...
    It runs fine on older hardware.

    We are very much working towards large amounts of entities being simulated & synchronised effectively.

    The screenshot is quite old while we were still in the middle of refactoring a bunch of code from main thread ComponentSystem to the new bursted Entities.ForEach. Please stop reading tea-leaves...
     
  40. johnroodt

    johnroodt

    Joined:
    May 30, 2012
    Posts:
    5
    Any chance, as part of the "sample", that Unity can deploy the FPS into a typical Google Cloud environment so we can see that partnership in action too?
    (Complete with Kubernetes management, ongoing CI/CD etc.)
     
  41. johnroodt

    johnroodt

    Joined:
    May 30, 2012
    Posts:
    5
    https://gcp.solutions/diagram/gaming-aaa-servers etc.
     
  42. johnroodt

    johnroodt

    Joined:
    May 30, 2012
    Posts:
    5
    Looks like 2019.3 has been released....maybe we'll get the new FPS drop soon too?
     
    Kender likes this.
  43. Jawsarn

    Jawsarn

    Joined:
    Jan 12, 2017
    Posts:
    96
  44. andreyefimov2010

    andreyefimov2010

    Joined:
    Oct 29, 2017
    Posts:
    19
    This doesn't seem to be possible now: GhostPrefabAuthoringComponent is gone in 0.0.4 and when adding ConvertToClientServerEntity to the ghost prefab set to Client only, the ghost is still just an entity without gameobject presentation.
     
    Last edited: Jan 6, 2020
  45. timjohansson

    timjohansson

    Unity Technologies

    Joined:
    Jul 13, 2016
    Posts:
    138
    The GhostPrefabAuthoringComponent is indeed gone, you now have to use the GhostCollectionAuthoringComponent for this instead, and the same GhostCollectionAuthoringComponent is used to generate the code for the collection instead of a menu item. It is mandatory to use it, the support for instantiating ghosts with an archetype has been removed.
     
    florianhanke likes this.
  46. andreyefimov2010

    andreyefimov2010

    Joined:
    Oct 29, 2017
    Posts:
    19
    So there's no way to instantiate a game object instead of/with archtype for ghost now?
     
  47. timjohansson

    timjohansson

    Unity Technologies

    Joined:
    Jul 13, 2016
    Posts:
    138
    Not directly, but you can instantiate a GameObject by creating a partial class of the SpawnSystem which implements
    UpdateNewInterpolatedEntities / UpdateNewPredictedEntities and creates game objects which you can link to the entities.
     
  48. andreyefimov2010

    andreyefimov2010

    Joined:
    Oct 29, 2017
    Posts:
    19
    Thanks for the reply. I've tried you solution, but I've run into a problem of linking an object to the entity:
    Code (CSharp):
    1. struct LinkedGameObject : ISharedComponentData, System.IEquatable<LinkedGameObject>
    2. {
    3.     public GameObject instantiatedObject;
    4.     public int id;
    5.  
    6.     public bool Equals(LinkedGameObject other)
    7.     {
    8.         return id== other.id;
    9.     }
    10.  
    11.     public override int GetHashCode()
    12.     {
    13.         return id;
    14.     }
    15. }
    16.  
    17. protected override JobHandle UpdateNewPredictedEntities(NativeArray<Entity> entities, JobHandle inputDeps)
    18. {
    19.     for (int i = 0; i < entities.Length; i++)
    20.     {
    21.         var entity = entities[i];
    22.         var position = EntityManager.GetComponentData<Translation>(entity);
    23.         var rotation = EntityManager.GetComponentData<Rotation>(entity);
    24.  
    25.         var obj = Object.Instantiate(prefab, position.Value, rotation.Value);
    26.         var linkedObject = new LinkedGameObject
    27.         {
    28.             instantiatedObject = obj,
    29.             id = entity.GetHashCode()
    30.         };
    31.         EntityManager.AddSharedComponentData(entity, linkedObject);
    32.     }
    results in error
    And building the LinkedGameObject array first and then passing it to the job also results in managed data being not allowed in jobs error.

    UPDATE:
    Calling
    Code (csharp):
    1. inputDeps.Complete();
    at the beginning of UpdateNewPredictedEntities seems to be doing the trick, but I'm not sure if this is the right approach.
     
    Last edited: Jan 6, 2020
  49. timjohansson

    timjohansson

    Unity Technologies

    Joined:
    Jul 13, 2016
    Posts:
    138
    If you want to setup the link in that direction you need to wait for the job performing the delayed spawn. It is passed in as an input dependency so you would have to do "inputDeps.Complete();" before the loop. Doing that will introduce a sync point which you can work probably work around in this case - but you will need sync points somewhere to work with GameObjects.
     
    andreyefimov2010 likes this.
  50. andreyefimov2010

    andreyefimov2010

    Joined:
    Oct 29, 2017
    Posts:
    19
    Yeah, that's what I did and it worked. Thanks.
    Also, as a side-note: I feel like this NetCode package takes more responsibility of implying how game state management and rendering should be done than necessary (Hybrid Renderer, really?). It would be nice to have a more straightforward way to sync the net state and the game objects even with implied performance cost (a lot of people will be doing this anyway). Right now all of this is going to be done manually.
     
unityunity