Search Unity

Questions on if Unity ECS is a good fit to start developing on now

Discussion in 'Data Oriented Technology Stack' started by cjddmut, Dec 10, 2019.

  1. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    120
    My team is in a position where we're trying to figure out whether Unity's ECS is in a good position for us to start developing on. The game we're working on has a lot of simulating state and ECS is a really good fit for it. We currently have our version of ECS that's built on Unity's Job System and Burst Compiler so I've been able to see just how efficient it can run. However we're lacking features that's causing me to reevaluate Unity's ECS before development really starts in earnest.

    However going over examples, API lists, and decompiling Unity's ECS I'm having a hard time answering if Unity ECS has the features that we'll end up needing in order to deliver on our game. A big wrench that's thrown is that our game is networked and uses a predicted deterministic lock step networking model (think GGPO but we won't predict the entire state of the game...similar to how Factorio does it based on their blog posts). We had been working to make our own ECS have easy support for this model but if Unity's ECS can support it as well then that would save us significant time.

    Checksum State
    If our game is deterministic then we'll need a way to detect that we are in sync at least while in development. In some ways ECS makes this easy because it packs state in such a easy to look at way. Would it be easy to checksum whole archetypes? This should also include being able to checksum dynamic buffers as well as a way to custom checksum components that have complex members (like a pointer since that can't be checksummed by memory). I'm aware of other issues such as padding that can make this tricky but there are ways around that.

    Also for sanity sake being able to dump a log of granular checksums so we can have tools to show which components, or even entity, had diverged can help with debugging. Are there sufficient hooks to do this kind of thing in Unity ECS?

    Archetype Rollback
    We're not doing full state prediction so some state will predict every frame while others don't. If in one world then you could imagine there being a Predicting component that specifies certain archetypes as the ones that predict and a predicting system would query to only work on those archetypes. However, once we get the input from other players we need to reset the archetypes that were predicting. I see that there is serialization/deserialization for whole worlds but what about archetypes chunks?

    It's possible instead that we have a predicting and non-predicting worlds instead. But I'll have to have a clean way to interact between the worlds, for example, EntityCommandBuffer might actually be from arbitrary worlds?

    Messaging
    This has less to do with our networking model and more of a question of how this would be accomplished in Unity ECS but how does Unity handle messages that sent from one entity to another (or one system that targets another entity). An example might be a system that explodes and damages entities. You could imagine this system gathers up a bunch of entities within radius of the exploding entity and registers damage to those entities. You could imagine it could do this by adding a component to the damaged entities with the damage amount. But what if two exploding entities overlap an entity, wouldn't we lose one?

    Split Up World Into Chunks
    Our world will be big enough that we won't want to simulate the entire thing constantly and can sleep chunks if the players are far enough away. As far as I can immediately tell there are two ways I could try to approach this.

    1. Split entities into separate worlds based on their location. Then I could only update the worlds that are 'awake'. I'll need a way of wholesale moving entities from one world to another when they move from one chunk to another.
    2. Have dynamic tags. I could tag entities with a chunk index for example which would put them into specific archetypes and only run the archetypes that are currently 'awake'. If tags have to be typed structs though then I'm not sure how I see how I would do this in Unity ECS. Filter based on a shared component maybe?

    Unsafe Component Writes
    We had to build our own collision system to ensure determinism. This collision system currently runs in its own 'physics' world and after it runs it'll run a sync system that takes its physics position and syncs it to the game simulation system. We do this unsafely by writing directly to the component memory in our own ECS system. This is safe to do because there's only ever a 1 to 1 relationship between physics entity and simulation entity and won't be subject to the lost update problem. I can't immediately find a way that we would do this with Unity ECS. I understand this is 'risky' behavior but is there a way to do this?

    Save/Load Versioning
    It looks like this is supported when looking at the APIs but they aren't fully fleshed out so I would want to confirm that deserialization/serialization of worlds can work across build versions. What if a components has a member added, removed, or rearranged? Assuming it's a memory serialization then I imagine this would have issues otherwise.

    Update
    This one looks supported but wanted to bring it up as it is essential to our model but we have to control the update function tightly. We'll have to pause it when there aren't any inputs and step it forward arbitrary amount of times based on the number prediction steps.

    Any guidance here would be awesome :) I'm excited and hopeful the Unity ECS is a good path forward as not reimplementing even a small part of the features would be an amazing time saver!
     
  2. loicbaumann

    loicbaumann

    Unity Technologies

    Joined:
    Mar 21, 2019
    Posts:
    4
    Hey there!

    I think I can bring some elements of answer, my colleagues can complete/give other point of views.

    Checksum State
    This is something that we are currently working on internally with the Determinism Framework that does exactly what you're talking about for the exact same reason. We want both runtime and serialized chunks/sub-scene to be deterministic and have Hash128 to detect changes.

    Archetype Rollback
    Well, I'm not quite sure of what you're trying to achieve here but the DotsSample that will be released soon is a client/server game with prediction/lag compensation on the network level, maybe it will be a good source of information.

    Messaging
    Why not having an "AccumulatedDamage" component that get added by your two systems and reset by another one when you apply the damaged on the entity?

    Split Up World Into Chunks
    You can for instance use SharedComponentData to partition your objects in a single world. Setting/changing the value of the shared component "Cluster" { X, Y } to a given entity will move it in a chunk where all others entities share the same X, Y. Then it's easy and very fast to do queries based on this.
     
    eterlan likes this.
  3. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    120
    Thanks for the reply!


    I must still misunderstand cause either only one of those explosions would add the system or I'm not sure how they would both accumulate the damage since I don't believe I have access to another entities component. If I kept the value outside ECS then I might hit race condition issues?


    The networking model we're following is a little bit different than the one you're talking about but there's definitely some overlap in concepts so maybe it would be a good source. The short of what I need is a way to rollback the state of archetypes chunks to a known good state. The same is probably true in the prediction in a replication style networking model.

    Basically if I want to do something along the lines of...

    // Pre prediction storing of state
    bytes = GetArchetypeBytes(archetype)

    // blah do some prediction with systems

    // My rollback for this archetype chunk!
    SetArchetypeBytes(archetype, bytes)
     
    Last edited: Dec 10, 2019
  4. dispatch_starlost

    dispatch_starlost

    Joined:
    Nov 17, 2017
    Posts:
    29
    You'll likely want to add a dynamic buffer to the entity. Or rather, add an entry into the buffer. You can specify custom buffers so you'd have something like Damage, DamageType in there, then you can have a system which applies damage and adds status effects based on the DamageType, then removes the buffer entry.

    Also you're not adding a system, you're adding a component - semantics :p
     
  5. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    120
    Sure I meant component :)

    The dynamic buffer idea would only work if it supports concurrent writes and I'm able to sort the buffer before consuming it (otherwise I might break determinism). I can get it as a NativeArray so I can probably sort it but I don't see how I could do concurrent writes to it.
     
  6. Enzi

    Enzi

    Joined:
    Jan 28, 2013
    Posts:
    248
    Messaging differs a lot in ECS.
    What you would do at this stage is create a combatEvent entity with all the necessary data that is consumed by a CombatEventsSystem
    These are fast-living entities which are, if your system order is correct, not even a single frame. So think of messages more like its own entity, completely decoupled from any dependencies but with all the data you need and not like buffers/arrays or anything like that. The reactive nature of ECS and archetype design can deal with that.

    On the topic of parallel writes:
    You can use NativeDisableParallelForRestriction to multi-thread different parts of the chunk.

    Right now, I think determinism is the biggest dealbreaker for you. I haven't seen too much problems myself but I also don't have a running network game out in the wild. When the Unity devs say it's not quite there yet, I trust them.
     
  7. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    120
    I understand that idea as well but it still kind of falls apart. Even if I create a 'combatEvent' component which contains the entity to target and the damage, then it doesn't seem especially clear to me how I would process those events other than manually through the EventManager. Which I can do! But feels somewhat unfortunate since, while I have to run the system in non-parallel (due to Lost Update problem) I could run different event systems in parallel against each other based on their comp read/writes.

    And if I'm gonna go that far I can also create my own command buffer handle that keeps a NativeQueue which I sort before processing by a message handling system. That at least solves the determinism issue but still has the unfortunate problem of being required to go through EntityManager instead of any job/burst friendly interface (as far as I can tell).
     
  8. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    750
    Entity as event and its brother component as event are popular patterns for their simplicity, but they don't really scale well. Actually, event systems are kinda overrated in DOTS as there are almost always better ways to solve the problem. (I'm not saying you should completely avoid them, just that you should not depend on them to solve every problem.)

    In your explosion example, instead of thinking about the explosion gathering all the entities and then applying damage to them (a write dependency), think of your entities gathering the explosions they fall into and have them damage themselves based on the data they read from the explosions (a read dependency which is safe in parallel).

    If you do find yourself truly needing events, NativeStream is almost always your answer if you care about performance.
     
    Opeth001, florianhanke, Jes28 and 2 others like this.
  9. GilCat

    GilCat

    Joined:
    Sep 21, 2013
    Posts:
    544
    I totally agree with @DreamingImLatios and that you should really think through your problem and see if you actually need events or not.
    In case you need events there is some work already done here using NativeStream.
     
  10. eterlan

    eterlan

    Joined:
    Sep 29, 2018
    Posts:
    111
    I'm reading dod-book these days, I think your solution on messaging & Split Up World is super fit to that book! Thanks! Messaging in this way is far better than use the entity as event.

    May I ask about how would you handle mutable state such as AI current action? Use SharedComponent? I'm not very sure about why document say SetSharedComponent is costly, it seems like doing exactly the same thing as other structural changes, like adding a tag component.
     
  11. eterlan

    eterlan

    Joined:
    Sep 29, 2018
    Posts:
    111
    It's working like this, there are two systems, Exploding system, ApplyDamage system.
    Let's say there are 2 bombs and a player nearby, the exploding system query all bombs and detect if there is any player in bomb range, if yes, then accumulate damage by using GetComponentDataFromEntity<AccumalateDamage>. Later (maybe 4 frames) the ApplyDamageSystem would query all AccumalateDamage and apply it to the Health component and clear it. In this way you can accumulate many damage by all sources, bomb or bullet, with one component.

    What other people suggest is using Entity as event. In Exploding system, after make sure which entity with health is nearby, using EntityCommandBuffer to create a new temp entity with a single component, Damage, with two fields, damage value and receiver entity.
    And later in the ApplyDamage System, you query all this kind of temp entity - component, Damage, and apply effect one by one on it's receiver.

    I use the latter method, but now I prefer the first one mucccch more. English is poor, hope this is understandable.
     
  12. dthurn

    dthurn

    Joined:
    Feb 17, 2015
    Posts:
    33
    It kind of depends on your timelines and stuff. I use ECS on a mostly "for fun" project, where I can deal with the occasional bugs/missing features/incomplete documentation. If I was paying a team of people to get a game done in a year, I'd definitely avoid ECS -- we'd be using 2018 LTS and banning anyone from even looking at the "show preview packages" option in the editor :).
     
  13. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,830
    We have EntityManager.CopyAndReplaceEntitiesFrom that we have used internally for a GGPO prototype at a hackweek. We basically keep a backup of the acked deterministic state in a backup world. And then continously reset the simulation back to the acked deterministic state and resimulate multiple steps forward. The method automatically using change version numbers to only copy data that has changed. Overall it worked surprisingly well & simplifies things a lot.


    The Internals of that function are not particularly complicated, so if for some reason you need the ability to copy chunks to a backup store based on some more precise filters, it should be quite straightforward for you to modify the code.

    That said, i think generally speaking i would always try to reset the whole world to a deterministic state. Its simpler to validate & enforce determinism with.

    Unity.Entities & Unity.Physics is forward deterministic & rollback deterministic on the same CPU architecture today.

    We have done some tests and experiments on making Burst be capable of cross CPU floating point architecture determinism (So you could have deterministic simulation between a PC & iOS for example, while using floating point math) but this is still quite far our. It is not a focus in 2020 Q1/Q2 to work on it. But likely we will look into it after end of next year.


    Overall based on you requirements you wrote, DOTS is a quite good fit and I would guess the couple of places where you would need changes is a whole lot easier to just modify the code rather than roll something completely on your own... Deterministic builds & simulation is definitely something we have set as a principle while developing DOTs.
     
    Last edited: Dec 11, 2019
    MNNoxMortem likes this.
  14. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,830
    I am not quite sure how that would work. Determinism when the input has changed, seems like an impossible problem to solve.

    I think for game state save, you have access to all the in DOTS directly. I would recommend just writing your own game save that saves the state that is necessary for your game.

    The main trick in a deterministic simulation is to seperate simulation state from presentation state. You can do this with multiple worlds. I would design everything so that the simulation world just references ID's of prefabs to instantiate in the presentation world, but the simulation world doesn't actually have any renderers or characters in it. This also makes rollback faster and gives you control over rollback of fatal & non-fatal events. eg. if you have predicted death but then want to rolll that back, thats not gonna work well. Better to tightly control and seperate things that can be rolled back, like movement vs things that can't and handle it explicitly in the transfer of data from simulation -> presentation world.
     
    Radu392 likes this.
  15. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    120
    I'll not respond to all the messaging ideas as it would be a little off topic :)

    Thanks for response Joachim_Ante!
    This would be our initial approach but I have my doubts it'll scale (our world can be quite large) but it's sufficient to know that it is possible to do even if it's not simple.

    This is interesting! We've already built a fixed-point math library and physics system but it does run slower than a floating point system. I guess the pointed question I want to dig at is is it deterministic on all PC CPU/OS that we could expect to support today? We're not looking to do cross platform play but on desktop we would need to rely on determinism.

    I guess a follow up question then is would Burst be deterministic on the same platform then?

    I think I must not have been clear. I'm trying to get a feel for how serialization/deserialization of a world would work when loading a game after a player has updated their build which means that components might have changed shape. I don't mean allowing players of different versions to play with each other. I'm not sure if determinism is a factor here.

    Thanks!
     
  16. TheOtherMonarch

    TheOtherMonarch

    Joined:
    Jul 28, 2012
    Posts:
    94
    It is unclear is floating point determinism on the same architecture close or far? Or just deterministic on the same individual PC.
     
  17. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,830
    Floating point determinism on the same CPU architecture is supported today.
     
  18. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,237
    I wouldn't feel comfortable starting development of any sort of *real* game with DOTS at this moment, but I do feel comfortable starting to develop "tools" for it.

    Basically the way I approach it right now is that I've just started work on building a library of gameplay tools (characters, cameras, hitboxes, weapons, projectiles, inventories, pickups, stats&buffs, etc...), and I intend to keep doing that for the next 1-2 years while DOTS matures. It's a great opportunity to start learning, and to be ready to make something with it as soon as it's ready.

    I'm personally not a fan of any sort of hybrid approach to DOTS, where parts of the game are implemented with the Monobehaviour/GameObjects workflow. It just feels a bit messy to me, and I'd rather go either full-Monobehaviour or full-DOTS. So I'm very much waiting for that to not be required anymore before I consider making a real game with DOTS (I'm excluding the conversion workflow stuff here, it's okay if that stays)
     
    Last edited: Dec 18, 2019
  19. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    120
    Would this be with/without burst or either? If I can comfortably drop our fixed point system then that would make some things MUCH simpler for us.

    EDIT: Trying to be very clear with my question as whether or not we can rely on floating point determinism could be a big win for us!

    Sounds like on console (all three) we can rely on floating point determinism for each individual console?

    On desktop are we able to rely on the following...

    1. Same architecture different OS?
    2. 32-bit Architecture vs 64-bit Architecture? (maybe not required for us but would be nice to know)
    3. Mono is deterministic? The il2cpp is?
    4. Does this include the Mathf library or just the math library?
    5. I guess really the question I'm digging at is for any expected desktop user should we be able to rely on determinism not just on the same computer but across all desktops?

    If these questions seem repetitive to above it's only because this is a critical piece of information for us and I'm trying to make sure I understand :)
     
    Last edited: Dec 20, 2019
    Jake99Jake likes this.
  20. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,830
    1. It will work on same architecture different OS
    2. 32 bit vs 64 bit architecture will require the full cross architecture determinism support
    3. No. All math code must be in bursted jobs
    4. only math library
    5. You should expect floating point math code to work across two different computers as long as they are on the same CPU architecture
     
    Jake99Jake likes this.
  21. cjddmut

    cjddmut

    Joined:
    Nov 19, 2012
    Posts:
    120
    Ah I see, it's deterministic when using Burst.

    How does that apply to Nintendo Switch? As far as I can tell Switch is not supported according to the Burst documentation. I have no idea if outside of Burst if floating point determinism can be counted on for Switch.
     
    Jake99Jake likes this.
  22. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,830
    Switch is supported in the 1.2, 1.2 is currently in preview.
     
    cjddmut, Jake99Jake and GilCat like this.