Search Unity

Official DOTS development status and next milestones - November 2023

Discussion in 'Entity Component System' started by ashley_unity, Nov 14, 2023.

  1. ashley_unity

    ashley_unity

    Unity Technologies

    Joined:
    Dec 18, 2016
    Posts:
    21
    Hello, everyone — we’re here with our Q4 2023 DOTS roadmap update!


    What is DOTS?
    Our goal with Unity’s Data-Oriented Tech Stack (DOTS) is to empower creators to build more ambitious games with Unity and serve their needs in the production of complex multiplayer, crossplay, and open-world games.

    DOTS consists of several packages based on the Entity Component System (ECS) architecture. The notion of DOTS is often extended to include the Burst compiler and C# Job System, two technologies that have been created to support the development of ECS but are independent of it. The Burst compiler and the C# Job System can be used in any Unity project, regardless of whether ECS is being used.

    This past June, we celebrated the Entities package officially entering Released status with Entities 1.0, ensuring support for production on projects using the 2022 LTS. More recently, we released Entities 1.1, compatible with the 2022.3 LTS.

    If you’re looking to get started with DOTS (or just need a refresher), we recommend watching our DOTS overview from GameDevGuild 2022 and checking out the following list of resources and studio production stories:
    1. Entities API Overview, ECS Samples and Learn Content: The Entities API Overview, ECS samples and Learn Content reflect the latest APIs available with ECS for Unity in a bite-sized format. These resources will be especially useful for new users to learn about fundamental concepts of data-oriented design.

    2. ECS Network Racing: If you’re looking for a smaller-scale multiplayer sample with ECS for Unity, ECS Network Racing showcases the capabilities of Netcode for Entities and how you can easily build a multiplayer game out of the box. Specifically, the sample exemplifies an implementation of client/server architecture with client-side prediction, interpolation, and lag compensation.

    3. Megacity Multiplayer: Megacity Multiplayer showcases how creators can build ambitious multiplayer titles that support more than 64 concurrent players, while leveraging UGS services like Game Server Hosting, Authentication, Voice Chat, and Matchmaker.

    4. Experimental Character Controller Package: For users looking to quickly jump into building their game, the experimental character controller package also contains bite-sized samples illustrating how you can quickly drag and drop the experimental character controller into your scene, and get prototyping quickly.

    5. Stickylock Studios - Histera (March 2023): Jamel Ziaty, producer from StickyLock shares their journey and experience using ECS for Unity, Netcode for Entities, Game Server Hosting, and many more Unity Gaming Services for the creation of Histera, a competitive FPS that mixes prehistoric settings with war-torn 20th-century cities through time travel.

    6. Sunblink Entertainment - HEROish (March 2023): Justin Larrabee, CTO at Sunblink Entertainment, shares how their clever use of ECS for the production of HEROish helped them achieve performance, reduce production risks, and empower designers.

    7. Walaber Entertainment - JellyCar Worlds (December 2022): Tim “Walaber” Fitzrandolph, Solo Developer at Walaber Entertainment dives into using ECS for Unity to create a soft-body physics system for the JellyCar title.

    8. Kasedo Games - IXION (August 2022): Technical Director Jérémy Guéry walks us through how ECS has powered heavy NPC simulation for their city builder in space.

    9. Stunlock Studios - V Rising (June 2022): In a 3h-long Twitch stream, lead developers Rasmus Höök and Fredrik Haraldsson presented their use of ECS, from world-building in the editor with custom visual scripting to scalable open-world streaming.

    10. Ramen VR - Zenith: The Last City (March 2022): CTO Lauren Frazier explains how ECS helped them scale up their gameplay for managing a VR MMO game architecture.

    11. Electric Square - Detonation Racing (March 2022): Lead Developer Jonas De Maeseneer covers the use of ECS for deterministic gameplay for QA and design loops, as well as streaming for a high-pace Apple Arcade racing game.

    What’s new since the 1.0 release?

    Since hitting release status for Entities, we’ve been keeping a close eye on community feedback in order to improve package stability, resolve bugs, and address high user pain issues. A small set of these improvements are available in the pre-release Entities 1.1 package and can be viewed in our changelog.

    There have been other significant updates in the DOTS realm outside of the Entities package:
    • With the new 1.1 version of Unity Physics, we've added a collider inspector that shows collider data in detail and allows you to analyze and modify options like collision filters, collider types, geometry, and more in real-time. We've also simplified the process for runtime collider modifications with new functions and components that make it easy to create unique colliders and disable the sharing of collider data. Lastly, the dynamic scaling of rigid bodies has been made more intuitive by carrying the uniform scale of a rigid body present at edit-time into its runtime LocalTransform component. You can learn more about these changes and other relevant additions via the Unity Physics changelog.

    • An Editor fix for crashes occurring when using AudioClip.GetData() will be arriving for 2022.3.13f1.

    • After releasing our first experimental version of Burst Occlusion Culling for the 2022 LTS, we conducted many tests and rounds of user feedback. These efforts helped confirm some of our approaches, but we also noted the performance gains in real-world scenarios have not met expectations. We are working on improvements for future versions, but to avoid confusion, we will be removing this experimental feature from general availability in the meantime. Many thanks to those of you who took the time to test the early release of this package. We’ll share details of our new approach in an upcoming roadmap update.
    For more details, please check out the detailed changelogs for each of the corresponding packages:

    Grateful for Your Feedback

    We’ve received a stellar amount of feedback from creators like you over the past months! Know that we are carefully reviewing your input as always for consideration towards our roadmap.

    In particular, this past August we hosted the 2023 DOTS Dev Blitz Day — a special event where experts from the DOTS team participate in live discussions and Q&A with users on the Unity Forums and Discord. We received many excellent questions from those who participated and continue to see predominant feedback themes in the areas of animation, open-world creation, physics, and overall DOTS ease of use.

    It’s always a gift to hear from you all, be it here on the forums, on social media, or at live events like GDC and Gamescom. Thank you and please continue to keep us posted on your thoughts and needs as you’re working with DOTS.

    As always, the best way to ensure your feedback is heard is by submitting and voting on ideas within our public roadmap page. Each card on the page can be clicked on to see details and voting options, and all feedback shared here is directly routed to the proper product teams.


    Roadmap
    We continue to expect an increasing volume of feedback as creators leverage ECS with Unity 2022 LTS. Our roadmap remains stable and on-track for the coming year as we continue to invest in the following areas:
    1. Consolidation of ECS workflows: We’re dedicating a significant amount of resources towards understanding and addressing feedback from users adopting ECS in production. Based on the feedback that we’ve received so far, we'll be focused on the following:

      ● We’re acting on opportunities to further consolidate ECS workflows so users have a better time iterating and discovering gameplay in their projects. These consolidations also intend to make working with ECS more familiar and approachable to traditional Unity users experienced with GameObject workflows.

      ● To support fast iteration for your gameplay, we're preparing to ship the official production-ready release of the ECS Character Controller package (formerly known as "Rival - DOTS Character Controller" on the Unity Asset Store). You'll be able to use this package to easily add an efficient and highly-customizable character controller solution for ECS, and it can be used with the Unity Physics or Havok Physics for Unity packages. We also have other controller styles in progress that we'll share more information on in future roadmap updates.

      ● We’re developing a unified authoring workflow that will allow you to select your preferred physics backend via the Project Settings. Game productions can be unpredictable and sometimes need a different physics engine than what they started with. The need to change backends shouldn't occur often, but productions have been finding it cumbersome when it does. With this workflow, you’ll be able to switch to the backend you need without having to redo your project. We’ll share more specifics in a future roadmap update.

      ● We're working on a new global entities store. Previously, entities were allocated per world and it was often confusing to determine which world an entity belonged to. Entities will be allocated globally so their values are unique when created in different worlds. When debugging, there’s no ambiguity anymore about which world an entity belongs to and this ensures that the data displayed by debug watches is always reliable.

      ● In addition, entity versions will always be odd. Even values represent unallocated entities, which makes it more convenient to determine what’s what when debugging the insides of the entity store.

      ● We will be restoring "Frame Selected" / "Lock View to Selected" (”F” / “Shift+F”) functionality in subscenes.

      ● Official DOTS support for Unity 2023 is coming soon.​

    2. Enabling cross-play multiplayer game creation: Multiplayer games continue to be a key focus, and our teams are working hard to provide efficient out-of-the-box support for a variety of multiplayer genres. This includes a netcode unification solution delivering all the features you need, a central location in the Editor for multiplayer tools and services, a Playmode for faster testing of multiplayer games, an upcoming multi-platform crossplay sample, and much more! See the details in our November 2023 Multiplayer Update.

    3. Enabling open-world game creation: Scalable open-world authoring and runtime capabilities continue to be important for the Unity editor. This is currently being developed on two main fronts, animation, and worldbuilding. On animation, we continue to progress toward a highly performant, customizable 3D animation system that’s powered by our ECS foundation. For worldbuilding, we’re working on better environment workflows more broadly.

    4. Improve parallel scheduling and configuration: In the current version of DOTS, all systems run on the main thread and rely on those systems to schedule jobs to enable parallelism. We’re working to improve the out-of-the-box parallelism of DOTS by enabling systems to run in parallel, whether or not they schedule jobs internally. Additionally, the job system is missing configuration controls needed for adjusting how jobs are run to best enable performance for individual game needs. We’re working on expanding the priority and control of how jobs run to prevent common issues users see, such as unintentionally waiting on long-running jobs on the main thread.

    5. Improve Entities Graphics performance: Another initiative we are looking at is the support of DOTS deformation in order to provide a mesh deformation pipeline that is performant and scalable - that works on any platform with GPU compute capability supported in DOTS. We’re currently working on improving the GPU performance, as well as the current workflow, for a better and more seamless integration into the engine/SRPs. The next versions will also bring broader material support and URP support for Motion vectors.

    6. New Education Content: Our ECS starter samples provide a strong foundation for new DOTS users to understand and bridge the concepts of the C# Job System and ECS. We’re currently working on expanding this repository of samples into guided Unity Learning pathways that give you the skills necessary to start building your multiplayer project with DOTS. Additionally, we will be introducing a new sample that extends the Megacity Multiplayer sample to showcase how those capabilities can be implemented in a cross-play environment, including mobile devices and with a significantly larger number of concurrent players. More details on this announcement will be communicated at Unite 2023.
    Our public roadmap page has been updated to reflect recent changes and what we plan to deliver next.


    See you soon at Unite 2023!
    Learn more about DOTS at our Unite 2023 conference this November 15-16 in Amsterdam. You’ll be able to hear relevant updates across two key sessions:
    You'll also be able to catch other DOTS-related sessions from our Multiplayer and Physics teams:
    Be sure to keep an eye out for the debut of an exciting new sample from the DOTS team! This upcoming URP-based multiplayer sample game features multi-platform crossplay and support for mobile devices, all based on Unity 2022 LTS features.

    Let us know if we’ll see you at the event, connecting with devs and creators like you is always our favorite part of Unite! For those not able to join us in person, the Keynote will be available to stream on our YouTube and Twitch channels. You'll also be able to send in questions during Unite's virtual Ask The Experts event taking place on Unity Discussions and Discord.


    Thank you!
    We look forward to seeing the ambitious creations you develop with ECS and connecting with you all again in our next roadmap update. In the meantime, keep us posted on your thoughts and questions here in this thread!


    - Ashley
     
  2. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,761
    Look forward to seeing how this does in practice. It'll certainly be interesting to see its limits.
     
    Last edited: Nov 14, 2023
  3. TheOtherMonarch

    TheOtherMonarch

    Joined:
    Jul 28, 2012
    Posts:
    866
    From the ECS 1.2 changelog
    From my bug report

    Since CopyAndReplaceEntitiesFrom is going away and never worked. Will there be a way to deterministically control entity iteration order if you want to roll back a world?
     
    msfredb7 likes this.
  4. Endlesser

    Endlesser

    Joined:
    Nov 11, 2015
    Posts:
    89
    This really helps, thanks!

    Does it mean there's no OC available for DOTS at the moment or near future till it's improved? Do we have an understudy or hybrid method to approach OC in DOTS Graphic? Where can I find report in detail about this deprecation?(PS: Just about to test it next montho_O)

    Another brilliant work! can't wait.

    That's okay, even it's more or less the same announcement for Anim and World Creation last year... personally I'd rather expecting Animation Status Update (Q4 2023).

    Questions about new upcoming world building tools:

    1. Is there going to be any converting workflow to adapt current Terrain Tools outputs and products to next upcoming new one?

    2. Is it out of beta yet?

    -----------
    enjoy the Unite~
     
  5. msfredb7

    msfredb7

    Joined:
    Nov 1, 2012
    Posts:
    163
    Thanks for the update!

    I'm raising a flag here regarding determinism. Our project relies on it for multiplayer and having global IDs would probably break it. What workaround do you suggest? Would it be possible to manually specify which store a World uses?
     
  6. Fabrice_Lete

    Fabrice_Lete

    Unity Technologies

    Joined:
    May 5, 2018
    Posts:
    55
    The global entity store was designed taking the needs of Netcode for Entities into account, and the way rollback is done there doesn’t rely on stable IDs. Doing a low level rollback by simply swapping the chunks like CopyAndReplaceEntitiesFrom was doing never worked reliably and keeping this function around, even without introducing the global entity store, was a liability.

    Now the good news is that entity creation is still deterministic. Chunk order remains deterministic per world. The only thing that shouldn’t be done anymore is to sort stuff based on entity ID. E.g. for parallel ECB needs, you should always prefer using the chunk index.

    We want to keep pushing entities forward. I cannot comment at this point on our plans, but the global entity store enables important improvements. Unfortunately, this means that we will need to eventually break a few things. We don’t make those decisions lightly, and we’re sorry for the inconvenience.

    That being said, we’d like to hear more about your use cases where you reckon that global entities would interfere. If you prefer to avoid discussing your project on the forum, please reach out to me: fabricel (at) unity3d.com. Don’t forget the “L” after my name in that address.
     
  7. msfredb7

    msfredb7

    Joined:
    Nov 1, 2012
    Posts:
    163
    Thanks for the reply.
    Our use case is: we don't use rollback. We use SerializeUtility.SerializeWorld/DeserializeWorld to send the world over to clients, then we just share game inputs during gameplay like RTS games do. (NB: SerializeUtility changes the entity ids + chunks so when a client joins, the server and other clients have to perform the same operation)

    You say Entity order is still deterministic, so maybe gameplay won't break. However we'll have to fix our "desync detector" that relies on entity IDs being the same to diff. But more importantly, I find it risky to accidentally add new non-determinisic gameplay code if IDs are not (e.g. Sorting by value, like you said).

    EDIT: at the moment, we also support Entity references when sending GameInputs to the server (I use my spell on THIS enemy). That will need to change to IDs we manage ourselves for all entities.
     
    Last edited: Nov 16, 2023
    Selmar and Thygrrr like this.
  8. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,267
    This is a major concern to me, as I rely on full determinism of test scenes to track performance regressions, and those algorithms require a way to order an arbitrarily-acquired list of Entity references in parallel. Sorting that list ensured determinism, but these changes will break that.

    If it is possible to recover a per-world truly deterministic run-to-run ID via EntityStorageInfoLookup, that would likely be sufficient for my use case.
     
    Selmar, NikiWalker, Thygrrr and 2 others like this.
  9. TheOtherMonarch

    TheOtherMonarch

    Joined:
    Jul 28, 2012
    Posts:
    866
    Unity’s net code is designed for a generic FPS game, and it is highly specialized for that application. It does not rely on determinism. It uses an authoritative server. By writing your own specialized netcode it is possible to create a tailored experience. I don’t want to be uncharitable, but Unity clearly has an incentive to push its own networking which has fees and is one of the ways Unity generates revenue and an area that Unity management views as one of its main growth drivers. The upper management clearly does not understand that high-end network games require tailored solutions, not this generic garbage.

    • It seems like Lock-step determinism is still possible. But deterministically rolling back / resetting a world to an old state is not possible. Or at least not without having complete control over all systems including default Unity systems.
    • Without deterministically rolling back when you join or rejoin a game everyone will need to rerun the simulation from the start, and you will not be able to combine Lock-step determinism with client-side prediction.
    • Entities relying on memory location to ensure determinism is problematic. This should be abstracted away and not relied on for determining the game state.
    The game I am working on was designed for Deterministic roll back with client-side prediction and lag compensation. It sounds like we will have to fall back to old fashion Lock-step determinism and comment out our more advanced features. For testing we already had to disable the broken CopyAndReplaceEntitiesFrom.

    I am not sure if the application will need to close and restart every time someone wants to join a new game but I guess yes. Which is bad. I don't think It is possible to reset to the defaults created by ICustomBootstrap.
     
    Last edited: Nov 16, 2023
    Selmar and UniqueCode like this.
  10. colinleet

    colinleet

    Joined:
    Nov 20, 2019
    Posts:
    189
    I'm curious about what this will actually mean in practice.

    Right now most of the unity system which do use jobs intensively (physics, animation rigging, HDRP/URP, CPU particles) are still mostly configured to run about at the same time as they would on the main thread (e.g. scheduling and then immediately calling complete on the job handles to make sure their work is done before the next part of the frame timing loop). My guess is that the above text is mostly talking about making it so jobs are scheduled earlier, and then allowed to fully run on the background thread to complete running before their (running state "finish") handles are ".complete()"-d (release to the main thread) later in the frame?

    I ended up making a presentation about this a couple weeks ago,
    specifically on how to use the job system without making it blocking on the main thread for anyone interested:
    https://docs.google.com/presentation/d/1NOMPJ0EnPqpR0rhwXWJKENZ1yakhHvoiBTHb_-Zwv-c/edit?usp=sharing

    Or is this more about changing the underlying dynamic of being able to finally schedule jobs off of the main thread?

    Or is this more about just making the individual dots systems which aren't as tied together through job dependencies, all get schedules at roughly the same time (e.g. update by default) so they just execute in relative parallel before all getting completed in something like late update by default?
     
    bb8_1 and koonm like this.
  11. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    283
    Thanks for the updates, glad to see consistent information coming through!

    I very much appreciate that all of the work you all are doing on ECS is open source. It means that when bugs arise, we can make a small tweak to the package. It's a night and day difference from working with GameObjects.

    I'm interested to hear more about the global entity store. We also rely on entity ID determinism for our regression analysis, however, it would be fine as long as we are able to manually reset the entity store, or have the ability to create a second store. I'm happy to have a global one by default.
     
  12. jivalenzuela

    jivalenzuela

    Unity Technologies

    Joined:
    Dec 4, 2019
    Posts:
    76
    Thanks for raising this folks - as @Fabrice_Lete mentioned, while we took care to stage out the changes here in order to maintain compatibility with our intended deterministic behavior, clearly there was other behavior we either should have taken care to preserve, or communicated from the start that the behavior never intended to be well defined.

    As a public API
    CopyAndReplaceEntitiesFrom
    induces a commitment. Having discovered the functionality was incomplete and fragile, however, our options for support became pretty limited. I'm hopeful we can provide support for the use-cases y'all have outlined without recourse to that specific API (or at least better define the behavior).

    To that end please take @Fabrice_Lete on his offer or email me directly joe.valenzuela -at- unity3d.com . You've spelled out some of the use cases here (great) but honestly I'm not sure I entirely follow what's going wrong and I'd appreciate more context.
     
    Faahhx, bb8_1, diakou and 3 others like this.
  13. TheOtherMonarch

    TheOtherMonarch

    Joined:
    Jul 28, 2012
    Posts:
    866
    Discussing this privately is definitely possible. I just feel a lot of confusion still exists and think it would be helpful to get input from some of the other affected users on this forum.

    @msfredb7 is using SerializeWorld world to synchronize his states. I find it somewhat surprising that he is getting deterministic results from that. The whole point of CopyAndReplaceEntitiesFrom was that it “Guarantees that the chunk layout and order of the entities will match exactly.” Depending on what he is doing his results are totally plausible, just probably not in the general case.

    A lot of different meanings of determinism exist. The kind that would be the most effective would be where the explicate state is cloneable. Simple clone the ECS data and you can recreate the future game state deterministically. This is more than what CopyAndReplaceEntitiesFrom offered. Because CopyAndReplaceEntitiesFrom is implicitly linked to the chunk layout.

    Unity physics, I already believe, supports stateless determinism and a clone function. Uncoupling the ECS data used by the built-in Unity systems from the chunk layout probably has performance implications. I have only spent the bare minimum of time looking through the ECS source code. Someone like @DreamingImLatios would know better than me. Since as far as I can tell Unity ECS does not guarantee cloneable determinism. It remains possible now or in the future that the transform system or build physics world system etc. could have code that embeds chunk layout implicitly into the ECS data. Two examples that come to mind are.

    1. An entity job that outputs a list where the order of the returned list can differ depending on the entity iteration order. Obviously, you can sort the list, but this would not be necessary if iteration order was deterministic. I don’t know if any Unity systems suffer from this issue.
    2. Cases where the ordering of commands in an entity command buffer causes desynchronization.

    Internet upload speeds are still slower than one would like. Affected users have the option of reducing the scale of their game and or slowing the update rate. This may allow switching to a custom client-server. RTS games are notoriously difficult to network because of the complexity of the interactions and the fact that often the entirety of the units will be moving and visible either on screen or on the minimap.

    I don’t expect deterministic lockstep and or cloneable determinism to be officially supported anytime soon. I would like that to be the case but honestly, I have not been expecting that for a while. We did spend extra time making sure our code base could support such a design and if it becomes more viable in the future, we can adopt it then. For now, we will likely be using a brute-force delta compression design. Which later can be adapted into allowing resynchronization or rejoining.
     
    Last edited: Dec 2, 2023
  14. Fabrice_Lete

    Fabrice_Lete

    Unity Technologies

    Joined:
    May 5, 2018
    Posts:
    55
    Thanks for everyone's feedback!

    So, I'm trying to identify our options here. Would the following suggestion from @DreamingImLatios address everyone's concerns? If not, could you come up with an example of something that was possible before but would not be with that new approach?

    As for CopyAndReplaceEntitiesFrom, that function was copying chunks from world to world and relied so much on the chunk layout that it could only ever do the right thing in the simplest of cases. So I'd be curious to know in which context some of you managed to use that function, so we can figure out a way to provide an alternative.

    Like I said before, if there are details about your projects that you'd rather not share in a public space, feel free to send me an email at fabricel (at) unity3d.com but if you're fine otherwise, keeping the discussion public is better.
     
    Last edited: Dec 4, 2023
    Anthiese and daniel-holz like this.
  15. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,267
    To be honest, I don't even like my suggestion all that much. Doing those entity lookups into some other data structure would involve lots of random accesses. Personally, I would prefer if a WorldIndex was encoded into the 64 bits of an entity that could be masked out to get a deterministic value (though if it were the most-significant bits, masking wouldn't even be required for sorting). I suspect someone is going to complain about a cap on NetCode thin clients if that were done though.
     
  16. Fabrice_Lete

    Fabrice_Lete

    Unity Technologies

    Joined:
    May 5, 2018
    Posts:
    55
    Sure, what I have in mind would be something that doesn't require a random access. I was more focused on the functional side of it : a deterministic per-world id per entity.
     
  17. TheOtherMonarch

    TheOtherMonarch

    Joined:
    Jul 28, 2012
    Posts:
    866
    I would want the sortkey to be independent of an global entity ID. Only depending on the creation/destruction order of entities within the world to which they belong. The key should also remain constant, after creation, and only be released after the entity is destroyed.

    I am not sure if it would be worth it, but I think it would be possible to make destruction order irrelevant and still maintain determinism. By not recycling/reusing sortkeys immediately and instead waiting until at least after the next update/fixed update. Then sorting the reused keys before reuse. This should keep destruction order from affecting the determinism of pure entities. However, it may not be worth it because entities that use physics probably still need a defined destruction order.
     
    Last edited: Dec 5, 2023
    daniel-holz likes this.
  18. Fabrice_Lete

    Fabrice_Lete

    Unity Technologies

    Joined:
    May 5, 2018
    Posts:
    55
    If the sort key needs to be reused, it would require a similar kind of logic to what the entity store is actually doing. Which means keeping track of all the sort keys in use and allocating them, etc. I'm not sure that's the right path to take, it sounds unnecessarily expensive.

    What if it was instead a simple 64 bits counter that gets incremented for every entity allocated?

    Usual disclaimer: I'm just spitballing at this point, don't take any of this yet as a "here's what we're going to do".
     
    daniel-holz likes this.
  19. TheOtherMonarch

    TheOtherMonarch

    Joined:
    Jul 28, 2012
    Posts:
    866
    A 64-bit counter is a lot, and it would work for me. I am not using millions of lightweight entities; so, I would not worry about the counter wrapping around and matching the same entity, if it was still alive, which could be the case for a static entity. I would like some way to set the counter so that it could be reset to an earlier world state. If reusable IDs are used some way to reset both the counter and reused datastore would also be required.

    I am actually already using a 32-bit nonreusable counter only for my root networked entities. Because of bandwidth limitations I don’t think a 64-bit counter would ever replace my network ID system. Which is fine but something to think about. In my case even a 16-bit counter with reuse could replace a 64-bit counter. But again, just using the key for sorting would be fine.
     
    Last edited: Dec 6, 2023
    daniel-holz likes this.
  20. daniel-holz

    daniel-holz

    Unity Technologies

    Joined:
    Sep 17, 2021
    Posts:
    278
    A 64-bit counter that wraps around eventually: Yes! Exactly what I was thinking as well when reading through this thread. That works well until you reach a ridiculous number of entities ;).
     
  21. Fabrice_Lete

    Fabrice_Lete

    Unity Technologies

    Joined:
    May 5, 2018
    Posts:
    55
    All good then. And wrapping around a 64-bits counter is not something to worry about, if a high end CPU was doing nothing but incrementing such a number as fast as possible, it would still take years to do so.

    Please reach out before sending a probe running DOTS code to Alpha Centauri though.
     
    vladlomakin, Ruchir, Samsle and 12 others like this.
  22. daniel-holz

    daniel-holz

    Unity Technologies

    Joined:
    Sep 17, 2021
    Posts:
    278
    :D LOL
     
  23. Mike37

    Mike37

    Joined:
    Oct 21, 2018
    Posts:
    26
    If continuously creating 1 million entities per second it would take... *checks calculator* 585,000 years to overflow.
     
  24. colinleet

    colinleet

    Joined:
    Nov 20, 2019
    Posts:
    189
    There really wouldn't be any need to get in contact about running dots code on a trip to Alpha Centauri -- It'll just take a measly 77,700 years for Voyager to get to Alpha Centauri (if it was heading that direction). So DOTS would run perfectly fine for about 3.8 round trips to our neighbor and back before getting a buffer overflow.

    https://www.wolframalpha.com/input?i=Alpha+centauri+distance+in+km+/+Speed+of+Voyager+1+Spacecraft
     
    daniel-holz likes this.
  25. Thygrrr

    Thygrrr

    Joined:
    Sep 23, 2013
    Posts:
    700
    I'm also working on a lockstep deterministic simulation and I would very, very much prefer entities stay ordered and identified deterministically without having to tediously query some "ID" component or housekeep some sort of mapping table across threads and create/destroy operations to find each affected entity one by one rather than using the entity ID itself to, you know, IDENTIFY an entity.

    System works with a special network command buffer, sending across command streams (basically parsed player input) with some supplemental data, such as state checksums and timestamps, to each other client in the pod. Command streams are delightfully light-weight even if issuing commands for dozens of entities at a time. (e.g. group orders)

    I have a deep aversion to rollback and server side reconciliation systems (only one game, Counter-Strike, ever got it right) and I also don't think FPS are the best use case for ECS. That would clearly be strategy games, economy games, and large RPG worlds.

    The current indication seems that Unity Netcode is unsuitable for all three of those. Yet you're catering directly to the needs of Netcode with this change, instead of adapting Netcode to the actual strong points of ECS.
     
    Last edited: Dec 22, 2023
    davenirline, Selmar and sngdan like this.
  26. UniqueCode

    UniqueCode

    Joined:
    Oct 20, 2015
    Posts:
    65
    Not to be rude but it sounds to me like the only change that has to be made is how entities are addressed over the network. So instead of [Input,Entity] you send [Input,ID] and then lookup the ID when processing the packets. Once that's done everything else should be able to use the local Entity value.
     
    Anthiese likes this.
  27. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,267
    That's far from the only issue here. First, there's lifecycle management, where as a user, you effectively have to double your structural change cost to add the unique IDs and clean them up afterwards, since you would probably want to make them cleanup components. Then there's the cost of working with these IDs. Any time you use any type of native container with a ParallelWriter that uses entity handles, you would also need the IDs in order to preserve determinism. That means you'll effectively need a 128-bit ID (maybe 96) gathered across two different ECS chunk arrays with two different type handles or lookups, or you need to do random lookups when trying to recover determinism. This is a significant regression in performance, usability, and boilerplate just to make the editor tools prettier.

    Determinism is critical for any kind of reproducibility when debugging advanced performance-sensitive systems. The moment your structural changes are allowed to get out of order, chunk component values and component arrays extracted from queries become random every run. It is the whole reason that ECB.ParallelWriter has a sortKey in the first place. But it is also apparent that aside from whoever designed ECB, no one really cares about determinism at Unity because otherwise this change to non-deterministic IDs wouldn't have happened. Plus, transform hierarchies have had non-deterministic structural changes for several years now.

    At this point, I'd just like Unity to be consistent. If they want determinism, then they need to write deterministic code and provide users performant determinism-preserving tools. Otherwise, they need to get rid of ECB.ParallelWriter's sortKey and any documentation related to determinism. The whole 1.0, and 1.1, and 1.2 release situation is such a mess right now. I just hope that in 2024 Unity can make me excited about new ECS releases again rather than dread them.
     
  28. joepl

    joepl

    Unity Technologies

    Joined:
    Jul 6, 2017
    Posts:
    87
    Now that entities 1.2.0-pre.6 has been out for a bit, I was curious if users had experienced any show-stopper issues with it (particularly involving the entity ID changes discussed above). It would be great to know before we leave pre-release.
     
    Anthiese likes this.
  29. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,267
    It is broken for me, as well as anyone who uses my tech.

    Something to keep in mind is that most people I've talked to about this don't have a strong grasp of the issue and would be unable to explain the tradeoffs in their own words. After explaining the tradeoffs as objectively as I can, most people ask me something along the lines of "is that a good change or a bad change?"

    The reality is that this change really only affects advanced usage and optimization. Most people aren't at that level yet, so they haven't even encountered what breaks by this change. But this straight-up ruins a major advantage that Unity's ECS had over other ECS solutions.
     
  30. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,761
    Personally I haven't even noticed the ID change for better or worse.

    I've had a pretty smooth experience on 1.2 since pre.4 fixed the major inspector issues in 2023.3 (yes I know it's not officially supported but I require it -edit- now supported!).

    I'm sure there are some entities hierarchy error spam that at this point I just completely ignore but overall pretty happy with current state.

    -edit-

    Seems like 1.2 is officially supported on 2023.3.0b1 now :)

    upload_2024-1-5_9-25-47.png
     
    Last edited: Jan 4, 2024
    bb8_1 likes this.
  31. WAYNGames

    WAYNGames

    Joined:
    Mar 16, 2019
    Posts:
    992
    I agree with @DreamingImLatios , I'm one of the user who can see the difference and I consider myself at least an intermediate to advanced user of ECS. (Not an expert)

    Given how Unity communicated from the early days of DOTS about determinism I kind of always assumed that unity code was deterministic. Maybe with the exception of some math involving float which is hard to get deterministic on all platforms. And as far as I know it was still one of the most difficult toptic the burst team was trying to achieve.

    Now I learn that the transform system isn't and apparently has never been deterministic and given the change about the global entity store it doesn't even seems that determinism is part of the goals.

    I think determinism is something good to have but like anything else it may have tradeoffs.

    And at this stage I, for one, don't understand what the trade-off is and why the change came to be.

    I've read the thread in it's entirety and the only thing I can see is that the global entity component store would make id unique to avoid confusion about which world and entity belongs to.

    I've never had that problem. The editor windows are pretty clear about what world you are visualizing.

    It's entirely possible (even likely) that I missed or miss understood something, so can someone please help clarify it for me ?


    What does "losing" determinism implies ?
    What problem would the global entity store solve ?
    Does that help a particular type of networked games ? If so why this one and not another ? What's the plan for others ?
     
  32. Laicasaane

    Laicasaane

    Joined:
    Apr 15, 2015
    Posts:
    361
    I hope you will release 1.1.0 separately so I'll get my hands on WithPresent. I've just beta released the current project, but there are still many things to do to polish it for a full release. So I have no time to experiment with 1.2.0. Plus the implied drop of performance for tech like that of @DreamingImLatios makes me feel a bit unwilling to try 1.2.0. FYI, I currently depend on Latios transform and physics for both code design and performance goals. Literally I'm just here to observe and try to understand the impacts of this change.
     
    dwulff and goncalo-vasconcelos like this.
  33. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,267
    Losing Entity ID determinism causes some really fast parallel algorithms to no longer preserve chunk order determinism if their result is used to perform structural changes on entities (which is often required for managing complex relationships efficiently). Once you break chunk order determinism, there's some really bad consequences:
    • Any bug associated with chunk components goes from reproducible 100% of the time to a totally random chance it will reproduce. Chunk components are a secret superpower to speed in Unity's ECS when applied correctly. I use them to make deforming meshes work correctly with culling and LODs (in a way that improves performance when offscreen) as well as to make my custom transform hierarchy system 10 times faster.
    • Physics prediction will be less deterministic leading to more mispredictions.
    • Any algorithm that considers multiple entities in a chunk at once will also have bug reproducibility issues.
    Fun fact, Unity Transforms breaks chunk order determinism when you dynamically change an entity's parent at runtime. If you don't do this, you've likely not encountered this particular determinism issue. The other determinism issue in Unity Transforms is a race condition that will cause some unchanged chunks to have their LocalToWorld change filter triggered for other systems.

    Also, Entity ID determinism is what allows you to quickly navigate to the same entity in a closed subscene after stopping and re-entering play mode. It is a pretty nice debugging trick,

    I suspect it is mostly because people working at Unity get confused by multiple worlds having the same entity IDs. Or some high-level manager got confused and complained. I don't actually understand the reason myself.
    It helps no networked games. It is just less bad for some types over others. Types Unity has chosen not to focus on become nearly impossible with this change.
     
  34. WAYNGames

    WAYNGames

    Joined:
    Mar 16, 2019
    Posts:
    992
    Thanks for the details.

    Based on that my conclusion is : the change will hurt more the debugging experience than it would help by making entity id globally unique. And it will also make it difficult or impossible certain game design.

    If you really want to make it clear which entity is in which world for editor debugging purpose there are probably far simpler and less impacting solution.
    But again I don't have the issue so it would be interesting to understand in what specific context the non global entity id is an issue.

    Maybe make editor only system that add an enabled tag component to every entity corresponding to the world they belong to (client/server/thinClient) ?
     
  35. FaithlessOne

    FaithlessOne

    Joined:
    Jun 19, 2017
    Posts:
    320
    Thanks for stating the consequences of the change. I also use Netcode and have never had any issues regarding distinguishing between entities in the server/client world. I'd also like to hear from Unity, what justifies losing the advantage of determinism here?
     
    WAYNGames likes this.
  36. NikiWalker

    NikiWalker

    Unity Technologies

    Joined:
    May 18, 2021
    Posts:
    316
    Netcode for Entities itself is free, and you're not required to use any Unity Services with it. We're very protective of that flexibility.

    I can say anecdotally that the multiplayer team is actually very encouraging of users bespoke Netcode solutions, and we advocate internally for those use cases where we can, including Deterministic Lockstep and GGPO.

    We want to move thin clients into a single world eventually to remove the memory and CPU overhead, so this is thankfully not a blocker.

    I mostly disagree, but unfortunately the only compelling argument would be to prove it, and we just don't have the resources to prioritize those kinds of samples.

    It is true that FPS and action games are our primary focus with Netcode for Entities today, but, we don't intend to stop there.
     
  37. joepl

    joepl

    Unity Technologies

    Joined:
    Jul 6, 2017
    Posts:
    87
    @DreamingImLatios
    >Losing Entity ID determinism causes some really fast parallel algorithms to no longer preserve chunk order determinism if their result is used to perform structural changes on entities (which is often required for managing complex relationships efficiently).

    Any chance you can write down some psuedo-code or simple steps of where this can happen and how it becomes a problem? Apologies to ask for you to do the heavy lifting, but I want to make sure I understand the problem fully and can repro locally in a little sample.
     
  38. TheOtherMonarch

    TheOtherMonarch

    Joined:
    Jul 28, 2012
    Posts:
    866
    Archetypes and chucks are per world only entity IDs are becoming global, correct? The 1.2 manual implies that they are not global. I just want to make sure my understand of what is happening is correct.

    https://docs.unity3d.com/Packages/com.unity.entities@1.2/manual/concepts-archetypes.html
     
    Last edited: Jan 6, 2024
  39. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,761
    Someone please correct me if I'm wrong but from what I understand chunks have already been shared for a while now. They're stored in a single sharedstatic

    Code (CSharp):
    1. internal static readonly SharedStatic<ChunkStore> s_chunkStore = SharedStatic<ChunkStore>.GetOrCreate<EntityComponentStore>();
    Obviously each chunk will only be used in 1 world at a time though.

    I believe archetypes are still allocated per world though (probably important considering the limited archetype count you can have).
     
  40. TheOtherMonarch

    TheOtherMonarch

    Joined:
    Jul 28, 2012
    Posts:
    866
    Ok that is probably related to why CopyAndReplaceEntitiesFrom got so broken.

    Basically, if you make a structural change in one unrelated world you can change the output in a different world. World encapsulation sounds more broken right now then I thought.
     
    Last edited: Jan 6, 2024
  41. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,761
    Just to be clear, it has been like this for a very long time (was like this in 0.17). I don't really see how this implementation would cause any issues like you describe.
     
  42. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,267
    That's really good news!

    I still think there's a couple of pieces missing before I would agree with you on this (especially regarding NetCode), but this is a conversation I'd prefer to have in PMs if you are interested.

    Simple? No!

    As I mentioned, the use cases where this comes up are for advanced usages. The solutions to these problems are far from simple. But I will highlight 3 real-world use cases and if that's not sufficient for you to understand, please ask more specific questions regarding what it is that you don't understand.

    Example 1: InstantiateCommandBuffer
    In my open source LSSS game project used to stress test several of my tech's features, I ran into a bottleneck where main thread ECB playback was taking a long time. This was mostly caused by ships spawning bullets when they fired. ECB playback instantiates entities one-by-one unless you do them in batch, but there's really only a few different kinds of bullets in the game. It would be way faster if I batch-instantiated all bullets of a specific prefab at once, and then initialized their components. But the firing logic which also knows what those component values should be is still in a parallel job. So I wrote an InstantiateCommandBuffer to do this. Every instantiation command records an entity, a sortKey, and a pointer to another buffer containing the component values (that last part isn't quite true, as the buffer data structure allows me to recalculate the pointers, but you get the idea). When it comes time to play back these commands, I sort the commands with the prefab entity as the MSB part of the key and the sortKey as the LSB part. This groups up all the commands by prefab, and then within each group, the commands are sorted by sortKey to retain their determinism (the sort is a stable radix sort by the way). After that, I identify ranges to batch-instantiate the prefabs, grab their chunk locations, and then use dynamic component type handles to write out the component values. This reduces the performance bottleneck I was encountering significantly.

    Now, if prefab entities don't have stable IDs, then the order these prefab batches get instantiated in become random, and I lose order determinism. So Entities 1.2 breaks the determinism of this feature. You can read a more in-depth article about the design of this feature here. And you can see the latest version of the playback code here.

    Example 2-pre: Unity Transforms Failure
    Before I discuss how I handle parenting in QVVS Transforms, let me take a little bit to expose how Unity Transforms tried to use a fast algorithm and failed to preserve determinism. And how up through Entities 1.1, there was a simple fix that could be made (at the cost of some performance) that brought back full determinism.

    There's only one parallel job in ParentSystem.cs that can break determinism, and it is called GatherChangedParents. It uses NativeParallelMultiHashMap to gather all added and removed children grouped by parent (changing a parent involves removing the child from one parent and adding it to another). Later in FixupChangedChildren, the children are added and removed from the buffers in the order they appear in NativeParallelMultiHashMapIterator. That order is non-deterministic, which means the Child buffer order is also non-deterministic. That's already bad, but the structural change determinism doesn't get broken until you then destroy the parent. UpdateDeletedParents() uses the job GatherChildEntities to get a list of entities that lost their parents, and it does so in the order of children in each Child buffer. So this list is also deterministic. And it gets sent straight away to EntityManager.RemoveComponent(). Ouch.

    The really simple fix I used to employ back in 0.50 days was to simply accumulate all the children in the NativeParallelMultiHashMapIterator into a small list, and then sort that list before modifying the buffers. This ensured that all add and remove operations on the Child buffers were in a deterministic order, thus keeping the Child buffers deterministic.

    Example 2: QVVS Transforms ParentChangeSystem
    Unity's ParentSystem was prone to causing lag spikes in some of my projects when instantiating a bunch of entities. QVVS Transforms use the same patterns for maintaining the hierarchy with Child buffers and PreviousParent components, but I completely redesigned the system to be much more parallel. That code is here.

    The trick to this system is that I process all children (new, dead, and existing) in a single parallel job where I write out required changes to a set of blocklists, which you can read about in that article I wrote for InstantiateCommandBuffer. The big optimization I made was that some of the structural changes (actually most of them) could be determined just from EntityQueries. Right before doing these structural changes, I schedule jobs the sort these blocklists into arrays using the entity value as part of the sorting. I also do some deduplication. These jobs run while the main thread is doing structural changes and are typically "free" which is why I do that instead of using NativeParallelMultiHashMap. But the sorting is the big step that keeps everything deterministic. After making those extra structural change adjustments, I then use those same lists to patch the Child buffers and PreviousParent on the affected entities in parallel jobs.

    Example 3: Skeleton Skinned Mesh GPU Bindings
    This one is complicated and evil, but those from my framework community know that the determinism from this is essential for bug repros.My Kinemation animation and rendering system supports two crazy features.

    First, I allow users to modify the skeleton bone transforms any way they want, and frustum culling will work correctly. This is done efficiently via a conservative minkowski sum algorithm, but relies on the skeleton caching per-bone influence bounds of all the bound skinned meshes. However, it is the skinned mesh that determines the skeleton it is bound to, similar to the Parent component. Naturally, I use a similar strategy as what I use for QVVS Transforms.

    Second, I ensure the number of skinned mesh parts in a character does not affect performance in a significant way. That is, it doesn't matter if a character is one mesh or split into multiple meshes, and the meshes don't even need read-write access at runtime. The purpose of this is to facilitate character customization and procedural characters efficiently at scale. For this, I use blobs to store all the deformation data the GPU needs, and I reference count them. While I could do this in a normal ECS job, I instead choose to do this maintenance work in a job that runs parallel to an EntityQuery-based sync point, because then that work happens for free. So naturally, I use the fast writes to my blocklist type and then sort them by entity in the job for determinism. Some of the data I need to compute and upload to the GPU is relationship information between each skinned mesh and its skeleton. This is because I compute skin matrices on the GPU, which is a big part of how I make this guarantee. Again, the determinism here is extremely helpful for being able to compare RenderDoc captures when debugging. And while for this particular case I could probably get away with storing the unfilteredChunkIndex, that increase the amount of memory I need to write prior to the sync point which slows things down. If you want to see this optimization monstrosity, you can find it here.
     
  43. TheOtherMonarch

    TheOtherMonarch

    Joined:
    Jul 28, 2012
    Posts:
    866
    CopyAndReplaceEntitiesFrom has been broken since at least 0.17 I made like 5 bug reports over the years.

    Or are you suggesting that you don't think ChunkIndexInQuery is global?
     
    Last edited: Jan 6, 2024
  44. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,761
    Pretty sure ChunkIndexInQuery is just a value of 0 to length-1
    If you do an NativeArray<ArchetypeChunk> chunks = EntityQuery.ToArchetypeChunkArray it'll just directly index into this.

    I suspect we've taken a bit of a tangent here. All I was saying was I think chunks are pooled globally.
     
    Last edited: Jan 6, 2024
  45. TheOtherMonarch

    TheOtherMonarch

    Joined:
    Jul 28, 2012
    Posts:
    866
    My own use cases. Where I had been sorting with entity IDs. I have never used ChunkIndexInQuery for any of my sorting.

    I have one world using a FixedRateSimpleManager for the sims synchronized code and a different world and or gameObjects for graphics. The unsynchronized graphics world affecting the deterministic world is illogical.

    The same issue applies with entity command buffers. For example, you do something on a client across the network that affects chuck order and then undo the change and update the unsynchronized world. But the chuck order has changed so when you apply your entity command buffer on the server the order is different between the client and server. On the server a variable gets set to 3 then to 5 and on the client the order is reversed 5 then 3. You can probably get around this by changing your command buffer execution order, such that the order does not matter, but in some cases, it may be very hard to understand. If you are relying on this command buffers would not even need a sort key. Likewise, for ParallelWriters.

    I don’t know what is going on in every line of Unity’s code, mostly it seems to be at a lower level and largely ordering agnostic. Gameplay code on the other hand has all kinds of relationships linking stuff together where sorting is sometimes required.

    It is easy enough to fix these kinds of issues in your own code. Not all entities need any type of sorting at all and adding a sort key component is possible. The issue is that a standardized way of ensuring determinism is required. EntityIndexInQuery is 32 bit and gets reused across worlds. A 64-bit resettable counter as suggested would do it for me.

    Basically, depending on chuck order for determinism is very fragile and does not support roll back well and Global IDs are subject to changes from different unsynchronized worlds that are not updated at a fixed rate. Also breaking roll back and encapsulation. You may be able to work around it for Deterministic Lockstep by providing your own sortkey. A traditional Deterministic Lockstep generally has unsynchronized elements that differ between the clients. At a minimum particle effects, trails, sounds and sometimes animations. When you want to host a new game, you would need to kill the program and use some kind of application launcher as a lobby. But GGPO seems too high risk.
     
    Last edited: Jan 7, 2024
  46. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    283
    It would help to understand what features the global entity store is intended to enable long-term. I'm guessing it's to help better integrate ECS into normal editor workflows? ie.
    • Making it possible to get a debug name from a simple entity reference.
    • Allowing GameObjects and UI to hold permanent references to entities.
    • Allow errors to cite exactly which entity is causing problem, rather than an opaque entity id.
    • Improve workflows with multiple worlds (ie. easily identify when entities accidentally store Baking World ids)
    I'm pretty on board with all of these improvements. They all logically need some sense of a global entity id. That said, I feel like a better solution might be to create a 'GlobalEntityId' reference that's simply:

    Code (CSharp):
    1. public struct GlobalEntityId {
    2.    Entity entity;
    3.    ulong WorldSequenceNumber;
    4. }
    GameObjects and UI could hold references to these. The name database could use them as well, as could any other general debugging data. It's a little less space efficient, but not by much, and largely used by non-runtime functionality.

    Game code could continue to use Entity ids as a performant identifier within the world, like everyone else here desires.
     
    rileyoc and JesOb like this.
  47. NikiWalker

    NikiWalker

    Unity Technologies

    Joined:
    May 18, 2021
    Posts:
    316
    Absolutely yes. PMs open, or email at niki.walker (at) unity3d.com.
     
  48. Fabrice_Lete

    Fabrice_Lete

    Unity Technologies

    Joined:
    May 5, 2018
    Posts:
    55
    Lots of things to unpack here, thanks everyone for your input.

    First, “determinism” is a loaded term. Certain things are and certain things are not. In any case, everything being deterministic everywhere is just not going to happen. What we want is a clear contract over what is deterministic and what is not, so people know what they can depend on when they need determinism. And for example, lock-step networking is a good use case we want to support.

    The change that triggered this discussion is the change in the entities allocation strategy. It was made in a preview release, and it can be disabled with a define (ENTITY_STORE_V1) that gets you back to the exact situation you had in 1.1 so the kind of feedback we’re getting now is exactly what we wanted from a preview release. We’re getting a clear signal about which use cases broke and what we need to address.

    That being said, this change makes entities themselves come from a global pool instead of a per-world pool but nothing else changed. The way chunks are allocated is the same, the order in which data is stored in chunks is the same. And the problems that were present before are unfortunately still present, like CopyAndReplaceEntitiesFrom being unusable and the parenting determinism issue.

    One thing improved, and it’s the speed of entity allocation. But it doesn’t matter much at this point.

    A question was asked about what the benefits of global entities were. If you ever tried to use a C# debugger with multiple worlds you probably already know that the watch information was useless, that’s not the case anymore. But the main goal is that global entities enable moving entities from world to world with much less remapping. This means faster streaming, and the ability to efficiently move workloads to different worlds for async processing.

    As a side note, I’ll add a small remark about sorting by entity index vs chunk index when working with parallel ECB. The preferred way has always been chunk index. Because even with the local allocation, you could have, let’s say, entities 1,3,5,7 in chunk 1 and entities 2,4,6,7 in chunk 2. If you sort by entity index, when playing back the ECB the memory access will ping-pong between the two chunks, whereas it will be ordered if sorted by chunk id.

    Another remark was made about being able to reload the same entity scene and get the same entity indices. This is absolutely not guaranteed (and never was), the scene streaming is asynchronous and if there’s something else allocating entities, the indices used by the scene will depend on when the scene gets merged into the world. This is part of the things that might look deterministic but are actually very fragile, that's why we’d prefer to have a clear delineation between what is and what is not. The unfortunate truth is that when we call out something as non deterministic, it’s not always a regression, it’s sometimes because it never was.

    In any case, we know what to do from here. We need to support entities allocation determinism for the use cases that need it, the current thinking is that it will likely take the shape of explicitly deterministic worlds which are provided with a fixed range of entities. This way people will have a choice between deterministic allocation but slow movement of data in/out of the world on one hand, and fast movement of data with global entities on the other. And it being a per-world option, you can mix and match. Putting your deterministic simulation in a deterministic world, and the data-heavy streamable content in another.
     
    dwulff, eizenhorn, Ruchir and 18 others like this.
  49. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,267
    I've always made subscene loading synchronous when I needed this. Usually I perform this kind of debugging when the bug can be recreated within the first few seconds of entering play mode. But you are right in that this will probably never be fully deterministic.

    This 100% feels like the right solution to me too! Thanks for listening to us and finding something that works!

    If I could just ask for one more thing. If it is possible, could you add a per-world deterministic ID to ArchetypeChunk? I'm not always iterating chunks by an EntityQuery.
     
  50. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    283
    Appreciate the clear communication. I can absolutely see the value of removing remapping. It's a huge performance cost right now for level streaming. Deterministic worlds sounds like a great solution, especially if we're able to control the ranges of entities they use.

    For the 'deterministic contract', to me that means:

    If you provide inputs to the simulation, on the exact same frames:
    - Disk state
    - Input Devices
    - Async Timings
    - Delta Times

    You get the same outputs from the simulation:
    - Determinism for all blittable component types and data.
    - (best effort) Determinism for managed types when comparing via equality, hashing, etc.

    For instance, Async Level Streaming always followed my 'deterministic contract' because we record frame timings for section loading, and force them to load at the original frames during replays.

    In our replay system, we can hash component state on each frame and determine whether it has desynced and more importantly what desynced. It's really great. :)
     
    PolarTron and koonm like this.