Search Unity

Feedback request: Envisioning new ways to debug in DOTS

Discussion in 'Entity Component System' started by KjartanK_Unity, Apr 22, 2020.

  1. KjartanK_Unity

    KjartanK_Unity

    Unity Technologies

    Joined:
    Apr 2, 2018
    Posts:
    4
    We’ve been hard at work designing and building the tools for how you navigate, inspect, and debug the data behind your DOTS-based game. We wanted to share some of that work with you here to start gathering some feedback on how it might impact your work. These features are in various stages of development so you may see feedback addressed either in the initial release or in a follow-up iteration. We’ll also be more active in the future posting about our plans for Editor workflows around DOTS.

    For us to best apply your feedback, it would be helpful if you can share motivations or goals behind what you’re trying to accomplish when you need a particular piece of information or the ability to perform a certain action.


    Systems Window

    We’ve heard and read a lot of feedback about how we visualize the Systems you have running and we’d like to take a first step towards addressing some of that. Some of the first things we’ve heard and are looking to address include clarity around what type (standard, group, command buffer) a System is, easier to read and distinguish query types, and lack of visualization of scheduling dependencies.

    Our design goals for this window include (1) giving you a clear view of the order in which Systems are running in your World and why, (2) better visualizing what a System is doing in terms of queries it runs, and (3) clearer indication of what component types make up a query. We’ve shared some mockups of various states below.

    Separating this information into a separate window also allows for more flexibility in how relationships to other information are shown, including for example which Systems have queries matching an Entity.

    Please share your thoughts on how this redesign of how you see your Systems affects your workflow, what value it brings, and what improvements you’d like to see moving forward.



    To address one potential question, this is the first step in creating a better experience for all of the information and workflows that you today find in the Entity Debugger. We’ll eventually be removing the Entity Debugger, but not until we have full feature parity and confidence that the new tools are more usable and useful.


    Entities Window

    We’ve heard your concerns about the lack of ability to browse through your Entities, so we’ve tried to bring a familiar perspective on your runtime data by putting together a view of the Entities in a World as a transform hierarchy (we’re also exploring giving you the ability to group and organize by other components). We’re looking to make some improvements to how you search and filter in such a data-heavy environment as well.

    Our design goals here are to (1) give you an Entity-first view into your World, (2) allow you to navigate your data in a way familiar to you as a Unity user (via transform hierarchy), and (3) to start showing you relationships that your runtime data (Entities) has with the authoring data (GameObjects) that created it and the Systems that act on it.



    For the Entities window, we’re interested to hear what value you think this will bring to your workflows, what you could imagine using it for, and if there are any other ways you mentally organize the Entities in your game. Additionally, are there any opportunities to build on this that this inspires?


    Thank you for reading and sharing your feedback! As mentioned, we’ll share more in the future to gather more feedback on other tools and workflows we’re crafting to improve your DOTS working experience.
     
    PhilSA, HeyZoos, Vacummus and 38 others like this.
  2. Orimay

    Orimay

    Joined:
    Nov 16, 2012
    Posts:
    304
    Wow, awesome!

    I hope there'll be no negative time displayed for slow systems. Even if it takes 3600+ seconds to run. No good system will run this long, but if one does, I need to see it in order to fix. And would've been awesome if the systems that wait for something weren't reporting they are taking this time, butt the slow systems that make everyone wait, were taking and reporting it instead.

    Entities window is also amazing. I'll be able to tag different groups of entities and filter them by tag! Show me everything trading-related! Now — everything ai-related!

    Will the entities be editable via the inspector?
     
    Cynicat likes this.
  3. TLRMatthew

    TLRMatthew

    Joined:
    Apr 10, 2019
    Posts:
    65
    Will the Systems window be able to display sync points as well? Being able to see where sync points are being triggered without opening up and deciphering the Profiler (and even better, WHY the sync point was created) would be an amazing workflow improvement!
     
  4. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    8,194
    Cool just what I was asking for in the API thread yesterday...

    If this provided CPU usage and Memory Bandwidth usage % as well as a system throttling mechanism* then we could fine tune our systems for optimal performance.

    * Thread limits and/or ms per frame heuristics and/or memory bandwidth priority or percentage limits.
     
  5. JakHussain

    JakHussain

    Joined:
    Oct 20, 2016
    Posts:
    318
    Selecting entities in the scene view is a big deal for a lot of workflows. I'm sure you guys are already aware of the open source solution someone made over a year ago and it's still supported. Selection of an entity in the scene view to highlight it in the entity window and vice versa is a must for production readiness.

    And as others have said, editing entities in the inspector during play mode. @tertle was first to discover how this could be hacked into working as of Entities 0.8 but exposing this by default would do wonders for iteration time.
     
    Nothke, Cynicat, toomasio and 9 others like this.
  6. Smilediver

    Smilediver

    Joined:
    May 5, 2011
    Posts:
    72
    Please add some way to visualize dependencies between entities as graphs. For example, let say you have this:

    Code (CSharp):
    1. struct Parent: IComponentData {
    2.     [EntityParentField] // this marks that field is used as entity's parent
    3.     public Entity Parent;
    4. }
    and you have Entity 2 with this component pointing to Entity 1. This could be shown in a graph like this:

    Code (csharp):
    1. +--------+
    2. |Entity 1|
    3. +--------+
    4.     |
    5.     |   +--------+
    6.     |   |Entity 2|
    7.     |   +--------+
    8.     +-->|Parent  |
    9.         +--------+
     
  7. JesOb

    JesOb

    Joined:
    Sep 3, 2012
    Posts:
    1,109
    Few notes:
    • Make full featured Inspector for entities:
      • With ComponentDrawers, FieldDrawers, PropertyDrawers (to show computed data in inspector) and ArchetypeDrawers(QueryDrawers). Last one will be very useful for drawing domain model e.g. mob entity in game with entire it state that is actually spread across many components and may be even entities. It is ok to see data from one component in few ArchetypeDrawers simultaneously. May be even draw all matching Archetype drawers on top of inspector following ComponentDrawes. And add switch to show or hide ArchetypeDrawers in inspector.
      • Entities can be composed from pretty big amount of components related to different systems so we need to be able to filter them in inspector. And allow to create named filters for fast workflow.
      • Mode to show all components of Entity (and all data inside) in inspector, not only public one and be able to edit it (it is just data overall) Additionally mode that show alignment and empty spaces inside components, size of each component in bytes and size of entity. Is essential for performance optimizations in DOTS. May be in this view show cache line size on platforms i.e. 32 or 64 bytes and now component fit it.
      • Entity can have pretty big amount of components and monitor space is finite so please make Entity inspector simple dont use empty lines where it is not necessary. e.g.:
        • sort tag compoennts to be first and in one wrapped line.
        • make component with single field 'Value' to not show field name and just show ComponentName following actual value e.g. upload_2020-4-23_13-19-21.png
      • Make able to switch to raw view (may be even description view and layout view) per component not only entire inspector. May be just if we have few drawers for component in project allow us to select default one and switch between them in inspector (may be by clicking component icon) . Then we can by our self create few drawers for Description, Layout and another useful info about component. Switch View for all components in inspector. e.g. we can switch inspector to DescriptionView and instead of component data will see description of every component following it name. It will greatly help understand what each component is for and why it on my entity.
      • Make IBufferElementData you know reorderable :)
    • Entities window
      • Get rid of duplication in hierarchy :)
        upload_2020-4-23_13-34-3.png
      • 4 pillars to work with big data (probably millions of entities)
        • Pillars:
          • Sorting
          • Filtering
          • Grouping (SubGrouing)
          • CustomTree
        • Make able us to sort entities by different criteria, filter them in any way we want, and group by our rules that we can define e.g. @Smilediver proposal.
        • Allow to save any sort and filter criteria for fast workflow, and may be even define in code so it will appear for every team member.
        • Custom Tree is tool for us to make custom tree like view of entities as plugin to entities window not as separate windows where we need to do all manually. May be custom tree will be compound from already sorted, filtered and pre grouped set of entities.
    • Systems Windows looks nice can not add anything for now :)
    • Authoring link
      • Actually dont want to have separate hierarchy and Entities window but dont know how to avoid this for now so it is just note. May be someone have ideas.
      • Want to have conversion on the fly. For now we have to open SubScene for editing and entities get removed replacing with GameObjects. Looks bad. Want game objects to be just like editor data representation but actual view to be rendered from entities. So I open subscene for edit. Start change some props in GO, system convert this on the fly into existing entity and I can see changed data in entity and actual changes in scene view. So GameObject will be just another way to change entity data, create and destroy them (primary way for Authoring). So MeshRenderer on Converted GO will not actually added to rendering, it converted to entity and entity added to rendering. Same with Physics and others.
      • One more time :) Please make GO Authoring to be tool only for changing actual entity data.
     
    Last edited: Apr 23, 2020
    HeyZoos, Flipps, SamOld and 6 others like this.
  8. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    8,194
    Will we also need some kind of system entities flow graph or in/out entity data?

    And links between entity types and systems that use them?
     
  9. JesOb

    JesOb

    Joined:
    Sep 3, 2012
    Posts:
    1,109
    We often have workflow to run game (enter play mode), tune GameObjects and then copy good tuned data back to editing stage. There is request for it to work for a decade. Developers just use some assets from store, to do this, or some hand crafted tools.

    With DOTS conversion workflow this become even harder to get changes back to Authoring representation.

    So we need some way to convert data back to authoring.
     
    Last edited: Apr 23, 2020
    BenzzzX, Orimay, TLRMatthew and 2 others like this.
  10. jdtec

    jdtec

    Joined:
    Oct 25, 2017
    Posts:
    302
    Good to see debugging getting some more attention!

    1) Click to select entities in scene and display data in the inspector, so what @JakHussain said. Also, strongly agree with @Jes28 point about being efficient with space for showing the component data and not wasting lines. It struck me already that it's not displayed that efficiently when using the Entity debugger.

    2) Editable data for entities in the inspector, like others have requested.

    3) Debug draw and log functions working from bursted jobs without having to rewrite or wrap them. (within minutes of me posting this I see @xoofx has posted saying the new burst has "partial support for Debug.Log" which is great.
     
    Last edited: Apr 23, 2020
    toomasio, SamOld, Orimay and 5 others like this.
  11. JakHussain

    JakHussain

    Joined:
    Oct 20, 2016
    Posts:
    318
    @jdtec touches on a great point with drawing debug lines in ecs! Another huge workflow necessity!
     
    toomasio, SamOld, Orimay and 2 others like this.
  12. Roycon

    Roycon

    Joined:
    Jul 10, 2012
    Posts:
    50
    Could we get support for custom worlds which are not updated via the built in ways?
    I'm updating my worlds manually with World.Update and the Systems window is blank for those worlds
     
    Orimay and NotaNaN like this.
  13. calabi

    calabi

    Joined:
    Oct 29, 2009
    Posts:
    232
    I'll just add my tiny feedback it would be great to be able to see the Systems and enable or disable them in the editor before the program is run.
     
    Orimay, NotaNaN and JesOb like this.
  14. KjartanK_Unity

    KjartanK_Unity

    Unity Technologies

    Joined:
    Apr 2, 2018
    Posts:
    4
    Thanks for this first set of responses! I want to summarize what I've read so far and ask a couple of clarifying questions:

    First, on the Systems window:
    • You'd like to see the Systems being scheduled without having to enter play mode
    • You want it to support a custom World manually ticked via `World.Update`
    • In order to optimize, you want to be alerted to Systems triggering sync points as well as the cause of those sync points
    • Also to optimize, you'd like Systems to accurately report excessively long execution times (no negative times) and if one System is waiting for something else to finish you'd prefer that "something else" to be the reporter of that wait time rather than the waiting System
    Next, the Entities window and browsing Entities:
    • In general it seems the ability presented here to browse the Entities in your World will be helpful
    • This needs to have robust filtering, sorting, and customizable grouping functionality (fortunately this is on our requirements list)
    • @Arowx you asked about "links between entity types and systems that use them," do you mean the ability to select an Entity and see Systems with matching queries (or select a System and see Entities matching its queries), or do you mean something else?
    • @Smilediver mentioned you'd like to visualize Entity dependencies as a graph; why would you prefer a graph over a hierarchical tree-view for the use case you presented? Do you have additional use cases where a hierarchical tree-view may not be sufficient?
    • You need to be able to select an Entity directly in the scene view
    Now, there's inspecting Entities:
    • You want a fully-featured Entity inspector including a few things:
      • Customized component inspectors
      • "Debug" mode (both full inspector and component-by-component)
    • Entities can have a lot of data on them, so you want to be able to navigate that data and you want us to economically use the space (the suggestion of components with only a 'Value' field is an interesting one)
    Then, iterating during play mode:
    • Before summarizing these items I want to set some context of our hypothesis:
      • We believe there are two primary cases for editing data during play mode: iterating on/experimenting with content and debugging data/functionality.
      • For the first, we think those are changes you want to make and potentially keep, and they are changes that you want to work with the authoring format of your data for (for familiarity, especially for people filling designer or artist roles)
      • For the second, we think you need to have fine-tuned control of the runtime data and you want to work with what the runtime Systems see but that you don't need to keep most changes.
      • That said, our goal is to create an experience where the optimal workflow for iterating on your content is through your familiar authoring-format data and the best workflow for diving in and debugging or figuring out how something works is by having access to your runtime data.
      • The biggest concern with editing runtime data is that those changes are even less permanent than editing GameObjects during play mode in classic Unity today because even a change to authoring data during runtime could cause a re-conversion and erase your runtime data changes. Because of that, we want you to be able to edit your runtime data directly, but we want you to be aware of the risks.
      • On to the feedback:
    • You need to be able to edit Entity data directly
    • @Jes28 said "please make GO authoring to be tool for only changing actual Entity data," can you clarify what you meant here?
    Finally, there were some things related specifically to profiling and debugging that I wanted to separate out:
    • You want more specific Systems profiling with things like CPU usage, memory bandwidth usage, and more
    • You want to see a component's data layout and the size it uses so you can optimize it
    • You need to draw debug lines and send debug logs

    Thank you for the great feedback so far! Please keep it coming and let me know if you feel I've missed or misunderstood anything.
     
  15. brunocoimbra

    brunocoimbra

    Joined:
    Sep 2, 2015
    Posts:
    679
    Hierarchical tree-view is good for parent-child relationships, but in DOTS I usually think of my data as I would when structuring a database, so having a visualization similar to a database relationship diagram would be a dream becoming true.
     
  16. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,271
    This is more of a complaint of the current Entity Debugger, but I figured I would post it here.

    So I add the TransformSystemGroup to multiple ComponentSystemGroups through code. It is important that I use the same instance of the group because of how change filtering works. The actual gameplay works fine, but the Entity Debugger complains about it.
     
  17. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    8,194
    Imagine I want to be able to :
    Track data through the systems maybe I've built or purchased a set of complex systems but there is a bug in the data. I can't find a bug in the system that "finds the bug", how do I quickly back track the data to find the system that created the buggy data?

    I'm hitting performance problems due to memory bandwidth limits but it's only happening as occasional glitchy FPS drops. I need to be able to see the systems memory and CPU bandwidth utilisation and also work out where the log jam of data is being generated e.g. I find the glitches are in a generic movement update system that just adds two vectors but then need to backtrack to find out what systems generated the extra load.
     
    quabug, SamOld and KjartanK_Unity like this.
  18. Kolyasisan

    Kolyasisan

    Joined:
    Feb 2, 2015
    Posts:
    397
    One thing that would help tremendously IMO is the ability to select entities in the scene view. Even if it were to be based on raycasting against colliders or meshes (naturally), it would still help nicely, especially alongside the entity window.
     
    flatterino likes this.
  19. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    8,194
    Please note when I refer to Bandwidth it's not just RAM bandwidth or PCI GPU bandwidth, as modern CPUs and GPUs have several layers of memory caches each one has bandwidth limits before the next cache is needed and the performance of operations drop. e.g.


    Notice the data access range and the latency in clock cycles. If you could highlight how the bandwidth of our DOTS processes and entity pools fit into these latency steps it could be very helpful to developers, although it might be complex to do as it's very hardware dependent.

    Also don't forget code takes up memory and cycling systems through cores uses up memory bandwidth. So in theory a DOTS systems with high system counts even with relatively small entity pools could hit performance issues due to code bandwidth. So there might be sweat spots for code size and data size on target hardware that your system should help developers find...
     
    NotaNaN, KjartanK_Unity and JesOb like this.
  20. JesOb

    JesOb

    Joined:
    Sep 3, 2012
    Posts:
    1,109


    • It looks like you already have something like this in bold text. It is mostly about get rid of middle state and go right from authoring components to Entity representation.

      So RigidBody MonoBehaviour will never create body in PhysX it will be converted to entity with Unity.Physics component right away. Same with MeshRenderer and others. So we will never see Mesh Renderer rendered through old system we will author MeshRenderer but it will be converted inplace into entity and rendered with HybridRendering right in Editor.

      This is base idea that GameObject authoring is just for authoring but under the hood everything will be converted inplace and work through entities. So we will see our Authoring data in Hierarchy and converted Runtime data in Entities while editing.

      P.S. It is not about editing but I have actually dont want to have [GenerateAuthoring] attribute. Actually want to be able to add IDataComponent to GameObject may be through MonoBehaviourContainer, so in code I have the only data type and methods that can author it without dealing with 2 similar data type one struct and one MonoBehaviour and convertion to each other in editor scripts.

      Conversion is good thing for complex systems but overcomplicated for simple ones.
     
    Orimay, NotaNaN and KjartanK_Unity like this.
  21. cultureulterior

    cultureulterior

    Joined:
    Mar 15, 2015
    Posts:
    68
    • I'd like the ability to see which entities correspond to which things visible on screen by having a decorator I could add onto my entities.
    • I'd like to run my application, and then get a graph of which entity tag transitions had led to which systems becoming active.
    • I'd like an optionally smaller entity window I could have in my sidebar
     
    Nothke, Orimay and bugfinders like this.
  22. Ziboo

    Ziboo

    Joined:
    Aug 30, 2011
    Posts:
    356
    Everything that was said :) mostly this 2 !!
    • Systems triggering sync points as well as the cause of those sync points
    • Selecting entities in scene view
    Also, improvement for the Profiler Timeline view.
    • Search bar
    • If we select a system in the Entities System windows it select it in the Timeline
     
    Nothke, Orimay, NotaNaN and 2 others like this.
  23. chlagou

    chlagou

    Joined:
    Jan 15, 2014
    Posts:
    13
    • I think it's very useful to be able to go back a little to see what happened, like a record feature where you are able to record every spawned, modified entities every frame and then pause your game and go back.
    • Be able to see system group order before entering play mode and reorder them from the system debugger then generate attributes accordingly, it's a bummer to reorder several systems from the script
     
    Last edited: Apr 26, 2020
    LudiKha, Orimay, JesOb and 2 others like this.
  24. Smilediver

    Smilediver

    Joined:
    May 5, 2011
    Posts:
    72
    The tree view only works for parent-child type relationships, but in ECS, another very common (if not the most common) relationship type is a simple link to another entity. For example, in my game there are two entities: `BuyerUnit` and `SellerUnit` that link to a third common entity `Contract`. Both `BuyerUnit` and `SellerUnit` entities are child entities of two separate `Firm` entities. If these relationships could be annotated, then in an entity graph viewer it would be possible to easily see these relationships and navigate between them.

    Btw, `parent of entity`, `child of entity`, `link to entity` are just a few relationships that comes to my mind, and probably there are many more. Maybe there could also be `list entry` relationship pointing to list aggregating entity, that could be displayed as a collection of other entities (case of item entity being in inventory entity).
     
    Orimay likes this.
  25. SamOld

    SamOld

    Joined:
    Aug 17, 2018
    Posts:
    333
    This looks really cool. I have some thoughts.

    It would be nice to be able to visualise the system dependencies as a graph. Clicking "Show dependencies" on one system and getting a list is all well and good, but sometimes you just want an overview of the whole picture. This could also be used to illustrate other profiling information, like showing sync points and critical paths. Maybe you could even integrate this with the profiler to line system nodes up on a timeline.

    You know how RenderDoc can record a frame of rendering and let you step through it? I want that for one (or more) frames of DOTS. Let us capture a frame, then move back and forward through it, and view the "blame" for an individual component's state. Ideally this should integrate with IDE debugging, so we can jump pack to a particular point in the frame, add a breakpoint, and step the frame at that point. The beauty of stateless DOTS is that this type of replay should be possible. We should be to able to capture on a rolling basis every frame and trigger the pause and inspect from code, so that we can detect arbitrary states we want to look at and debug them when they occur.

    This last one is a big ask and I don't think I'm going to get it, but here goes. I want authoring components that are not based on
    MonoBehaviours
    , or even
    UnityEngine.Component
    . Those are legacy technologies with some very questionable design decisions baked into them, and I want free of their tyranny. Particularly, I want no magic method names and no messaging. DOTS is about the future of Unity and moving towards writing more standard and good C# code. I'm worried that the choice to build the future on the foundations of the past is missing an opportunity to put past mistakes behind us and instead locking us into them for another decade.

    More pragmatically, I would also accept an
    AuthoringComponent
    base class that inherits from
    MonoBehaviour
    , but that the engine explicitly excludes from messaging and magic names. Virtual methods could be provided for any callbacks needed, and the junk that's not needed for authoring components like the messaging methods and coroutines could be marked
    [Obsolete]
    and hidden with
    [EditorBrowsable]
    .

    I also want the ability to use a completely different serialisation system based on explicitly reading and writing data to a store, so that the authoring components are themselves stateless in the style of DOTS without any reflection magic, but I think I've been radical enough for one day.
     
  26. SamOld

    SamOld

    Joined:
    Aug 17, 2018
    Posts:
    333
    Actually, I can still be more radical.

    We have lots of tools in DOTS for serialising and storing data. We have the ECS itself, as well as things like blobs. Let us use those tools to store our authoring data.

    Here's a quick prototype sketch off the top of my head. It probably has some fundamental design problems.

    Code (CSharp):
    1. // Would be attached to a regular GameObject and part of the conversion workflow
    2. abstract class AuthoringComponent
    3. {
    4.     // IComponentData components are used for backing storage.
    5.     // They may be the same types used in gameplay, or special authoring only components if more convenient.
    6.     // It would be common to define them privately within the AuthoringComponent.
    7.     protected void SetStore<T>(T component) where T : struct, IComponentData => throw new NotImplementedException();
    8.  
    9.     protected bool TryReadStore<T>(out T component) where T : struct, IComponentData => throw new NotImplementedException();
    10.  
    11.     protected T ReadStoreOrDefault<T>() where T : struct, IComponentData => throw new NotImplementedException();
    12.  
    13.     protected T ReadStoreOrDefault<T>(Func<T> defaultFactory) where T : struct, IComponentData => throw new NotImplementedException();
    14.  
    15.  
    16.     // This is like a hybrid of a MB component and an editor.
    17.     // The user always manually lays out the inspector.
    18.     // This is more powerful than using [Range] etc on fields.
    19.     // With the right helpers, it's nearly as succinct.
    20.     // Now that authoring components are only editors,
    21.     // it doesn't make sense to have Editor be a separate layer anyway.
    22.     protected abstract VisualElement DrawInspector(VisualElement container);
    23.  
    24.     // Does normalish conversion stuff.
    25.     protected abstract void Convert(ConversionContext context);
    26. }
    27.  
    28. class DemoAuthoringComponent : AuthoringComponent
    29. {
    30.     protected override void DrawInspector(VisualElement container)
    31.     {
    32.         var authoringData = this.ReadStoreOrDefault<AuthoringData>();
    33.  
    34.         // This ref based UIElements API doesn't exist, but hopefully something like it can.
    35.         // I'm not sure about that, I haven't looked into the UIElements implementation much.
    36.         // I know binding is done differently from IMGUI, but there's probably a way to make it work...
    37.         container.Add(new Vector2Field("Initial Velocity", ref authoringData.InitialVelocity));
    38.  
    39.         // Store changes back
    40.         this.SetStore(authoringData);
    41.     }
    42.  
    43.     protected override void Convert(ConversionContext context)
    44.     {
    45.         var authoringData = this.ReadStoreOrDefault<AuthoringData>();
    46.         // Read from store, do conversion, produce gameplay data.
    47.     }
    48.  
    49.  
    50.     private struct AuthoringData : IComponentData
    51.     {
    52.         public float2 InitialVelocity;
    53.     }
    54. }
    Obviously facilities would need to be provided to manage storing references to non DOTS types and things like that. Just imagine it though - a world where we're free of Unity's serialisation engine.

    Perhaps this has gone a little off topic for this thread. I might move it later if I don't think it's a stupid idea after I've slept.
     
  27. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,460
    Have you tried the 2020.1 beta? We don't yet have dependency tracking between jobs in the Profiler, but the flow visualization, shown in this video at 7:22, already shows when a job was scheduled, and if it was synced upon and leading to a wait, you get a flow line to the sync point to. Also with Call Stacks collection option on, you get callstacks for the WaitForJobGroupID sample. Any feedback on that and how to improve that for DOTS would also be appreciated.
     
  28. SamOld

    SamOld

    Joined:
    Aug 17, 2018
    Posts:
    333
    I've seen the video, but I've not tried it.

    I would particularly like a system oriented graph view. I'm picturing system nodes on a canvas with dependency lines. Like a readonly version of ShaderGraph, displaying data flowing through the ECS. It's basically a complete alternative view to the systems window at the start of this thread. It feels like the more natural way to visualise complex arbitrary dependencies. The "Show dependencies" filter proposed in this thread seems like a bad way to reason about the bigger picture, because you're only seeing one step at a time.

    I'm not imagining this as primarily a profiler feature, it's more of a DOTS editor feature. It could have all of the same filtering and selection features already proposed. However, having that profiling data overlayed over it when profiling would be nice. I think it could be a general canvas on which lots of types of metadata like that could be displayed when needed.
     
    Nothke, Orimay, Bivens32 and 2 others like this.
  29. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    987
    It's not mentioned or maybe I have skipped it, but I'd like to be able to select an entity through the scene just how it works for GameObjects. Like if I have hundreds of characters in the scene and I saw a misbehaving one, I'd like to be able to select that and the editor scrolls and selects it in the Entities window. Even if we have to add some kind of bounds component for this to work, that would be fine for us.
     
  30. Orimay

    Orimay

    Joined:
    Nov 16, 2012
    Posts:
    304
    +1 for entity decorators! I want to be able to select an entity or several entities and see their outlines, colliders and custom gizmos!
     
    Nothke likes this.
  31. JesOb

    JesOb

    Joined:
    Sep 3, 2012
    Posts:
    1,109
    I Assume more direct way for storing ICD on GameObjects that suffice for simple workflows and we can use any tools created for runtime ICD right in editor without need to write duplicates for authoring components.


    Code (CSharp):
    1.  
    2. namespace UnityTemplateProjects
    3. {
    4.     public class EcsComponentsContainer : MonoBehaviour, IConvertGameObjectToEntity
    5.     {
    6.         [SerializeReference] List<IComponentData> _components = new List<IComponentData>( 8 );
    7.  
    8.         [ContextMenu("Test Code")]
    9.         private void TestCode( )
    10.         {
    11.             gameObject.SetEcsComponent( new MobTag        { } );
    12.             gameObject.SetEcsComponent( new MobMoveData    { Acceleration    = new float3( 5 ), Velocity = new float3(9)} );
    13.             gameObject.SetEcsComponent( new Position    { Value            = new float3( 2 ) } );
    14.  
    15.             Assert.IsTrue( gameObject.GetEcsComponent<MobTag>( ) != null, "can not retrieve tag component" );
    16.             Assert.IsTrue( ( gameObject.GetEcsComponent<MobMoveData>( ).Value.Velocity == new float3(9) ).x );
    17.         }
    18.  
    19.         public T? GetEcsComponent<T>( )            where T : struct, IComponentData
    20.         {
    21.             foreach ( var component in _components )
    22.             {
    23.                 if( component is T data )
    24.                     return data;
    25.             }
    26.  
    27.             return null;
    28.         }
    29.         public void SetEcsComponent<T>( T value )    where T : struct, IComponentData
    30.         {
    31.             for ( var i = 0; i < _components.Count; i++ )
    32.             {
    33.                 var component = _components[i];
    34.  
    35.                 if( component is T )
    36.                 {
    37.                     _components[i] = value;
    38.                     return;
    39.                 }
    40.             }
    41.          
    42.             _components.Add( value );
    43.         }
    44.      
    45.         public void Convert( Entity entity, EntityManager dstManager, GameObjectConversionSystem conversionSystem )
    46.         {
    47.             //just copy components to entity
    48.         }
    49.     }
    50.  
    51.     public static class EcsComponentsAuthoring
    52.     {
    53.         public static T?    GetEcsComponent<T>( this GameObject go )                where T : struct, IComponentData
    54.         {
    55.             var cont = go.GetComponent<EcsComponentsContainer>( );
    56.  
    57.             if( cont == null )
    58.                 return null;
    59.  
    60.             return cont.GetEcsComponent<T>( );
    61.         }
    62.         public static void    SetEcsComponent<T>( this GameObject go, T value )        where T : struct, IComponentData
    63.         {
    64.             var cont = go.GetComponent<EcsComponentsContainer>( );
    65.  
    66.             if( cont == null )
    67.                 cont = go.AddComponent<EcsComponentsContainer>( );
    68.  
    69.             cont.SetEcsComponent<T>( value );
    70.         }
    71.     }
    72. }
    Even in serialized source it looks better and all components inside one mono behavior and with readable names. I dont test it but think that this representation will behave better in merge conflicts:
    upload_2020-4-27_17-17-20.png

    I thinks Unity serialization will be always with us just because engine need default way to serialize Authoring to/from storage scene type in Version Control suitable way :)
     
    SamOld likes this.
  32. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    6,364
    D stands for Data
    i need to see the data being transformed
    select an entity and see all its data being transformed
    bonus point with a timeline we can scrub
     
    quabug, Nothke and SamOld like this.
  33. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    8,194
    Will this include direct editor support for creating/editing entities and systems?

    Aren't a lot of DOTS systems so small and simple they could be edited (without boilerplate code) as a few lines of code within the Editor?
     
    Last edited: Apr 28, 2020
  34. Sylmerria

    Sylmerria

    Joined:
    Jul 2, 2012
    Posts:
    369
    Inspect blobs datas is a real need too.
    I create my blob at runtime and I link it via a component so see them would be great
     
    Last edited: Apr 29, 2020
    quabug, Orimay, calabi and 4 others like this.
  35. Per-Morten

    Per-Morten

    Joined:
    Aug 23, 2019
    Posts:
    119
    Graph View
    I really agree on the graph view. I was envisioning something like the image below.
    I think it would give clear indications of what systems interact with what components, in what order they are run, and where potential synch points are, giving a good way to get a proper overview. Next to the system names, we could put the time it took the system to run (similar to today).
    System groups could be collapsable (FirstSystemGroup as opposed to SecondSystemGroup), only showing the components that passed through them and their "combined" read-write actions.
    ("SystemB" and "SpawnerOfDSystem" are both scheduling jobs that can run in parallel.)
    ECSVisualization (1).png

    Maybe we could have a list of all archetypes, and when you clicked on them, you were given a graph like above, but that only contained the systems that interacted with the components in that archetype. edit: or potentially allow us to create entity queries and get a graph with all systems that had similar entity queries

    Somewhat going off on a probably unrealistic tangent. It would be really cool if we could have snapshots of the values that the entities had at various stages of a frame so that we could track data changes throughout the various systems. Like, selecting an entity, then clicking on the arrow between SystemA.ComponentA and SystemB.ComponentB and see what the value of that component was for that entity at that particular time.

    Data Visualization Tools
    It would also be great to have some simple data visualization tools like histograms, pie charts, or plotting graphs, that we could use to more easily look for patterns in the data that we could take advantage of for optimization purposes.
     
    Last edited: Apr 30, 2020
    rauiz, quabug, Nothke and 9 others like this.
  36. SamOld

    SamOld

    Joined:
    Aug 17, 2018
    Posts:
    333
    You've illustrated pretty much exactly what I was thinking! Thanks!

    I wasn't actually imagining lines for each individual component type, but that's a neat idea, although possibly slightly busy. There are also the
    [UpdateAfter]
    etc dependencies to consider.

    In my ultimate dream world, this would be a very customisable canvas. Various features could draw themselves onto it when they wanted, including user tools. Profiling? Now it's scaled by time and has profiling data shown. Want to see individual jobs? They can be toggled. Want to filter so that only the dependencies of the physics system are shown? Easy. You could also have an equivalent to custom editors, so that systems could supply a custom UIElements
    VisualElement
    to display whatever data they want on their node. When thought of as a general view, there could be lots of types of visualisation that use this.
     
    Orimay likes this.
  37. Per-Morten

    Per-Morten

    Joined:
    Aug 23, 2019
    Posts:
    119
    No problem.

    My thought was that the systems could appear in order within the SystemGroup's based on their dependencies and update sequence attributes, but also be ordered above and below each other if they didn't have any dependencies.
    Also, I somewhat agree that the lines for components can be a bit much (at least if they are on all the time, for the dataflow for archetype example I gave I think it would be useful). I had a small prototype of this sort of visualization (without system grouping) and it grew complicated very fast. Although, I didn't get far in my prototype, so it might be that I would have found a better way to group stuff if I worked on it longer.
     
    SamOld likes this.
  38. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    3,356
    Ability to customize the inspector I think needs to be complete.

    I haven't looked but is the UI markup driven? Other references to ui elements made me think so.

    My ideal would be it's driven by markup. Which would almost inherently mean it would have some data/view abstraction layers. So then I could override/replace pretty much whatever I want. Include non ECS data, add inputs, buttons, etc..
     
  39. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    6,364
    and while at it, add a prev/next arrows... there was a proto years ago what happened to it...
     
  40. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    8,194
    What about hybrid systems if you have a new amazing dots viewing/profiling system should it also provide a view and profilining for monobehaviours?

    If the 80:20 rule applies then most games can get away with 80% Mono and only need 20% DOTS?
     
  41. Flipps

    Flipps

    Joined:
    Jul 30, 2019
    Posts:
    51
    Great work!

    Some suggestions / comments:

    Entities Window

    I think here is the biggest potential in improvements (comparing to the system window).

    There should be several ways to inspect / select the entities of interest:

    1. Filter by Component (Entities Window Screenshot 4 OP)
    This is the only useful way to inspect entities in bigger projects right now. It takes some time to get used to but it follows the idea of ECS the most. The ability to query for NOT a component is really nice, too.
    Useful if you need all entities of an archetype.

    2. Filter by Entity Name
    This seems really simple but is not implemented yet (or i did not found it). I think this can be useful if you have entities like enemies etc which share all the same components but differ by name.

    Problems general to Entity names: You can not use SetName for an entity in an ecb. So there is not really a possiblity to set a entity name in a bursted job. In addition there are a lot of unnamed entites created by Unitys own packages like netcode. Problematic for a good hierachical view, too.

    3. Hierarchical View (Entities Window Screenshot 1 in OP)
    Big fan that this is coming! Even though Entities != GameObject the Parent/Child relation is still there and dominates in environments etc. REALLY helps to get a clear overview of the entities and hides all child entities floating around in the same level right know.
    As mentioned beside Parent/Child it whould be really nice to filter by other relations, too:
    LinkedEntityGroup: Entities which are destroyed together
    References Entities in components: Dont know how to show these exactly, but right now you have the option to get to the referenced entity by a button (better than nothing)

    It would be great to toggle on / off the different relations for a selected component!

    4. Select Entity in SceneView
    Already mentioned in other posts --> great improvement

    5. Matching by Query in System (Entities Window Screenshot 3 OP)
    Great!


    6. Selected by converted GameObject (Entities Window Screenshot 2 OP)
    Would be nice to help people to get a better understanding of the conversion workflow. But seems to be only useable with subscenes right now ?

    Other things with Entity Window:

    Prefabs (IMPORTANT)
    Really wondering why this was not mentioned by anyone. A LOT of confusion of the entity window comes from prefab entities showing up in the same window as the "real" entities. The prefabs share the same components + name (except the prefab tag) as the "real" entities. Only the "real" entity gets queried by systems and is rendered; the prefab entities do nothing except waiting to be copied for instantiation.
    It would really help if prefabs and "real" entities would be seperated in the Window by two views (but still under the same world).
    Especially if you doing things with netcode and there are several prefabs created for a ghost.

    Editable Components in playmode
    Mentioned in other posts. Adding BufferElementDatas to DynamicBuffers would be great, too.

    Mark different Component Types
    Mark different data types and managed types (struct, class) with an icon or color
    - ComponentData (struct, class)
    - ComponentObject (MonoBehaviour, class)
    - SystemStateComponentData (struct, class)
    - BufferElementData

    Mark destroyed Entities
    Please mark all entities which were destroyed but still exists due to SystemStateComponentDatas. Would be nice to see entities which did not get cleaned up correctly!

    System Window
    Really great!
    Cant add much here:
    - Show Sync points
    - Show all Systems (right now only shows systems which already run) in play and editor mode
    - also agree at graph view (but last prio)



     
    Last edited: May 1, 2020
    Cynicat, DotsCreative, Orimay and 3 others like this.
  42. florianhanke

    florianhanke

    Joined:
    Jun 8, 2018
    Posts:
    426
    So much +1 for the Prefabs, but especially also for Ghosts in NetCode. Or at least allow for a default query of -Prefab -Ghost X.
     
  43. nicolasgramlich

    nicolasgramlich

    Joined:
    Sep 21, 2017
    Posts:
    231
    Not sure if anyone mentioned it before, but something like this is already in concept and/or development:



    upload_2020-5-17_21-15-13.png

    It's from November 2018, so it should be done any day now :D

    Jokes aside, this would be amazing to find bottlenecks and really understand dependencies and be able to optimize them by rethinking which R/W access is necessary and deciding to split things into multiple jobs.
     
    Nothke, thelebaron, Cynicat and 2 others like this.
  44. SamOld

    SamOld

    Joined:
    Aug 17, 2018
    Posts:
    333
    I hadn't seen this before. That looks like exactly the graph view that I was requesting! 2018 is a long time ago, I hope this is still being planned.
     
    nicolasgramlich likes this.
  45. nicolasgramlich

    nicolasgramlich

    Joined:
    Sep 21, 2017
    Posts:
    231
  46. JoNax97

    JoNax97

    Joined:
    Feb 4, 2016
    Posts:
    611
    That graph could be the single most important piece of DOTS. I currently have to develop with a whiteboard next to me just to keep the big picture of my systems. Of course that gets outdated quickly.

    If you guys can pull it off and provide a graph that shows both data dependencies and sync points, DOTS approachability would go up tenfold.
     
    Nothke, Cynicat, brunocoimbra and 3 others like this.
  47. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    8,194
    Could I ask that contextual debug warnings could be raised during DOTS development as a learning/training aid.

    DOTS is complex and has odd built in behavioural patterns that are not apparent to noob developers e.g. the LocalToWorld system which may or may not give you access to Scale/NonUniformScale and also multiple system updating this data can cancel each other out.

    A smarter Warnings and Debug system along with, improved documentation with examples, clear warnings and advice could really help.
     
    Cynicat and cultureulterior like this.
  48. KillHour

    KillHour

    Joined:
    Oct 25, 2015
    Posts:
    49
    A few things:

    1. It would be nice to be able to filter the component inspector to only show the values of the components you want to see. Entities with dozens of components get busy really fast.

    2. Being able to see a cross section of component data - so all the values of a single component type across all entities versus all the values of the components on a single entity

    3. Being able to select a component and get a list of all the systems that updated that component in the last frame or in a given period would be an absolute godsend for debugging. Bonus points if you can see the history of the values as it passes from system to system.
     
    JakHussain, Cynicat, JesOb and 4 others like this.
  49. KillHour

    KillHour

    Joined:
    Oct 25, 2015
    Posts:
    49
    Just thought of a couple more:

    - This is probably on the list, but the current inspector for nativeStrings is REALLY bad. I have a component with a native array of native strings and it's so terrible to do anything with. Please just display the .ToString() result of the data, since we can't edit them during runtime anyways. Maybe have a foldout for the raw byte data if it's important to people.

    - We need to be able to filter systems by not just the components that are part of the initial query, but by what components they use GetComponent<T>() to access. Same thing with systems that access buffer element data or entity command buffers.
     
  50. RogueStargun

    RogueStargun

    Joined:
    Aug 5, 2018
    Posts:
    296
    I think there is too much attachment here for a tree hierarchy. There is no hierarchy anymore. Systems should be depicted with a database graph representation. I think the attachment to trees is due to a desire to cram in data into a sidepane as with the current system. This will probably confuse people. Instead give users the ability to group and collapse graph node clusters for organization.