Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

API usability

Discussion in 'Data Oriented Technology Stack' started by Adam-Mechtley, Sep 4, 2018.

  1. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,494
    You can't (in safe way), SCD is not about data storing in general idea.

    Don't forget - it's in preview, and player loop not in final state
     
    FROS7 likes this.
  2. Shinyclef

    Shinyclef

    Joined:
    Nov 20, 2013
    Posts:
    364
    Hmm that's a shame. I figured you could because of things like shared render components, but I suppose those are not accessed after assign to GPU or something.

    Yeah, just figured it's therefore important to provide the feedback early. Also the more the play loop changes, the more important it is to be able to easily reorder your own systems.
     
    GliderGuy likes this.
  3. vitautart

    vitautart

    Joined:
    May 3, 2018
    Posts:
    29
    Not sure if this is right thread. It would be cool to have sticky thread like "General ECS feadback from users". But anyway.

    I have proposal to add little function to EntityQuery API that calculates chunk count, like it has already for entity count - EntityQuery.CalculateLength().
    Usecase for example, when you need to allocate some temporary native container with capacity of chunk count, for this I need first to allocate nativearray for chunks, than take length, and than dispose(), and sometimes this chunk array is not used anywhere except only for getting length from it.

    Instead of this:
    Code (CSharp):
    1. var query = EntityManager.CreateEntityQuery(queryDesc);
    2. var chunks = query.CreateArchetypeChunkArray(Allocator.TempJob);
    3. var array = new NativeArray<int>(chunks.Length, Allocator.TempJob);
    4. chunks.Dispose();
    Will be cool to have:
    Code (CSharp):
    1. var query = EntityManager.CreateEntityQuery(queryDesc);
    2. var array = new NativeArray<int>(query.CalculateChunkCount(), Allocator.TempJob);
     
  4. BenzzzX

    BenzzzX

    Joined:
    Jun 4, 2018
    Posts:
    6
    I've been trying to implement a ecs framework with cpp recently.
    And I found that inline job via lambda is so well suited to express the work flow.
    Some concept code.
    Code (CSharp):
    1. void ScheduleFrame(ecs::Flow& flow)
    2. {
    3.     static ecs::Entity e[1];
    4.     static ecs::Entity es[10'000'000];
    5.  
    6.     flow.Sync([](ecs::Context & context)
    7.         {
    8.             context.CreateEntities<Test, TestInternal>(e, TestMeta{ 1,2 });
    9.             context.WriteComponent<Test>(*e) = { 1 };
    10.             context.WriteComponent<TestInternal>(*e) = { 13,2 };
    11.             context.Instantiate(*e, es);
    12.         });
    13.  
    14.     flow.ForEntitiesPar(/*filter,*/[](Test& test)
    15.         {
    16.             test.a += 1;
    17.         });
    18.  
    19.     flow.Task([](ecs::Accessor<const TestInternal> test)
    20.         {
    21.             /*async*/
    22.         });
    23.  
    24.     flow.Sync([](ecs::Context & context)
    25.         {
    26.             context.DestroyEntities(e);
    27.             context.DestroyEntities(es);
    28.             context.RemoveComponent<TestInternal>(e);
    29.         });
    30. }
     
    Jes28 and illinar like this.
  5. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    7,119
    What about discoverability, e.g. when working with the API sometimes you will literally be searching for the feature you want with the tooltips provided.

    Just recently I spent a good half an hour trying to find a way to Scale an entity, as the default .Scale(float s) function was all I could find until I dug into the documentation and found .NonUniformScale(float3 s)

    You could make the API more 'discoverable' by naming the functions like this by what they do first then how they do it e.g. I would have found ScaleNonUniform(float 3) at the same time I found Scale().
     
    SirIntruder and Shinyclef like this.
  6. illinar

    illinar

    Joined:
    Apr 6, 2011
    Posts:
    586
    @Arowx most of the major naming conventions suggest to use more "english" sounding names, more readable names. And Unity ECS is made following NET standards as far as I know.

    The current API should be pretty discoverable with IDE like VSCode at least.

    upload_2019-5-6_20-23-52.png

    In any case things should be okay when the stable api and Documentation are in place. That transform documentation looks really crazy though. Could use more clearly highlighted definitions and key points and maybe fewer examples.
     
    Last edited: May 6, 2019
    AndrewKaninchen and TeagansDad like this.
  7. illinar

    illinar

    Joined:
    Apr 6, 2011
    Posts:
    586
    GetSingletonBuffer<T>()
    would be nice to have.
     
    siggigg likes this.
  8. LazyGameDevZA

    LazyGameDevZA

    Joined:
    Nov 10, 2016
    Posts:
    80
    Having a more flexible singleton api would be great. The team from Blizzard that built Overwatch said they use singleton components as a means to abstract systems away from one another. One of the key things being that only the system that owns the singleton is allowed to write to it. In my view exposing the physics world for Unity Physics would be a great example of this as it's a data structure that needs to be queried in some way by other systems.
     
    illinar likes this.
  9. vitautart

    vitautart

    Joined:
    May 3, 2018
    Posts:
    29
    I have proposal and question about NativeQueue.

    Proposal. It would be nice to have method
    NativeQueue.ToNativeArray(Allocator)


    Question. Is concurrent version of this queue realy deterministic? Will order of elements be the same on different processors, or at least on x86?
     
  10. illinar

    illinar

    Joined:
    Apr 6, 2011
    Posts:
    586
    Rename ISharedComponentData to IChunkComponentData?
     
  11. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,494
    It'll confusing, cos we have chunk components
     
  12. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,683
    I think shared component makes more sense.
     
    Shinyclef likes this.
  13. GliderGuy

    GliderGuy

    Joined:
    Dec 14, 2018
    Posts:
    29
    Now it's technically not API, but It'd be nice if when you're hovering your mouse over a variable that is a struct that derives from an ECS interface like IComponentData or IBufferElementData, instead of displaying the type of the variable it would display the interface the struct derived from. Because we all know an IComponentData is a struct, but we don't know if a struct is an IComponentData!

    Of course, I'm not actually sure if it's even possible to do this because it might be code-editor based (and thus you have no control over it), plus it might not be a good idea from a consistency standpoint... But it's something i noticed and it bugged me. :oops:
     
    Last edited: May 17, 2019
  14. Ryetoast

    Ryetoast

    Joined:
    Mar 8, 2015
    Posts:
    40
    This isn't direct feedback on Unity's ECS, but I was inspired to try making my own ECS library in C++ to see what it would turn out like. It's not as fast as Unity's ECS for various reasons (lacking many batch optimizations, and this is basically my first dive into multi-threading). I am, however, quite proud of its API and user experience, so I thought I'd post it here for the Unity DOTS team to look at:

    https://github.com/RyeToastyO/Ecs
     
    GliderGuy likes this.
  15. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,265
    I think the "data" is more misleading than "shared", it make you think you could store and use that data at will. It need to feel more auxillary-ish.

    Maybe something like IHashedComponent / ISharedHashComponent / IHashedTagComponent ? When removing "data" it is more explicit that your data were transformed and is not technically there with the entity anymore.

    Then the old GetSharedComponentData that could get the real thing with SCD index/with associated entity could be instead named GetDataOfHashedTag(...). By reserving the "data" wording to use here it may be more explicit that you are converting the index to real data. (and that you couldn't do this in the job)
     
    illinar likes this.
  16. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,683
    In my opinion adding Hash word to the Component name, will get confused with other "hashes".
    But I know where you come form and why this proposition. However, I am not convinced, "Hash" is more descriptive in this case.

    Yet data is data, weather is shared or not. In fact all is data, as DOD indicates :)
     
  17. sngdan

    sngdan

    Joined:
    Feb 7, 2014
    Posts:
    928
    To me it is an ArchetypeSplitterData (without restriction what data, ie can be managed, etc)

    And it’s split by an index to the actual data / object (whereas only the index is accessible in a job)
     
    illinar likes this.
  18. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    483
    technically, it's not strictly a component (it's associated with a chunk, not on an entity). it is data though.
    what about
    IChunkPartitionData
    /
    IArchetypePartitionData
    ?
     
    illinar likes this.
  19. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,683
    Isn't "partition" associated with "division" into parts?
    Which may lead to potential confusion. Maybe?
     
  20. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    483
    indeed it divides/partitions the entities into different chunks
     
  21. recursive

    recursive

    Joined:
    Jul 12, 2012
    Posts:
    598
    I like ISharedGroupingData myself. Says exactly what it does on the box without getting too technical.
     
    jdtec, Sarkahn and Antypodish like this.
  22. Arowx

    Arowx

    Joined:
    Nov 12, 2009
    Posts:
    7,119
    Would converting a game from No API to ForEach to Fluent and then to Chunks be a good way to test the API?

    The aim would be to see how much the code changes between versions and to see how much boilerplate is requried for each version of the API.

    Also it could highlight elements that do not translate well between APIs.
     
    GliderGuy likes this.
  23. Ryetoast

    Ryetoast

    Joined:
    Mar 8, 2015
    Posts:
    40
    So I went back through the Unity API to compare big differences:
    1. Singleton Components
      • The Unity implementation is basically just a helper function to assert there's one of a component and to get it. This places the burden of guarantees onto the user.
      • My implementation requires a singleton component to inherit a separate interface. It guarantees that GetSingleton and SetSingleton always succeeds, and that exactly one exists per World. It also allows jobs to access them in basically the same way that normal components are accessed, affecting the read/write dependencies, but not the entity query being iterated.
      • I also believe singleton components are significantly more useful if they can contain non-POD data. Storing things like a native multi map built in one system for several others to consume is a relatively common occurrence in my experience. The current unity method of getting the other system and manually applying input deps again places the burden of guarantees on the user. This is easier to allow if they are treated as actually separate concepts instead of just shortcut accessed normal components.
    2. Job Data Access
      • Component access in unity is very inconsistently accessed based on the type of access, type of job, etc. I solved this by basically making the entity query and the data access the same thing, managed by the job itself. Shared, Singleton, require only, and exclude components are accessed using the same, or extremely similar APIs.
    3. Job Ordering
      • Tagging for update before or after should pass the deps handle returned by the depended upon job.
     
    illinar likes this.
  24. TJHeuvel-net

    TJHeuvel-net

    Joined:
    Jul 31, 2012
    Posts:
    413


    Watching this talk right now. I could imagine complex objects having many different components, yet SetComponentData only accepts one at a time. Would it make sense to have an EntityManager.SetComponentData overload that accepts a `params` as componentdata? That way we could write:

    Code (CSharp):
    1. manager.SetComponentData(Entity entity,
    2.     new Translation { Value = transform.position },
    3.     new Rotation { Value = Quaternion.Euler(rotation) }
    4. );
     
    pvloon, GliderGuy and starikcetin like this.
  25. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    4,283
    You can do that as an extension method pretty easily. Note that params allocates an array, so it's probably not ideal.
     
    Cynicat and Jes28 like this.
  26. starikcetin

    starikcetin

    Joined:
    Dec 7, 2017
    Posts:
    245
    In the meantime, Eric Lippert is not convinced that generic templates are useful enough to be implemented.
     
    GliderGuy likes this.
  27. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,592
    I'm erring on the side of templates not being as useful as well, because with a little boilerplate management, the program is the same work. The programs are absolutely tiny so I would like an actual use case to see the benefit of this argument.

    Viewing latest Unite (videos), I saw very some great signs which indicate I would probably be best off writing code to change data as opposed to writing code to change code.
     
  28. starikcetin

    starikcetin

    Joined:
    Dec 7, 2017
    Posts:
    245
    The only viable alternative of generic templates is code generation.
    I hate code generation.
     
  29. scobi

    scobi

    Unity Technologies

    Joined:
    May 14, 2014
    Posts:
    32
    > Note that params allocates an array, so it's probably not ideal.

    The typical workaround for this is to provide overloads with different param counts. Typically there's not a huge amount of components being messed with at a time on an entity, so probably don't need too many overloads. You can use T4 for this if you want, like we do in Entities.

    > I hate code generation.

    We unfortunately have to do a lot of things that we don't like in order to balance perf, maintenance, safety, etc. Author-time codegen (.tt and .gen.cs files) and compile-time codegen (demoed at last week's keynote) are tools that can help us with this balance.
     
  30. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,592
    I only want to write code that transforms data. Anything else should be code gen or hidden from me. Why would anyone ever care about the things that do not affect their program execution?

    So count me in for as much code gen as possible so my hours are spent transforming data not wasting it preparing to transform data.

    So it's this thinking that allows me to accept codegen :)
     
    Cynicat, GliderGuy and optimise like this.
  31. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,683
    @hippocoder just to add, in the end we don't much care, how on atomic level machine code writes to memory (sure we can look if is optimized). Is already code gen-ed for us, if form of compiler :) At least for most of us?

    Times when I loved write assembly code by hand are long gone ..
     
    Cynicat, GliderGuy and hippocoder like this.
  32. starikcetin

    starikcetin

    Joined:
    Dec 7, 2017
    Posts:
    245
    Codegen is magic. It is not configurable, something gets broken in the generated code and you have to wait for people who developed it to fix the generator. I want to be in control of the code I write.

    On a side note, I am not against compile-time code generation, that is fine. What I am against is the use of development-time code generation.
     
    wobes likes this.
  33. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,592
    But this "magic" is not optional. If it breaks here, then it would've broken if you did it. The magic here is mandatory magic. I don't see why we should concern ourselves with mandatory and repetitive code. It's the same as accessing an API and find it's wrapped stuff that has to be wrapped no matter what.

    Now if we talk about the bastard that UNET became then I agree, that's bad codegen. That's codegen you should have control over.

    But this is not the same thing & I'm sure Unity's learned their lessons where to draw the line on it (essentially if its mandatory then it makes sense).
     
    Cynicat, GliderGuy and Antypodish like this.
  34. bac9-flcl

    bac9-flcl

    Joined:
    Dec 5, 2012
    Posts:
    779
    I was skeptical about codegen until we started using Entitas ECS framework. It uses attribute-driven code generation that supercharged our workflow, and over the past year we haven't hit a single problem with that generated code. It exists in the background, it doesn't interfere with anything we do and it just gives us a rich, useful surface to write systems on.

    What's more, it's important to remember that most of Unity is not a black binary box. Almost everything, especially all these new packages, is either out in the open C# or in IL that can be shown as C#. Whenever I'm unsure about how something is implemented, I hit F12 in Rider and get a clear view of a part that interests me, even if it's inside some library. Furthermore, knowing where certain functionality is and how it looks, I can easily patch it without source access using tools like Harmony. So I don't foresee being in a situation where I'll be a hostage of some bug in Unity code, with nothing to do but wait for a fix - there are plenty of nice workarounds that can keep a Unity project chugging until patches are out.
     
    MNNoxMortem likes this.
  35. Lurking-Ninja

    Lurking-Ninja

    Joined:
    Jan 20, 2015
    Posts:
    4,489
    Codegen is not magic. Especially if you have clear rules what you're generating and you have tests to ensure hat the end product code is what you expect.
     
  36. starikcetin

    starikcetin

    Joined:
    Dec 7, 2017
    Posts:
    245
    Let's hope you guys are right.
     
  37. esc_marcin

    esc_marcin

    Joined:
    Sep 30, 2016
    Posts:
    18
    What do
    ComponentType.ReadOnly<T>()
    /
    ComponentType.ReadWrite<T>()
    in a GetEntityQuery actually do? I tried to verify my assumptions in the docs and it's not clear or at least easy to find.
     
  38. wobes

    wobes

    Joined:
    Mar 9, 2013
    Posts:
    677
    Meaning that a component in a query is only for the reading or for the reading and writing? It is self descriptive..
     
    Last edited: Oct 15, 2019 at 1:08 AM
    GliderGuy likes this.
  39. MikeMarcin

    MikeMarcin

    Joined:
    May 15, 2012
    Posts:
    13
    How to use it is self-evident, what it actually does is not at all obvious. What additional features / performance / saftey etc do you get from using ReadOnly instead of just ReadWrite everywhere?
     
  40. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,707
    It's for jobs so the scheduler knows which jobs can be executed in parallel for better performance.

    And for what jobs need to be completed before a system is run.

    -edit- an example

    SystemA : JobComponentSystem, with a Job
    SystemB : ComponentSystem, with a query
    C : IComponentData
    SystemA executes before SystemB

    Case 1
    If SystemA has a Job with a ReadOnly<C>
    If SystemB has a query with ReadWrite<C>
    Main thread needs to be stalled and Job completed before SystemB can run to ensure the data in job is valid.

    Case 2
    If SystemA has a job with a ReadWrite<C>
    If SystemB has a query with ReadOnly<C>
    Again main thread needs to be stalled and Job completed before SystemB can run but this time to ensure data in SystemB is up to date.

    Case 3
    If SystemA has a job with a ReadOnly<C>
    If SystemB has a query with ReadOnly<C>
    This time the Job does not need to be completed and therefore main thread does not need to be stalled as SystemB can run in parallel to the job in SystemA without concern.
     
    Last edited: Oct 15, 2019 at 2:34 AM
    GliderGuy likes this.
  41. Micz84

    Micz84

    Joined:
    Jul 21, 2012
    Posts:
    240
    I have some small feedback about create menu.
    When we are using Unity physics then things related to it are in:

    create>DOTS>Physics>[...]

    And ECS are in the separate menu down below. I think it would be more consistent if all DOTS related things were it DOTS menu where physics is.