Official API usability

Hi all!

Various suggestions for enhancing API usability have come up in a few different public channels and personal conversations, so we thought it would be a good idea to talk a little more about our intent in a centralized place.

It is no secret that jumping into all of the new high-performance APIs can be challenging! There is a lot of new information to learn, and users are still responsible for managing a lot of complexity and boilerplate code themselves, even in simple use cases. While many factors contribute to how easy it is to learn and use things like ECS, API usability is a major one.

To help focus our efforts on this front, we've tried to come up with a more structured approach to evaluating our different usability affordances, so we can identify which ones merit continuation/development, which areas of the API are lacking, and so on. I won't go into full detail here, but it includes things like

• How much information developers need to keep in mind for a given task
• How well it conforms to idiomatic C#
• How quickly developers get feedback on the correctness of their code
• How it can mitigate integration discontinuity as developers need more manual control

Applying our usability criteria internally has already helped us identify some areas to clean up, as well as proposals that weren't helping us move toward our goal of making it easier to write correct, high-performance code than it is to write non-performant code. We'll use this thread to try to keep you up-to-date about our plans.

Ref Returns

It has been suggested in some places that ref returns could be used to help minimize typing.

Code (CSharp):

1. // today you need to do something like this
2. var position = Positions[index];
3. position.Value.y = someValue;
4. Positions[index] = position;
5.  
6. // ref returns would enable something like this
7. Positions[index].Value.y = someValue;

We have currently decided against pursuing this affordance for a few of reasons, such as:

• It would not be possible to do ref returns with elements in an SOA array, which introduces more inconsistencies in language feature support within a single codebase, and would increase the refactoring burden when migrating to NativeArraySOA
• It provides little advantage over the status quo (e.g., it is fairly idiomatic C#, and developers currently have to do copy/modify/set of structs in MonoBehaviour C# code; the change would only save 2 lines of typing per struct type)
• We would have to statically analyze for safe usage, which is doable, but there are other more impactful places for us to invest in enhancements right now

Instead, our goal is to make more code look like IJobProcessComponentData, which bypasses the annoyance introduced by indexing.

 

In my recent thread Joachim has mentioned you will provide an IJobProcessComponentData that exposes a four component version. Personally I happen to need more (*) on certain occasions and thus have to fallback to the IJobParallelFor and pass multiple ComponentDataArrays of components to the respective jobs which I dislike a lot.

In general the way IJobProcessComponentData is designed with the signature of Execute with all components of a single entity being passed is much cleaner IMHO if you don't need the index at all.

Is there any information on your stance regarding this scenario? Most notably any explanation whether you will consider adding more component versions akin to System.Action?

(*) More in the sense of two or three components that hold their own data while having multiple other components as "marker components". An example would be the following set of components on one AI controlled entity where I run a different job per state of entities (a job for all idle grunts, one for all fleeing ones etc.):

• AgentController
• AgentGrunt
• AgentGruntIdleState
• Animation
• Position
• Rotation
• Scale

 

It would be interesting if you could do something similar to a component group

Code (CSharp):

1. IJobProcessComponentData <Group>
2. struct Group{
3.   DifferentComponentDataTypes Data
4. }

 

It would be nice to be able to group ComponentDataFromEntity just like ComponentDataArray.

 

The most iteration time I took is the errors related to `EntityCommandBuffer`. (Mostly duplicated Add, Remove when it does not have one, etc.) The difficulty is that the log does not capture where in the code the command being played back causing the error came from. This means I must not forget to commit more regularly than usual to reduce the possible problem space (by viewing changed files in SourceTree), or else I would have to search for the entire project's ECB usage.

Currently a solution that dramatically cuts down my debugging time is to go into EntityCommandBuffer source code and add some log what is the type of command being play back right now, to which Entity, and adding what. It floods the log with thousand entries and slow down the game to almost unplayable but at least I have some clue where is the target entity. (With help from Entity Debugger) Still, this way I don't have the exact originating point of the command.

One another thing which I heard you are already working on is the static analyzer, so the error is revealed before we have to go into play mode. (Which take considerable time to get the game to the point that will trigger that job and reveal the error) I had hit several work rollbacks because I have some job design in my head, spend hours writing it, and I made mistakes like aliasing, something cannot be parallel, etc. and it invalidates my initial design.

 

We agree IJobProcessComponentData is much cleaner for all the cases it can currently solve. We haven't settled on specifics yet, but our goal is to improve it to solve more cases and/or build out future scaffolds in similar ways.

Our current inclination is actually to explore alternatives to ComponentGroup structs and move away from them, rather than build out new features for them, but these are useful suggestions so consider them noted

@5argon Thanks for all these feedback items! We discussed your first two points during our team meeting today (e.g., whether redundant removals should even be an error, how we can improve diagnostics for ECB). As for your question regarding static analysis, our goal is indeed to improve on this front over time. From the standpoint of the usability criteria we are applying now, we are going to strive to identify and inform users of potential problems prior to run-time, in as many cases as it is possible to do so.

 

That being said, after a quick glance at the chunk iteration archetype, it would be interesting if we could just build a job from that query rather than having to do a ParallelFor. Something similar to the IJobProcess

 

In terms of api usability, I find math.select() counter-intuitive. The C# ternary operator uses the first expression if true otherwise the second. On the other hand, math select returns the second expression (or value) if true.

Given the lack of proper intellisense documentation, I'd prefer if the two would behave the same way.

 

I like to think that math.select is analogous to math.lerp as you basically just lerp with 0(false) or 1(true) so it's more important that those two behave the same way as the operator syntax is very different anyway.

 

Providing more options to the developers is the way to go.

Make ref returns available, and warn people about the cache misses. Whether or not people use them should be up to them, not the framework designers.

 

you can use out instead ref, can you not?
I like implemented ECS restrictions, less error prone, which allow closer to unify scripts across the globe.

 

My two cents: I prefer when an API maps to the problems I'm trying to solve in an obvious way. For my experience with the ECS framework so far, most of the problems I want it to solve with my systems have these dimensions:

• Component dependencies
• N groups of entities
• Data dependence/independence between groups

From this perspective, IJobProcessComponentData<> has a really nice interface for declaring limited component dependencies for 1 group of entities operated on independently of other groups.

I haven't yet seen any nice patterns for processing N>1 groups, nor for handling data dependence between groups.

I think that particularly as performance is such a critical constraint, and given that games are very complicated and frequently need to process interactions between disparate groups of entities, that nailing the API for N>1 dependent groups will be essential for ECS to deliver on its promise of "performance by default".

For me, this means making it simple and clear, and not having dozens of possible ways to solve the same problem (with highly variable performance characteristics).

As @Antypodish mentioned above, a limited API that addresses the above dimensions would make it much easier to reason about how to solve our game problems, and would result in consistent tutorials, examples, and stack overflow answers, which would lower the learning curve and accelerate uptake.

 

I've got quite a lot of thoughts about this that I'm planning on (have started) writing up which I'll try to dump here soon.

Will the Burst compiler always run on ECS code, such that an API design that has performance (boxing) issues in standard C# but that could be optimised by Burst would be acceptable? I suspect the answer is sadly no, but I'd like a clarification on that please.

 

What makes you think no? Can you clarify? Chances are that I misunderstood your question.

 

Additionally, is it acceptable for the API to have a dependency on C# 7.2? I know there's talk of moving of moving to C# 7.

 

Well currently the Burst compiler can (it seems) be enabled and disabled. With the API I have in mind, performance would be horrible without Burst enabled, because there'd be a high rate of boxing of value types. Is that going to cease to be optional at some point? Additionally I'd assume that ECS code may sometimes be distributed in a DLL, and I'm unclear on whether Burst can process a DLL to further optimise.

Basically it wouldn't have performance-by-default. It would only have performance-after-Burst.

 

Number of ECS libraries are already wrapped with Assembly Definition files, reassembly dlls.
I don't that would be a problem, as long code is not obfuscated (or maybe event then?). But don't know, how burst works behind a scene.

I don't see reason really for forcing burst to be enabled. It should be optional, if someone want to disable it. Also, you need explicetely declare, which jobs should run on burst using [Burst]. I assume there is reason behind, why is not by default. But this way, it keeps ensure, that if burst is for some reason not compatible with your system / job, you can simply do not activate burst. Which is convenient.

I think working without burst is good starting point for optimization. Then when happy with optimized scrips, aim for enabling burst.on a job(s). That is my opinion.

Also performance is relative term. Still is likely supersede OOP by far, for most, if not all case. Even in bad optimized code, with tons of boxing.


But best, if someone with better expertise, could clarify upon.

I think this option comes with next release of Unity 2018.3
Weather it will be executed straight away, I don't know.

 

Perhaps I'm misusing the term Burst if it's specific to jobs only, I was thinking of it as being a more general optimised compiler for all Unity code that simply currently focuses on jobs.

I'm talking about potentially making every component-read operation boxing. That is definitely not okay. Boxing operations create garbage. There's a relatively simple inlining that could be done to fix that, but the compiler doesn't do that on its own and it would need to be done as an additional optimisation in a 100% reliable manner, otherwise you'd have potential GC hell.

It's not worth going too far into the details of that now, I'll post further if/when I get my prototype into a decent state. If a Unity dev could comment on the feasibility of that kind of compile optimisation that would be great though.

 

ECS piratically does not produce GC, if executed correctly. This is one of its advantage. At least that its aim.

And burst yes, it apparently works outside jobs or ECS. But don't know how effective is there.

 

I must say I'm with @Antypodish on this. ImInot completely sure why you're expecting the components to be boxed and unboxed. The API is using the IComponentData interface merely as a generic constraint. Ref and out keyword also won't cause boxing and unboxing. The potential is there that this can happen, but if you're writing jobs you'll be relatively safe from managing to do this yourself.

 

I feel like I'm hijacking the whole thread here and I never meant to. My question makes sense in context, but that's a little complicated and I haven't explained it well. I wasn't talking about the current API, I was talking about some changes I'm considering proposing. I also wasn't talking about boxing of the components themselves, rather an iteration helper type.

I never meant to have this whole discussion, and while I do appreciate you guys trying to help, I'm worried we're hijacking the thread now. That's my fault for tiredly asking incoherent questions.

What I was aiming for was just asking a quick question of the devs. Specifically, is an API design acceptable if that design is only fast (not heavily boxing all the time) after it's been through some form of transformation as a compilation step?

Sorry admins, feel free to delete the mess I made in this thread if you'd like.

 

Don't worry, non need for deleting. You raised a question with your concerns and is relevant to a topic.
I hope you received some of clarification, even if maybe your questions wasn't accurate as meant to be.
You can always open separate thread to if you feel, you want find specific details.

 

A couple of years ago I messed about with prototyping an ECS API in C#. This was actually in response to me being unsatisfied with Unity's Monobehaviour system, so I'm thrilled to see my thoughts on that vindicated with the way Unity is going now. I'm aware that this reads as fairly critical of the current direction, so I just want to be clear that I really love that this is happening and this new direction for Unity is very impressive. The performance you guys are achieving with this is fantastic, I'm very impressed by all that. I have strong opinions on this stuff and I'm going to express them because I'd love the API to be the best it can be, but I'm not meaning to be down on the great work that's happened here.

My conclusion when I built an ECS myself was that C# as a language is not well suited to this type of API, primarily due to a lack of union and intersection types or similar. ECS is largely about doing queries on collections of types, and C# is bad at presenting the results of those queries in a type-safe manner. I did however find some nice-ish workarounds to that which I will hopefully be posting here soon.

I've also prototyped a similar API in Typescript, a language with a more powerful type system that I found very beneficial. The ability to deal in collections/sets of types, as well as to perform conditional type transformations such as producing readonly versions of types, resulted in a very nice API that is now my point of comparison. C# will never be able to achieve anything quite as beautiful, but I'd love to emulate as much of that functionality as possible.

My views on what makes a good API

1. READABILITY, not only writeability, is key. Verbose or boilerplatey code is harder to grok.
2. Wherever possible, correctness should be checked by the compiler. Exceptions are a last resort.
3. It should be hard to write wrong or buggy code. The API should guide you here as much as possible.
4. As a consequence of 2 and 3, everything that possibly can be, should be strongly typed.
5. Features should be easily discoverable via Intellisense. A really nicely designed API can be used without ever looking at external documentation because everything is intuitive and in an obvious place.
6. The language should be used idiomatically.
7. Attributes are a last resort because they (mostly) can't help the compiler check your code.

Thoughts on the current Unity ECS API
Auto/reflection filled group structs
The pattern of using a struct with fields that get filled according to their type is, IMO, in breach of every single one of the good API rules I suggested above. You have to be very familiar with Unity's ECS API in order to follow this at all.

It's verbose. You're defining a struct in order to do a query. That struct definition is probably not next to its use, so you have to jump between the struct and its use to grok the code.

In group structs the field names are also excessive, as you only need to get the components by type. This presents a maintenance problem, especially when working in larger teams. You need to agree on a naming-scheme for the fields. Do you just make them the same as the component name? Okay, so what happens when we rename the component type? The IDE can't (easily) auto-rename all of our group-struct properties. This isn't the worst problem, but it's undesirable complexity and effort to maintain. I suspect that this will result in messy code in some cases.

The compiler can't check any of this. You can put invalid fields in the structs. That's a runtime error that would be much better as a compile-time error. We're enforcing field access with attributes ([ReadOnly] etc) and again the compiler doesn't know what this means so can't enforce it.

Wait, so the engine is reflecting in and populating those for me? What happens if you put two fields of the same component type in a group struct? Does it work if the component fields are private? It does, but I only know that from trial and error, it's not at all obvious. How does the injection work? Does the length field get populated by name or because it's an int? These are the things I was thinking when I first saw the API. It's far from intuitive or idiomatic C#.

Existing static analysis tools don't understand that these structs are being filled by reflection, so if you have a readonly field or something they complain.

Accessibility attributes
This is mostly covered in the previous section, but anywhere where [readonly] and similar attributes are being used, there's an issue. The compiler can't check that for us, it's easy to get wrong on some rare code path, and it's a bit ugly to have attributes everywhere. I believe this can be designed out.

Field injection
Injected fields are, IMO, undesirable. The compiler and static analysis tools don't know that they're injected, so lots of dev environments will complain about them. If I put a

[Inject] private readonly OtherSystem m_SomeOtherSystem;

field in a class I'll get warnings that it's never assigned to. I could turn these off, but I don't want to, I usually want to be warned about that. I could put a suppression comment by every injected field, but that's ugly. I could replace that rule with a custom analyzer that knows not to complain about injected fields, but that's silly. This pattern leads to a poor dev experience.

Use of typeof()
There are a few places where the API makes use of typeof(). The main ones I know of are an overload to EntityManager.CreateArchetype() and in various attributes. The CreateArchetype overload is presented as a convenient alternative to ComponentType.Create<T>() but I would suggest that it's objectively worse and should be removed or heavily discouraged. Whenever a type needs to be provided to an API and there are constraints on what that type should be, the type should be supplied generically so that it can be checked at compile time. Non-safe alternatives, if required, should be discouraged unless needed.

ComponentGroup
I believe that, given the above, an API based on a form ComponentGroup may have the most potential. This pattern is more powerful, expressive, and idiomatic. You don't need to declare structs, name unnecessary fields on them, deal with attributes, or use injection. They also support things like SetFilter() which I don't think injection does.

The obvious downside with this API right now is that it is not strongly typed. I believe this can be mostly fixed with some clever use of generics. I've built an API a bit like that before, and have been prototyping a generic version of ComponentGroup over the last day or so. I'll hopefully post that suggestion up in a bit when I've got it a little more refined.

Naming
This is pretty low down on the priority list, but now that this is almost a completely new Unity API, would you perhaps consider moving to more typical C# naming schemes? I'm assuming that Unity's current naming is a hangover from the Javscriptish support, which is no longer relevant. It would be nice if it used more typical conventions.

Static analysis is not the answer
I've seen quite a few suggestions from the Unity team that a lot of these issues can be solved with additional static-analysis tools. While Roslyn Analyzers and the like are great, I think you've got a problem if they're required to make your API decent to use. They should be complementary only. I see two obvious problems with this approach.

Firstly, IDE support is limited. Not horribly so, both VS and Rider support Analyzers, but it's a restriction.

Secondly, proper use of the current Unity API clashes with a fair few of the sensible default rules that other static analysis tools provide. This is already an issue with the existing Monobehaviour API, for example Monobehaviours are all about public fields, which every static analyzer quite rightly screams at you for. There are few places where the current ECS API introduces more of these issues. The obvious examples being that subtractive and injected fields are considered non-used and not-initialised respectively by most tools. You can work around this by turning those rules off or disabling them with comments on every occurrence, but it's not a smooth dev experience.

Static analysis that clashes with idiomatic C# has not solved the problem.

Misc. thoughts
C# 7.2 introduces some new niceness that could fit well with this API. The unmanaged generic constraint restricts structs to having only unmanaged fields. That would be nice to be able to use.

Proposal
WIP.

I have been messing about prototyping a version of this API that I believe improves upon a lot of these issues. It's not quite ready yet, I still need to work some things out, but I hope to post that here fairly soon. Is that type of suggestion welcome here?

 

Thanks for providing this detailed breakdown! It actually looks very similar to the list of criteria we are using now. I didn't go into full detail because I wasn't sure it would be interesting, but our current criteria for scaffold evaluation include:

• Simplicity (i.e. opposite of complexity): how a scaffold impacts the amount of information developers need to keep in mind for a given task
  
• Idiomaticity: how well the scaffold conforms to idiomatic C# (expect some deviation with e.g., mutable structs, but ideally an experienced C# dev should be able to look at code and reason about its behavior and performance characteristics)
  
• Immediacy: how quickly developers can be confident they are creating correct code (e.g., compiler error is better than static analysis message, which is better than a runtime error)
  
• Incrementality: the extent to which a scaffold results in code that could survive partial removal of the scaffold
  
• Transparency: the ability to reason about data and any transformations happening to it
  
• Readability and Locality: the ability of a developer to understand the behavior of a given excerpt of code at a glance
  
• Distributed Lifetime Value: a scaffold’s utility to newcomers spread across the user base as a whole, over time
  
• Individual Lifetime Value: a scaffold’s utility to an individual developer as that developer achieves mastery

This was also a conclusion of our internal evaluation. We will be phasing these out (it will start with migration of our examples), but they will not likely be fully removed until we have a new replacement in place.

I suppose it depends on the precise cases you have in mind, but at least in the case of parameter modifiers for job methods, our goal is to have a system in place where you will use ref or in to indicate permissions intent, so you can get a compiler error if you try write to something readonly. (Ideal here is to also use static analysis to warn if you aren't actually writing to something for which you have specified write permissions.)

I agree. This specific example actually came up yesterday, but I've also remarked about the likelihood of error it introduces for us to implicitly cast System.Type to ComponentType (e.g., typing PositionComponent instead of Position, which has the added problem of actually looking correct at a glance). Many of the System.Type-based API points may still need to exist, but there should always be generic alternatives that are encouraged.

Per the point on immediacy above, our ultimate preference is that correct usage can be enforced by the compiler. There may still be some cases that we can only detect via static analysis, but the ideal is that its primary use will be to help you improve your code, rather than to fix errors as such (see my point above about warning if you are not writing to something where you have requested write permissions).

By all means! @Joachim_Ante already has some ideas we have tossed around internally, but these proposals still have some kinks with respect to some of our criteria. We will benefit from looking at the problem from as many angles as possible.

If it is useful to keep in mind, our ideal is that it should be easier to read and write correct/performant code than it is to read and write incorrect/non-performant code. Although we may not ultimately make it easier to write Capsicum-based game code than it is to write MonoBehaviour code, it at least gives us a direction for alignment (e.g., are we moving more that way, or just moving laterally).

 

Thank you very much for that comprehensive response!

It's great to see that our thinking is aligned on these things, that's very pleasing. I'll admit that I had some concerns about the direction but you've largely alleviated them here.

That sounds great, and is one of the options I was considering. I was leaning towards that perhaps not being possible given the final section of your original post here, so it's great to see that I misunderstood that. That sounds like a pretty decent way of handling it.

Absolutely. I tried to ask above (and made a complete tired mess of it) whether you have control of the build pipeline to the extent that optimisable code that may not be fast if only compiled using standard means is acceptable. Part of the prototype I'm working on uses a method that looks roughly like the following

Code (CSharp):

1. // The name Bag needs to be changed to something better.
2. // This is a simplification of the actual method for demonstrative purposes.
3. public static T Read<T>(this IBag<T> bag) => ReadFromBag<T>(bag);

In any normal use-case the IBag<> would actually be a struct defined as part of the API. This is theoretically boxing that value type, which is a no no for performance. I suspect the JITer may inline this in many cases and avoid the boxing, but that would need to be done reliably for this to be an acceptable API. Is that possible with the compile pipeline that Unity uses?

I have a multi-stage proposal in the works (it's just prototype code at the moment, I'll try to turn it into a write-up) that starts off as a proposal just for a generic ComponentGroup and builds up to being a bit more radical. I'll probably post the first part first, as that's more developed.

 

We have potentially quite a bit of flexibility. For example, a proposal we're currently evaluating basically uses lambda syntax but actually gets compiled into a job struct instead of a delegate instance. The caveat with compiler tricks like this, however, becomes whether they result in idiomatic violations/surprises. For example, our guidelines on idiomaticity entail:

• (+) Syntax or pattern results in code whose behavior and performance characteristics could be predicted by a C# developer outside of Unity
• (=) Syntax or pattern produces behavior or performance characteristics that are unexpected or counterintuitive if similar approach were used in non-Unity C#
• (-) Syntax or pattern produces behavior or performance characteristics that are unexpected or counterintuitive if a similar approach were used elsewhere in Unity

So bending the compiler to our will is a bit like trying to wield Sauron's ring. If we use something like lambda syntax to achieve high-performance code in one place, but it actually results in low-performance code if someone tries to use it elsewhere in their code base, then we have an obligation to mitigate that pitfall (e.g., some how make it immediate and clear that this optimization does not apply everywhere).

Not being part of the team working on the compiler, I unfortunately can't be more specific than that, other than to say it's not a bad idea to have a plan for how to communicate the limits of a compiler trick, if it is ultimately something that could not be applied across the board.

 

That's great to know, thanks. Yeah this specific example would just be a built-in extension method on a built-in value type being non-boxing fast, I doubt that would cause any developer confusion.

 

what if, instead of generating the job based on the system (your lambda approach), you generate the system based on a job definition?
e.g. if you have a job like this:

Code (CSharp):

1. [GenerateSystem]
2. public struct MyJob : IJobProcessComponentData<...> {
3.     // declare stuff to inject, e.g.
4.     [InjectGlobal("Time.deltaTime")] public float deltaTime;
5.     // or, for a type-safe and auto-completable alternative:
6.     [Inject] public DeltaTime deltaTime;
7.     public void Execute(...) {...}
8. }

this system gets automatically generated:

Code (CSharp):

1. public class $generatedName$ : JobComponentSystem {
2.  
3.     protected override JobHandle OnUpdate(JobHandle inputDeps) {
4.         return new MyJob {
5.             // inject stuff
6.             deltaTime = Time.deltaTime
7.         }.Schedule(this, inputDeps);
8.     }
9. }

(you define how to inject stuff accordingly.)

 

I'm going to start by just dropping my ComponentGroup proposal in here.

Author's note: I use the term "unsafe" to refer to non-generic non-compiler-checked access, rather than to pointery unsafe code. I need a less confusing word for that.

Ideally, I'd love to see ComponentGroup be a generic class, where the type system knows what components it contains and what their accessibility (readonly, writeonly, etc) is. It should provide easy access to the component types within it, disallow access to components it doesn't contain, and regulate the available actions according to the accessibility of each component. It should also support groups of arbitrary sizes and be unordered, although I don't believe this to be possible.

I believe I have a proposal that achieves all but the last of these goals, albeit in a somewhat round-about way. The basic premise is to support generic ComponentGroup<T1, T2, ... Tn> up to some size (number of components), and then to provide an unsafe fallback if someone needs a very large group. This is inspired by how things like Action<T1, T2, ... Tn> already work. It's a little ugly that we need to define each version, but it can be lived with. We then provide the bulk of the API using extension methods on these types.

This uses some new collection types. ReadonlyComponentDataArray<T> and WriteonlyComponentDataArray<T> would be needed. They're pretty self-explanatory. Implicit casting can be done to the main ComponentDataArray<T> type. These would all be structs.

Show me the code
I'll start with a big dump of the API, then look below that for a separate code block with example usage.

The API mockup

Code (CSharp):

1. // These are wrapper types that provide guidance to the extension methods.[/B]
2. [B]
3. // IReadable<PositionComponent> corresponds to a PositionComponent that can be read from.
4. // The base interface, IComponentAccessTag would ideally not exist, but is used to help generic constraints.
5. // It should be considered meaningless.
6. public interface IComponentAccessTag { }
7. public interface IReadable<T> : IComponentAccessTag where T: unmanaged, IComponentData { }
8. public interface IWriteable<T> : IComponentAccessTag where T : unmanaged, IComponentData { }
9. public interface IReadWriteable<T> : IReadable<T>, IWriteable<T> where T : unmanaged, IComponentData { }
10.  
11.  
12. // This is an internal interface that the user should never be implementing.
13. public interface IComponentGroup
14. {
15.     int Length { get; }
16.  
17.     // These throw exceptions if the component type is not in the group and should generally not be used by user code.
18.     // I think that I'd prefer these not to exist.
19.     // They could be removed if the extension methods simply throw exceptions if the user tries to implement any IGroupn themselves.
20.     ReadonlyComponentDataArray<T> ReadonlyComponentsUnsafe<T>() where T : unmanaged, IComponentData;
21.     WriteonlyComponentDataArray<T> WriteonlyComponentsUnsafe<T>() where T : unmanaged, IComponentData;
22.     ComponentDataArray<T> ReadWriteComponentsUnsafe<T>() where T : unmanaged, IComponentData;
23. }
24.  
25.  
26. // These are just markers that allow extension methods to safely call the unsafe methods on Query.
27. // These should never be implemented by the user.
28. // Implementing IGroupN<T> means you guarantee the unsafe methods return safely for this type.
29. // The Nth version of this interface corresponds to the Nth type parameter.
30. // Sadly we can't have a single interface implemented multiple times due to language restrictions.
31. public interface IComponentGroup1<out T> : IComponentGroup where T : IComponentAccessTag { }
32. public interface IComponentGroup2<out T> : IComponentGroup where T : IComponentAccessTag { }
33. public interface IComponentGroup3<out T> : IComponentGroup where T : IComponentAccessTag { }
34. public interface IComponentGroup4<out T> : IComponentGroup where T : IComponentAccessTag { }
35. // ... As many as implemented.
36.  
37.  
38. // This would be roughly equivalent to the current ComponentGroup.
39. // It's the weakly typed base class for the strong ComponentGroups.
40. public abstract class ComponentGroup : IComponentGroup
41. {
42.     public int Length { get => throw new NotImplementedException(); }
43.  
44.     private protected ComponentGroup() { }
45.  
46.     // These would tie into ECS internals
47.     ReadonlyComponentDataArray<T> IComponentGroup.ReadonlyComponentsUnsafe<T>() => throw new NotImplementedException();
48.     WriteonlyComponentDataArray<T> IComponentGroup.WriteonlyComponentsUnsafe<T>() => throw new NotImplementedException();
49.     ComponentDataArray<T> IComponentGroup.ReadWriteComponentsUnsafe<T>() => throw new NotImplementedException();
50. }
51.  
52.  
53. // This is what the strong generic ComponentGroup versions look like.
54. // There's N versions of these, where N is the maximum supported group size.
55. // These would be the types the user would use directly. Most of their functionality will be supplied with extension methods.
56. // Each type parameter has its own tag interface. This allows the extension methods to address individual type parameters.
57. // This makes up for the languages lack of ability to have types of the form IComponentGroup<*, IReadable<PositionComponent>, *>.
58. // There may be a way to achieve that with covariance or contravariance, but I don't believe so.
59. public sealed class ComponentGroup<T1> : ComponentGroup, IComponentGroup1<T1>
60.     where T1 : IComponentAccessTag
61. { }
62.  
63. public sealed class ComponentGroup<T1, T2> : ComponentGroup, IComponentGroup1<T1>, IComponentGroup2<T2>
64.     where T1 : IComponentAccessTag where T2 : IComponentAccessTag
65. { }
66.  
67. public sealed class ComponentGroup<T1, T2, T3> : ComponentGroup, IComponentGroup1<T1>, IComponentGroup2<T2>, IComponentGroup3<T3>
68.     where T1 : IComponentAccessTag where T2 : IComponentAccessTag where T3 : IComponentAccessTag
69. { }
70.  
71. public sealed class ComponentGroup<T1, T2, T3, T4> : ComponentGroup, IComponentGroup1<T1>, IComponentGroup2<T2>, IComponentGroup3<T3>, IComponentGroup4<T4>
72.     where T1 : IComponentAccessTag where T2 : IComponentAccessTag where T3 : IComponentAccessTag where T4 : IComponentAccessTag
73. { }
74. // ... As many as implemented.
75.  
76.  
77. // These extension methods provide the bulk of the actual API.
78. // There's N versions of each of these to handle the N type parameters.
79. public static class GroupExtensions
80. {
81.     // Provides readonly access from groups that support it.
82.     public static ReadonlyComponentDataArray<T> Readable<T>(this IComponentGroup1<IReadable<T>> group) where T : unmanaged, IComponentData => group.ReadonlyComponentsUnsafe<T>();
83.     public static ReadonlyComponentDataArray<T> Readable<T>(this IComponentGroup2<IReadable<T>> group) where T : unmanaged, IComponentData => group.ReadonlyComponentsUnsafe<T>();
84.     public static ReadonlyComponentDataArray<T> Readable<T>(this IComponentGroup3<IReadable<T>> group) where T : unmanaged, IComponentData => group.ReadonlyComponentsUnsafe<T>();
85.     public static ReadonlyComponentDataArray<T> Readable<T>(this IComponentGroup4<IReadable<T>> group) where T : unmanaged, IComponentData => group.ReadonlyComponentsUnsafe<T>();
86.     // ...
87.  
88.     // Provides writeonly access from groups that support it.
89.     public static WriteonlyComponentDataArray<T> Writeable<T>(this IComponentGroup1<IWriteable<T>> group) where T : unmanaged, IComponentData => group.WriteonlyComponentsUnsafe<T>();
90.     public static WriteonlyComponentDataArray<T> Writeable<T>(this IComponentGroup2<IWriteable<T>> group) where T : unmanaged, IComponentData => group.WriteonlyComponentsUnsafe<T>();
91.     public static WriteonlyComponentDataArray<T> Writeable<T>(this IComponentGroup3<IWriteable<T>> group) where T : unmanaged, IComponentData => group.WriteonlyComponentsUnsafe<T>();
92.     public static WriteonlyComponentDataArray<T> Writeable<T>(this IComponentGroup4<IWriteable<T>> group) where T : unmanaged, IComponentData => group.WriteonlyComponentsUnsafe<T>();
93.     // ...
94.  
95.     // Provides full readwrite access from groups that support it.
96.     public static ComponentDataArray<T> ReadWriteable<T>(this IComponentGroup1<IReadWriteable<T>> group) where T : unmanaged, IComponentData => group.ReadWriteComponentsUnsafe<T>();
97.     public static ComponentDataArray<T> ReadWriteable<T>(this IComponentGroup2<IReadWriteable<T>> group) where T : unmanaged, IComponentData => group.ReadWriteComponentsUnsafe<T>();
98.     public static ComponentDataArray<T> ReadWriteable<T>(this IComponentGroup3<IReadWriteable<T>> group) where T : unmanaged, IComponentData => group.ReadWriteComponentsUnsafe<T>();
99.     public static ComponentDataArray<T> ReadWriteable<T>(this IComponentGroup4<IReadWriteable<T>> group) where T : unmanaged, IComponentData => group.ReadWriteComponentsUnsafe<T>();
100.     // ...
101.  
102.     // Helpers for reading from a specific index without dealing with the full array.
103.     public static T ReadAt<T>(this IComponentGroup1<IReadable<T>> group, int index) where T : unmanaged, IComponentData => group.ReadonlyComponentsUnsafe<T>()[index];
104.     public static T ReadAt<T>(this IComponentGroup2<IReadable<T>> group, int index) where T : unmanaged, IComponentData => group.ReadonlyComponentsUnsafe<T>()[index];
105.     public static T ReadAt<T>(this IComponentGroup3<IReadable<T>> group, int index) where T : unmanaged, IComponentData => group.ReadonlyComponentsUnsafe<T>()[index];
106.     public static T ReadAt<T>(this IComponentGroup4<IReadable<T>> group, int index) where T : unmanaged, IComponentData => group.ReadonlyComponentsUnsafe<T>()[index];
107.     // ...
108.  
109.     // Helpers for writing to a specific index without dealing with the full array.
110.     public static void WriteAt<T>(this IComponentGroup1<IWriteable<T>> group, int index, T value) where T : unmanaged, IComponentData { var c = group.WriteonlyComponentsUnsafe<T>(); c[index] = value; }
111.     public static void WriteAt<T>(this IComponentGroup2<IWriteable<T>> group, int index, T value) where T : unmanaged, IComponentData { var c = group.WriteonlyComponentsUnsafe<T>(); c[index] = value; }
112.     public static void WriteAt<T>(this IComponentGroup3<IWriteable<T>> group, int index, T value) where T : unmanaged, IComponentData { var c = group.WriteonlyComponentsUnsafe<T>(); c[index] = value; }
113.     public static void WriteAt<T>(this IComponentGroup4<IWriteable<T>> group, int index, T value) where T : unmanaged, IComponentData { var c = group.WriteonlyComponentsUnsafe<T>(); c[index] = value; }
114.     // ...
115. }

Example system

Code (CSharp):

1. class Demo : ComponentSystem
2. {
3.     // Ideas for how to initialise this nicely are in the works.
4.     // I'm unsure how I feel about this fairly verbose syntax.
5.     // Factories and the like provide interesting alternatives to explore.
6.     // Consider this to be placeholder for now.
7.     private readonly ComponentGroup <
8.             IReadable<PositionComponent>,
9.             IReadWriteable<AgeComponent>,
10.             IWriteable<HealthComponent>
11.         > enemyGroup;
12.  
13.     // This is the basic version using the lowest level of the API.
14.     // I've fully specified types on this one for illustrative purposes, but these would normally be vars.
15.     void BasicOnUpdate()
16.     {
17.         // Get the arrays.
18.         // Each has its access limited (by the type system) according to the specification in ComponentGroup.
19.         ReadonlyComponentDataArray<PositionComponent> positions = this.enemyGroup.Readable<PositionComponent>();
20.         ComponentDataArray<AgeComponent> ages = this.enemyGroup.ReadWriteable<AgeComponent>();
21.         WriteonlyComponentDataArray<HealthComponent> healths = this.enemyGroup.Writeable<HealthComponent>();
22.  
23.         for (var i = 0; i < this.enemyGroup.Length; i++)
24.         {
25.             // Die if we're too old or fall off the bottom
26.             if (ages[i].value > 100 || positions[i].value.y < 0) healths[i] = new HealthComponent() { value = 0 };
27.  
28.             // Get older
29.             ages[i] = new AgeComponent() { value = ages[i].value + Time.deltaTime };
30.         }
31.     }
32.  
33.     // This is using the ReadAt/WriteAt methods to save a few lines.
34.     void HelperOnUpdate()
35.     {
36.         for (var i = 0; i < this.enemyGroup.Length; i++)
37.         {
38.             // These could of course be inlined if we wanted, broken out for readability.
39.             var age = this.enemyGroup.ReadAt<AgeComponent>(i);
40.             var position = this.enemyGroup.ReadAt<PositionComponent>(i);
41.  
42.             // Die if we're too old or fall off the bottom
43.             if (age.value > 100 || position.value.y < 0) this.enemyGroup.WriteAt(i, new HealthComponent() { value = 0 });
44.  
45.             // Get older
46.             age.value += Time.deltaTime;
47.             this.enemyGroup.WriteAt(i, age);
48.         }
49.     }
50.  
51.     // Foreach support is not actually implemented in the API at the moment, so this is just a preview.
52.     // I took it out because it made the code dump a lot bigger and more verbose, but it's easy to see how it would be added.
53.     // There would be generic iteration type equivalent to ComponentGroup<...>.
54.     // I'll probably explain that in a later post.
55.     void ForeachOnUpdate()
56.     {
57.         foreach (var enemy in enemyGroup)
58.         {
59.             var age = enemy.Read<AgeComponent>();
60.             var position = enemy.Read<PositionComponent>();
61.  
62.             // Die if we're too old or fall off the bottom
63.             if (position.value.y < 0 || age.value > 100) enemy.Write(new HealthComponent() { value = 0 });
64.  
65.             // Get older
66.             age.value += Time.deltaTime;
67.             enemy.Write(age);
68.         }
69.     }
70.  
71.     void ThingsThatTheCompilerPrevents()
72.     {
73.         // This is a compiler error, as the group is defined with IReadable<PositionComponent> and IWriteable<HealthComponent>.
74.         // These extension methods do not exist on these types.
75.         var positionsWrong = this.enemyGroup.ReadWriteable<PositionComponent>();
76.         var healthsWrong = this.enemyGroup.Readable<HealthComponent>();
77.  
78.         // We're doing it right now. We get back a ReadonlyComponentDataArray<PositionComponent> and WriteonlyComponentDataArray<HealthComponent>;
79.         var positions = this.enemyGroup.Readable<PositionComponent>();
80.         var healths = this.enemyGroup.Writeable<HealthComponent>();
81.  
82.         // This is another compiler error. Positions is readonly and healths is writeonly, so we get our red squigglies.
83.         positions[0] = new PositionComponent();
84.         var healthWrong = healths[0];
85.  
86.         // Same here.
87.         this.enemyGroup.WriteAt<PositionComponent>(0, new PositionComponent());
88.         this.enemyGroup.ReadAt<HealthComponent>(0);
89.  
90.         // Ages give us full access, so all of this works fine.
91.         var ages = this.enemyGroup.ReadWriteable<AgeComponent>();
92.         var age = ages[0];
93.         ages[0] = age;
94.         this.enemyGroup.ReadAt<AgeComponent>(0);
95.         this.enemyGroup.WriteAt(0, age);
96.     }
97. }

I have more ideas in development for providing IEnumerable support (that's where the boxing issue comes in), for various APIs that could be used to nicely build the ComponentGroup objects, possible changes to the system API, alternatives to injection, and ways this ComponentGroup API could be integrated with jobs. However, that's all very much still in development and this is already long enough, so I'm dumping this here now.

I'd be interested to hear feedback on whether you consider this design to be feasible and interesting.

 

I you plan provide more samples, I think it would be worth it, if you setup git (github) repository.
It would guarantee keeping latest version and allow to keep all scripts in one place.
Otherwise, you will get scattered across pages on forum thread. Which may become tedious, to keep track what is the the most actual.

 

Good point

Good point, thank you. I'll throw together a repo or a gist or something at some point.

 

Thanks! I raised a possibility like this internally a couple weeks ago, so it's on our radar. We're all heads-down on some other stuff right now so I haven't had a chance to formally evaluate it with the new criteria, but I think my immediate reaction is that it quickly becomes pretty attribute-heavy when done in a way that has much individual lifetime value (e.g., while you might pass some common things like time/deltaTime, it needs to a more generic approach to be useful in other cases, or it runs out of steam pretty quickly).

Something like that would be great! Let me know on here if you do and I'll make sure we take a look. For whatever it is worth, I'd say our current thinking is to try to find a way to ultimately get rid of ComponentDataArray (and related scaffolds). Among other things, it presents like an array, which invites misuse that is detrimental to performance. E.g., if a user is doing anything other than moving through it linearly, one index at a time, they're introducing performance problems. Part of what we like about IJobProcessComponentData in comparison is that all of the iteration is handled for the user, so this type of problem doesn't arise.

 

That makes a lot of sense, perhaps there's an argument for moving to only providing the foreach capability (that isn't in the version posted here yet but works roughly the same way) and skipping over providing the arrays. Is there ever a scenario where it makes sense to still be able to access by index? Maybe in some situation where you're handling relations between components/entities and you don't want to copy data into a new structure? That would be non-sequential access though anyway. I agree that direct access to the arrays should probably at least be discouraged for general use, as well as being nicer code.

In a more radical departure, I've been toying about with (though haven't developed) the idea of moving to a more callback based system API, as inspired by how you write Roslyn Analyzers. So chuck out the

OnUpdate()

method, throw in an

OnInitialise()

and do things like

OnGroup(enemyGroup, enemy => {});

That could be nice if you want to be able to handle additional event types and various overloads, and allows you to move some things that are currently private fields in as locals. Thought would need to go into how you handle per-frame state, but I reckon that's solvable with some clever generics and a data struct. I'm not sure if I've made that clear, I'll explain it in more detail when I've thought it through further.

Is a callback based API like that something that's been considered internally?

 

Unless there's an alternative explicit api to describe "grab a single/the first one of these", then yes. In my current project, I use entityGroup.component[0] all the time, particularly for "spawning" (removing the Disabled component from) all sorts of entities.

 

Could you expand on this a little?

Do you mean you know that you only have one matching entity, so you just take the first? That seems a little less maintainable and harder to debug if something goes wrong. What if you suddenly (through bug or design change) have two of them? I don't believe they're in any guaranteed order, so you're into undefined behaviour there.

My instinct would be to treat it as a collection even if you only ever expect it to have a single item. I may be missing something though.

 

I mean I know there are many matches, but I only ever want to use the first match in my system. For example, my current project is an infinite runner. As the player progresses, every X distance I'll spawn an obstacle. Just one. But I have many matching obstacles "pooled" via the Disabled component. When the player passes the obstacle, it is returned to the pool by having Disabled added back onto it.

If the system API abstracted away the indices, or always iterated over all matches, it would become inefficient, as I'd have to implement specific logic to do nothing after the first entity. Right now, most of these spawning systems of mine don't iterate over matches at all: they just check some condition to determine if they should spawn anything, then spawn one thing (the first matching entity).

My point was that the API shouldn't preclude that sort of logic. Nor should it preclude iterating over one group within an iteration over another group. If the API for systems were simply a lambda that would be called for each element of a matching entity query, much of what I need to do for my game would be difficult/impossible, and many systems would become very inefficient without an early exit option.

 

Can we take this as confirmation that both of these 'APIs' are going away:

Code (csharp):

1.  
2. struct State
3. {
4.   public readonly int Length;
5.   public ComponentDataArray<EnemySpawnCooldown> Cooldown;
6.   public ComponentDataArray<EnemySpawnSystemState> S;
7. }
8. [Inject] State m_State;
9.  

??

 

That is the eventual plan, yes.

 

How's the solution you going to apply? Directly generate the final machine code without able to see the generated code at Project tab or able to see the generate code at Project tab like lambda syntax will generate job struct code? I suggest that generating code at project tab that will tell user that the lambda syntax will turn into generated code and it will execute based on the generated code instead of lambda syntax. Even better u can log clear message at Console tab to tell which part of the code has been converted into generated code. I believe with this approach we can get even simplified and better API for Reactive System and also many other ECS related APIs.

 

So, this is good I suppose - but at the same time it puts a bit of doubt in my mind on how long it will take for the ECS to be ready for prime time use. If you're reworking such core APIs already and most of the rest of the engine isn't ECS-pure compatible... yeah, I mean I understand why it's being changed, the current API is anything but user friendly.

 

I've learned to accept the idea that the ECS core probably won't be stable and ready for full production until the end of the 2019.X cycle / beginning of the 2020.X cycle. And it's likely that'll only cover the basics (core data structures and ECS API, probably basic physics/collisions/AI navigation/rendering), it'll be years before the full Unity API stack is converted. There's already movement to expose jobified script access to stuff like Particles and the Playable Graph, so at least that's already ahead in some areas.

You may be able to ship something with ECS this and next year if you just use it to supplement something via hybrid and/or can roll with the changes as they come.

 

So basically as always -don't depend on 'waiting' features ;

 

Apologies, I know that I said I'd post a repo of my suggestions and have gone dark since. I've not been able to work on this too much during the week, but I hope to get something up over the weekend. I have a quick technical question though.

@Adam-Mechtley or other Unity devs, is there ever any need for systems to have write-only access to components? Presumably read-only systems can run in parallel with jobs, but write-only (as I understand it) does not need to be supported by any API? To be clear, by "access to components" I mean through groups/injection/whatever.

As far as I've seen,

[writeonly]

is only ever used for native collections in jobs, is that correct?

 

Without any current examples on hand I suppose your guess is as good as mine If it's helpful to clarify your thinking for now, let's just assume it's not necessary

 

After using the api for a while I have been noticing a desire for better readability to distinguish from what injected data is actually being read from vs written to vs filtered by. For example, let's say I am injecting the following data:

• OutputData
• [ReadOnly] InputData1
  
• [ReadOnly] InputData2
  
• [ReadOnly] TagData1
  
• [Subtractive] TagData2
  
• EntityArray

And doing the following transformation on it:

Code (CSharp):

1. PostUpdateCommands.SetComponent<OutputData>(
2.     entity,
3.     new OutputData { data = InputData1.data * InputData2.data }
4. );

From the code above, the only data that I am actually reading from is InputData1, InputData2, and EntityArray. The other data, TagData1 and TagData2, is only used for filtering. And OutputData is only used for writing to.

From a readability perspective, I think it would useful query data like so:

• ReadFrom: InputData1, InputData2, EntityArray
• WriteTo: OutputData
• FilterBy: TagData1, [Subtractive] TagData2

This tells me at first glance (without digging deeper into the code) what data is actually being read from, what data is being written to, and what is used only for filtering. In the Update function I would expect to only being able to retrieve data from what I am reading from.

 

Here's one suggestion I would make:

As I understand it, IComponentData implementers need to be blittable structs. However, outside of throwing exceptions I don't believe Unity enforces this. It could if IComponentData was changed to:

Code (CSharp):

1. public interface IComponentData<TSelf>
2.     where TSelf : unmanaged, struct, IComponentData<TSelf> { }

This couldn't be implemented at the moment since unmanaged was added in C# 7.3. But this should ensure IComponentData is used as intended.

 

Ignoring IJobProcessComponentData in the example, one pattern/code style that works really nicely with injection is as follows:

Code (csharp):

1. class MySystem : JobComponentSystem {
2.   struct JobA : IJob {
3.     public struct GroupData {
4.       public ComponentDataArray<Foo> Foo;
5.     }
6.     public GroupData Group;
7.     public void Execute() {}
8.   }
9.  
10.   struct JobB : IJob {
11.     public struct GroupData {
12.       public ComponentDataArray<Foo> Foo;
13.     }
14.     public GroupData Group;
15.     public void Execute() {}
16.   }
17.  
18.   [Inject] JobA.GroupData _jobAGroup;
19.   [Inject] JobB.GroupData _jobBGroup;
20.  
21.   protected override JobHandle OnUpdate(JobHandle inputDeps) {
22.     inputDeps = new JobA { Group = _jobAGroup }.Schedule(inputDeps);
23.     inputDeps = new JobB { Group = _jobBGroup }.Schedule(inputDeps);
24.     return inputDeps;
25.   }
26. }

The advantage compared to ComponentGroup here is that the data declaration remains localized to the Execute function. I hope whatever replaces injection maintains the input-spec / execute locality.

Still pretty verbose, though.

 

Is there any necessity of existing of the Entity.Null?
It's just a wrapper static property of new Entity()/default(Entity).
Just writing default(Entity) is the most performant way because there is no function call.
We can write simply 'default' in some cases by C#7.2.

 

https://github.com/Unity-Technologi...ation/content/ecs_in_detail.md#icomponentdata

This still states you'll use ref returns, is that true?