Official API usability

Oh no, I understand why it's there. And why it's named as such. It doesn't mean I like it though.

 

For the same reason why there is no more than 6 components allowed for a job ForEach. Too many use cases to implement, so only the popular ones have been implemented by the Unity guys. ECS doesn't play nice with the 'params' keyword. You can get around this by using chunks so have a look into that.

 

I have 2 suggestions that would greatly improve general ECS workflow.


1. Allow callbacks on jobs. Since there is quite a bit of stuff that can only be done on the main thread, it would be great to be able to have a callback to a job so that we can execute code on the main thread that relies on that job without having to call complete. Alternatively, add a parameter to .ScheduleSingle() that forces the dependent job to run on the main thread and the safety system shouldn't complain about it anymore.

2. Improve communication between systems and monos at startup. Currently, OnCreate on systems is called before the Awake on monos which seems backwards to me. You can't even use GameObject.Find(string) in OnCreate, even if the gameobject you're looking for exists in the scene at compile time. You can theoretically disable auto create on systems and create them after the monos have been initialized, but having to manage the order of system creation can become a pain in a big project. I think you can also make it so that monos start first and systems second, but that just reverses the problem. Some sort of OnStart method (after mono awake for example) would be amazing.

I don't know if any of these suggestions are possible. I'm hoping they are!

 

.Run() executes the job on the main thread

To be honest, I'd have to see how you're trying to do your initialization but this sounds more like poor implementation of initialization.

I find the concept of using GameObject.Find(string) in a systems pretty gross considering systems are generally scene independent. I assume it's where you store initialization data? Have you considered using ScriptableObjects?

I could be wrong and am simply overlooking good use cases.

Alternatively maybe you are looking for OnStartRunning? Though be careful with this.

 

Run() does execute the job on the main thread, but you can't pass a dependency to it. Attempting to run a job that depends on another job throws errors.

For the second part, I use GameObject.Find() to cache a reference to a mono that holds data, which is indeed a poor implementation. I eventually plan on loading data from files, probably xml, but it would be nice to have the capability for quick prototyping.

 

Oh. So you want to queue up and chain main thread tasks as you build up a job chain? It's a cool concept and would be very useful in an ES architecture, but I suspect this may be a poor design in the context of an ECS. Care to elaborate your use case?

 

Why would it be poor design? Unity is encouraging this whole hybrid approach, which naturally involves using the main thread in cases where it's simply impossible not to.

This is one such case: https://forum.unity.com/threads/set-mesh-data-inside-job.766625/

But it can apply to anything dealing with hybrid code. As it currently stands, you can't truly make a pure ecs game without using some components from the old days, unless of course the game in question is of a simple nature. It simply lacks a lot of the old features. Maybe in the near future, but not now. So imo, being able to chain dependencies to the main thread should be a thing.

 

Thanks for your advice, I know I can use chunk, just lazy & wondering why. Actually you can use OnStartRunning instead of OnCreate, hope that solve your problems.

 

OnStartRunning can be called multiple times in the lifetime of a system. It's sort of like the OnEnable of the monos. Even if it covers my second suggestion, you'd always have to make sure to create a boolean so that you don't run the code inside that method if you don't want it to run more than once. Not a big deal, but also not ideal.

 

You most likely want to run designated DOTS entity conversion workflow, instead find entity GameObject. Specially in the system.

 

Why do you have this constant back and forth between main thread and job tasks? To me that suggests unnecessary dependencies or you not allowing defer-able operations to be deferred. Or perhaps your system order wasn't thought out at all. In the case of mesh updates, you don't need to update the meshes until right before rendering where there is already a sync point. You are going to have to elaborate in quite a bit of detail if it is a real problem and not an avoidable one.

 

The whole point of that first suggestion is simply just quality of life. I know it's possible to code your way around every problem, but it should not be that way. You should not have to write dozens, maybe hundreds extra lines of code every time you need to execute something on the main thread, especially when a callback solution would eliminate that. This may be a bit of a hyperbole, but it would be like lacking a way to iterate through a hashmap. This used to not be a thing in earlier versions, but it is now. It wasn't the end of the world because you could just implement it yourself just like you're suggesting I should do now with my own issue. So I'm arguing from a quality of life perspective. Call me lazy, but aren't programmers like that?

I also know this shouldn't be abused because you'd lose performance and trust me, I'm all about that performance. But in some cases it does make sense.

I recall seeing suggestions to leave Unity figure out the best order in which systems execute and not mess it up myself, which according to some of my own experiences, holds some value.

 

I'm asking for a real-world use case. Not a hypothetical. Ask yourself this: When would such a main thread task run relative to the other systems running?

I simply cannot think of a use case that would avoid race conditions on the main thread, not stall worker threads if system updates lag, and be a measurable speedup over running everything immediately and sequentially.

Lazy implementation is easily fixable. But lazy design opens up a wormhole to a nightmare that the designer may rightfully be fired for.

As for system execution order, it really depends on how you track dependencies. Letting Unity fully decide the order only works if your dependencies don't need to be fulfilled within the same frame.

 

For your second problem, I'm wondering why not just use mono behavior and EntityManager to set your data? Also, conversion workflow is easy to use, you can use the code template to avoid too much typing.

 

A small suggestion: It would be great if methods that take a transform as a parameter (mostly thinking about Unity.Physics here) had more descriptive parameter naming than 'transform', ideally a standardized format like aFromB or bToA.

 

It appears the

EntityManager.RemoveComponent( query, type )

operation will corrupt data. In my case we tracked down physics collision breaking for the world to this call operating on a query which returned one entity and one completely unrelated to physics at that..

 

If you can reproduce it, please file a bug.

 

(Disclaimer: I understand this is still experimental code, subject to change, not for beginners, etc., etc. I'm describing an ideal end-state and noting some of the areas I've run into trouble with that I expect should be useful notes when the final code and documentation is released.)

The biggest problem I've found with the usability of the ECS APIs is gaps in documentation. It's a lot of little things that add up to a lot of frustration, for me at least. The general concept of data/entity/system was a pretty easy switch for me. I'm 20+ years enterprise line-of-business software, inversion of control/dependency injection, flexible and configurable systems, and I think the years of IoC/DI really got me into the mindset. I've never liked the whole MonoBehavior/every object carries it's data and code around concept.

It's the little gaps in documentation that trip me up:

• https://docs.unity3d.com/Packages/com.unity.entities@0.0/manual/shared_component_data.html: um, okay? So how do I actually make use of it?
  
• Or this from https://docs.unity3d.com/Packages/com.unity.entities@0.0/manual/entity_command_buffer.html:
  • "ComponentSystem subclasses which update on the main thread have one available automatically called PostUpdateCommands. To use it, simply reference the attribute and queue up your changes", followed by an out-of-context block of code
    
  • Sure, I know what a main thread is, but where do I find that in Unity? My MonoBehaviour Update? I thought I was getting rid of those with ECS
• Something as simple as creating a spawner system: I want to randomly select from a set of prefab entities (I managed to figure out that bit), and I've got some "fixed" data on each (e.g. prefab1 has maxSpeed = 1, prefab2 has maxSpeed = 1.5) that I'd like to have available as VehicleData:IComponentData on the instances so I can use that in the VehicleMoverJob. But there's nothing I've found to explain how to accomplish what to me seems like it should be both a) simple and b) pretty common. What I have found suggests that if the original entity prefab has a data component on it, it should be copied to the instance on Instantiate (it isn't), or that I need to use
  GetComponentDataFromEntity and then figure out how to avoid the "object is deferred" error by using PostUpdateCommands, which isn't available in a JobComponentSystem (I'm going to make a separate post for help on that - I'm only including it here to illustrate what I think is a not-uncommon use-case - I don't want to hijack this thread with non-usability support).

Those are just a couple examples. There are tons of examples I've found where something is "explained" often with a single line of code, or a very short block, with zero-context about how you would call it, what set up requirements are, that sort of thing.

One of the best articles I've found is https://gametorrahod.com/world-system-groups-update-order-and-the-player-loop/ from @5argon (their posts and blogs are generally excellent) which really fills in the missing pieces (Oh! It's just like MonoBehaviour in that Unity is automatically providing the main thread and calling my systems for me, unless I do something to do it manually).

I realize I may not be the typical target audience for this early stage of ECS (i.e. Unity novice albeit very experienced developer), but I thought it was worth sharing my experience in the hopes it helps others as this code is finalized and shaped into a fully supported package. Because when things do work, it's awesome.

Also, you should hire @5argon to write docs for you

 

Quick note: You may not have seen the latest docs, which are under the 0.1 path, not 0.0, e.g. https://docs.unity3d.com/Packages/com.unity.entities@0.1/manual/shared_component_data.html

 

or just use https://docs.unity3d.com/Packages/com.unity.entities@latest to get the latest one

 

Thanks @florianhanke and @rizu , that's very helpful.

It would be even more helpful if there was something on the documentation site that indicated if you were looking at the latest documentation or not (I mean besides having to decipher the URI).

A quick review still shows the same lack of context issues, so still something for that ideal future state.

 

@colin_young Thanks for the comment!

In reality from the start that blog wasn't meant to share knowledge, but I am always a type that can't learn without writing things down since high school. So they are all lecture notes of sorts. (hence there is no new articles recently since I am taking a detour to non-ECS project + I am waiting for the new conversion workflow) I am glad it works equally when someone read them.

 

Completely agree with you, and your other points.

In addition, I notice that there is a wealth of information in the forums, by many people, including Unity engineers, that does not make it into the docs. In the forums, it's quite hard to find, especially if one does not constantly read the latest entries. A technical writer that could and would compile these bits of information would be golden. Something which @5argon basically did, and btw, many thanks for doing it, helped me out a few times!

 

Agreed on the wealth of info in the forums, and that's a bit to be expected at this point.

On the official docs a simple (I think) fix would be to offer a version selector as is done on the main Unity docs. The Google results go to the 0.0 version, and I expect many people may not notice or even be aware there are newer versions. I figure that's easier for Unity to fix from their end than trying to get Google to update things.

 

My google searches finally go to 0.1 now, but with 0.2 around the corner which seems to bring some significant changes we're likely to run into this issue again.

I think a small bar up top with version selector but also a warning, go to latest documentation, would come in very handy.

 

I am commenting from a long-time Unity/Monobehavior programming background. In fact, I pretty much learned C# through making things with Unity. I have dabbled with some of the example DOTS projects and I'm very impressed with the performance that is possible!

From my perspective, the biggest drawback and why I am unlikely to attempt a DOTS project on my own is the lack of feedback in the Editor while the game is running. Is that going to change in the future? I've seen the debugger output of code that sort of does this, but it's not the same as seeing things in the Editor as they are propagated and running.

This is probably my own terrible way of debugging, but I like to hit 'Play', and then select the instantiated object and check if it's doing what I want in terms of coordinates, script linking, etc. Is that workflow incompatible with the DOTS way?

 

From this thread:

So it's on the roadmap to handle those things.

 

Back in Burst 1.1.3, I was compiling function pointers for burst functions that used NativeArrays passed by ref. After updating to 1.2, a console error told me to switch to using pointers, which I did. There appeared to be no change in performance or results, but getting it to work with the pointers required enabling unsafe code, in addition to using some of the unsafe utilities, and I assume bypasses the safety that comes with using NativeArrays. While this is fine, it's still a lot more work than just passing by ref as I did before. Further, later on I'd like to make a package using the code I'm writing to be distributed on the asset store, and would rather not require the users to have to manually enable unsafe code. Are there plans to add back support for function pointers passing NativeArrays by ref, or a way to not require enabling unsafe code for such things?

 

Is there any reason why DynamicBuffer<T> and NativeArray<T> do not implement System.IList<T>/System.IReadOnlyList<T>? It seems quite weird for array-like data structures to implement IEnumerable only.

 

For instance both type don't look like they would be able to implement the required Insert and retain their intended design

 

The intention of IEnumerable also isn't quite to be some standard interface wrapper around collection types, but rather closer to the iterator pattern. Coupled with this knowledge it thus allows a C# compiler to imply how to iterate the results which allows us to use foreach instead of a for loop to iterate array-like structures.

I think this is the only reason why Unity's collection types implement IEnumerable. @5argon also makes a very valid point that these structures don't conform to the C# interfaces which mean they're just not intended to function in the same way.

 

While this is true for NativeArray<T>, DynamicBuffer<T> allows insertions/removals by index. Still IReadOnlyList<T> would not break the design in any way.

Actually compiler does not require IEnumerable to allow foreach, it requires GetEnumerator() method only.

That said, there is one reason not to implement C# collection interfaces. Both of the types are structs, which means any cast to an interface would cause boxing. So the less interfaces used, the less chances for unintended boxing. The only problem with this explanation is that both types implement IEnumerable, so for instance Linq usage would be a problem...

 

I spaced on the fact that the C# compiler uses duck typing for the foreach construct. Using IEnumerable is an easy way to be "forced" to implement it for the purposes of foreach, especially if there isn't any cases in your library code using it as such.

Regarding the boxing situation it is possible to leverage interfaces as a generic constraint and then not have the issue with boxing. This is the main reason why our components are defined as IComponentData. It allows the compiler to enforce that something conforms to a specific interface while still keeping around enough metadata to prevent boxing when working with the type within the generic context. I do think that had the collection types implemented the .NET collection interfaces it would've meant a whole lot of unwanted .NET APIs would be compatible with the collection types making it difficult to enforce the "Performance by Default" nature of these types because of the boxing issues you mention.

 

1. Should we be open about "AddReaderWriter" (implicit/explicit) ?

GetEntityQuery and such adds a permanent state to the system. I think we should not abstract away what that is and let the user know it did something to the system design. For instance, it is not clear that GetSingleton would add reader dependency instead of "cheat its way" to get a one-off component data for us. If in the code docs mention this I think it might be a good idea. I was under an impression that I must RequireSingletonForUpdate to add real dependency. Also that method has problem on its own, it is not clear either that it overwrites all implicit ARW with this one explicit ARW, where GetSingleton appends to the implicit ARW. (Therefore all your previous GetEntityQuery are now ignored of ARW effect) I wouldn't have known about all this without seeing the source code and it affects system design decision.

2. GetAllUniqueSharedComponentData

The summary says "Gets a list .." but the argument is all about passing a list so it get added with result. The Get on the method name is fine, but the summary I think is weird that the argument is not an `out List` or just return a `List`. (I am not objecting to the design that we must pass a list in due to all the reasons, but the description or method name may need to reflect this design more)

Though the remark said "fill" the list, I don't know about others but still "fill" sounds like it make my list became purely a list of the answers. (Fill = "fill an entire capacity", but we know List can expand so we shouldn't use the word "fill" in the first place as it is more like a thing with fixed-length array) But in reality it performs an .Add to the list. (i.e. calling repeatedly appends the same information to the end.) So maybe change it to "append" to make it clear that you are supposed to .Clear the list if you are going to do this in every OnUpdate. (Use .Clear when SCD order version changes, etc.)

 

Not API strictly speaking, but Entity Debugging has a few usability gaps. The first is that the timing and "not run" info appears to be for the most recent frame or something. So if you have a job that only has entities to process once every n frames, where n is sizable, it shows a system as "not run" which can be misleading. Maybe a min/max/last or something on the timing.

Another issue is identifying entities at runtime. If I've got 20 AI vehicles active in the scene and I'm trying to figure out why they aren't showing up where I expect them to be, it's difficult to find the one I'm interested in and then to map the data points it is following to the entity those came from. In my situation I'm working with Open Street Map data that has been imported and mapped to a terrain in a world of about (X,Z) (-1000, -1000) to (1000, 1000), and vehicles will be at locations like (-791.23123, 1.983, 542.3423) (and it is likely interpolated between 2 points) and now I need to figure out if the lane entity that the vehicle claims to be following is anywhere near that location. I don't have a real good suggestion that's general purpose, although even being able to visualize any component data on the selected entity that looks like a 3D position in the editor when you've paused play would be a huge help. Also being able to highlight the current entity (assuming it has a visible component attached to it) would also help.

 

Are we doing something about how it is possible to choose wrong choice of Allocator? You know in many places like .ToComponentDataArray, Allocator.Temp is not a choice since the implementation use jobs. How about turn .Temp into .TempJob automatically inside? I think semantically that wouldn't make any difference, knowing just the "temp" aspect is enough for user.

 

eq.AddSharedComponentFilter this method has no documentation, and it is not obvious that the capacity for the SCD filter is 2. It was understandable for SetSharedComponentFilter since the argument goes up to 2.

 

Agreed ! i'm still not 100% sure what happens with GetEntityQuery and Jobs in a JobComponentSystem (wasn't there a connection some versions ago or is still there, where IJobForEach used the query you got in the system ?)
Anyway it would be nice if that background "magic" would be removed and we had to use an explicit call to something like "UseQueryForSystem(query)" or to correspond with Singleton Api "RequireQueryForUpdate(EntityQuery query)" and for Jobs i should also be able to explicitly state if i want them to work on a Query i pass in (maybe to Run()/Schedule() ) or if i don't use that parameter the job works on its internal Query (from the type-parameters) or from the Lambda parameters when using the Entities.ForEach-lambda function. Passing the query to the job would also allow for a job to run only on a subset of entities that fit its type-parameters, you could reduce the query selection with additional tag-components or set filter on it and such.

 

A little bit about documentation :

While this kind of sentence make perfect sense now, I remembered it wasn't convincing when I was a beginner. I get it that it was constructed to sounds like BaaS service where they would say it let you focus on developing awesome thing instead of housework. But by reading this (as a beginner) I didn't feel like data is the actual problem I am solving. I didn't realize that opening a chest or performing attack is a data, for instance. And instead feel like ECS is an even more roundabout way of doing them. I don't know how could this be improved (first you have to convince that everything are actually data) or should we left it as is as a riddle so beginner could get a light bulb moment later.

 

ecb.SetBuffer I think this command is confusing.. when what it actually does is more like ecb.GetBuffer. I get it that the purpose is so you finally modify the buffer's content returned (setting it) but because this command didn't do that just yet it feels awkward to call. (Also EntityManager only has Add and Get, better mirror the EntityManager one?)

While ecb.AddBuffer make sense since you are adding a new component and at the same time get a buffer ref for adding contents.

 

Good point and agree with ecb.SetBuffer / AddBuffer.
Is weir that all components are set after data been created, while buffer is set before data is created. Is very none concise.

 

I don't understand how ECS is good for stateful data, when there are a lot of specific relationships and sequential / dependent steps (I understand all data is 'stateful') but I mean the flow and transitions of behaviours on specific entitie types. Say if one was making a flow like a statemachine, and there were many such dependent flows in the game, is it usual to just make a component for every state and remove it, add a new 'tag state' component on that object? In a lot of cases there are specific relationships between entities that ends up looking a lot like monobehehaviour reference mess. I'm I the only one who is completely stuck with this way of programming? I understand how it all works perfectly nice for systems that run on independent entities the exact same way, but say even some game flow logic, or stateful entity behaviours etc. seem quite difficult. For any sort of complex input system would you have an Input entity on which to put all the different components like CursorPosition, Touch objects etc that all other entities can have access to, or just use monobeheviours for that?

 

I am working on a complex story driven game using ECS that requires a lot of state machine like logic, relational data, and sequentially data. Including seamlessly loading and unloading data as a player progresses through out the game. And from my experience, I have found that ECS makes this a whole LOT easier then OOP. But I totally get that it's hard to wrap your mind around how to deal with relational data with ECS. It took me a while and quite bit practice to get the hang of it. More tutorials and samples to teach people how to deal with relational data in ECS is very much needed. But here is a good talk on relational data that unity gave that will give you a little more insight how to work with relational data:

 

I agree. This caught me out before.

 

I'd like to have managed and buffer singletons. Not just struct : IComponentData.

Currenlty the API for getting my single array of something is more awkward than it needs to be.

I have to make an extra component type to mark the entity with the buffer I need, then call

EntityManager.GetBuffer<ByfferType>(GetSingletonEntity<ExtraComponent>) 

instead of just doing

GetSingletonBuffer<BufferElementType>()

I need managed singleton for things like NativeHashMap and some references to managed objects.

 

With the new FixedList variants, you can at least have an IComponentData singleton with a buffer.

 

What if NativeArrays could be activated with a compile directive [UseNativeArrays]?

 

Would like to subscribe to your newsletter. If you consider doing a blog or anything about it I think it will be very valuable for a lot of people.

 

I'd really like to see more idiomatic C#. As a long time C#/dotnet developer it's always been a bit of a culture shock switching over to unity.

It's mainly naming inconsistencies. I might try and compile a list of stuff that I've run into. I also loath using the new mathematics api. It's C# code it should follow C# conventions.

 

In newer API's int he "Unity" namespace Unity using the C# naming convention... with the ecxeption of the new mathematics package, which is difficult to use for programmers who are not familiar with shaders.