State of ECS & DOTS

Yup, that's exactly how it works. And great explanation on the 4 layers. I am very interested to see what Unity's answer will be for the scripting/orchestration layer. From what I gather, dealing with scripting and relational data is the biggest hurdle people have with adopting ECS.

 

That code:

Code (CSharp):

1. Condition.If(() => bugaMoveToLighthouseTopPatioAction.HasPlayed() && distance(gigi, buga) < 5)
2.     .Execute(() => SceneData.LoadSnapshot<LighthouseScene, BoatRescueSnapshot>());

Could be rewritten like this:

Code (CSharp):

1. void Update() {
2.     if(bugaMoveToLighthouseTopPatioAction.HasPlayed() && distance(gigi, buga) < 5) {
3.         SceneData.LoadSnapshot<LighthouseScene, BoatRescueSnapshot>();
4.     }
5. }

The original code is more complex, less readable, and less performant than the equivalent non-ECS code. It's like generating CSS from a SQL query.



Which was basically what that Will dude was saying in that video -- the data structures of ECS are not appropriate for all use cases. The issue with ECS is that it's been advertised (and is being evangelized) as some sort of universally better programming paradigm that is going to solve all your problems, and that hybrid ECS workflows are so difficult.

EDIT: Sorry if this came off as attacking you; that wasn't my intention. Your code looks like a pretty awesome hack, especially if there's not too many scenes that need it. But it could be better if Unity had more support for hybrid workflows.

 

That's not actually the same thing. The Condition.If doesn't evaluate the expression. Instead, it stores the expression inside the ECS data structure for another system to pick up and execute. All the system is doing is generating these things and setting data. It only runs once, not every frame.

What you are suggesting is a different form which would have to exist inside a coroutine. Trying to express high level behavior in that form (ticks versus conditions) inside an ECS is a very bad idea unless you have some form of coroutine abstraction. I may take a stab at building something like that once source generators arrive and after I finish my animation system.

Once again, the issue isn't ECS, it is the premature state of DOTS today.

 

I got that.

But it seems like that condition is a lambda that needs to be evaluated somewhere, right? And I assume it's being evaluated every frame. On the main thread, since the condition evaluator system won't know the set of dependencies.

So you're still evaluating it every single frame, on the main thread, only with an extra virtual call, more allocation, more layers of abstraction, and direct access to the EntityManager which is going to be slower than the equivalent Transform code. It gains nothing in terms of performance or readability (and is now 10x harder to debug and teach to new developers).

The other alternative (that I believe is not supported in IL2CPP) is that it is using the Expression tree of the Lambda like LinqToSql and somehow building queries out of that. Which would be neat, since those queries could be executed in jobs. I'm still not convinced that would be easier to read, write, or debug for general scripting code, but it might be better than chaining systems and query-result entities like you have to now in ECS.

 

Coroutines are trivial to translate into ECS.

Code (CSharp):

1.     public class timerSystem : SystemBase
2.     {
3.         protected override void OnUpdate()
4.         {
5.             //update reload timers
6.             Entities.WithBurst()
7.             .ForEach((ref timerComponent timer, ref weaponComponent weapon, ref weaponComponentAmmo ammo) =>
8.             {
9.                 if (timer.reloadWeaponTimer > 0.0F)
10.                 {
11.                     float t = timer.reloadWeaponTimer - worldLoop.MyFixedTimeStep;
12.                     if (t < 0.0F)
13.                     {
14.                         t = 0.0F;
15.                     }
16.  
17.                     if (t == 0.0F)
18.                     {
19.                         //do something
20.                         weaponFunctions.whenReload(ref weapon, ref ammo);
21.                     }
22.  
23.                     timer.reloadWeaponTimer = t;
24.                 }
25.             }.ScheduleParallel();
26.         }
27.     }
28.  
29.     public static class weaponFunctions
30.     {
31.         //call when weapon needs reloading
32.         public static void onReload(ref weaponComponent weapon, ref timerComponent timer, in weaponComponentAmmo ammo)
33.         {
34.             if (weapon.ammoInClip != ammo.clipSize && timer.reloadWeaponTimer == 0.0F)
35.             {
36.                 weapon.reloadState = reloadStates.reloadState.reloading;
37.                 timer.reloadWeaponTimer = ammo.reloadDelaySec;
38.             }
39.         }
40.  
41.         public static void whenReload(ref weaponComponent weapon, ref weaponComponentAmmo ammo)
42.         {
43.             weapon.reloadState = reloadStates.reloadState.loaded;
44.             float ammoLeft = ammo.amountOfAmmunition;
45.  
46.             if (ammoLeft > ammo.clipSize)
47.             {
48.                 ammoLeft = ammo.clipSize;
49.             }
50.  
51.             ammo.amountOfAmmunition = ammo.amountOfAmmunition - ammoLeft;
52.             weapon.ammoInClip = ammoLeft;
53.         }
54.     }

 

I don't know the details, but I'm willing to guess this was a non-DOTS ECS where these things were the norm. All those problems used to be problems with Entities.ForEach until Unity added codegen. There's really nothing stopping the same to be true here in the future.

I take it you are either unfamiliar with or dislike declarative programming, because that's what this is. And I find it to be a quite readable implementation.

The good news is that declarative scripts and coroutines are both valid technique for describing high level behavior, despite having very different thought processes. And both of them can easily be made jobs and burst friendly with source generators. Heck, the declarative scripts might even be possible today if someone can figure out how to recover the concrete type from a void* in Burst using a concrete generic writer at store time.

My point is that just because the DOTS Entities ecosystem is incomplete, that doesn't mean ECS is bad or that DOTS won't be able to support these genres and use cases in the future. But people who try to do high-level scripting of Gigis and sunsets using per-frame systems, queries, and chunk iteration are going to feel a lot of unnecessary pain.

 

I really hope they don't add coroutines. They would break network roll back. ECS should remain stateless.

 

With the implementation I imagine, the actual state of coroutines would live on components attached to entities, which means the coroutines could be rolled back too.

 

Imagine future where your mostly normal async ECSTask method would be converted into helper IComponentData struct with Switch Case states from method so we can write burstable jobified async logic for ECS

 

What about

yield return

?

 

I'm not against declarative programming where it's needed; back in the day I wrote a whole binding framework for WPF converters to solve some issues with state management. But in this case, the "dclarative" aspect is just adding noise. It might be better to go back to my original example:

Spoiler: code

Code (CSharp):

1. private IEnumerator<IActivity> main(IScriptContext act)
2. {
3.     // turn the player towards the NPC and the NPC towards the player
4.     act.turnTowards(transform, act.player, true);
5.  
6.     // The NPC is playing an animation with another NPC in the background; pause that animation until the script completes
7.     act.pause(backgroundScript);
8.  
9.     // Get some data about the dialogue boxes (eg positioning them relative to the person talking,
10.     // face sprites to display, etc)
11.     var mPlayer = act.dialogueBox(act.player);
12.     var mSelf = act.dialogueBox(transform);
13.  
14.     // this shows a dialogue box and waits for the player to press a button to continue
15.     yield return mPlayer.say("Hello!");
16.  
17.     // now the NPC will jump up in the air and say some text. we wait until both actions are complete before
18.     // moving on with this script
19.     yield return act.animate(transform, AnimationId.JUMP).background(out var anim);
20.     yield return mSelf.say("I'm jumping and saying something at the same time. Isn't that cool?");
21.     yield return act.waitFor(anim);
22.  
23.     // at this point of the script, we check the player's inventory to see if they have a key item
24.     yield return mPlayer.say("Uh oh, you must be possessed by an evil spirit!");
25.     if(act.inventory["beer"] > 0)
26.     {
27.         yield return mPlayer.ask("I don't have any holy water, but this could be a good substitute...",
28.             "Give Pabst Blue Ribbon", "Keep Pabst Blue Ribbon");
29.         if(mPlayer.choice == 0)
30.         {
31.             yield return mPlayer.say("Here try this");
32.             act.inventory["beer"]--;
33.  
34.             // play 2 animations in parallel so it looks like the player is giving an item to the NPC
35.             yield return act.animate(act.player.transform, AnimationId.GIVE_ITEM).background(giveAnim);
36.             yield return act.animate(transform, AnimationId.TAKE_ITEM).background(takeAnim);
37.             yield return act.WaitFor(giveAnim, takeAnim);
38.  
39.             // once those animations are complete, do these things in order
40.             yield return act.animate(transform, AnimationID.DRINK);
41.             yield return mSelf.say("My soul has been saved thanks to the refreshing power of PBR!");
42.             act.persistentData["expelled_evil_spirit"] = true;
43.         }
44.         else
45.         {
46.             yield return mPlayer.say("I'm sorry; I don't have any holy water to give you right now");
47.         }
48.     }
49.     else
50.     {
51.         yield return mPlayer.say("I'm sorry; I don't have any holy water to give you. I wonder if I could find a substitute...");
52.     }
53. }

Which isn't really appropriate for declarative programming even if you drank a supermarket worth of Kool-Aid.

Yes, if you sprinkle enough code generation on DOTS, you can translate code that already feels natural in OOP world into something that works in DOTS. But if you need tons of code generation, isn't that basically what I've been arguing -- that the programming paradigms of DOTS are not appropriate for this type of code?

I would also posit that it's going to be less efficient. Let's say you translate this (sorry for stealing your names,
Vacummus, but Gigi and Buga are just too catchy; mind if I name my firstborn that?):

Code (CSharp):

1. Condition.If(() => distance(buga, gigi) < 5).Execute(() => sunset())

Into some kind of set of systems. I'm guessing one would be a "distance system" which would take pairs of Position components and emit distances, then some sort of system that would emit events for true conditions (or skip the first and do the distance lookup using the EntityManager), and then a system that would consume those and execute sunset. Well, that's a ton of bookkeeping on the main thread and 3 jobs that each do almost nothing.

It wouldn't matter for only a few scripts, but if you had hundreds of scripts with unique sets of conditions over small entity sets, that would be hundreds of jobs, each of which gum up the main thread with scheduling and many would need to run sequentially since they'd be accessing the EM. In database terms, the cardinality of distinct data types and queries is high while the actual series are tiny. Which gets way back to that Brian Will dude's comment that ECS is not a universally perfect data structure.

Anyways, I'm not really sure what I'm arguing at this point. Just that I tried with ECS to make that type of game work, and it was way more trouble than it was worth. While with certain genres/tasks, the programming in ECS is more natural, the common case is that using ECS is trading developer productivity for runtime performance. Which is a fine trade to make if you get better runtime performance. But for games with lots of one-off experiences, ECS as it currently stands is both awkward to program in and slower than the equivalent in MonoBehaviour/OOP land. Kinda wish Unity would make hybrid ECS/MB work better (that would also open ECS up to all the many asset store things, etc).

 

That's the thing I was trying to make work last year, but didn't get past a prototype. The underlying mechanics of ECS - in particular handling references and entity lookup - are unsuited for what I needed it for, so I gave up. But I can see situations where it would be useful even in current DOTS (for example if you're making a bullet hell game and you wanted to yield return a set of patterns for the enemy to shoot out, eg...

 

Yes. The way you reasoned about this cutscene isn't appropriate for declarative programming because you reasoned about it as a coroutine. It is just like how a signed-distance field renderer isn't appropriate when your scene is composed of polygons. My point is that both SDF renderers and polygon renderers can handle AAA graphics, the former authored using thumbsticks. In the same way, both declarative programming and the coroutine approach can handle complex cutscenes you often find in RPGs. Yet how you work with them and reason about them is completely different, and different people are wired differently.

I would agree with that for the state of DOTS today, because currently code gen is done with cecil. However, source generators make code gen part of the language, which isn't that much different than how polymorphism really just code gens switch cases in the form of vtables.

I would translate it into a single system which parses the condition expression structure and evaluates the expression using an operation table or function pointer, then if true queues the reacting events.

Which ECS were you using? Because if it was DOTS, then I agree it was probably more trouble than it was worth. But that has more to do with DOTS being unfinished and not having the tools in place to make that comfortable.

There's a difference between DOTS Entities and ECS architectures in general. And there's a difference in what DOTS Entities is today and what it can become if it continues down its current trajectory. You are 100% right that today it is easier to make RPG cutscenes using MonoBehaviours over DOTS Entities. But with regards to RPG scripting logic, no one else here was talking about DOTS ECS today. We were talking about ECS in general and the potential future of DOTS Entities. Maybe it is just me, but in that light your dismissive comments strike me as ignorant.

 

No; I don't think you could write a cutscene like that in a declarative manner and have it not be awkward. You could "declare" it with a different scripting language (not a C# coroutine). You could "declare" it with a visual/node-based editor. But the cutscene consists of a linear series of activities and branches.

And that system would have access to the entity manager and be able to query every component directly? Then it's essentially no different than evaluating the code directly every frame (eg in a MB update), except it's a bit slower because it's an interpreter. This simply adds a layer of complexity.

I am saying that if it continues down its current trajectory it still will not be a good fit for this type of system or provide value. It needs to change trajectory to make hybrid ECS/MB more viable.

My comments are based on an attempt to actually implement it in DOTS ECS (as of about a year ago). If they are ignorant, it's an ignorance born from experience, from actually attempting to do this in idomatic ECS, to translate OO code into Burst-compatible code and run it in jobs, The data structures and systems of DOTS both make coding it harder and make the underlying accesses slower.

The solution that you seem to be suggesting - and the one AAA teams would take - is to add layers upon layers of abstraction. If you're evaluating Lua/XML/whatever scripts, some visual scripting language, or a codegenned rewrite of your C# scripts, then you can build up a set of systems upon which to execute scripts. I question what value ECS would add to the script execution, but the value of other aspects of ECS can be worth it. That is an argument for hybrid ECS/MB, though.

Again, though, the value propositions of ECS - in particular, the data layout and multithreaded access - do not apply to the scripting scenario. At *best* you find ways to work around them, at worst, they make coding more difficult and are a performance drain.

 

I don't think coroutines are great (or ever been).
They're a workaround made by Unity to allow deferred execution. Complex one can be acting like a FSM.

They aren't great to support and maintain.
Moreover, coroutines can be decomposed into multiple smaller logic blocks which can be chained to execute one by one. Implementation wise - it may depend.

There's no 1:1 translation of coroutines to Entities, and that's a good thing.
Coroutines are a mess:
- They bind their current state (data to logic), and you can't serialize them if they're complex enough.
- GC allocation from StartCoroutine;
- If implemented incorrectly -> GC from every yield.

Sure they're useful for prototypes, but once you're out of that phase - you'd better use something more appropriate for the specific task.

Timers / delays can be easily implemented in Entities. FSMs a bit more tricky, but still doable.
The main problem here, is an attempt to fit OOP logic into DOD as is. It doesn't work that way.

In DOD timers would be nothing but a data, "attached" to the entity.
And FSM would be a "list" of data with conditions to evaluate and switch at.

Problems arise when attempting to fit all the logic into one single system, where in reality it should be multiple data processing steps.

 

You can also look at a cutscene as a DAG of events and responses. This better represents "parallel tracks" like what you have in Timeline. With such a representation, a single-frame script (no a coroutine) declares these event/response interactions.

Does a compiled single-threaded Burst job for all entities processed by the system sound slower than a MonoBehaviour to you? With source generators, you can perform dependency injection at compile time.

What about its trajectory makes it bad? Why would a co-routine-like mechanism implemented similarly to Entities.ForEach be insufficient? Why would an event-response mechanism implemented similarly to Entities.ForEach be insufficient?

The ignorance I am referring to is when you refer to a successful implementation of a technique you dislike as "adding noise" or "drinking Kool-Aid". I don't have a problem with your technical arguments but these sorts of comments need to stop if you want people to respect your opinion.

Just one layer is enough, but yes that is what I am suggesting. And those "systems" are ECS systems. And the interop between the scripting layer and the lower layers takes place via ECS mechanisms from inside ECS.

I don't see how MBs provide a better interop experience. Care to articulate?

The benefit is that they make everything the scripting layer relies on fast, predictable, and well-behaved.

 

I also think that sometimes people get too optimistic about DOTS/ECS and too quickly dismiss real-world cases where this tech doesn't provide any significant benefits, only drawbacks.

Say, if you develop 2d games, of puzzle or other genres that don't require hordes of gameplay objects, you have your production pipeline working, tried and tested, with lots of reusable code accumulated over the years. There are no performance problems, Unity is plenty fast for such scenarios, and if some extra effects are needed - particle-like hordes of objects may be easily added using Job System, without the need to convert the whole gameplay code to ECS.

In that context, a possibility of Unity to become ECS-only with GO/MB deprecated seems definitely not positive: you loose your tried and tested approach, throw all your accumulated reusable code in the window (as it was GO/MB based) and you're forced to learn completely new approach to coding, writing a lot of code again, messing with trivial stuff like events and state machines that were solved and just worked in GO/MB land long ago.

 

Do we have any estimation on the next entities/hybrid renderer update? Its been almost a year now.

 

On SIGGRAPH 2021 Unity has presented new version of HybridRenderer so they actively work on it.
But update, I think, will arrive around release of entites 0.20 or something

 

I already wrote this but again: Unity ECS is not DOD by default it is just pattern and Unity GO Component is the same pattern. So even is Unity (for some reason) deprecate GO MonoBeh approach for runtime (which is unlikely to happen) then We (Community) can easily create top level abstraction for ECS that will have API of old good GOs because ECS have all needed tools for that. I even propose to replace low level layer of GOs from native C++ implementation to ECS to support only one low level system instead of 2 ones.

 

I haven't tried to work with Unity Entities, only examined some examples, but I did work with Job System and Burst compiler, and as Entities package is based on Jobs/Burst, at least currently it's quite an unorthodox way of C# coding with all those native containers, severe limitations on accessing reference type objects in jobs, etc. I have hard time imagining how it may be used as a backend with a top level GO abstraction that would work in a usual C# way, even more unrealistic to expect old GO-based code to work with the new abstraction as is or with slight changes.

Long story shorts - all this DOTS stuff is solving a problem many of us just don't have, instead we want our old code to work, that's it.

 

It's solving a problem that many in the community don't have, but that many others do have.

Performance of Unity games has been a widely-known & widely-criticized issue for over a decade. So much so that seeing the Unity logo at the start of a game has now become synonymous with "ugh this game isn't gonna run well" in most non-dev people's minds. My day job gives me the opportunity to work on all kinds of projects with all kinds of studios, and I've rarely seen projects whose development wasn't severely slowed down by performance issues. Even projects that look simple and don't have visible "hordes of things". There's a Unite presentation about the game 'Inside' somewhere (a simple-looking sidescroller with very few things on screen), and they go in detail about the many performance roadblocks they've encountered while creating this game. Level streaming was a big one, and this is something that DOTS solves. The GameObjects scene format simply wasn't made for efficient streaming

DOTS is a tool that makes so many problems go away, and can cut several months of development time off of your project (which would've previously been spent on optimization). It will allow us to make more impressive games more easily. For those who aren't limited by vanilla unity's performance, DOTS may seem like it's not worth it. But for those who are, it's a game changer. The extra dev time cost of DOTS boilerplate is a tiny insignificant grain of sand compared to the time cost of performance problems & spaghetti OOP architectures that often necessitate large refactors

And then, there are the other "indirect" advantages of DOTS:

• the ECS pattern has its advantages over OOP in terms of ease of architecting your code & keeping things clean/scalable
• the source code of almost everything is fully available and modifiable (ECS, rendering, physics, animation)
• DOTS's APIs tend to be better designed than their monobehaviour counterparts (for example; DOTS physics lets you do a lot of crucial things that were impossible in mono physics, because it didn't properly expose a lot of PhysX stuff)

Adding OOP programming to a DOTS project is a trivial task and is already supported, but adding efficiently-implemented ECS/DoD modules to a Monobehaviour project is much more difficult, limiting and/or wasteful due to data conversion steps. For that reason, I think the best way to support both possibilities is by making a DOTS-first project rather than GameObjects-first. A GameObjects-first project will never be able to benefit from DOTS instanced renderer, or fast level streaming, or DOTS physics which performs about 10-20x better according to my tests

 

I really like the whole ECS idea especially since you have access to all the data regardless if you own it or not. This makes integrating new things into it or reusing some parts so much easier.

But my big issue with DOTS is modularity within a system. I more or less design everything I make in a modular way with replaceable interfaces or similar so it's super easy to adjust in a plug any play kind of fashion. In a few cases you can do that using generics but more often than not its not enough.

You would have to copy paste the whole system for each different type and hard code the adjustments. For me that simply not maintainable.

 

Don't have time for a full response right now, but yes.

EntityManager.GetComponentData<Translation>(entity).Value

is slower than

transform.position

. If there's an additional function pointer or lookup step before the evaluation, that just adds runtime.

Burst is a ton faster than Mono, but that's Mono being an old technology (and Unity using an older version of it); if/when .NET 5 lands, the advantages of Burst will shrink: https://forum.unity.com/threads/hpc-vs-net-core-3-benchmark.762857/

 

A lot of the problems we solve in real games don't fit ECS well. And this is where Unity has always gone off the rails. They don't solve high level flows well. Their goal is to make their framework work, and make everything fit it. That seems like the right approach from their side, after all the want it to work for most of what we do.

In practice that results in Unity doing things that work directly against what game developers need. The priorities are in conflict way more often then would be ideal. And this has been true of Unity basically forever. Pick an engine feature you can see this pattern.

So looking at DOTS from the perspective of waiting for it to be finished before using it makes absolutely no sense. There is a lot here that is worth leveraging right now. You pick and choose what fits well, find your own patterns for what doesn't, and move on.

 

MS runtimes don't have embed api's. And cross platform/IL2CPP also diminish where mono is used. Add burst to that mix and it's not obvious why they would ever move to a MS runtime. It seems more likely to me that they just keep moving more to their own runtime and gradually add more language support to burst. At some point IL2CPP isn't even needed.

.Net 5/6 are fast but for memory management native is just a better fit on the client. GC isn't all about object creation. It's also a lot about how much is in gen 1/2 and how you deal with memory fragmentation. Because those impact stop the world collection times. If you use tools like the new pinned object heap appropriately combined with performance first design you can get amazingly far in .Net 5, but it's still just not as good as the strategy Unity has taken with burst and native collections.

 

Just ran an embarrassingly dirty test (Editor + Mono + Entities 0.16, can't be bothered to properly measure this in an IL2CPP build) and EntityManager.GetComponentData<Translation> is 40% faster than transform.position if you're iterating over 5k objects, and only 28% slightly slower if you're accessing the same object every time. Either way, is that really a hot path in your code? Have you considered using ECS? I hear it makes component access real fast.

I vote that we nuke the DOTS forum and get back to programming

 

...or instead of accessing directly on main thread, access and logic can be just offloaded to the job and cost only job schedule cost. (if there's free time on other threads ofc)

Working with TransformArray (allocation, reallocation for Transforms) on the other hand... is fast as punching a turtle. Then inability for the engine to compute transforms in parallel if they've got parent. It adds up very quick. Transforms suck tbh.

Transform component as entity data when?

 

First off, you are unfortunately incorrect about this. You see, transform.position isn't a field, it is a property. None of the transform's data actually lives in the C# object. It is all deep inside the engine, meaning that there must be a second address you follow from the C# object to get the real data. So unless you have source access and can tell us whether Transform contains a direct pointer or has to jump through even more hoops, you have no idea if it is actually slower or faster. Just that it appears to be less operations from C# because there's an abstraction layer.

However, let's pretend that there is a world where Unity's C# runtime was on par with Burst and Transform was twice as fast as EntityManager. How often are you going to touch Transform in your scripting layer per frame? It is going to be pretty small, right? Perhaps, it is even negligible compared to physics step that checks for collisions and contacts across all potential pairs of colliders and rigidbodies in the world. Then there's animation sampling. Also there's the hierarchy update. And there's frustum culling which has to check culling states not just for the main camera but also shadows. And that's not counting core gameplay logic and tweening and whatever else. All of those things are doing far more computation on far more objects than your high-level scripts.

Here's the thing, there's tradeoffs. If you try to optimize the high-level script by letting the high-level objects store direct addresses to the underlying data, then suddenly the underlying systems can no longer move that data around, which means it has to work with fragmented arrays. That introduces logic and branches in the inner loop. You've now sacrificed measurable performance on your heavy data just to get more performance in scripts that have a near negligible impact.

This is why I propose that ECS can work for a large variety of genres and why it is also important to keep scripting on a separate layer. There's a lot of performance-critical stuff that happens in a game, and ECS can help make this go very fast. Some of its gains are lost due to the suboptimal scripting, but overall it is a net gain.

So really, I have no idea why you argue for MonoBehaviour for performance. That just isn't true unless you are doing something wrong.

If you want polymorphism and references, class components exist in DOTS. If you want coroutines, source generators are coming. If you want compatibility with existing assets, well, you have a valid complaint there. But you should be complaining about how hybrid breaks things that it shouldn't. But any arguments that an ECS should not power an RPG are simply not true. That doesn't necessarily mean it is the best fit in all cases. There's other cool tech out there which makes different performance tradeoffs. You can't expect Unity to provide the best solutions for all scenarios. Check out some of Bobby Anguelov's videos. I think his engine is a lot closer to what you want than Unity will ever be.

I absolutely agree with you on everything you said other than this point. I don't think it is the problems themselves that don't fit. I think it is Unity's implementations that don't fit. They've sacrificed a lot of modularity and customizability in their high level systems by designing those systems to work without requiring custom code. But I am quite curious what problems you think don't map well? The only problems I haven't been able to map are really OOP-style solutions trying to be adapted to DOTS.

 

I think I can name 4 main types of things that are complicated in DOTS right now:

1. Problems that involve calling a variety of different functions in a specific order that can't be pre-established (ex: an ordered events system)
2. Lack of "instant changes". The transform system is where this is most painful. If you change the pos of a parent, you can't immediately get the updated world pos of a child
3. Difficult to create a job that users can extend with custom functionality. Sometimes it can easily be done with generics, but sometimes it can be more complicated (like if users want to read a component type on another entity, but that component type was already chunk-iterated in the original job)
4. You may often end up with a solution to a problem where you'd have to schedule so many ForEach jobs in order to make it work that the scheduling cost alone would kill your performance. The alternative in this case is to use a switch case instead of scheduling all of those jobs, but that's manual work. This might actually be the broader problems category that encompasses #1

But at the same time, I also think the benefits of DOTS outweigh the difficulties it creates. Especially since we always have the OOP fallback when we can't figure out how to implement a thing in ECS. So the worst possible scenario is that our thing we're trying to implement will have the same performance as monobehaviour

 

Can you give a specific example of such a problem that requires ordered events or functions to be called within a single frame? I'm sure there's one that exists, but oddly enough, I can't think of anything other than behavior trees, and that's a solution to AI, not really a problem.

I don't think this is so much an issue with it not mapping well to ECS as much as it is an issue with Unity not providing a utility for this, so you have to do it yourself. Of course you can't get free reign over all aspects of the transforms in parallel without some form of restriction. But you can have free reign single-threaded. And for multi-threaded all it takes is a simple restriction like limiting the scope of accessible transforms to a group of hierarchies per thread and maintaining a hierarchy cache. I've had to do this in C++ before, and I will be doing it again in DOTS for IK.

That's a really tough problem multi-threaded because there's a lot of safety concerns. However, if the job is single-threaded, or you can structure the API such that users can only get a readonly CDFE, and you only read from chunks, then you can use [NativeDisableContainerSafetyRestriction].

This is a problem I haven't encountered yet but I could see being a problem. There might be something you can do with generics of generics and chunk arrays to get a single IJobEntityBatch which automatically filters each chunk to a query and invokes an approapriate IJobForEach-like interface on each entity. But it is also a problem I believe Unity is working hard at improving.

Anyways, thank you for taking the time to list these out. It is very insightful.

Perhaps I should clarify how I define whether a problem maps cleanly to DOTS. I don't consider it solely based on whether or not Unity provides that utility. Anyone who does I think will be disappointed by DOTS 1.0. Instead, I base it on whether or not I can write a utility once that solves the problem cleanly for every future instance I encounter it.

With that said, if anyone has more issues that they have encountered (problems, not generalities or solutions), I would love to be stumped! Plus this style of conversation is much more insightful than the complaining that has been going on lately (not that I disagree with the complaints, but still).

 

You're repeatedly going to great lengths to list DOTS benefits, but my point is that there is a lot of developers that don't expect any benefits, they expect their old code to work, period. It's also frustrating how in "things that currently are complicated in DOTS" list most items are trivial and solved in GO/MB land.

In case if Unity will go fully DOTS/ECS and current GO/MB approach gets deprecated, it will be almost as complex task to adapt to DOTS/ECS as to adapt to a completely new engine (except for the editor familiarity that will probably be kept to some degree at least). And if forced with a need to adapt to new engine, it may become more reasonable to look at some engine different from Unity. Say, both UE and Godot are using traditional OOP approach and can be felt more familiar. Btw, in both cases, authors are vocally against ECS:

Tim Sweeny (UE): https://twitter.com/timsweeneyepic/status/1009269785320984576?lang=en

Juan Linietsky (lead Godot dev): https://twitter.com/reduzio/status/1365325838720389124

And there are quite complex games that are done without using ECS, like Teardown: https://teardowngame.com
and their authors are skeptical about ECS as well: https://twitter.com/tuxedolabs/status/1174432346063429633

The bottom line is that the picture that is often being painted here about how the future is DOD and OOP is fading away is actually far from being the truth. It's looks more that when you've got a hammer everything starts to look like a nail.

 

Just let you know you still can expect old code to work.

 

Epic will use ecs for ai(EDIT : actually it seems it will be a plugin u can use for what ever u want not only ai) in ue5 and Unity officials said many times(hope nothing changed meanwhile ) both tech(GO classic unity and dots) will continue to be developed in parallel : https://twitter.com/SiggiGG/status/1443480260633366531

 

Have to agree with @PhilSA here.
DOTS/ECS would solve a lot of issues for multiplayer games in Unity.
Virtually everything in GameObject world gets in the way.

• Character controllers won't scale past a few hundred CCU.
• Navigation is fixed on N iterations per frame.
• Hierarchy can only handle so many GameObjects - consider thousands of monsters & players.
• Impossible to have in-memory worlds for host mode, instanced dungeons, etc.
• Dependency injection isn't easily doable, if at all.
• Tests are super hard.
• Servers are launched into a giant GameObject black box. If your server encounters a critical Unity bug, you are screwed.
• Performance, obviously. It just won't scale much past 300-500 CCU. Endless headaches.
• Managed types get in the way of delta compression. Decide between performance (FixedString) or usability (string) for networked types.
• Nearly religious OO patterns that everyone argues about all day.

For virtual worlds, it's really just data transformations if you think about it.
For 1000 players, update their movement / combat / skills.
Gather the N monsters around them.
Delta compress everything.
Send it to everyone.
etc.

I still have high hopes for DOTS/ECS, it would give small indie teams the ability to do large scale multiplayer games.
Think DayZ, which has to update thousands of players/zombies/weapons/bushes.
Or EvE with their gigantic worlds.
MMOs with in-memory dungeons, all the monster AI etc.
It's endlessly exciting to think that we'll be able to do these types of games by ourselves.

Performance aside, I am even more excited about open source packages.
Finally the freedom to fix critical bugs ourselves.

That being said, the way Unity completely ignores the community who helped them grow, defended them against Unreal and stuck with them through thick and thin is disappointing to say the least.

People should not have to beg for status updates.
Threads asking about DOTS should not be closed.

 

I've created a "Stats & Stat Modifiers" system, where you can have rpg-like character stats like strength, intelligence, magic resistance, attack damage, etc....

A stat can have modifiers that have different operation types, so there must be a concept of applying them in the correct operation order

A stat can also react to the change in value of another stat, and have a custom evaluation rule. For example:

• my Intelligence stat is raised
• since my MagicResist stat depends on Intelligence, we trigger a recalculation of the MagicResist stat too
• MagicResist = (0.5 * Intelligence) + (0.2 * Strength); (and then apply all other regular modifiers for MagicResist)
• if we have a spell that buffs our Intelligence or our Strength, our MagicResist must be recalculated based on the final buffed Intelligence/Strength values

So there are two things that use order here: the order of modifier operations, and the order of stats that depend on the final buffed value of other stats. And since some units can have buffs that make StatA depend on StatB, and other units can have buffs that make StatB depend on StatA, we can't pre-determine an order of stat calculation by stat type. And also; stat A can depend on B, which depends on C, which depends on D, etc.... the depth of stat dependency can go far

One more thing: there could be thousands of stat entities in a scene (easily dozens per unit / item), and the system must be optimized for relatively infrequent stat changes. So minimal frame cost when nothing is being changed

There are many different ways we can implement this, but after thinking about it for a while, I've settled on:

• a stat is an entity with a dynbuffer of "StatModifier" and a dynbuffer of "DependantStats"
• we keep a singleton buffer of dirty stats entities
• triggering a stats update means going through each dirty stats in the buffer, applying their stat modifiers in order, and adding the stats that depend on it to the end of the dirty stats buffer for recalculation
• stat modifiers a struct that do a switch case based on their operation type, and are applied in order
• all stat change operations are done through a buffer of "change commands", which automatically handles setting the stat dirty
• a person could add some jobs to run after the stat update, if they want to write back the final calculated float value of the stats directly into components, for maximum efficiency (instead of always getting stat value through entity). But this is a lot of manual work
• custom stat evaluators (like in the magic resist example) are just another type of stat modifier that we must add to the StatModifier struct's switch case

This implementation doesn't feel perfect to me, but it seems to be pretty efficient in practice at least. It was pretty dificult to solve this problem though. I must've gone through at least 3-4 different implementations before this, and I was noticing flaws every time

 

I would do this by maintaining a number of different nativelists in a system; no entities required. One list for each physics type of projectile or spell. You call a method in the system and add a spell or projectile data struct to the list. This is how my rollback temporal physics works I can enter a time stamp and shoot projectiles back into the past deterministically. As you iterate over the list you recreate the lists with the current alive projectiles.

You could do this with entities, but I found no need to roll back my anti-lag roll backs. In fact I wanted to maintain this data only in my lock step world. After it gets synced across the network.

 

My main motivation for stats as entities is to make it easy for users to create custom OnStatChanged behaviours for specific stats. We can add a component to them that gets picked up by a job that looks for this component type + an OnStatChanged tag (or bool)

There's also the fact that each stat must hold its own stack of modifiers and its list of dependant stats

Sometimes we also need to be able to "keep a reference" to another stat, as is the case with MagicResistance needing to know the value of Intelligence

 

In my case I don't care about third party programmers. Cant you maintain modifiers and case statements in static modifier functions?

 

My impression is that this would be too limited if the onStatChanged behaviour needs to hold instance data and/or have read/write access to user-created component types

 

Well then MagicResistance and Intelligence needs to be on your ComponentData. But that would be attached to your impact target entity, I assume not to your projectile.

 

What about the use case of the buff that makes my Strength be 125% the final value of another unit's strength? In this case we need to reference a stat that can't live on the same component or entity

 

I would add two fields an buffing Entity ID and a bool field "buff125Precent" true or false.

But updating a buff on less then a per frame basis is not logical. Really you should have a buff timer.

 

I dont want to be too annoying with my "but what ifs", but.... what if the % of the modifier can scale procedurally based on other stats, and doesn't just stay hard-coded at 125%?

It sounds like I have all the special requirements that make this difficult to implement, but that's really what I do need for my project

 

Ya you would need a buffing Entity ID and a float buff value.

But I don't think it is a logical use case. Transforms don't even get updated like that. For ECS to make sense you need to update your logic per frame.

 

You can just imagine a Diablo character that has a spell that does "Target ally's strength becomes 100% of your strength + a bonus that scales with your Intelligence". And when your Intelligence changes after casting the spell, the target ally's strength must react & update automatically

If those are my requirements; those are my requirements. I can't just change my design to make it fit the tech; it's the tech that must adapt to my design instead. I need to find a way to implement this no matter what tech stack I use.

Now, just because this one system in my game isn't a great fit for ECS doesn't mean my game as a whole can't benefit from ECS. This would be a case where I've established that my game as a whole would strongly benefit from ECS, but there's this one Stats system that needs to fit in somehow, even though it's not a great fit for ECS. I could just end up choosing to make this Stats system use OOP, but I wanted to give an ECS implementation of it a good try. I still think what I have now would perform better than an OOP implementation, due to being in a burst job instead of on the main thread. The burst math optimizations of stat modifier calculation probably outweigh the cost of DataFromEntity access

 

@PhilSA @TheOtherMonarch To be perfectly honest, I see this example as a false one. Stats change infrequently. This is a great red flag for not caring about random memory access too much. It is not part of the hot path by any means. So I would probably store them in an array or something and produce data structure which can be consumed easily by systems inside entities when they are using them.

But cool puzzle.

 

The way I do similar is to have a NativeList inside a system so projectiles are outside of ECS. But once they are launched they only change stats based on hitting stuff and are not based on if the firing tank is alive. You can definitely check every frame if the spell caster is alive. But checking every time you update each projectile does not sound logical to me.

Magic is not based on cause and effect in some cases.

 

Yeah. But sometimes I can't help but to let my imagination run wild and think about: what if I have a game where I can shoot some kind of moving magic orb that has a "slow-movement-speed buff area" around it, and that magic orb can move through large armies of RTS units, adding/removing stat modifiers at a pretty high frequency, etc....

I'm trying to build this system as a tool to keep in my DOTS toolbag, even though I'm using it on a specific project currently. So I kinda have to plan for the unplanned in some way

 

Your orb cannot move faster then once per-frame.