Do complex DOTS systems start to develop OO style architectures?

OOP is an attempt to group code and data into domain or problem relevant spaces, and I believe an outshout of simulation programming.

If so do we need intermediary structures in DOTS e.g. between a World and Entities that would allow us to group our systems to domains or problem spaces?

Or should OOP programmers just think of Worlds as Objects and Systems as methods?

 

one of OOP's biggest mistakes is the way it couples data to a specific context (class). If I add an OOP style component to an entity then all that encapsulated data is fixed and applies only to the context of that component. Because of this we end up jamming functionality into this one class that also seems to share a context with that class, once we reach the limits of that we attempt to break it down into a hierarchy. This is a fundamentally flawed approach as it assumes that there is a correct place for data somewhere within that hierarchy. With data oriented design a large number of separate systems could be building up completely unique contexts for that data based on the specific subset that they require. Data oriented design allows you to create new collaborations of data at any time depending on the needs of the specific problem you are trying to solve.

OOP is a human friendly abstraction that allowed large companies to coordinate development on large projects. It is not a performant or malleable approach to software architecture. To properly understand data oriented design you must first forget everything you know about OOP.

 

That's like saying I should go back to functional programming and by the way we have re-banded it DOD.

All problem domains have processes and data that are unique to themselves in OOP you have a handy toolset for grouping that data into objects an abstraction that lets you think and work at a higher level.

The thing is once DOD systems get above a certain level of complexity then you are going to need to partition or link the areas of the systems that work within certain sub-domains. Some form or heirarchy or sub-division will be needed and good designers will set out to establish that within their own projects.

The subdividing could be in the form of namespaces or folders but a fundamental aspect of solving any problem is the divide and conquer strategy of breaking big problem domains down into smaller and smaller problems.

Does mainstream DOD not have a process for this e.g. do DOD developers build the database first then the solution or how do they break down the problem domain/game design into small enough manageable chunks to write systems.

There seems to be this big gap in designing and working with DOD that is built into OOP. In OOP you can start out mapping domain subject into objects then start to figure out how these objects interact to solve the problem.

In DOD what do you do make a list of all the data in the problem domain then try to figure out what systems solve the problem one at a time?

 

what works well, is to just try it, put some effort...and when you hit a wall with a concrete problem, post it here and ask for help

 

I used that approach with OOP Unity and then found there were scaling design problems once you start working on more complex games in Unity with OOP.

So your saying that by taking the bottom up approach of DOD I will magically be able to deal with the more complex design problems when I tackle more complex games...

I suspect there is a big missing void of game design and game architectural patterns that is not being captured by the unity game engines API's be it OOP or DOTS.

In another game engine I noticed the Blackboard design pattern whereby global information can be shared by NPCs allowing them to communicate and respond to the the player in a more believable way. Something as simple as a 'Global' Dictionary with key/value pairs can allow NPCs to communicate and change their behaviour.

I have not seen anything similar in any of the Unity material yet it is the kind of design solution developers would come up with when making an actual game with NPCs.

Before I spend weeks or months building up a more complex DOD game system I would really like to know how DOD programmers deal with the complexity?

 

The complexity can be built up through dependencies of data transformations made through previous systems. The entire flow of the ecs "system" architecture is like the price is right plinko board. Your data falls in from the top the transformations happen in stages defined by your dependencies. The notion that complex code requires complex hierarchies like folders, namespaces, or objects is an absolute fallacy. Ecs as a paradigm is so different from OOP that any attempts to shoehorn your prior patterns in will only lead to disappointment.

Try this book: https://www.amazon.com/Data-oriented-design-engineering-resources-schedules/dp/1916478700

 

Pointing me at a $30 book that only has a couple of reviews and who's author I cannot find any online lectures for (good authors often do lectures or talks alongside the books e.g. at GDC) is not really answering my question.

Is there a video or blog that talks about the complexities of DOD and more specifically DOTS and the problems more complex games/systems can have and more importantly how to solve them?

I know there are a few developers pushing more advanced games with DOTS who are deep into the technology so we should start seeing them hit problems with DOTS that will occur due to the complexity of the systems they are making.

Meta problems I suspect will happen with DOTS:

• Flow conditions where simpler faster tasks perform poorly due to a previous task being longer and slower.
• Event handling and signalling in real time that the developer wants to happen as a priority having to be expressed as a separate higher priority system. Due to DOTS not having any kind of in built priority system.
• Dynamic prioritisation of Jobs or rescheduling systems depending on workload/game state.
• To my knowledge DOTS does not seem to have a way to dynamically control/overview the flow of data through the systems and how those systems are allocated, admittedly I am a visual person so some kind of flow graph/meta node graph would be ideal here.

As well as the design battles between small light weight atomic systems and large complex potentially cache overloading systems for optimal performance and the best solution.

 

That book is a great resource to learn about data oriented programming. He also has most of the content available online at http://www.dataorienteddesign.com/dodbook/.

Overwatch was made using ECS and there is a great talk about that here

 

Thank you that's much more helpful.

So they adopted Singleton, Deferment and Utility Functions as solutions to DOD game complexity when working on a AAA game over 3 years. This is the sort of in depth information you need before you start working with new systems.

• Creation and Destruction of Entities is Deferred.
• Shared code lives in Utility Functions.
• Complex side effects should be Deferred.
• Systems can't call other systems (should massively reduce complexity as a project progresses).

Interesting the AI 'Navmeshing' is not an ECS system "as that problem space should not be shoehorned into ECS".

ECS is glue code that brings together your games systems. How some systems are like icebergs to ECS where only the aspect of the systems that interact with ECS like an iceberg is above the surface with the remainder of the system outside of ECS.

So does Unity need to move all of it's systems into DOTS/ECS or would an iceberg approach where more complex systems have their own solution space e.g. Navigation, Animation and only need to surface their control features within DOTS.

Interesting that the speaker mentions spinning up jobs dynamically to deal with workload e.g. lots of projects by a player, does DOTS have any dynamic load control features?

 

While this talk is really good, the overwatch ECS approach is somewhat different in 'mentality' than the Unity one, there are more talks from this GDC where they dig into some deeper parts of their architecture. But they have a lot of 'fat' components, i.e. where one component contains a whole object hierarchy, etc.

While it probably costs them a bit of performance, it allows for much more flexibility than what is currently possible in Unitys ecs.

 

Unity.Physics is already an example of implementing the solution outside of ECS space and exposing an ECS surface.

 

I think OO and ECS solve different problems and are complimentary, similar to how Relational (e.g., database) and OO architectures compliment each other. Each fits as a layer/architecture area in the overall architecture. The Unity Engine can even be abstracted as a Presentation Layer. Objects work well in the business domain, and in dealing with "one-off" cases. Entities live more in the Rules and technology domains, and excel at bulk operations on Records. Databases support long-term persistence.

Along those same lines, I'm playing with "OEM's" - Object to Entity Mappers, and "ROEM's" - Relational, Object, Entity Mappers. I use OO Classes to represent business objects, exposing service-oriented interfaces. Operational data lives in the ECS layer. Basically, the OO Classes are Controllers and wrappers for groups of Systems. Presentation talks to both layers. If a collision occurs, ECS handles that directly, but a UI button press could trigger a business process. The OO layer would take that, maybe adjust some system parameters, and create a "seed entity". ECS would take over from there. Creating the "seed" would be done through the Mappers. If a business process needs information (like "How many exist?"), it uses the Mappers. Mappers are essentially Systems that expose a few properties and methods that can be used by the business layer.

 

One of the most problematic barriers between a well-designed OO software and an average programmer today is lack of essential knowledge. This is something that DOD comes to solve but at the cost of well-known OOP tools (or pillars if you prefer), and in many cases, this trade is not acceptable (especially for game developers). The thing is, you can make software with great architecture, performance, and keep it maintainable using OO if you do things right (OOP is dead, long live OOP for example).

To break the barrier you need to understand the problems:

1. Memory access
2. Data structuring

Here's the most basic example (86 SLOC) that intended to demonstrate a difference between DOD, OOD, and OOP. Believe it or not, there's no difference in performance if we follow simple rules, but there's a difference in design. To keep the OO design, but to achieve the DOD performance we need to eliminate encapsulated writing and structure fragmentation (both things make the difference by a factor of 20 in the benchmark).

 

What you are describing is what I like to call an "Entity-System" architecture, where an "Entity" contains all the data and each "Entity Archetype" has either associated systems or interface implementations for systems to be processed in batch. This architecture isn't very commonly used or understood, which is unfortunate because it is excellent for small games and fills in a lot of the missing knowledge gaps between GameObjects and ECS.

However, an ES suffers heavily at large scale in terms of code-reuse. And that's because it lacks modularity. Modularity comes from the breakdown of components and compositions, something GameObjects had (you can think of the GameObject architecture as an EC). Naturally, ECS is an evolution and convergence of those architectures which takes the best from both. It uses DoD for data and OOP for logic (systems are classes and jobs are objects that contain all data they process).

The main downside of ECS is that it requires quite a bit of architectural plumbing to really get going, but Unity is doing a lot of that plumbing for us so it is really just learning the next evolution that is causing a lot of pain.

 

Well, the abstraction can certainly get similar. But where OOP focused around something like a monster and what it can do, with ECS I think about the mechanics and what those can do with the monster.

I mean, the games don't change. We'll implement age-old mechanics over and over again so some resemblance of systems that can be viewed like objects in an abstracted way will stick out.

The implementation is turned around with composition over inheritance so instead of a monster who can be damaged, can attack, walk, find targets, etc... those mechanics are split up into systems that revolve around those single mechanics. There is exactly 1 layer of abstraction, so pretty direct. And when you suddenly want a door you can damage, you just add a healthComponent to it.

The really great thing is that I could use the same pathfinding, walk code in 3 different projects. I was never able to do this with any kind of OOP approach, like, truly copy/pasting/done.
And I think after the initial weirdness of ECS settles, it's much more direct to break down problems and processes and program any kind of mechanic.

And something about the simulation thing. In OOP, objects would know that gravity exists and they apply it themselves. In ECS, entities have no clue about the GravitySystem, it's happening but they are not directly aware. In that regard, ECS works much more like our universe.

 

I agree about the similarities at the abstracted level. Even more so when we look at OOAD and early OO, especially from Jacobson's perspective. In the early days, the idea of a "Data Object" (as a Class) made no sense. Classes and Structures came out together. Classes only exposed behaviors, and Structures only contained data. Classes did contain data, but not in the sense of "public properties", it was intended to be private (for Identity and State). ECS takes this a step farther. It says that instead of working on individual instances, Classes (Systems) should work on types (Entities - collections of structures). To do this, we just need to move Identity and State out of the Class and into their own Structures. Jacobson's (pre-UML) work and SDL show signs of ECS thinking. He even used the terms System and Entity in a similar way.