Search Unity

  1. How has 2019.2 and the beta been for you so far? Give us feedback in this thread.
    Dismiss Notice
  2. We would like to hear your feedback about Unity and our products. Click here for more information.
    Dismiss Notice

Memory layout for ECS components?

Discussion in 'Data Oriented Technology Stack' started by Nelvin123, Nov 30, 2018.

  1. Nelvin123

    Nelvin123

    Joined:
    Apr 4, 2014
    Posts:
    4
    Hi all,

    I'm trying to understand how the ECS in Unity works as I plan to move to Unity and make some simulation games with a lot of simulation objects and I think the ECS is the way to go but I'd really like to understand how it actually works (I used to use C/++ and simply know exactly where and how my data is stored).

    What I primarily wonder about is, how is component memory managed internally if different Systems use a combination of different components but some of those are overlapping?

    Let's say, we do have a world and there are countless types of entities, but let's start with a very simple setup, like, rocks and trees.

    Rocks will have a position component, trees will have a position component but also a simple growState.

    Now let's assume we have a system to "render" objects (just imagine, we're creating a mesh for each object) and a system to update growing trees but the growth depends on their actual position.

    As far as I understand it, Systems get linear arrays of all the components they need to work on so the System can do it's calculations in the most efficient way. But how is this achieved? There's no way to have a linear representation of the position including rocks and trees but also for just trees (when updating the growState)?

    An entity is a simple (integer) ID but as soon as not all entities are exactly the same, the storage for the components will have gaps. I.e. if entity 1, 2, 4 are rocks but 3 and 5 are trees, then the growState component of the trees obviously can't have the same mapping from entityID to component data it does to it's position component.

    So how does this work? Is Unity creating a linear array copy of each components for each entity a system works with each frame and, after the system has done it's work, the resulting data is copied back? Or is there any other way to handle this that I have not thought about yet?

    Thanks for any enlightment ...
    Nelvin
     
  2. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,304
    They not overlapped :) Their chuks in different archetypes, which mean archetype {Tree, Position, GrowState} have own chunks array with Position and Tree components arrays inside, same for {Rock, Position} and when you query only Position component (for exampe you must iterate through all positions of all trees and all rocks) system gives to you chunks from different archetypes. Entity always unique, and it not just ID it's Index and Version combination, which mean Entity with Index 1 and Version 2 it's not the same as Index 1 Version 3 (actually Entity with Version 2 destroyed if you have Version 3). And it how they looks in memory (in simple imagination)
    upload_2018-11-30_14-3-23.png
     
    Nelvin123 likes this.
  3. Spy-Shifty

    Spy-Shifty

    Joined:
    May 5, 2011
    Posts:
    519
    Well, each "component set" is defined in a unique ArchetypeChunk. It's a container there all entities with the same set of components are stored.

    E.g. you have the following ArchetypeChunks for:
    [Rock, Position, Rotation, Scale]
    [Tree, Position, Rotation, Scale, Growstate]
    [Grass, Position, Rotation, Scale, Growstate],
    ...

    Those are packed tightly together in an array. So you get the perfomance boost of your hardware on iteration...

    To iteration over all Entities with e.g. Position and Rotation, the ECS will take the first chunk, that contains Position and Rotation components and iteration over it. After that it will take the next chunk and so on.

    Basically there is a pointer which points to the current chunk in memory and a offset which points to the current element in that array.

    [Rock, Position, Rotation, Scale],[Tree, Position, Rotation, Scale, Growstate],[Grass, Position, Rotation, Scale, Growstate], ...

    If you remove or add a component to an entity, will change it's ArchetypeChunk. So that this entity will be removed from the old chunk and added to a new chunk.

    If you remove a component or entity from a chunk, the last entity in the chunk will be moved to the gap.

    Hope this will help you a little bit.
     
    Nelvin123 likes this.
  4. SubPixelPerfect

    SubPixelPerfect

    Joined:
    Oct 14, 2015
    Posts:
    180
    entities that have the same components set (same archetype) are grouped together and stored in a linear way without gaps, split into chunks 16kb per chunk, so all entities in the chank have the same archetype

    when you make a query for some set of components, you get a list of chunks.
    for example, your stones may be defined as Stone + Position, and trees as Tree + Position,
    and if you query everything that has Position, you'll get both tree chunks and stone chunks
     
    Nelvin123 and Spy-Shifty like this.
  5. Spy-Shifty

    Spy-Shifty

    Joined:
    May 5, 2011
    Posts:
    519
    Ah S*** you were faster :D
     
    SubPixelPerfect likes this.
  6. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,231
    I wonder would it be possible for "zero-sized component" add-remove to not cause chunk movement at all? Like, just pretend that zero sized components are tags attached to each entities in a chunk. That way I could mass attach/detach a tag component for filtering entities without worrying about add component performance drop.. but thinking carefully the "filtering" is precisely because there are chunk movement on adding removing a tag then ECS could select the whole new chunk without thinking much. Maybe it is impossible.
     
    Nelvin123 likes this.
  7. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,304
    Daft-Punk-2017.jpg
     
    Spy-Shifty likes this.
  8. SubPixelPerfect

    SubPixelPerfect

    Joined:
    Oct 14, 2015
    Posts:
    180
    I think it works this way - when you adding a zero-sized component to all entities in a chunk, looks like they don't move to another chunk, just archetype of a chunk changes
     
    Last edited: Nov 30, 2018
    Nelvin123 likes this.
  9. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    1,231
    So we will need an another set of ChunkManager.Add/Remove/Destroy(ArchetypeChunk) and ChunkCommandBuffer.Add/Remove/Destroy(ArchetypeChunk) ... o_O
     
    Nelvin123 likes this.
  10. Nelvin123

    Nelvin123

    Joined:
    Apr 4, 2014
    Posts:
    4
    Thanks a lot for the fast and decent answers - the info about the list of chunks a system get's to do it's work instead of just a single chunk was somehow missing in my mind and obviously caused me to wonder how this could work. But now it all makes sense :)
     
    Spy-Shifty likes this.
  11. Nelvin123

    Nelvin123

    Joined:
    Apr 4, 2014
    Posts:
    4
    I do have one additional question ... I fully get the entity ID with it's version/generation alike counter (it's the common way to manage handles) but how does cross referencing entities of different archetypes typically work?

    Is there a way to reference entities and not know their actual archetype but still access the components of those I'm interested in?

    Say an entity that's a projectile that got locked onto a target entity - but that target could be an enemy spaceship, a building on the map etc. and all the projectile movement system is interested in is the position component of it's current target entityId?

    One option would be that each component type has a table mapping from the entityId to the actual component data, but that does not sound like a good idea in big projects as the combination of many components and big amounts of entities result in exponential growth of resources needed for those tables.

    So how does it work? The only way I could imagine is multiple lookups. First one to just find the archetype of the target entity, then do a test if the archtype includes the requested component and then do a lookup (maybe using hash table?) from the entityId to the "local" index to finally get the component data within the chunks of the given archetype.
    Sounds complicated but, as with the multiple chunks, I might again just miss an important detail.

    Sorry if this sounds like asking for implementation details, but I'm actually coding since a long time (i.e. last millenium) and I'm just used (and wanting) to know and really understand what my code does and what's going on.
     
  12. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,304
    No. Always when you create Entity it's in some archetype.
    And this why Chunk Iteration exists.
    ArchetypeChunk.Has(ArchetypeChunkComponentType acct), ComponentDataFromEntity<T>(Entity entity) / ComponentDataFromEntity.Exists(Entity entity)

    Chunk Iteration is low level management of memory layout in ECS (lowest only direct working with pointers, Chunk Iteration too use many pointers work, actualy most part - pointers) you always can check contains chunk some specific ComponentData or not, you always can check exist entity in chunk and get it if yes, you can check changed chunk with previous state or not (DidChange\DidAddOrChange)
     
    Last edited: Nov 30, 2018
    Nelvin123 likes this.
  13. M_R

    M_R

    Joined:
    Apr 15, 2015
    Posts:
    425
    EntityManager
    has a "map" (actually an array) of [entity index => version, chunk pointer, index in chunk]
    so to get the data from an Entity it does (pseudocode)
    Code (CSharp):
    1. data = entityArray[entity.Index];
    2. if (data.version != entity.version) abort (invalid/destroyed entity)
    3. return data.chunk->arrayOfRightComponent[data.indexInChunk]
     
    Nelvin123 likes this.
  14. Nelvin123

    Nelvin123

    Joined:
    Apr 4, 2014
    Posts:
    4
    Thanks once more for the additional answers - cleared things up a lot and helped me to search/find for some additional information/threads. I think I have a good understanding of how everything (of interest to me, at this moment) works which is great.
     
  15. davenirline

    davenirline

    Joined:
    Jul 7, 2010
    Posts:
    450
    This is very educational. The chunk model is so damn clever. I'll steal this idea if I'm to make my own entity system.
     
  16. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,304
    In addition.
     
    Nelvin123 likes this.