Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Exposing lower level access to chunk memory.

Discussion in 'Entity Component System' started by fholm, Jan 18, 2019.

  1. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,052
    I'm trying to figure out a way to access the raw chunk memory for a specific component type (or a whole entity would be preferable).

    I've been digging through the internals of the entities code and for the GetComponentData<T> call internally it uses the EntityDataManager.GetComponentDataWithTypeRO method which in turn calls another internal class called ChunkDataUtility.

    You can clearly see that these calls access the pointer to the raw memory of the chunk, and calculates offsets/indices which later is used to copy the pointer data to the IComponentData struct.

    So two questions:

    1) Is it possible to get access the chunk memory in another way using the public API today?
    2) If it's not possible, is there any chance we could have internal classes like EntityDataManager and ChunkDataUtility exposed to us?

    I'm aware this is of very limited use for most people, but it would still be nice to have the option to do this.
     
  2. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,753
    I haven't looked through it thoroughly, but they actually added a bunch of public chunk APIs today that might interest you.

    -edit-

    in particular

    Code (CSharp):
    1. public void LockChunk(ArchetypeChunk chunk)
    2. public void LockChunk(NativeArray<ArchetypeChunk> chunks)
    3. public void UnlockChunk(ArchetypeChunk chunk)
    4. public void UnlockChunk(NativeArray<ArchetypeChunk> chunks)
    5. public void CreateChunk(EntityArchetype archetype, NativeArray<ArchetypeChunk> chunks, int entityCount)
    6. public ArchetypeChunk GetChunk(Entity entity)
    7.  
    8. public void RemoveChunkComponent<T>(Entity entity)
    9. public void AddChunkComponentData<T>(Entity entity) where T : struct, IComponentData
    10. public void RemoveChunkComponentData<T>(ArchetypeChunk archetypeChunk) where T : struct, IComponentData
    11. public void AddChunkComponentData<T>(ArchetypeChunk archetypeChunk, T componentData) where T : struct, IComponentData
    12.  
    13. public T GetChunkComponentData<T>(ArchetypeChunk chunk) where T : struct, IComponentData
    14. public T GetChunkComponentData<T>(Entity entity) where T : struct, IComponentData
    15. public void SetChunkComponentData<T>(ArchetypeChunk chunk, T componentValue) where T : struct, IComponentData
    16.  
    17. public NativeArray<ArchetypeChunk> GetAllChunks(Allocator allocator = Allocator.TempJob)
    18.  
    19. public void SwapComponents(ArchetypeChunk leftChunk, int leftIndex, ArchetypeChunk rightChunk, int rightIndex)
    No idea how helpful this is to you, but someone might find it useful.

    (All in EntityManager)
     
    Last edited: Jan 18, 2019
  3. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,052
    Today? I'm not sure where you mean
     
  4. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,753
    I just made an edit with all the new methods.

    It doesn't get you direct access to memory but does give you a bunch of new methods for manipulating chunks.
     
  5. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,052
    Thanks for the edit! Sadly this doesn't seem to help me at all, all of the APIs to access the data seem to be generic taking a T : IComponentData parameter sadly.

    I just want access to the void* and all of the methods to calculate component type offsets, etc. so I can read data directly out of the chunk at will.
     
  6. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,052
    So I'm able to work around this by simply exposing the classes that I need in the source code for the entities package, but it's obviously a pretty nasty workaround (having to modified source code). It would be great if this was exposed for us by default in a nice way, as right now I have to make something like 20 internal types public to make it all compile again.

    I understand that this is a very limited use-case, but it'd be cool for those of us that need to to be able to do this, maybe 'hidden away' in a utilities class somewhere or something.
     
    Last edited: Jan 18, 2019
  7. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    ArcheTypeChunk.GetNativeArray(ArchetypeComponentType).GetUnsafePtr()

    Essentially gives you the pointer to a single component stream in the chunk.
     
  8. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,052
    I dug into the code and saw this, but the problem for me here is that this is a generic API as ArchetypeComponentType<T> is a generic struct (this is what I'm doing at the moment).

    This means that I have to have unique serializer and delta compressor 'instances/jobs' for each component type, and there's no way to create a serializer for a whole archetype in one go, and it massively increases the overhead for me. It'd be great to just have access to the methods you internally use to access the chunk memory exposed.

    I understand this is a very limited use case, but maybe it could be hidden away in some utility class or something.

    Edit: As I said, I simply exposed the internal pieces of the entities package, and it lets me do exactly what I want.
     
  9. Spy-Shifty

    Spy-Shifty

    Joined:
    May 5, 2011
    Posts:
    546
    Sure you can do ;)
    I also use a generic approach for this.
    I simply handle it by a common use of an interface
    https://forum.unity.com/threads/error-on-generic-ijobprocesscomponentdata.613432/
     
  10. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,052
    Maybe I'm missunderstanding you, but this requires using one job type per component type doesn't it? That's a massive amount of overhead (at least in my benchmarks). Having the ability to 'get' into the component memory via the pointer and just some integer offsets makes calculating things like delta compression much faster because you don't need the intermediate step of going through archetype/job.
     
  11. Spy-Shifty

    Spy-Shifty

    Joined:
    May 5, 2011
    Posts:
    546
    Well this depends on what you are doing exactly.

    As @Joachim_Ante sad in an differend thread: