Search Unity

Serialization. How to iterate over all components of a given entity?

Discussion in 'Data Oriented Technology Stack' started by SubPixelPerfect, Apr 1, 2018.

  1. SubPixelPerfect

    SubPixelPerfect

    Joined:
    Oct 14, 2015
    Posts:
    222
    How to serialize/desalinize all data components associated with entity ?

    this obviously should be super easy as long as all data components are structs...

    Code (CSharp):
    1. public class SaveSystem:ComponentSystem{
    2.  
    3.     public struct Saveable : IComponentData{}
    4.  
    5.     public struct Data{
    6.       public ComponentDataArray<Saveable> Saveable;
    7.       public EntityArray Entities;
    8.       public int Length;
    9.     }
    10.  
    11.     [Inject] private Data data;
    12.  
    13.     protected override void OnUpdate(){
    14.  
    15.       if( Input.GetKeyDown( KeyCode.S ) ){
    16.         Debug.Log( data.Length );
    17.      
    18.         for (int i = 0; i != data.Length; i++){
    19.          Entity e = data.Entities[i];
    20.          var types =  this.EntityManager.GetComponentTypes( e );
    21.           for( int j = 0; j != types.Length; j++ ){
    22.             ComponentType t = types[j];
    23.             ... ??? ...
    24.           }
    25.           types.Dispose();
    26.         }
    27.      
    28.       }
    29.    
    30.     }
    the problem is that the only public method to get component data i've found is
    Code (CSharp):
    1. EntityManager.GetComponentData<T>(Entity)
    and it require a compile time Type
    but i have a Type in a variable

    i've found a vary ugly solution using reflection:
    Code (CSharp):
    1. MethodInfo methodInfo = typeof(EntityManager).GetMethod("GetComponentData");
    2. MethodInfo genericMethodInfo = methodInfo.MakeGenericMethod(t);
    3. var parameters = new object[]{e};
    4. object componentData = genericMethodInfo.Invoke( EntityManager, parameters );
    There must be a better way to get all entity data....
     
    Last edited: Apr 3, 2018
  2. SubPixelPerfect

    SubPixelPerfect

    Joined:
    Oct 14, 2015
    Posts:
    222
    i see that EntityManager has few interesting internal methods

    Code (CSharp):
    1.  
    2. internal unsafe void SetComponentDataRaw(Entity entity, int typeIndex, void* data, int size)
    3. internal unsafe void* GetComponentDataRaw(Entity entity, int typeIndex)
    4. internal unsafe object GetComponentBoxed(Entity entity, ComponentType componentType)
    5. internal unsafe void SetComponentBoxed(Entity entity, ComponentType componentType, object boxedObject)
    6.  
    is it right direction?
     
  3. SubPixelPerfect

    SubPixelPerfect

    Joined:
    Oct 14, 2015
    Posts:
    222
    I wonder why nobody commenting here, is it bother only me?

    Easiness of implementing saving system is one of key benefits of data-oriented paradigm,
    but for me it looks like there is no easy way for that in unity ecs... (right now at least)

    Am I wrong? Or I missing something?
    Please, please don't ignore me )
     
    Last edited: Apr 3, 2018
    deus0 likes this.
  4. ParadoxSolutions

    ParadoxSolutions

    Joined:
    Jul 27, 2015
    Posts:
    272
    structs need to be their own class in order to be serialized, for example this will serialize in the inspector:
    Code (CSharp):
    1. [Serializable]
    2. public class SOTest : ScriptableObject//or monobehaviour
    3. {
    4.     public Test test;
    5. }
    6. [Serializable]
    7. public struct Test : IComponentData
    8. {
    9.     public int testInt;
    10. }
    but not this:
    Code (CSharp):
    1. [Serializable]
    2. public class SOTest : ScriptableObject
    3. {
    4.     [Serializable]//SerializeField doesn't work either.
    5.     public struct Test : IComponentData
    6.     {
    7.         public int testInt;
    8.     }
    9. }
    10.  
    So as of right now Entity can not be serialized but IComponentData structs can be if they are not declared within another class, so you might just have to segregate the code and be careful not to include any non-serializable variables in your component data structs.
     
    Last edited: Apr 3, 2018
  5. SubPixelPerfect

    SubPixelPerfect

    Joined:
    Oct 14, 2015
    Posts:
    222
    there is no problem to serialize struct

    the problematic part is to iterate over all components on a given entity
     
    Last edited: Apr 3, 2018
  6. ParadoxSolutions

    ParadoxSolutions

    Joined:
    Jul 27, 2015
    Posts:
    272
    I'm still trying to wrap my head around parts of ECS but how I understand it is Entities that act as a collection of components are not accessible "objects" until the game starts whereas components are just containers of data thus accessible in edit mode. You might not be able to iterate through an entities components until it has been activated in play mode; at least in pure ECS. I guess I'll just have to try and do it and see what I come up with, maybe try [ExecuteInEditMode] or system reflection if possible.

    You are trying to see all an entities components in the inspector correct?
     
  7. SubPixelPerfect

    SubPixelPerfect

    Joined:
    Oct 14, 2015
    Posts:
    222
    no, i'm not new to ECS concept
    I understand how it works

    i have a pure ecs setup with few entities.

    those entities have different data components attached including:
    Code (CSharp):
    1. public struct Saveable : IComponentData{}
    those entities belong to different archetypes (have different set of components)

    game is running

    i can see my entities in the entity debugger window

    also i have a SaveSystem from first comment
    that system iterates over all entities containing Saveable comonent

    for each entity i'm getting a list of component types and iterate trough types

    Code (CSharp):
    1. var types =  this.EntityManager.GetComponentTypes( e );
    2. for( int j = 0; j != types.Length; j++ ){
    3.   ComponentType ct = types[j];
    4.   Type t = ct.GetManagedType();
    5.  
    6. }
    7. types.Dispose();
    next i need to get a component data by its type
    unity ECS has a generic method
    Code (CSharp):
    1. GetComponentData<T>(Entity)
    that is useless in this situation
    because i have a type as a variable

    the only workaround that i found is to use a reflection api to hack generics
    Code (CSharp):
    1. MethodInfo methodInfo = typeof(EntityManager).GetMethod("GetComponentData");
    2. MethodInfo genericMethodInfo = methodInfo.MakeGenericMethod(t);
    3. var parmeters = new object[]{e};
    4. object componentData = genericMethodInfo.Invoke( EntityManager, parmeters );
    but this is a super ugly solution
    and i asking a question if i missing something?
    i refuse to believe that there is no better way to do that
     
    Last edited: Apr 3, 2018
  8. SubPixelPerfect

    SubPixelPerfect

    Joined:
    Oct 14, 2015
    Posts:
    222
    question is not answered, i'm ready to hear: "it is not implemented yet" but please say something
     
    Last edited: Apr 6, 2018
  9. sahildhanju

    sahildhanju

    Joined:
    Mar 27, 2018
    Posts:
    6
    I am currently in the same boat as you. After scouring the EntityManager API, it seems that at the moment this isn't possible with the public API. Inside GetComponentData<T> is

    Code (CSharp):
    1. public unsafe T GetComponentData<T>(Entity entity) where T : struct, IComponentData
    2.     {
    3.       int typeIndex = TypeManager.GetTypeIndex<T>();
    4.       ...
    5.     }
    You can get the TypeIndex of the desired component from the ComponentTypes array. So if Unity simply let us pass in a TypeIndex instead of a generic parameter, it would easily work. Not sure why it's not an option at the moment.
     
    SubPixelPerfect likes this.
  10. GabrieleUnity

    GabrieleUnity

    Unity Technologies

    Joined:
    Sep 4, 2012
    Posts:
    116
    @sahildhanju: we want as much as possible of the API surface to be GC free. If we expose
    GetComponentData(Entity e, int typeIndex)
    , how would you return the
    IComponentData
    without boxing? We could expose some lower level APIs that deal with pointers and preallocated Temp memory I guess, but I'm not a big fan of it.

    That said, we are working on a way to serialize/deserialize Entities very efficiently, which will likely lend to new APIs to operate with IComponentData without knowing the concrete type.
     
  11. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    507
    Hi @GabrieleUnity, is that prefab will be reworked based on this and Game Object will also be reworked to Entity too in Scene?
     
  12. GabrieleUnity

    GabrieleUnity

    Unity Technologies

    Joined:
    Sep 4, 2012
    Posts:
    116
    Yes, once we are done we will support all (or similar) concepts to what we have today, and also allow easy transition between Unity's OO and ECS concepts (where possible).
     
  13. floboc

    floboc

    Joined:
    Oct 31, 2017
    Posts:
    89
    Do you have an idea about when (de)serialization will be available for testing ? Is it a matter of weeks, monthes, years ?
    Thanks :)
     
  14. GabrieleUnity

    GabrieleUnity

    Unity Technologies

    Joined:
    Sep 4, 2012
    Posts:
    116
    We are working on it right now. I'd say "weeks" unless something goes badly wrong - but don't quote me on this ;)
     
  15. floboc

    floboc

    Joined:
    Oct 31, 2017
    Posts:
    89
    Quoted ;)
    I am really looking toward it !
     
    SubPixelPerfect likes this.
  16. SubPixelPerfect

    SubPixelPerfect

    Joined:
    Oct 14, 2015
    Posts:
    222
    I can't resist to quote it as well )
     
  17. starikcetin

    starikcetin

    Joined:
    Dec 7, 2017
    Posts:
    268
    Might as well join the hype train.
    PS: Now I see why you avoid giving ETAs. :)
     
  18. capyvara

    capyvara

    Joined:
    Mar 11, 2010
    Posts:
    77
    Any news on this?
     
  19. SubPixelPerfect

    SubPixelPerfect

    Joined:
    Oct 14, 2015
    Posts:
    222
    now you can serialize the whole world to a binary file, and deserialize that file back to an empty world

    but you still can't
    - serialize single entity
    - serialize single component in a generic way
    - iterate over all entity components without reflection
    - save world partially
    - merge saved data with the existing world (without using second world)
    - use inspector to tweak serialized data
     
    kerjemanov and rakkarage like this.
  20. capyvara

    capyvara

    Joined:
    Mar 11, 2010
    Posts:
    77
    Seems that all public interface shields you from accessing any raw memory.

    I've noticed EntityContainer at Unity.Entities.Properties that seems to allow to serialize entities (at least to JSON in the example), internally it uses EntityManager.GetComponentDataRawRW.

    I wonder if they are planning to use it for flexible serialization or just for editing mode inspection.
     
  21. rigidbuddy

    rigidbuddy

    Joined:
    Feb 25, 2014
    Posts:
    35
    I've implemented quite flexible serialization logic to serialize on entity or on component basis (for custom netcode and replays). But it uses SetComponentDataRaw, GetComponentDataRawRW so I had to modify Entities package (InternalsVisibleTo).

    Hope Unity will make separate Entities.Unsafe API to access it on low level for such use cases. Or bring asmdef feature to make "friend assemblies" to access certain package internals (inverse to InternalsVisibleTo)
     
    Guerro323 likes this.
  22. e199

    e199

    Joined:
    Mar 24, 2015
    Posts:
    100
    Can you show some examples of how you use them, please?
     
  23. rigidbuddy

    rigidbuddy

    Joined:
    Feb 25, 2014
    Posts:
    35
    Sure. I've created simple data structure and extension helper methods.
    Also beware that you need to fix TypeIndex for Types you want to store. Please refer to this answer:
    https://forum.unity.com/threads/is-...ween-platforms-instances.581098/#post-3875950

    Code (CSharp):
    1.     internal unsafe struct RawCompData
    2.     {
    3.         public int TypeIndex;
    4.         public void* Data;
    5.     }
    Code (CSharp):
    1.     internal static unsafe class RawCompDataApi
    2.     {
    3.         public static int Size(ref this RawCompData rc) => TypeManager.GetTypeInfo(rc.TypeIndex).ElementSize;
    4.  
    5.         public static void WriteRaw(ref this RawCompData d, EntityManager em, Entity e)
    6.         {
    7.             if (ComponentType.FromTypeIndex(d.TypeIndex).IsZeroSized)
    8.                 return;
    9.  
    10.             em.SetComponentDataRaw(e, d.TypeIndex, d.Data, d.Size());
    11.         }
    12.  
    13.         public static RawCompData ReadRaw(int typeId, EntityManager em, Entity e)
    14.         {
    15.             if (ComponentType.FromTypeIndex(typeId).IsZeroSized)
    16.                 return new RawCompData() {TypeIndex = typeId};
    17.  
    18.             var data = em.GetComponentDataRawRW(e, typeId);
    19.             return new RawCompData() {Data = data, TypeIndex = typeId};
    20.         }
    21.     }
     
    e199 likes this.
  24. pointcache

    pointcache

    Joined:
    Sep 22, 2012
    Posts:
    537
    Hi Gabriele, what solution did the team come up with?
    Im looking to implement several things like runtime entity inspector and serialization, but i can't find an api to get all components from an entity.
     
  25. Tony_Max

    Tony_Max

    Joined:
    Feb 7, 2017
    Posts:
    74
    What about performance of this solution? Do you use it?
     
  26. dzamani

    dzamani

    Joined:
    Feb 25, 2014
    Posts:
    110
    Is there any updates on this or at least a solution in the backlog ?
    Just want to know if it will stay as is (so by design) or it has not been looked at yet.
     
  27. SubPixelPerfect

    SubPixelPerfect

    Joined:
    Oct 14, 2015
    Posts:
    222
    at some point, we'll get an entity asset, hope for good news on GDC :)
     
  28. dzamani

    dzamani

    Joined:
    Feb 25, 2014
    Posts:
    110
    Well I'm more interested in a way to get all components of an entity (without a hack) but not for serialization.
    And yes, it's coming just look at UTiny.
     
  29. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    This is doable already, but you need to use dynamically bound generics for each component type
     
  30. andrew-lukasik

    andrew-lukasik

    Joined:
    Jan 31, 2013
    Posts:
    142
    Did anything changed on this topic? I want to deserialize component data but entityManager.SetComponentData is still for compilation-time only.
     
  31. Guedez

    Guedez

    Joined:
    Jun 1, 2012
    Posts:
    258
    I'd create specialized systems whose only mission is to serialize a type of data. I'd also set all kinds of dependencies to them, so that for instance, a oven's current recipe data can only be saved after the oven data was saved, so that the recipe knows where it have to go after it's deserialized.
    I've already checked ages ago that a system can access basically anything in the current world, so a totally unrelated system to the data being saved can easily access it and serialize. I just am not sure how to set the dependencies
     
  32. Turnipski

    Turnipski

    Joined:
    Jun 27, 2015
    Posts:
    9
    +1 for exposing `object GetComponentData(ComponentType type)` somehow. I need this because I need to serialize to/from JSON (for web server validation) so the binary serializer is useless to me.

    Until then I suppose the reflection method posted above is good enough, it's not like I care about performance when serializing to JSON anyway! :p
     
    vestigial and Seb-1814 like this.
  33. MostHated

    MostHated

    Joined:
    Nov 29, 2015
    Posts:
    906
    I heard you can get some pretty good results from taking a screenshot of the JSON, then using OCR to capture the data, then fax it to your local courthouse and have someone dictate it to the stenographer, etc, etc.
     
    Chmyke, thelebaron and wobes like this.
  34. wobes

    wobes

    Joined:
    Mar 9, 2013
    Posts:
    736
    Will add to my TODO list.
     
  35. Guedez

    Guedez

    Joined:
    Jun 1, 2012
    Posts:
    258
    "And therefore, if you ever need a IComponentData from a Type variable, you should just handle it with Reflection"
    What if we need it for something that is not serialization? Being forced to use Reflection because the API does not want us to box variables is way worse
     
    RandyBuchholz likes this.
  36. Brendon_Smuts

    Brendon_Smuts

    Joined:
    Jun 12, 2017
    Posts:
    50
    If you give people an easy method to do something a bad way they will do it the bad way. ECS priority is performance first and boxing goes very strongly against that.

    There are almost always performant, ECS friendly methods to solve the problems people want “easy APIs” for that just need a little more thought.
     
    Tony_Max likes this.
  37. Guedez

    Guedez

    Joined:
    Jun 1, 2012
    Posts:
    258
    And if you don't, they will be forced to do even worse and use Reflection

    Very nicely put, ALMOST always. What if there is not a way to do it? And don't just assume that "ECS friendly" and "performant" goes hand in hand, ECS is terrible for any hierarchical structure and the solution is always "Don't use hierarchical structures in ECS".
    Sometimes we just want to get stuff done without having to dismantle 80% of something else
     
unityunity