Search Unity

  1. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  2. Improved Prefab workflow (includes Nested Prefabs!), 2D isometric Tilemap and more! Get the 2018.3 Beta now.
    Dismiss Notice
  3. Want more efficiency in your development work? Sign up to receive weekly tech and creative know-how from Unity experts.
    Dismiss Notice
  4. Participate with students all over the world and build projects to teach people. Join now!
    Dismiss Notice
  5. Build games and experiences that can load instantly and without install. Explore the Project Tiny Preview today!
    Dismiss Notice
  6. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice
  7. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

Speed up GameObject position/rotation before using ECS

Discussion in 'Entity Component System and C# Job system' started by MattiHietanen, Dec 3, 2018.

  1. MattiHietanen

    MattiHietanen

    Joined:
    Nov 24, 2018
    Posts:
    13
    Hi
    I'm working on an complex engine simulation. Before starting to use ecs I'd like calculate all complex parameters and relations in a common way. If the simulation works perfectly then I want to start over with ecs to have control of what is going wrong and have a correct comparable simulation engine code in the first place.

    The simulation engine runs currently in one function. I cache position/rotation/etc of all parts into massive vector3/4 arrays and apply them back to the engine objects.

    But what is fastest method in common unity system to apply the result transform to the objects.

    Calculate in update and apply result transform values in LateUpdate()?
    Or does it not mater to apply the transform values directly after a single calculation to the game objects?
    Any hints on this?
     
    Last edited: Dec 3, 2018
  2. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    785
    The fastest is you don't apply back but use Graphics.DrawMeshInstanced/DrawMesh to draw from data. And so ideally the result from calculation should be encoded as a transformation matrix array Matrix4x4[]

    https://docs.unity3d.com/ScriptReference/Graphics.DrawMeshInstanced.html

    If you can do that, ECS has float4x4 matrix component usable with ECS drawing code so you should be able to port the data easily.
     
    MattiHietanen likes this.
  3. MattiHietanen

    MattiHietanen

    Joined:
    Nov 24, 2018
    Posts:
    13
    Thank you. Seems to use an entity manager directly and add components of transform and mesh but didn't find any methods to store and restore the contents of an entity manager.Takes so long prepare 150K objects on each start. Is this undocumented or any other solution on this?
     
  4. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    785
    For store/restore ECS has basic support for serialize/deserialization which is not documented that much. Check the class SerializeUtility or SerializeUtilityHybrid if you got shared component data. Use ECS lib class StreamBinaryReader/Writer to write to file. Or implement BinaryWriter/Reader (not .NET ones) interface to get to handle each written byte manually. (requires unsafe asmdef)

    SCD will be baked to SharedComponentDataWrapper with name ___Component to a new game object, where ___ is your ISharedComponentData name. On deserialize you then need to present this game object first.

    The catch you have to remember is that on deserialization all the types has to be of exact assembly qualified name + exact same struct layout in order to produce matching hash, because it read back chunk data exactly without any processing. (can rename struct field, but cannot change/add/remove or arrange field order)

    It serialize an entire world. Use EntityManager.MoveEntitiesFrom in the world made for serialize with overload that uses ComponentGroup made from EntityArchetypeQuery to select only subset of entities to serialize from your main world.

    (I didn't have any experience with any of that, just read about them yesterday. I am currently trying to migrate from Protobuf for certain things to reduce hassle of restoring ECS state)
     
    MattiHietanen and florianhanke like this.
  5. MattiHietanen

    MattiHietanen

    Joined:
    Nov 24, 2018
    Posts:
    13
    Oh thank you! But seems not in my range of acknowledge currently about ecs. An easy example to handle this would be absolute great. Just wondering there's no require on this before, cause ecs exist about ~8 month. I read about another threads some guys try but fail on. I think would be really great and helpful to have this as feature or example code.
     
    Trainzland likes this.
  6. Trainzland

    Trainzland

    Joined:
    Nov 28, 2018
    Posts:
    2
    Count me in.
     
  7. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    785
    When I can get my game back to running with ECS serialization I may have something more to say. But in the mean time you can read this article : https://gametorrahod.com/unity-ecs-serializing-ecs-data-252231e5b16d

    I remembered Unity said this part of API is pretty much work in progress 3-4 months ago. So it is very rough. But I would like to onboard knowing that it enables me to not write a tedious save system which I have to make a replica of each IComponentData in some other format..
     
  8. MattiHietanen

    MattiHietanen

    Joined:
    Nov 24, 2018
    Posts:
    13
    Thanks for trying to help. Unfortunately your blog about the ECS serialization is interesting to devs that already know about all the stuff already and puzzle it from the images of code together. For me it's a bit hard to understand all the stuff for you is naturalness. :)

    You should be not afraid about hacking your game serialized chunks by changing the byte code of any file. Thee are many ways to encrypt the complete data block before writing depends on load times and machine id by jobs in less than ~4ms ;)

    For sure I will have a look at your blog time by time if it's done so far.
     
  9. N3V-Developer

    N3V-Developer

    Joined:
    Jun 7, 2014
    Posts:
    19
    Hi MattiHietanen, we can colaborate on this solution if you like. Pm me.
     
    MattiHietanen likes this.
  10. Quatum1000

    Quatum1000

    Joined:
    Oct 5, 2014
    Posts:
    628
    (Simplyfied) Does SerializeUtility.DeserializeWorld read the file data chunks, search for the references (by any InstanceID in world) and and pass the correct results into the entity.manager entities?
    But why to insert the components into the world previously, when the serialize data chunk contains all information about a entity including the shared components etc.?

    For me the behavior of De/SerializeWorld is not that logical and not programmers friendly in termas of "We wana do it as simple as possible". If I have the contents of an entity manager, it's a pretty nice data chunk containing all required information. If saving all the information into a file, and reload them, then all mesh, material transform, etc components must be already in the asset database, comparable by the unique IDs was saved. The De-SerializeWorld should connect all memory references (or references that pass to the asset database) at least.

    Currently everyone must deal with it's own serialization prodcedure to get the stuff to work anyhow. But in the end its all the same for everyone.
    For sure if someone requires a very special solution, then he must do it manually on it's own.
    Why we cannot simplify the process by:

    Code (CSharp):
    1. FileError = EntityManager.SaveAsync(fromEnity, toEnity, path.filename)
    2. *byte[] = EntityManager.SaveAsyncMemory(fromEnity, toEnity)
    3. bool = EntityManager.SaveHasDone()
    4.  
    5. EntityManager.LoadAsync(ref/inout fromEnity, ref/inout toEnity, path.filename)        // ... absolute
    6. EntityManager.LoadAsync(ref/inout fromEnity, ref/inout toEnity, unsafe *byte[])       // ... absolute
    7. EntityManager.LoadAddAsync(ref/inout fromEnity, ref/inout toEnity, path.filename)
    8. EntityManager.LoadAddAsync(ref/inout fromEnity, ref/inout toEnity, unsafe *byte[])
    9. EntityManager.LoadHasDone()
    10.  
    11. Perhaps:
    12. byte[] = EntityManager.PrepareDataMemory(GameObjectParent, [recursiveInChilds true/false])
    13. byte[] = EntityManager.PrepareDataMemory(GameObjects[])
    14. FileError = EntityManager.PrepareDataFile(GameObjectParent, [recursiveInChilds true/false])
    15. FileError = EntityManager.PrepareDataFile(GameObjects[])
    16.  
    It's an easy and common way to handle a basic serialization. Correct me if I'm wrong.
    If the unity team does anything in that way, would be great!

    I think the mega city ECS solution code is quite complex.


    Would love to have some comments on this. Thank you.
     
    N3V-Developer and MattiHietanen like this.
  11. MattiHietanen

    MattiHietanen

    Joined:
    Nov 24, 2018
    Posts:
    13
    Yes must admit this. Fairly enough for our needs. Cool!
     
  12. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    785
    "it's a pretty nice data chunk containing all required information."

    The chunk is missing shared component data. It has only integer representing the real shared component data which can be of any type. Only those indexes goes into the serialized chunk not the shared data + archetype representing the shared component data goes in the serialized chunk. Imagine I have `Animator` in the shared component data, that could not possibly be serialized together.

    "and reload them, then all mesh, material transform, etc components must be already in the asset database, comparable by the unique IDs was saved."

    On reload what's coming back are just some integer numbers and while they are unique they are just small running number (not like GUID which maps to an asset, which you might be thinking) so not comparable to anything in the asset database. Which unique value comes first will claim the lower number. If the value that comes later is not unique then internal ref count increased while using the same number.

    For them to link to the same thing again you got to have all the shared components lined up (added sequentially to empty world) in the exactly the same order. That is achieved by the GameObject which holds shared values. It is added back in order of components attached to the game object. Also it does not need to be prepared "before" if you use SerializeUtilityHybrid class. It is a wrapper which load shared in one step.

    (I just managed to serialize back and forth after ~20 hours, it is a hell lots of bugs and awkward API. Definitely requires improvement and I had to give up keeping IShared because too much hassle)
     
    Quatum1000 and eizenhorn like this.
  13. Quatum1000

    Quatum1000

    Joined:
    Oct 5, 2014
    Posts:
    628
    Thanks for your answers. Okay I try to explain.

    What MattiHietanen and me discussed is, to have the ability to load/save the contents of the entity manager.

    Small simple code example: 1 mesh will be add to the entity manager and should be drawn 10 times in the scene at different positions.

    Code (CSharp):
    1.  
    2. for (int i = 0; i != 10; i++) {
    3.   var Entity = entityManager.CreateEntity(
    4.                      ComponentType.Create<LocalToWorld>()
    5.                    , ComponentType.Create<MeshInstanceRenderer>()
    6.                    );
    7.   mesh.transform.position = random.xyz;
    8.   entityManager.SetComponentData(Entity, new LocalToWorld { Value = mesh.transform.localToWorldMatrix });
    9.   entityManager.SetSharedComponentData(Entity, new MeshInstanceRenderer {...});
    10. }
    entityManager.CreateEntity(
    ComponentType.Create<LocalToWorld>(),
    ComponentType.Create<MeshInstanceRenderer>()


    (I think) This will allocate un-managed memory for 2 components in memory for rendering from linear arrays. (native?) And the index referenced to these arrays are returned to the chunk from where your talking about.

    So my and MattiHietanen intention was to save/load the entity-manager.entity chunks and the data to the un-managed array where component data are linearly stored.

    Now on perhaps EnityManager.LoadAsync(,, filename) the complete EntiiyManager will be rejected and restored, including the un-managed memory array with the component data and it's references to the real asset database, or where ever they are located in unity.

    * EnityManager.LoadAsync() does exactly the same what the small code snippet above does.

    That gives everyone the ability to prepare any scene or situation directly after loading a scene, and in build as well. This has several advantages:
    • Save/load the current situation of all entities are involved in one EM.
    • Ability to handle real and large worlds and loading EM sets on entering any new area by player. with the async behavior.
    • Does not required to inserted shared components into the world before using SerializeUtility.DeSerialize or the complete SerializeUtility system at all.
    That was my intention to in this case.
     
    Trainzland and N3V-Developer like this.
  14. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    785
    It is 1 element of managed list of `object` and 1 unmanaged chunk memory.

    Since MIR is an ISharedComponentData you will get one managed memory holding that one mesh's reference fields, plus (unmanaged) one chunk for the archetype (LTW + MIR) containing linearly 10 LTW + 1 shared index of type MIR. If this is the first thing in the game, that could be the number 1.

    Entity Manager : List<object> : [MIR]
    Chunk#1 : Archetype MIR + LTW
    1 [LTW][LTW][LTW][LTW][LTW][LTW][LTW][LTW][LTW][LTW]


    This means the content of List<object> has to be serializable together to this `filename` somehow. How would you serialize arbitrary thing in a file? Unlike chunk, which is well defined what that chunk of memory means just by looking at the "header" archetypes attached with it.

    Naively doing that through the List<object> as it is would serialize you 2 integers of pointer value of mesh and material contained in MIR. That would be incorrect when loaded back, the correct serialization in this case should be keeping mesh and material's AssetDatabase GUID. (Similar approach to the Addressables Asset System). For other class/struct fields not representing Unity asset, it could be serialized using a standard C# [System.Serializable] scheme.

    Unity already have the said logic which serialize GUID of Unity things plus C# serialize for the rest (If it is public or [SerializeField]), the GameObject.

    Unity's answer to serialize the "unknown" currently is that if things in List<object> can become attachable to GameObject (That is constrained by the SharedComponentDataWrapper required for the operation), then it can piggy back on the classic game object serialization code as a storage.

    I feel that this is a "for the time being" solution by wiring to the existing GO code. The next good step for Unity would be serialize this GameObject together to the chunk file so the whole process is transparent like you said. (Should be possible with AssetBundle code?) And then finally make a dedicated code to serialize that List<object> which does not require you to prepare the alter ego ___Component : SharedComponentDataWrapper for every ___ : ISharedComponentData just so that it can piggy back the GO.
     
  15. Quatum1000

    Quatum1000

    Joined:
    Oct 5, 2014
    Posts:
    628
    Okay, but were here on well defined components that are restricted to the ecs system. We do not talking to lists containing delegates, functions calls or similar complex objects or tasks.

    I cannot see any real problems here. Indexing these 2 pointers on deserialize would be possible with an additional index that represent the original stored index. Same with mesh and material's asset database IDs. I'm sure it's solvable.

    But who can count on this? ;)

    Anyway, currently it's a performance pain to prepare an EM in a loop for eg. static GOs that have nothing special attached to, because all the required value pointers references and asset database IDs already exist. Currently there is no other deal to on loading a scene to crawl, select all GOs, peek all the stuff and poke'em into EM one by one.

    Perhaps the mega city project will bring some new options on loading async static objects with a scene. In the end I would really recommended to have a ecs script sub scene tool, you can select a static parent GO/prefab and all childs are prepared as well on for static scene rendering on SceneManager:LoadAsync()

    Thanks 5argon for your time & explanation.
     
  16. MattiHietanen

    MattiHietanen

    Joined:
    Nov 24, 2018
    Posts:
    13
    I also prefer any global ecs tool or script that does easify the process to load and save game objects sets into the em quatum1000 described.
     
  17. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    785
    Ok, I forgot to explain that List<object> I am talking about is the backend of ISharedCompoentData deep in the library, not a thing you intended to serialize. You store just a single int or complex mesh in IShared, they will be all together linearly in that list of EntityManager nonetheless. Are you talking about IShared should understand itself whether it contains simple defined component easily serializable, or complex object component? It can't do that now, it throws everything together as managed list of boxed unknown objects. That put difficulty on the serialization as they are arbitrary, out of strict typing system of ECS.

    Indeed it is solvable, because that's how GameObject serialization works together with .meta files. The problem is just that Unity do not want to write new code on this area of ECS yet (which I understand as other important area are quickly improving), so a fast and dirty solution is to use GameObject's serialization code as-is resulting in awkward API. The most important that enables this (dirty solution) is SharedComponentDataWrapper, which bridges GameObject code and ECS together.
     
    N3V-Developer likes this.
  18. Quatum1000

    Quatum1000

    Joined:
    Oct 5, 2014
    Posts:
    628
    Only simple components: mesh renderer, LocalToWorld and LOD perhaps, that's all. Same as MattiHietanen and N3VDeveloper told me. MattiHietanen animate it's stuff by jobs. I and N3V-Developer prefer displaying statically only. We throw all out is not required for animated objects. (eg. update matrix position calculation and y-culling on each frame are unnecessary).

    If it would possible to do by our self without unity support, then great. But I must admit I know the most common stuff.but not all the internal dependencies and complex ecs structures. Here in the forum there are only a handful of ppl have deeper acknowledge, including you as well.

    Now we have that great ecs system, but I'm really afraid this will leaved at some small use cases because of missing
    rudimentary methods. Pull out as much as possible return on capital is the most killer on complexer systems based on short time technological hypes. The next is Nvidia RTS, AI at all, transform64 and shaders64bit, that list is endless but not the lifetime of the developers.:)
     
    N3V-Developer likes this.
  19. N3V-Developer

    N3V-Developer

    Joined:
    Jun 7, 2014
    Posts:
    19
    Hi @5argon, can we access the list<ob> and examine the fields by reflection and value?
     
    Last edited: Dec 7, 2018 at 11:37 PM
  20. 5argon

    5argon

    Joined:
    Jun 10, 2013
    Posts:
    785
    While not recommended with reflection of course anything is possible.. please take a look at SharedComponentDataManager.cs
    The instance "m_SharedComponentManager" field (internal) is in each world's EntityManager, then "m_SharedComponentData" field (private) is that list.