Search Unity

Fastest way to find which entities had a component changed

Discussion in 'Data Oriented Technology Stack' started by fholm, Feb 5, 2019.

  1. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    I've been trying to figure out a fast way to find out which entities had a specific component changed, for example if you have a 'Health' component and you update the value of 5 of them out of 10000 I'm looking for a good way to pick that up and process it.

    I know about Chunk.ChangeVersion for example, but this means I'd have to iterate every entity in that chunk doesn't it?

    Reactive systems only support added/removed as far as I can tell, which is not what I want (I think?) - as I'm modifying an already existing component not adding or removing one.

    Adding a separate flag component called 'HealthChanged' or something isn't a viable solution either since it means i'd have to create a flag component just to signify that the health value changed, the same with creating a separate entity which tells you the health value of another entity changed. I've not been able to find a way to do this.
     
  2. sngdan

    sngdan

    Joined:
    Feb 7, 2014
    Posts:
    987
    who changes the health component value? could this system not simply spawn 5 event entities that have a reference to your 5 entities who's value got actually changed? Those event entities get consumed by the other system that cares about this.

    Other than that, I think Chunk.ChangeVersion is not too bad. Check in the entity debugger, how many entities you have in a chunk. if you have 100 chunks with 100 entities you reduce to worst case 100 chunk checks + 500 entity checks (if the 5 changed ones are in different chunks)
     
  3. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    So this is a generic system intended for networking games, it's not a specific component i'm dealing with (the health component was just an example).

    Having to iterate over every entity in every chunk that possibly contains a changed entity (which isn't even sure, ChangedVersion only says that it might have changed) is just such a rough solution.

    Target goal here is to be able to have 100k live entities (of not the same archetype) in the world. I just see no way of solving this in the ECS as is today, been trying to turn this inside out for a few weeks.
     
  4. sngdan

    sngdan

    Joined:
    Feb 7, 2014
    Posts:
    987
    and there is also something I have not used, each chunk now has a chunk component. Maybe this allow to set offset positions in the chunk for the changed components?

    edit: cross post - did not see your reply when I wrote this
     
  5. PublicEnumE

    PublicEnumE

    Joined:
    Feb 3, 2019
    Posts:
    173
    It sounds like you may have already gone down this route, but it sounds like you might use a combination of generic systems, IJobProcessComponentDataWithEntity, the ChangeFilterAttribute, and manual system creation to get what you want:

    Code (CSharp):
    1. using System;
    2. using System.Reflection;
    3. using UnityEngine;
    4. using Unity.Entities;
    5. using Unity.Jobs;
    6.  
    7. public interface IChangeableData : IComponentData
    8. {
    9. }
    10.  
    11. public class CatchChangesSystemGroup
    12. {
    13. }
    14.  
    15. [UpdateInGroup(typeof(CatchChangesSystemGroup))]
    16. public class CatchChnagesSystem<T> : JobComponentSystem where T : struct, IChangeableData
    17. {
    18.     struct CatchChangesJob : IJobProcessComponentDataWithEntity<T>
    19.     {
    20.         public void Execute(Entity entity, int index, [ChangedFilter] ref T c0)
    21.         {
    22.             // do work on entities with changed T components here.
    23.         }
    24.     }
    25.  
    26.     protected override JobHandle OnUpdate(JobHandle inputDeps)
    27.     {
    28.         CatchChangesJob job = new CatchChangesJob();
    29.  
    30.         return job.Schedule(this, inputDeps);
    31.     }
    32.  
    33.     public static class IChangeableDataUtilities
    34.     {
    35.         [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.BeforeSceneLoad)]
    36.         public static void CreateEntityTemplateInstantiationSystems()
    37.         {
    38.             Type iChangeableDataType = typeof(IChangeableData);
    39.             Type catchChnagesSystemType = typeof(CatchChnagesSystem<>);
    40.             Assembly[] assemblies = AppDomain.CurrentDomain.GetAssemblies();
    41.  
    42.             for (int iAssembly = 0; iAssembly < assemblies.Length; iAssembly++)
    43.             {
    44.                 Assembly assembly = assemblies[iAssembly];
    45.                 Type[] types = assembly.GetTypes();
    46.  
    47.                 for (int iType = 0; iType < types.Length; iType++)
    48.                 {
    49.                     Type type = types[iType];
    50.  
    51.                     if (type.IsValueType && iChangeableDataType.IsAssignableFrom(type))
    52.                     {
    53.                         Type genericCatchChnagesSystemType = catchChnagesSystemType.MakeGenericType(type);
    54.  
    55.                         World.Active.CreateManager(genericCatchChnagesSystemType);
    56.                     }
    57.                 }
    58.             }
    59.  
    60.             ScriptBehaviourUpdateOrder.UpdatePlayerLoop(World.Active);
    61.         }
    62.     }
    63. }
    64.  
    Here I'm essentially creating a new System of type
    CatchChnagesSystem<T>
    for each ComponentType that might change. Each of these systems then runs a job that filters for components that have actually changed. Then, due to the generic nature of the System, you can act on these changed components using their specific type. The only real downside should be that you have a lot of systems...but I think that's just going to be a reality of ECS for now. :)

    Apologies, I didn't have time this morning to make sure this is only actually catching changed components. But it compiles, and may be worth checking out. GL!
     
  6. e199

    e199

    Joined:
    Mar 24, 2015
    Posts:
    100
    You can use generic component for that, to track changes
    SyncedState<T> : icomponent
    {
    public T Value;
    }

    It means you need to compare component instances, so you need to override IsEquals
     
  7. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    This is a good idea, but my understanding of [ChangedFilter] is that it just uses the version number behind the scenes and this will still trigger for every entity? Or at least there will still be a comparison done for every entity in every chunk?
     
  8. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    Also I'm getting this error:
    Code (csharp):
    1.  
    2.  
    3. ArgumentNullException: String reference not set to an instance of a String.
    4. Parameter name: s
    5. System.Text.Encoding.GetBytes (System.String s) (at <ac823e2bb42b41bda67924a45a0173c3>:0)
    6. Unity.Entities.TypeManager.CalculateMemoryOrdering (System.Type type) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/Types/TypeManager.cs:178)
    7. Unity.Entities.TypeManager.BuildComponentType (System.Type type) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/Types/TypeManager.cs:206)
    8. Unity.Entities.TypeManager.CreateTypeIndexThreadSafe (System.Type type) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/Types/TypeManager.cs:159)
    9. Unity.Entities.TypeManager.GetTypeIndex (System.Type type) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/Types/TypeManager.cs:122)
    10. Unity.Entities.ComponentType..ctor (System.Type type, Unity.Entities.ComponentType+AccessMode accessModeType) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/Types/ComponentType.cs:73)
    11. Unity.Entities.IJobProcessComponentDataUtility.GetComponentTypes (System.Type jobType, System.Type interfaceType, System.Int32& processCount, Unity.Entities.ComponentType[]& changedFilter) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/IJobProcessComponentData.cs:244)
    12. Unity.Entities.IJobProcessComponentDataUtility.GetComponentTypes (System.Type jobType) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/IJobProcessComponentData.cs:219)
    13. Unity.Entities.JobProcessComponentDataExtensions.GetComponentGroupForIJobProcessComponentData (Unity.Entities.ComponentSystemBase system, System.Type jobType) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/IJobProcessComponentData.cs:397)
    14. Unity.Entities.ComponentSystemBase.InjectNestedIJobProcessComponentDataJobs () (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/ComponentSystem.cs:113)
    15. Unity.Entities.ComponentSystemBase.OnBeforeCreateManagerInternal (Unity.Entities.World world) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/ComponentSystem.cs:104)
    16. Unity.Entities.JobComponentSystem.OnBeforeCreateManagerInternal (Unity.Entities.World world) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/ComponentSystem.cs:522)
    17. Unity.Entities.ScriptBehaviourManager.CreateInstance (Unity.Entities.World world) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/ScriptBehaviourManager.cs:21)
    18. Unity.Entities.World.CreateManagerInternal (System.Type type, System.Object[] constructorArguments) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/Injection/World.cs:137)
    19. Unity.Entities.World.CreateManager[T] (System.Object[] constructorArgumnents) (at Library/PackageCache/com.unity.entities@0.0.12-preview.21/Unity.Entities/Injection/World.cs:208)
    20. Bootstrap.Start () (at Assets/NetCode/Bootstrap.cs:50)
    21.  
    When trying to add this system:

    Code (csharp):
    1.  
    2.   public class ChangedSystem<T> : JobComponentSystem where T : struct, IComponentData {
    3.     struct Job : IJobProcessComponentDataWithEntity<T> {
    4.       public void Execute(Entity entity, int index, [ChangedFilter] ref T c0) {
    5.         Debug.Log(entity);
    6.       }
    7.     }
    8.  
    9.     protected override JobHandle OnUpdate(JobHandle inputDeps) {
    10.       Job job = new Job();
    11.       return job.Schedule(this, inputDeps);
    12.     }
    13.   }
    14.  
     
  9. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    Okey so updates to latest ecs preview, but getting this error now instead:

    Code (csharp):
    1.  
    2. NullReferenceException: Object reference not set to an instance of an object
    3. Unity.Entities.TypeManager.HashStringWithFNV1A64 (System.String text) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/Types/TypeManager.cs:465)
    4. Unity.Entities.TypeManager.CalculateStableTypeHash (System.Type type) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/Types/TypeManager.cs:475)
    5. Unity.Entities.TypeManager.CalculateMemoryOrdering (System.Type type) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/Types/TypeManager.cs:425)
    6. Unity.Entities.TypeManager.BuildComponentType (System.Type type) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/Types/TypeManager.cs:485)
    7. Unity.Entities.TypeManager.CreateTypeIndexThreadSafe (System.Type type) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/Types/TypeManager.cs:443)
    8. Unity.Entities.TypeManager.GetTypeIndex (System.Type type) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/Types/TypeManager.cs:318)
    9. Unity.Entities.ComponentType..ctor (System.Type type, Unity.Entities.ComponentType+AccessMode accessModeType) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/Types/ComponentType.cs:88)
    10. Unity.Entities.JobProcessComponentDataExtensions.GetComponentTypes (System.Type jobType, System.Type interfaceType, System.Int32& processCount, Unity.Entities.ComponentType[]& changedFilter) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/IJobProcessComponentData.cs:78)
    11. Unity.Entities.JobProcessComponentDataExtensions.GetComponentTypes (System.Type jobType) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/IJobProcessComponentData.cs:53)
    12. Unity.Entities.JobProcessComponentDataExtensions.GetComponentGroupForIJobProcessComponentData (Unity.Entities.ComponentSystemBase system, System.Type jobType) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/IJobProcessComponentData.cs:314)
    13. Unity.Entities.ComponentSystemBase.InjectNestedIJobProcessComponentDataJobs () (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ComponentSystem.cs:137)
    14. Unity.Entities.ComponentSystemBase.OnBeforeCreateManagerInternal (Unity.Entities.World world) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ComponentSystem.cs:127)
    15. Unity.Entities.JobComponentSystem.OnBeforeCreateManagerInternal (Unity.Entities.World world) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ComponentSystem.cs:612)
    16. Unity.Entities.ScriptBehaviourManager.CreateInstance (Unity.Entities.World world) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ScriptBehaviourManager.cs:27)
    17. Unity.Entities.World.AddManager[T] (T manager) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/Injection/World.cs:220)
    18. Unity.Entities.World.CreateManagerInternal (System.Type type, System.Object[] constructorArguments) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/Injection/World.cs:128)
    19. Unity.Entities.World.CreateManager[T] (System.Object[] constructorArgumnents) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/Injection/World.cs:195)
    20. Bootstrap.Start () (at Assets/NetCode/Bootstrap.cs:50)
    21.  
    ... sigh, i'm adding the manager with this:

    Code (csharp):
    1. _world.CreateManager<NetCode.ChangedSystem<Health>>()
    just as a test
     
  10. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    Okey so I got it running by making the system non-generic, but it will only react to add/removed, not to actual value changes on the component... so it's basically useless for what i want to do.
     
  11. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    It will, a chunks version number for a component is increased whenever someone acquires a write access to that component type from the chunk, so even if you change nothing the whole chunk+componenttype will be flagged as changed.
     
  12. sngdan

    sngdan

    Joined:
    Feb 7, 2014
    Posts:
    987
  13. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    6,939
    What about having single entity with buffer array, or even native array, which stores only latest changed properties, for relevant matching entities. Then you can have separate system, which iterates through populated elements of this store, and clears its content, when complete. As result, the length (not capacity) will be changing dynamically as needed, every frame, or system execution.

    However, this solution may require additional thinkering, if same property is added to store more than twice, in a frame. For example health of entity 3 was changed multiple times in single frame.
     
  14. PublicEnumE

    PublicEnumE

    Joined:
    Feb 3, 2019
    Posts:
    173
    @fholm Are you getting your latest error when you create the System? Or when it Updates?

    I was able to create them without errors on 2019.1.0a14, fwiw.

    I also got no errors while the game was running, although I also had no IChangeableData components in my World at the time.

    I've done things like this before with generic systems, except without using the ChangedFilterAttribute. Try removing that attribute, just to isolate that as the problem.
     
  15. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    This just isn't a valid option for a generic system, basically asking the users of the library to not only change the value but remember to spawn an event entity every time... that's just not a nice workflow or something i'd even want to work with in my own game.


    But how would I get the 'latest changed' properties? The problem I'm trying to solve is how I figure out *what* changed, without having to iterate over large chunks with potentially just a few changed entities in them.


    I saw only able to make it run without the generic parameter, basically hardcoding it to the "Health" component, otherwise it would not create properly (crashes during creation, not update).

    But again, ChangeFilter only seems to execute for added/removed as far as I can tell.
     
  16. PublicEnumE

    PublicEnumE

    Joined:
    Feb 3, 2019
    Posts:
    173
    Hmm. You should definitely be able to at least create the Systems (I was, using the code sample).

    But if [ChangeFilter] only works with Added/Removed components, then that's no good. :/

    Man...are you sure? I'm trying to find it's usage in the source code right now, but I'll need to leave soon. If that's true, Unity may want to change the name of that attribute.
     
  17. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    I can only get it to trigger on add/remove at least... maybe I'm doing something wrong (I hope I am as this would be the perfect solution to my problem)
     
  18. jooleanlogic

    jooleanlogic

    Joined:
    Mar 1, 2018
    Posts:
    332
    Yes

    There is no way around this. Ecs can't know about component changes at the entity level unless it were to cache before and compare after, or insert itself in front of every memory write operation. It'd then have to store a list of which indexes changed for each component or use a bitflag array or some such.
    Maybe they will actually implement an opt in feature like this in the future but there's no free lunch. A check has to be done on every entity one way or another.
     
  19. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    6,939
    You would be required in such case, for each system that makes change, to selected entities, to write into the store.
    This way you record keeping, if that makes sense.

    As result, you may end up in one frame store length of just 5 elements, another 80k (I.e. Initialization). Depends, how many changes to entities you do per frame.
     
  20. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    Okey... yeah I understand there's no free lunch, but a system which is intended to track delta changes for a large amount of entities in the world doing this on the chunk level has potentially massive performance issues.

    I can and have implemented very fast delta state tracking myself, where I can track exactly which entities changed very easily and cheaply, but my way of doing it isn't compatible with the way the ECS keeps its data as I need lower level access to the data as it's being modified.
     
  21. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    Yeah but this is a no-go for a generic networking system, asking people to add the changes they do to properties by hand
     
  22. PublicEnumE

    PublicEnumE

    Joined:
    Feb 3, 2019
    Posts:
    173
    I can't remember the specific case, but I believe in the past Unity has auto-refactored user code, to convert type members into properties. Those properties had extra code in their Setters that would report when they had been called to a higher observer.

    I remember being surprised by this at the time - just that they would make that design choice. I wonder if they would ever do it again.
     
  23. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    This is what UNET did with its weaver to track delta changes, yeah. But this only works in an OOP setting easily where you can have access to the 'container' (gameobject/whatever) and can store delta tracking on that
     
    PublicEnumE likes this.
  24. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    6,939
    Sorry, I can not think of better solution.
    Since it will required already by dev, to mark which properties should be checked for changes.
    At best, we could have special IComponentsDataChanged, which flags automatically, which entities with such component has changed. But that would be special feature request, to Unity devs.
     
  25. PublicEnumE

    PublicEnumE

    Joined:
    Feb 3, 2019
    Posts:
    173
    Yeah, wouldn't work in a Job. Not without some new ECB APIs....aaaand not even sure why I'm going down this rabbit hole. :p Never mind.
     
  26. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    The problem is further amplified because when dealing with networking systems for dedicated servers you can't just throw threads (jobs) at the problem to make it go away, as this will completely screw over scalability in terms of how many instances of a dedicated server executable you can fit on each machine, at most you want to use maybe 1.5 cores of CPU per instance of the game on average.

    If I could just throw threads at the problem then I could probably accept the [ChangedFilter] solution.
     
  27. jooleanlogic

    jooleanlogic

    Joined:
    Mar 1, 2018
    Posts:
    332
    I agree it would be costly on a global scale but you could perhaps do it for a custom solution. You would have to shadow duplicate the Health component which you could do with an IStateSystemComponent, so anytime you add a HealthComponent, it's shadow gets added as well. This would double the size of that component within the chunk but if you're doing delta comparison, you have to store the copy somewhere anyway.

    Then you just iterate, delta compare and update as you go.
    Code (CSharp):
    1. IJobProcessComponentData<Health, ShadowHealth>
    2. {
    3.     void Execute([ChangedFilter] ref Health health, ref ShadowHealth shadowHealth)
    4.     {
    5.         if (health != shadowHealth)
    6.         {
    7.             FilteredExecute(ref health);
    8.             shadowHealth = health;
    9.         }
    10.     }
    11.  
    12.     void FilteredExecute(ref Health health)
    13.     {
    14.     }
    15. }
    Maybe you make that into a parametized system or something.
    I can't see a lot of cpu overhead to this vs a non ecs delta compare scenario. You only have one comparison and one mem copy and only on changed entities. The memory/chunk overhead is the cost of the shadow, be that what it may.
    Between that and [ChangedFilter] it's maybe not so bad.
    If Health is a heavy component, then it might be better to add a byte sized shadow which you'd have to manually set to true whenever you change the HealthComponent. Per chunk, a byte sized filter is only going to take up a few dozen cache lines. I use this approach in my own systems that have to filter out individual entities where the main component is heavy.
     
  28. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    So this type of delta compression where you store shadow states (and potentially several per entity, depending on what each client has ack:ed back to you) is only really viable for games with smaller amounts of entities, etc. It goes completely out the window when dealing with 100k networked entities in a large world as the memory overhead is too great. There are much more efficient ways in terms of both CPU and Memory to do delta compression of networked state.

    Yeah I'm sad to say but I can't see a way to make my system work within the confines of the ecs... it sucks because I really want it to.

    I will say tho that your idea is the best one I've seen so far, but it's still not going to be enough I think. I'm gonna keep plinking away and thinking about it, so we'll see where it ends up in the end.
     
    Last edited: Feb 5, 2019
  29. AndesSunset

    AndesSunset

    Joined:
    Jan 28, 2019
    Posts:
    60
    Is this actually true? I believe Unity treats Zero-sized components differently - storing them as part of an archetype, without actually storing data for them in a Chunk. I certainly wouldn't tell you you're wrong, but this might be a good assumption to verify.
     
  30. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    It's not a zero sized component tho, it's a clone of the health data
     
  31. AndesSunset

    AndesSunset

    Joined:
    Jan 28, 2019
    Posts:
    60
    Oh! Of course. :p Well that makes it potentially simpler. Instead of a copy of the Health data, you could use a Zero-sized:

    struct ChangedComponentData<T> where T : IComponentData


    ...and require that the System which alters the component to Add one of these.

    ^...I hear you. I know you've mentioned above that asking this type of thing from the alterer would be too large of a requirement. A few thoughts to that point:

    1. There may be ways to automate or simplify this work, using things like Properties, extension methods, Utility methods- it's worth looking into. Is *any* requirement too much of a requirement? How simple could this 'reporting' code be?

    2. This may get simpler in the future, if/when Unity adds support for Arrays in IComponentData. You could just have a single Component on the Entity which tracks all the component types which have changed. That component could be included as a type in a ComponentGroup, or Job-related filter.

    I absolutely, totally understand not liking certain solutions. Though from my own dumb experience, being dumb: There's a big difference between "no possible solution in ECS", and "More cumbersome than I'd like in ECS". Thankfully, "more cumbersome" brings it into the realm of an optimization problem, which might improve over time. I'm talked to myself as much as anyone, there. :p

    Sincerely, good luck.
     
  32. sngdan

    sngdan

    Joined:
    Feb 7, 2014
    Posts:
    987
    I have no experience with networking, maybe thats why I have difficulties seeing your issue with the event approach. At least as a work around...

    I recall that there were two forum posts here with networking approaches, one from @Spy-Shifty and the other I forgot - maybe some inspiration there.


    You would not do this - they would not change values directly but would request value changes, if I stick with you Heath example...
    Health System -> changes health directly

    becomes...

    Health Request Change System -> creates event entity referencing the entity in question & the change in health
    Health Apply Chang System -> applies the change
    Other System -> does something with the changed entity
    End of Frame -> destroys event entity

    possibly not super performant but if the logic is that only very few of many (ie. 5 of 10000) change, possibly ok...
     
  33. AndesSunset

    AndesSunset

    Joined:
    Jan 28, 2019
    Posts:
    60
    The solution in my last post wouldn't allow for delta-compression. But don't give up! There's probably something out there.
    This wouldn't allow for delta compression. Nevermind.
     
  34. jooleanlogic

    jooleanlogic

    Joined:
    Mar 1, 2018
    Posts:
    332
    Yeah it's a heavy approach and not a solution that would work generically for a bunch of reasons, but the only other way I can see is to perform the check on every component access.
    I.e. You can make your own job types (IJPCD and IJobChunk) and cache and check on a per entity basis. E.g.
    Code (CSharp):
    1. Execute(ref health)            // Behind the scenes Execute
    2.     var shadow = health;
    3.     Execute(ref health);    // user Execute
    4.     if(shadow != health)
    5.         .. dump to command buffer or something.
    This avoids all the wasted memory but you're now performing the delta check on every access. Plus you'd have to use this custom system wherever you use HealthComponent so you didn't miss a change. I don't know which is worse.
    I think the only solution you have is a custom one where the system that changes the health fires an event then and there.

    Zero sized won't work in a delta solution cos they aren't stored per Entity, they're just part of the archetype.
     
  35. AndesSunset

    AndesSunset

    Joined:
    Jan 28, 2019
    Posts:
    60
    "It goes completely out the window when dealing with 100k networked entities in a large world as the memory overhead is too great."

    I keep wondering how true this is. Since it should be a functional (if slow) solution, I would be inclined to try it out and verify just how bad the performance is. Then spend some time optimizing, before deciding it's just not viable.
     
  36. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    So, a lot of replies here right now... instead of qouting all of you separately I'll try to respond in one go, so the reason the event system idea isn't feasible is simply that having the requirement of firing an event for every property thats networking that you want to change is just not a viable solution that's going to work with every game or that a lot of developers are going to accept.

    I have seen the other attempts at networking the ecs on the forums yes, and neither would scale to the level I am after. I know a few people are working on new stuff also, but nowhere near the scope I am chasing.

    Additionally using the job system isn't a viable option simply because I can not have a networking system that potentially will chew up all the threads on the server machine, you usually try to keep each instance of the server executable around 1.5 cores of CPU max - less is better of course. This essentially means it needs to be single threaded or it cuts to much into the bottom line (hosting becomes to expensive).

    On top of this, I have designed and built a system which can do all of my requirements (memory efficient, extremely fast delta compression, handling hundreds of thousands of entities) but it of course lives outside of the ECS. Now I'm fair and squarely aware that the ECS is most likely the future of Unity, so I would like to port this system to the ECS and I have just flat out failed so far, turning it inside out and on its head several times over, and I just can not make it work.

    I've been down the route of allocating memory by hand, writing my own custom collections which are blittable (hashmaps, queues, stacks, etc.) to let it all interface with the ECS properly, but the way the ECS is designed currently I can not make it work the way I want to/need to.

    Anything that involves scanning chunks of entities for changes when dealing with 100k entities on a single core just doesn't cut it to put it bluntly.
     
    AndesSunset likes this.
  37. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    You'll have to store maybe half a second (so 30 in a 60 tick game) snapshots for each entity back, depending on what the different clients have acked, after 30 ticks you start replacing them (basically a ringbuffer of snapshots), but this also means that if a client comes around that had previously seen the entity but then lost scope of it for more than 30 ticks he needs to delta compress his update towards the default zeroed out state (essentially sending full state for the entity).

    So the memory overhead is massive, and the bandwidth implications also. It's not a viable system for this level of scale.
     
    AndesSunset likes this.
  38. AndesSunset

    AndesSunset

    Joined:
    Jan 28, 2019
    Posts:
    60
    What you're describing is a pretty common need for networked games. If the current ECS design doesn't allow for a nice solution, Unity would likely want to know...and having recently had a team build the FPS networked demo in ECS, they probably have some detailed thoughts on this.

    Now would be a great time for knowledgeable folks on the ECS team to chime in. @Joachim_Ante?
     
  39. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    Yeah it absolutely is, but the FPS sample is a limited use case with a purpose built system for one game, they are using the delta compression technique of storing old snapshots... which is a great system for a game like that, its relatively simple to build, its easy to delta compress, etc.

    But that type of system will not scale to having thousands upon thousands of entities in the world, both memory wise and if you need to scan and serialize all entities to preserve snapshots, etc.

    Edit: I also want to say that the FPS sample is great, and is a really good implementation of the full world delta snapshotting system commonly used for competitive fps games, so great place for people that want to learn more about networking to look.
     
    Last edited: Feb 5, 2019
  40. jooleanlogic

    jooleanlogic

    Joined:
    Mar 1, 2018
    Posts:
    332
    What does your optimal solution actually look like?
    How are you currently checking for changes outside of ecs?
     
    AndesSunset likes this.
  41. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    So it has the ability to track what has changed as you update it, which itself isn't that hard (can use bitmasks or other techniques for that) but also can do delta compression without having to store any additional snapshots on the server or having to merge/track expensive bitmasks per client.

    Now this is a purpose built system, which uses similar techniques to the ecs for performance reasons (i.e. native memory, structs, pointers, etc. very few good ol classes, etc.) - which is why I thought that porting it to the ECS would be a breeze.

    But so far i've failed at every turn simply because it's too hard to inject code into the components to track the state changes as they happen, and even if I did there's no way to hold on to a reference to another component inside of an existing component or in other small state on the entity itself (i know an entity is just an index/version, i know they dont have state)

    I get that the ECS currently is very focused on batch processing of tons of entities, and it's a good first target to hit, but it'd be great to at least have the ability to more easily deal with entities on a singular level and in a more fine grained matter.
     
  42. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,839
    What you want is to double buffer all component data, that you want to synchronize.

    The change version is a per chunk early out. If anything on the component has changed you diff against the double buffer and generate the delta. This is 100% robust and quite fast. Dealing on a per entity basis in high perf code is not useful anyway, you will just lose perf by jumping around in memory.
     
    rigidbuddy and eizenhorn like this.
  43. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    Is this in relation to the code in the IJobProcessComponentDataWithEntity I write myself or something else in the ecs code itself that early outs? Because I'm getting a callback to [ChangeFilter] for every entity in a chunk that has a any changed entity in it no matter what, and I can't see a way of earlying out when checking the chunk versioning by hand outside of a IJobProcessComponentDataWithEntity job.

    So from my testing with the ECS, and while it is fast, trying to do this all on a single thread the overhead of the way the chunk versioning works becomes an issue.

    I've set up a simple test case in my prototype networking system in the ecs and my own system, again it's probably not 100% fair simply because my system is obviously purpose built - but I need to at least get close to the performance for the ECS variant to be viable. The case is this:

    1) 100 000 entities of varying complexity (2-16 properties each, all entities are spawned in the same positions for both systems)
    2) Sweep a sphere in a fixed pattern over the world that changes entities in a set radius around it (to simulate a player moving around affecting the world)
    3) Find all changed entities

    On average around 1000 entities are changed with each simulation tick. Now for finding all the changed entities it doesn't matter if I need to jump around in memory or not - I still need to visit all places in memory that has a changed entity in it. So no matter if I can jump directly to a specific entity (like I can in my system) or I have to scan the whole chunk the entitie(s) are in to find it... I'm just spinning my wheels checking for changes for all entities in a chunk that have not changed simply because the ECS can not provide me a list of 'changed' entities and only provide 'possibly changed chunks'.

    My system uses a similar type of linear/cache aware memory layout as the ECS, so they are comparable in that regard. Obviously the ECS is more fully featured in terms of what it can do, but the rough memory layout system is similar.

    Again this is 100% single threaded simply because throwing threads on this problem isn't acceptable when running a dedicated server for a game with a CPU core budget of ~1.5.

    Don't missunderstand me here, I want to make this work in the ECS and would love to port my system over - but I just can't see a way of doing it efficiently with the way the chunk versioning currently works.
     
    Last edited: Feb 6, 2019
  44. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    Okey so to provide some performance numbers, here's a simplified setup:

    1) I create 100k entities with a total of 5 components spanning total 16 properties (total byte size of all components are about 96 bytes)
    2) Each frame modify the position component of 1000 entities
    3) Use archetypechunkquery API to do chunk iteration, using DidChange to check for which were modified
    4) If a chunk was modified, then compare the position and positionshadow values with each other to detect a change, and store the new value in the shadow

    When built in *release* mode this takes around 1 millisecond to execute on my modern workstation (threadripper 2950x, 32gb ram). There's no other load on the system. When compiling with IL2CPP it's slightly faster, around 0.8ms

    This is 1 millisecond to just detect that something changed for one component type, I've got no idea how I'd do to scale this to many component types and more modified entities.

    I can of course throw all cores on my CPU at it (using IJobProcessComponentDataWithEntity), and then it completes in something like 0.15ms, but that's still only for checking if one components values changed, and this is hard coded to specifically check the position component and its shadow - it's obviously going to cost more when I need to make the system generic and to be able to only sync changed properties on a component, add serialization etc. on top.

    Don't take this as direct critique of the ecs, I am simply trying to find a way to make this scale and I would love if someone told me that I'm doing something wrong, or maybe do it like this, etc. instead
     
  45. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,839
    Are you using burst for doing the comparisons? You can run C# jobs on the main thread using .Run() method if you dont want your server to run multithreaded.
     
  46. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,839
    >I've got no idea how I'd do to scale this to many component types and more modified entities.
    The point is that there is cost two places that needs to be traded.

    1. Cost of writing a value. (A callback would clearly be unacceptable for perf, builtin comparison etc all requires complex code to run on every assignment and really prevents good batch processing) It also means that the needs of netcode like into game code in terms of perf.

    2. Cost of figuring out what has changed. Clearly having exact information already prepared would be useful for networking.

    In all tests we have done when looking at the whole picture of simulation code + netcode, doing things chunk based is better.
     
    rigidbuddy likes this.
  47. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    Not using burst, I will try after lunch... I did have some issues with my generic serializer where burst would crash when I tried to cast a byte* to a float3* ... I know alignment was correct and so. It worked fine with floats and ints.
     
  48. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    I'm running into an issue trying to Execute() the job on the main thread:

    Code (csharp):
    1.  
    2. Assertion failed on expression: 'jobData->range.numJobs == 1'
    3. Unity.Jobs.LowLevel.Unsafe.JobsUtility:ScheduleParallelFor_Injected()
    4. Unity.Jobs.LowLevel.Unsafe.JobsUtility:ScheduleParallelFor(JobScheduleParameters&, Int32, Int32)
    5. Unity.Entities.JobChunkExtensions:ScheduleInternal(Job&, ComponentGroup, JobHandle, ScheduleMode) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/IJobChunk.cs:53)
    6. Unity.Entities.JobChunkExtensions:Run(Job, ComponentGroup) (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/IJobChunk.cs:32)
    7. NetCode.ChangedSystem:OnUpdate(JobHandle) (at Assets/NetCode/BufferSystem.cs:104)
    8. Unity.Entities.JobComponentSystem:InternalUpdate() (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ComponentSystem.cs:588)
    9. Unity.Entities.ScriptBehaviourManager:Update() (at Library/PackageCache/com.unity.entities@0.0.12-preview.23/Unity.Entities/ScriptBehaviourManager.cs:83)
    10. Bootstrap:FixedUpdate() (at Assets/NetCode/Bootstrap.cs:118)
    11.  
    The job is just a IJobChunk, maybe its the wrong jobtype?
     
  49. fholm

    fholm

    Joined:
    Aug 20, 2011
    Posts:
    2,043
    Absolutely there's two sides of this coin, i.e. the overhead that the implicit change tracking causes - but I've seen the opposite in my tests, while assignment of a value is slightly more expensive (x2 vs just setting a regular field on a component) the cost of having to check all the objects in potentially changed chunks is still more.

    I'm under no illusion that doing automatic change tracking is always going to be cheaper, but where these two trade-offs cross each other on the graph of cpu cost probably varies a lot depending on which game and test scenario being run.

    BTW it'd be amazing if we could burst compile a static method and call that from the main thread without going through jobs, is that a feature that's being planned? I know there's some delegate API in relation to burst but not sure how to use it or if it's ready at all?
     
  50. jooleanlogic

    jooleanlogic

    Joined:
    Mar 1, 2018
    Posts:
    332
    Perhaps problem with parallel aspect of it? I don't know what Run does under the hood.
    If you're just trying to perf test, you could try an IPCD.ScheduleSingle() version which may run on another thread but won't parallel process.
     
unityunity