ECS for RTS - A few questions and feedback

Hello,

I am current working on an RTS game using the ECS.
I came across a few interesting topics which i solved more or less hacky and i'd like to hear if there are better ways to do it.

The first thing is gaining fine-grained control of the world updates, in my game i got 2 worlds - one for simulation and one for rendering.
The issue i am facing however is that i want to invoke the ECS updates from inside my own deterministic fixed lockstep implementation, which also should support being paused (stop invoking the ECS, in order to for example resync the game after a hash mismatched, more on hashing later) and be executed at fixed framerates.

What i am doing right now is creating my simulation world and keeping track of all created managers like this:

Code (CSharp):

1.             systems.Add(this.simulationWorld.CreateManager<UnitMovementCS>());
2.             systems.Add(this.simulationWorld.CreateManager<UnitCommandMoveCS>());

I disabled the player loop update for my simulation world only by calling ScriptBehaviourUpdateOrder.UpdatePlayerLoop(new World[] { renderingWorld });

At the point where i want to update the ECS i then call ScriptBehaviourManager.Update() on all previously created managers.

Code (CSharp):

1.             foreach (var system in this.systems)
2.                 system.Update();

This works, but it doesnt feel quite right and i cant imagine thats the intended way of keeping manual update control.
It would be very nice to have some World.Update() method that updates / ticks the world and also properly handles UpdateAfter / UpdateBefore / UpdateInGroup without being forced to execute the ECS updates in the player loop. Is something like this planned or are we supposed to update the systems ourselves?

Another topic i came across is verifying multiple game states are in sync.
In order to do this im hashing my game state into one hash that is exchanged between sessions and used to test if the own hash matches the others.

Right now im hashing quite some management stuff outside of ECS, but thats not an issue. The issue is hashing the ECS data (which is a lot, that was the point i looked into ECS after all ) is quite inefficient this way:

Code (CSharp):

1.             var em = ECSUtil.simulationEntityManager;
2.             ComponentDataFromEntity<UnitData> ud = em.GetComponentDataFromEntity<UnitData>(true);
3.             ComponentDataFromEntity<UnitMovementData> umd = em.GetComponentDataFromEntity<UnitMovementData>(true);
4.             ComponentDataFromEntity<DeterministicRenderTransformData> dtd = em.GetComponentDataFromEntity<DeterministicRenderTransformData>(true);
5.  
6.             foreach (var unit in this._units)
7.             {
8.                 unsafe
9.                 {
10.                     var entity = unit.entity;
11.                     if (ud.Exists(entity))
12.                     {
13.                         var _ud = ud[unit.entity];
14.                         context.HashCRC32("UnitData", (byte*)&_ud, sizeof(UnitData));
15.                     }
16.                     if (umd.Exists(entity))
17.                     {
18.                         var _umd = umd[unit.entity];
19.                         context.HashCRC32("UnitMovementData", (byte*)&_umd, sizeof(UnitMovementData));
20.                     }
21.                     if (ud.Exists(entity))
22.                     {
23.                         var _dtd = ud[unit.entity];
24.                         context.HashCRC32("UnitTransformData", (byte*)&_dtd, sizeof(DeterministicRenderTransformData));
25.                     }
26.                 }
27.             }
28.  

This results in pretty bad timings however, see this profiler screenshot:



It would be very nice if we could either directly recieve byte array references or pointers on the locations where ECS stores its data, or even better if the ECS itself could hash all data for us.
Is something like this planned or is there a better approach?

I have been using the ECS intensively now for some days and checked out most of the features and i have to say, so far it has been a really interesting way of solving problems.
Apart from those 2 things i explained above i didnt encounter any issues at all with the ECS (I would just be happy about a bit more documentation, even if just C# summaries). It works, performance is insane and it seems perfect for projects like RTS games

 

- update order: for now you need to update manually each system, as the UpdateBefore/After calculations are done inside UpdatePlayerLoop (and technically you can UpdateBefore(PlayerLoop.FixedUpdate) and it syncs with physics, so it's not simply an ordering)

- hashing: world should be able to serialize/deserialize into binary format. you could get the resulting byte[] and hash that

 

The problem about the update order is that i will have situations in which i dont want ECS to be updated at all, which is decided in one of my highlevel logic gameobjects.
An example reason for the ECS simulation to stop would be that in MP a client didnt submit the commands he wants to execute in a lockstep yet. In that case i have to wait for the client to submit the command before simulation can go on.

For now this workaround seems to be working just fine, im just fearing this will be broken at a later stage of ECS since i dont think this is the intended way for us to gain control over when ECS is updated.

The serialization approach seems interesting, i've quickly hacked together a test for this.
BinaryWriter for recieving serialization callbacks:

Code (CSharp):

1.         class StreamWriter : Unity.Entities.Serialization.BinaryWriter
2.         {
3.             public static StreamWriter instance = new StreamWriter();
4.             public byte[] data = new byte[4096];
5.             public int ptr = 0;
6.  
7.             public void Dispose()
8.             {
9.  
10.             }
11.  
12.             public void Reset()
13.             {
14.                 this.ptr = 0;
15.             }
16.  
17.             private void ExpandIfNeeded(int bytes)
18.             {
19.                 int spaceLeft = (this.data.Length - this.ptr);
20.                 if (spaceLeft < bytes)
21.                 {
22.                     int spaceNeededTotal = this.data.Length - (spaceLeft - bytes);
23.                     int needMoreBytes = spaceNeededTotal - this.data.Length;
24.                    
25.                     int growSteps = Mathf.CeilToInt((float)needMoreBytes / 4096f);
26.                    
27.                     Array.Resize(ref this.data, this.data.Length + (4096 * growSteps));
28.                 }
29.             }
30.  
31.             public unsafe void WriteBytes(void* data, int bytes)
32.             {
33.                 ExpandIfNeeded(bytes);
34.                 byte* dataB = (byte*)data;
35.                
36.                 fixed (byte* _data = this.data)
37.                 {
38.                     for (int i = 0; i < bytes; i++)
39.                         _data[this.ptr++] = dataB[i];
40.                 }
41.             }
42.         }

Actual serialization test code:

Code (CSharp):

1.             int[] serializedSharedComponents;
2.             Unity.Entities.Serialization.SerializeUtility.SerializeWorld(ECSUtil.simulationEntityManager, StreamWriter.instance, out serializedSharedComponents);
3.             byte[] d1 = new byte[StreamWriter.instance.ptr];
4.             Array.Copy(StreamWriter.instance.data, d1, d1.Length);
5.             StreamWriter.instance.Reset();
6.             Unity.Entities.Serialization.SerializeUtility.SerializeWorld(ECSUtil.simulationEntityManager, StreamWriter.instance, out serializedSharedComponents);
7.             byte[] d2 = new byte[StreamWriter.instance.ptr];
8.             Array.Copy(StreamWriter.instance.data, d2, d2.Length);
9.  
10.             for (int i = 0; i < d2.Length; i++)
11.             {
12.                 if (d1[i] != d2[i])
13.                     Debug.LogError("Difference detected at " + i + " - " + d1[i] + "|" + d2[i]);
14.             }

It looks as if the written serialized data is not deterministic, the error is triggered on every frame:



The serialization also doesnt seem to be optimized for runtime usage. The memory allocation is rather large (which is understandable, serializing stuff at runtime isnt a very common thing ):


I think the only way to achieve the performance needed to hash large amounts of entities is avoiding any memory copies at all and directly hash the data that is stored by the ECS using some cheap algorithm like CRC32 or xxHash.

 

I think you should be able get good the hashing performance using the new component versioning using 'ChangeFilter':

This way you could only recalculate and update hashes of entities that actually changed and update a world hash.

 

Very nice talk, thanks for linking it. I didnt see that one yet

The idea to use change filters seems to be pretty good, but the attribute is not yet in the current version retrievable by the package manager (0.0.12-preview8).

However i was able to speed up hashing performance a fair bit, up to a point that is usable for me at the moment.
In order to do so i added a component that keeps track of the hashes of every component data (this could be decreased up to only a hash per entity, but im keeping them atm to be able to better track down desyncs):

Code (CSharp):

1.     /// <summary>
2.     /// Hashing data for units.
3.     /// </summary>
4.     public struct UnitHashData : IComponentData
5.     {
6.         /// <summary>
7.         /// Hash of <see cref="UnitData"/>
8.         /// </summary>
9.         public int unitDataHash;
10.  
11.         /// <summary>
12.         /// Hash of <see cref="UnitMovementData"/>
13.         /// </summary>
14.         public int unitMovementDataHash;
15.  
16.         /// <summary>
17.         /// Hash of <see cref="UnitCommandMoveData"/>
18.         /// </summary>
19.         public int unitCommandMoveDataHash;
20.  
21.         /// <summary>
22.         /// All hashes combined
23.         /// </summary>
24.         public int unitHash;
25.     }

And in a job component system i hash the component data:

Code (CSharp):

1.     /// <summary>
2.     /// Hashing component system.
3.     /// Hashes the ECS data that represents the game state.
4.     /// </summary>
5.     public class HashingCS : JobComponentSystem
6.     {
7.         /// <summary>
8.         /// Hashes <see cref="UnitData"/>
9.         /// </summary>
10.         private struct JobUnitData : IJobProcessComponentData<UnitData, UnitHashData>
11.         {
12.             public unsafe void Execute(ref UnitData data, ref UnitHashData hashData)
13.             {
14.                 UnitData ud = data;
15.                 uint crc = CRC32.Compute(0, (byte*)&ud, sizeof(UnitData));
16.                 unchecked
17.                 {
18.                     hashData.unitDataHash = (int)crc;
19.                 }
20.             }
21.         }
22.  
23.         /// <summary>
24.         /// Hashes <see cref="UnitMovementData"/>
25.         /// </summary>
26.         private struct JobUnitMovementData : IJobProcessComponentData<UnitMovementData, UnitHashData>
27.         {
28.             public unsafe void Execute(ref UnitMovementData data, ref UnitHashData hashData)
29.             {
30.                 UnitMovementData ud = data;
31.                 uint crc = CRC32.Compute(0, (byte*)&ud, sizeof(UnitMovementData));
32.                 unchecked
33.                 {
34.                     hashData.unitMovementDataHash = (int)crc;
35.                 }
36.             }
37.         }
38.  
39.         /// <summary>
40.         /// Hashes <see cref="DeterministicTransformData"/>
41.         /// </summary>
42.         private struct JobTransformData : IJobProcessComponentData<DeterministicTransformData, UnitHashData>
43.         {
44.             public unsafe void Execute(ref DeterministicTransformData data, ref UnitHashData hashData)
45.             {
46.                 DeterministicTransformData ud = data;
47.                 uint crc = CRC32.Compute(0, (byte*)&ud, sizeof(DeterministicTransformData));
48.                 unchecked
49.                 {
50.                     hashData.unitCommandMoveDataHash = (int)crc;
51.                 }
52.             }
53.         }
54.  
55.         /// <summary>
56.         /// Combines all hashes on <see cref="UnitHashData"/>
57.         /// </summary>
58.         private struct JobCombineHashes : IJobProcessComponentData<UnitHashData>
59.         {
60.             public void Execute(ref UnitHashData data)
61.             {
62.                 data.unitHash = Essentials.CombineHashCodes(Essentials.CombineHashCodes(data.unitMovementDataHash, data.unitDataHash), data.unitCommandMoveDataHash);
63.             }
64.         }
65.  
66.         protected override JobHandle OnUpdate(JobHandle inputDeps)
67.         {
68.             // First hash all components
69.             var job1 = new JobUnitData().Schedule(this, inputDeps);
70.             var job2 = new JobUnitMovementData().Schedule(this, job1);
71.             var job3 = new JobTransformData().Schedule(this, job2);
72.  
73.             // And lastly combine all hashes
74.             var job4 = new JobCombineHashes().Schedule(this, job3);
75.  
76.             return job4;
77.         }
78.     }

This seems to be a lot more efficient than the previous approach (it can handle ~5k units + all other game simulation at stable 60fps in editor, with the ChangeFilter attribute most likely even more in most use-cases), however i think an ECS internal hashing method with direct access to the data will beat the performance of this by a wide margin.

 

Thanks for looking into this, Joachim!
I've checked again but i cant find ChangeFilter in the current release:




Is there another source for getting even earlier test packages of ECS outside of the package manager or am i doing something wrong (Im using AssemblyDefinition files, maybe a missing namespace i need to add manually)?

I have extensively profiled the hashing code and i see now that it actually is quite fast, the way i combined all hashes on the main thread just was suboptimal (ComponentDataFromEntity<UnitHashData> over all unit entities).

I replaced this with this component system:

Code (CSharp):

1.     /// <summary>
2.     /// The final hashing component system running after <see cref="HashingCS"/>, combining all hashes of:
3.     /// - <see cref="UnitHashData"/>
4.     /// </summary>
5.     public class HashingFinalCS : ComponentSystem
6.     {
7.         struct Group
8.         {
9.             public ComponentDataArray<UnitHashData> hashData;
10.             public readonly int Length;
11.         }
12.  
13.         [Inject] Group group;
14.  
15.         /// <summary>
16.         /// Combined <see cref="UnitHashData.unitHash"/>
17.         /// </summary>
18.         public int unitHash;
19.  
20.         protected override void OnUpdate()
21.         {
22.             int hash = 0;
23.             for (int i = 0; i < group.Length; i++)
24.             {
25.                 hash = Essentials.CombineHashCodes(hash, group.hashData[i].unitHash);
26.             }
27.  
28.             this.unitHash = hash;
29.         }
30.     }

And performance is a lot better! (I assume ComponentDataFromEntity has some overhead / non-linear memory access due to the order look up data for my unit entities).

I also tested burst and the results are absolutely insane!
Before (without [BurstCompile):



After (with [BurstCompile]):



On JobMovementData that is an acceleration of 13,6x. Absolutely amazing :O
I attached the full hashing system code file if anyone is interested in the code.

Last thing i would like to know is if unity is planning anything to change the manual update thing i mentioned above. The current solution clearly works very well (i saw it being suggested by some unity dev here in this forum but cant find the thread anymore), but something like World.Active.Tick() would be amazing (taking into account system orders based on UpdateBefore / UpdateAfter).

As final statement i can just say im blown away by the ECS, Jobs and the Burst Compiler. Very nice work

 

In our RTS (fully refactored to ECS, on first ECS versions, couple months ago) we also use own Update wrapper for manually call update on some systems (created manually and marked DisableAutoCreation), I think now is only one "right" way.

 

It seems that the ChangeFilter attribute has been renamed to ChangedFilter at some point.

 

@kennux I'm working on a simple RTS as well using a deterministic lock-step approach. This is how I have implemented it. Let me know what you think:

I derive all my systems from a base class

Code (CSharp):

1.  
2. class BaseSystem
3. {
4. override OnUpdate()
5. {
6. if (TickManager.ReadyToMoveToNextTick())
7.    OnTick();
8. }
9.  
10. protected virtual OnTick()
11. {
12. }
13. }
14.  

Code (CSharp):

1.  
2. class MovementSystem : BaseSystem
3. {
4. protected override OnTick()
5. {
6. }
7. }
8.  

In this approach, all systems inherit from BaseSystem and they all override the OnTick() method rather than OnUpdate(). OnUpdate() gets called every frame, but OnTick() gets called only when it's actually a new tick. You can use TickManager.ReadyToMoveToNextTick() to pause or slow down the tick frequency. The advantage of this is that you don't have to maintain your own list of systems, and this honours the UpdateBefore, UpdateAfter attributes.

 

@EvansT
The problems I've experienced with UpdateBefore / UpdateAfter were some inconsistencies about when they were called compared to regular monobehaviour methods (i tried to debug this but gave up at some point, in an empty project it seemed to work fine just not in my game simulation for some reason).

My code was split into GameObjects (for user input, ai, levels, ...) and ECS (for units, projectiles and game sim in general).
So i absolutely needed a way to control the exact time at which the game simulation was updated. If you can manage to keep your simulation in sync with the above approach and by only using ComponentSystems this seems fine to me

However i personally like ECS very much, but i find GameObjects for stuff like Actors (Player / AI), Levels, ... a lot more comfortable so i do not want to work in ECS only

So what i ended up with is (roughly) this running on a MonoBehaviour:

Code (CSharp):

1. public FixedUpdate()
2. {
3.     if (this._justFinishedLockstep)
4.     {
5.         // Check for advancement
6.         if (!this.canAdvanceLockstep.Check())
7.         {
8.             Debug.Log("Lag detected! Cant advance!");
9.             return;
10.         }
11.  
12.         this._justFinishedLockstep = false;
13.     }
14.  
15.     // New lockstep
16.     if (this._currentLockstepFrame == 0)
17.         this.StartLockstep();
18.  
19.     // Frame update
20.     FixedTimestep();
21.     this._currentLockstepFrame++;
22.  
23.     // End lockstep
24.     if (this._currentLockstepFrame == this.gameFramesPerLockstep)
25.         this.FinishLockstep();
26.  
27.     // Update ECS
28.     this.UpdateECS();
29. }
30.  
31. private Dictionary<int, ScriptBehaviourManager> systems = new Dictionary<int, ScriptBehaviourManager>();
32. protected virtual void UpdateECS()
33. {
34.     // Sort systems (TODO: cache)
35.     List<int> orders = ListPool<int>.Get();
36.  
37.     // Write all orders into list
38.     foreach (var order in this.systems.Keys)
39.         orders.Add(order);
40.  
41.     // Sort list
42.     Utils.InsertionSort(orders);
43.  
44.     // Execute systems in order
45.     foreach (var order in orders)
46.         this.systems[order].Update();
47.  
48.     ListPool<int>.Return(orders);
49. }

So i think it depends on your game what way you should go, but i think giving people the ability to update their systems manually in a more "official" way would be a good idea

 

Do you get the system information in the inspector (showing how long each system took in the frame [in ms]) using this method? I experimented with manually calling systems and did not see this information, which is a shame. Hopefully whenever they add a standard procedure for manually calling systems, this information will be available.

 

In case you mean the entity debugger stuff, no unfortuantely that doesnt seem to be available when using this kind of updating method.

This actually is another good reason to get an official way for manual update calls

 

I don't mean to necro an old post; however, I needed a similar system for rollback networking (checksums for verifying state divergence), and I've made a slightly more generalized, but still hardcoded solution to this problem. It makes use of the built in XXHash class in the ECS system for speed and uses full 64-bit hashes for both speed and lower rate of collisions. It disables a good number of safety checks and uses raw pointers everywhere, but it should be both stable and safe to use.

This solution utilizes a generic IJobChunk to generalize queries for specific components, and includes a entity ID based sort to ensure that reshuffling of entities via destruction or archetype changes will not affect the resultant hash.

My full implementation can be found here: https://github.com/HouraiTeahouse/F.../src/Runtime/Match/Systems/HashWorldSystem.cs

 

Interesting thanks.