How to store state in a system

I am trying to visualize real-world prediction data. I have ~160 objects, and for each of them, I have their positions for every minute from now until the next 3 hours. I read these positions from a rather large text file via an HTTP request.

Currently, I have a system that will read this data every hour. After it parses all the data, it adds it to a dictionary in my movement system. The dictionary key is an ID associated with the game object to a Queue<PositionData>, containing 180 fairly simple PositionData structs. I also have a 2nd dictionary that maps the game object ID to the current PositionData struct.

My movement system calls Entities.ForEach and loops through all of my objects, stepping closer to the next data point. Once the minute is up and the next data point is reached, the movement system dequeues the next position data item and puts it the 2nd dictionary containing the current data.

Outside of interacting with those dictionaries and queue, all I'm doing is updating the transforms and rotations by adding the pre-cacluated step sizes. My game runs SSLLOOWW. Much slower than other ECS demos and benchmarks I've used. Am I doing something horribly wrong by storing so much data in the movement system? Should I be storing all of those values in the PositionData component? Or in another component that represents the queue/future values? Or maybe an external store like redis or something? I really have no idea what to do with all this extra data for the future positions, if that is the cause of my problems.

 

So I've read this a few times now, but I suppose it's never sunk in... should I be using SystemStateComponents?

https://docs.unity3d.com/Packages/com.unity.entities@0.14/manual/system_state_components.html

I'm struggling to find good examples of how to use this. Would I assign one SystemStateComponent to every entity that contains the future positions that I was previously storing in a Queue?

Or do I just use a Dynamic Buffer Component?

 

Did you use profiler, to check, if the issue is on the expected side?

I suggest to check for garbage collector as well.

Any code snipets, that you can provide?

 

If you are using the actual Dictionary and Queue + GameObject, then I should assume that you are doing things WithoutBurst and using Run, correct? Is so, probably this is why it is much slower than the samples, as everything is single-threaded. To take full advantage of DOTS speed you need to give up on those managed collections and GameObjects.

ISystemStateComponentData/ISystemStateBufferElementData are special component/buffer types that tell Unity that they shouldn't be automatically removed when calling Destroy on an Entity. They are great to store system states in a per-entity basis, but for state unrelated to entities I recommend to store with a common singleton ComponentData/BufferElementData, see this small thread about that: https://forum.unity.com/threads/singleton-vs-variables-in-the-system.960147/

 

Thank you very much for the response. I'm just starting out and have read the docs many times before starting my first project. Now that I have my hands dirty, going back and reading again sheds a whole new light on things. I quickly realized that while I got my code working, I was definitely stuck in some old OO paradigms. Things I'm doing wrong:

1) I had no idea how to write systems that don't need to capture local variables... everything was running on the main thread without burst
2) I didn't grasp the concept of system groups and when to make things run where.

So here is my new strategy:

1) In the initialization group, have my parser system run that checks the external sources for a) potential new game objects that need to be spawned (this will RARELY ever happen, but it will happen) and b) on a schedule, creates an http request to pull down all the prediction vector data. I have a struct with only blittable types called `MovementData` and a dynamic buffer that stores up to 60 `MovementData` structs. When a new batch of data is retrieved, I set the buffer component:

Code (CSharp):

1. DynamicBuffer<SatelliteMovementDataBuffer> buffer =
2.     commandBuffer.AddBuffer<MovementDataBuffer>(entityInQueryIndex, entity);
3. foreach (MovementData dataPoint in data)
4. {
5.     buffer.Add(dataPoint);
6. }

2) In the simulation group, my movement system checks to make sure the timestamp in the attached MoveData component is still valid. If so, I update the transform. If not, I grab the next MovementData struct from the buffer.

In my head, this works, and I've started reworking my code for this design, but now I'm having another weird issue. If I write any method in a system in the Initialization group that uses `Entities.ForEach`, the system will not initialize and it will not be added to the initialization group with no warnings or errors. Am I making another rookie mistake here? Should I just do everything in the simulation group and just use [UpdateBefore/After]?

 

Is it not initializing/being added, or is it just not running? I believe that by default the Entity Debugger doesn't show inactive systems (it being inactive, ie the query never matching, could be the reason you think that it is not being added to the group).

 

Ok that was it! But I don't understand...

So I can add a randomly named method to my system that NEVER gets called and has a really dumb ForEach query... and if that query doesn't match, the system is disabled?

 

Yes, because it codegens the GetEntityQuery to OnCreate.

If you want to force it to update always, I believe there is an "AlwaysUpdateSystem" attribute (or something like that) that you can use.

Or you can use IJobChunk + EntityManager.CreateEntityQuery instead of using Entities.ForEach.

 

This is so helpful. AlwaysUpdateSystem does the trick and I'm making progress now! I hope you don't mind if I ask one more question

This parser system needs to make ~300 HTTP requests to fetch all the data. Each request is about 10MB, so it takes a bit of time to get through all of them. I'm using the standard System.Net.Http HttpClient and calling async methods to invoke the client. After "await" returns, I create a new entity with the EntityManager. This worked in the past, although very slowly. It seems like only 2 requests were being made every few seconds and it took quite a while for all my game objects to pop into the scene. I never liked the fact that I have no idea how those async requests play with the job system.

Now that I'm actually designing this thing purposefully, I think this has become an issue. After I grab that first chunk of data, parse it, and try to create a new entity, I get the message:

`Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak`

I'm just guessing this is due to the long-running nature of the HTTP requests? I would love any advice on the right way to perform these requests in this system.

 

Do you have any undisposed native arrays?

Best if jobs are complied within a frame.
But shouldn't be running more than 4 frames. Put watchdog, to see, how long jobs are running.

Maybe try load external data first, from http request, then wen that is complete, execute the job. Personally I am not fan of forcing continuous running.

 

Can you share the actual code? Without it, I could only guess some random stuff.

 

I suppose I should also mention that I'm using DOTSNET, thus the [ServerWorld] attribute. Here is my code:

Code (CSharp):

1. using System;
2. using System.Collections.Generic;
3. using System.Net.Http;
4. using System.Threading.Tasks;
5. using DOTSNET;
6. using Unity.Collections;
7. using Unity.Entities;
8. using UnityEngine;
9.  
10. namespace ME.Server
11. {
12.     [ServerWorld]
13.     [DisableAutoCreation]
14.     [UpdateInGroup(typeof(InitializationSystemGroup))]
15.     [UpdateBefore(typeof(SpawnSystem))]
16.     [AlwaysUpdateSystem]
17.     public class ParserSystem : SystemBase
18.     {
19.         [AutoAssign] protected SpawnSystem spawnSystem;
20.         private List<string> _knownIds = new List<string>();
21.         private float _timeSinceLastUpdate = Mathf.Infinity;
22.         private int _updateInterval = 60 * 60; // in seconds - pull files every hour
23.         private static readonly string URLTemplate = "...";
24.         private readonly HttpClient _httpClient = new HttpClient();
25.    
26.         public EntityCommandBufferSystem CommandBufferSystem;
27.  
28.  
29.         protected override void OnStartRunning()
30.         {
31.             CommandBufferSystem
32.                 = Bootstrap.ServerWorld.GetExistingSystem<EndInitializationEntityCommandBufferSystem>();
33.         }
34.    
35.         protected override void OnUpdate()
36.         {
37.             _timeSinceLastUpdate += Time.DeltaTime;
38.             if (_timeSinceLastUpdate >= _updateInterval)
39.             {
40.                 _timeSinceLastUpdate = 0;
41.                 Update();
42.             }
43.         }
44.  
45.         async void Update()
46.         {
47.             Debug.Log("UPDATING!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
48.             await FetchEtls("myType", Static.IDS);
49.         }
50.  
51.         async Task FetchEtls(string type, string[] ids)
52.         {
53.  
54.             foreach (string id in ids)
55.             {
56.                 string url = string.Format(URLTemplate, id);
57.                 var data = await GetEtlAsync(url);
58.                 if (data != null)
59.                 {
60.                    MovementData[] dataArray = Parser.Instance.ParseData(type, id, data);
61.                     if (dataArray == null || dataArray.Length == 0)
62.                     {
63.                         Debug.Log(id + " returned no data.");
64.                         continue;
65.                     }
66.  
67.                     if (!_knownIds.Contains(id))
68.                     {
69.                         _knownIds.Add(id);
70.                         spawnSystem.Spawn(dataArray[0]);
71.                     }
72.  
73.                     UpdatePositionData(id, dataArray);
74.                 }
75.             }
76.         }
77.  
78.         async Task<string> GetEtlAsync(string url)
79.         {
80.             try
81.             {
82.                 var data = await _httpClient.GetStringAsync(url);
83.                 return data;
84.             }
85.             catch (Exception e)
86.             {
87.                 Debug.Log("EXCEPTION IN HTTP CLIENT. " + e);
88.             }
89.  
90.             return null;
91.         }
92.    
93.    
94.         public void UpdatePositionData(string id, SatelliteMovementData[] data)
95.         {
96.        
97.             EntityCommandBuffer.ParallelWriter commandBuffer
98.                 = CommandBufferSystem.CreateCommandBuffer().AsParallelWriter();
99.             FixedString32 ID = new FixedString32(id);
100.             Entities.ForEach((Entity entity, int entityInQueryIndex, in MovementData movementData) =>
101.                 {
102.                     // trying to update one single entity with the given id -- skip if this entity doesn't match
103.                    //  this means i'm calling ForEach in a for loop.  not sure if this is ok
104.                     if (!movementData.ID.Equals(ID))
105.                     {
106.                         return;
107.                     }
108.  
109.                     DynamicBuffer<MovementDataBuffer> buffer =
110.                         commandBuffer.AddBuffer<MovementDataBuffer>(entityInQueryIndex, entity);
111.  
112.                     foreach (MovementData dataPoint in data)
113.                     {
114.                         //buffer.Add(dataPoint);
115.                         buffer.Add(1);
116.                     }
117.                 })
118.                 .ScheduleParallel();
119.             CommandBufferSystem.AddJobHandleForProducer(this.Dependency);
120.         }
121.     }
122. }

^^ I might add that due to the nature of the error, I changed my buffer to be of type int, just for testing, to make sure that using the MovementData struct as the type wasn't causing problems.

Not sure if my spawn system is interesting. It's interesting to me because I'm just using entity manager to create entities and letting the DOTSNET server handle the magic (this is per DOTSNET examples):

Code (CSharp):

1.  
2. public void Spawn(MovementData data)
3. {
4.     Debug.Log("Spawning " + data.ID);
5.  
6.     if (prefabs.Get(spawnPrefabId, out Entity prefab))
7.     {
8.         Entity entity = EntityManager.Instantiate(prefab);
9.         EntityManager.SetName(entity, data.ID.ToString());
10.         float3 position = new float3(data.position.x, data.position.y, data.position.z);
11.         SetComponent(entity, new Translation {Value = position});
12.         SetComponent(entity, data);
13.         // spawn it on all clients, owned by no one
14.         server.Spawn(entity, null);
15.     }
16. }
17.  

There is more to to this class, but it's pretty DOTSNET specific.

I just did some more experimentation and it appears the main culprit is the Entities.ForEach loop in ParserSystem.UpdatePositionData, not in the spawn system. Actually, it appears to be specifically caused if I try to use the commandBuffer to call AddBuffer. If I remove the call to the commandBuffer and use Run() instead of ScheduleParallel(), the `InvalidOperationException: <>c__DisplayClass_UpdatePositionData_LambdaJob0.JobData.data is not a value type. Job structs may not contain any reference types.` error goes away and all my game objects spawn. but they're just missing the buffer with all the future positions i need

 

I currently never "new" any NativeArray. As a noob, I'm not entirely sure if there is anything I've done that would have created any.

I do like the idea of executing the http request first, then running the job. However, this needs to continuously run on an interval, from the second the server starts until forever. Every t minutes I'll see if I need to create any new game objects (this will be a long-running visual tool for my company), which should only happen a couple of times a year. Every hour I need to make the http requests and load the individual data for all of the game objects that I have created.

So my "Parser" system is just trying to keep my component data in sync with the real world data I'm fetching. Totally open to ideas here!

 

OK, I understand your approach. But do you really read http request every frame? Or just periodically. If I gather right, you don't build a game and you don't need real time response? Hence even reading data every second, that should give your app plenty time, to finish reading data, then process it in next frames, using for example jobs. That assuming, reading does not take longer than a second. Or whatever time frame. Then lag between reading data and processing it, should t be an issue iny understanding.

 

No, not every frame. That would be pretty impossible . Let me see if I can explain this better.

My company builds and launches satellites. We have about 160 currently in orbit. I don't want to manually manage game objects and convert them into entities, but instead, I want the "game" to detect when we've launched new satellites. For every active satellite, we generate this really nasty file. That file contains a list of future predicted positions for each satellite. Each line is one record, and the records are 60 seconds apart. There is 10 days worth of data in a single file.

My desired workflow is something like this:

1) Server starts.
2) Immediately, and then again one time per day, the server fetches list of all operational satellites.
3) For each satellite that doesn't currently have a game object, the server instantiates a prefab for each satellite.
4) On a separate schedule, I fetch the file for each satellite that has the 10 days worth of future positions. This data gets less accurate the further you go into the future, so I'm only going to use the first hours worth.
6) Grab the 60 data points and populate the dynamic buffer on each entity.
7) The movement system looks at the current `SatelliteMovementData` component and interpolates between the start and end positions for that minute.
8) When that full minute's worth of interpolations is done and we move on to the next minute, i grab the next `SatelliteMovementData` struct from the buffer and copy it into the standalone `SatelliteMovementData` component, where it will be used for the next minutes worth of interpolations.

So the goal is, the movement system blissfully always has a dynamic buffer with the next 60 positions for upcoming hour, and uses them 1 at a time, minute by minute, to interpolate the positions for that minute. It has no knowledge of schedules or when/how the buffer data is populated.

The parser system will continue to run forever in the background, making sure that any new satellites we might launch just show up within 24 hours, and ensuring those buffers are always full of 60 data points.

 

Ok, so I'm in a good place now

I found that if I don't try to use a separate entities query to set the buffer, it works fine. In my spawn system, I just add the buffer and populate it when I spawn my prefab.

Code (CSharp):

1. Entity entity = EntityManager.Instantiate(prefab);
2.                 EntityManager.SetName(entity, data.SatID.ToString());
3.                 var buffer = EntityManager.AddBuffer<SatelliteMovementDataBuffer>(entity);
4.                 foreach (SatelliteMovementData d in dataArray)
5.                 {
6.                     buffer.Add(d);
7.                 }

Now everything is loading and spawning great! AND... performance is WAY better! Before, my computer was rendered almost useless by how hard it was chugging. Now it runs without effort!

My next issue is..........

My movement system is not matching queries, so it's not active, so no movement. The movement system is being updated in the simulation systems group, while the buffers are being added in the initialization group. I can 100% see my buffers on my entities in the entity debugger, along with my SatelliteMovementData component. Here is my query in the movement group:

Code (CSharp):

1. Entities.ForEach((ref Translation translation, ref Rotation rotation, in SatelliteMovementData movement, in DynamicBuffer<SatelliteMovementData> buffer) =>

Is there a special mechanism to access the buffer? There is a GetBuffer method on EntityManager, but none on CommandBuffer.ParallelWriter. Not sure if this is useful info, but I only REALLY need to access that buffer once a minute per game object... so if I can't schedule the job nicely when calling EntityManager.GetBuffer, maybe I don't need to do that on every update?

 

Is it SpaceX by any chance?

Can you confirm if I am right?
Your buffer SatelliteMovementData, holds 60 points per satellite. And each entity is a satellite.

You can process each entity on its own thread.
But my question is, why do you want access buffer in parallel jobs?
With IJobParallelFor for example, you can make buffer, with [NativeDisableParallelForRestriction].
Like that

Code (CSharp):

1. [BurstCompile]
2. public struct MyJob : IJobParallelFor
3. {
4. [NativeDisableParallelForRestriction].
5. public BufferFromEntity <SatelliteMovementData> buffer ;
6. ...
7. // execute
8. {
9. }
10. }

Now you can write in parallel. But be careful, for race conditions.
I suggest, to not use Add, but instead, preallocate size of the buffer.

 

I work for Planet Labs, but we work closely with spacex. I honestly don't know exactly what I want! But your assumptions are correct. I just know that I want to schedule the job and not run it on the main thread.

I don't know if I fully understand the solution, but I'm on mobile right now so I will play with it soon!

 

That sounds nice

I suggest you look into DOTS samples, if you haven't yet.
You can learn a lot from them.

 

So I'm spending lots of time between DOTS samples and the official documentation. As I continue to make progress, it makes more and more sense. Until now, I've used Entities.ForEach and Job.WithCode, but never created a job using an IJobParallelFor. I'm definitely going to look into it and give it a shot though, if it will let me get access to the dynamic buffers. Hopefully it won't take me too long to get the hang of it!

If there is a solution that makes more sense without going parallel, I'm all ears. I just assumed I'd need to use a command buffer to avoid capturing the entity manager in my ForEach lambda, and the command buffer examples I've been referencing all use `CommandBufferSystem.CreateCommandBuffer().AsParallelWriter();`. I know I've seen another version that seems to allocate memory in some way and uses `Schedule()`, but I can't find that example again.

Should I just be using IJob instead? And would you mind explaining why Jobs can get away with using the entity manager, and I would be able to access my dynamic buffers with them and still schedule the job, and why that isn't possible with Entities.ForEach?

 

According to the documentation, I should be able to just add the DynamicBuffer as a parameter in my ForEach lambda. However, I can't get my movement system to match a query. I decided to take the DynamicBuffer parameter out, and only keep the SatelliteMovementData component in the query, and it still doesn't match and the system does not appear in the entity debugger. Here is my query:

Code (CSharp):

1. ntities.ForEach((ref Translation translation, ref Rotation rotation, in SatelliteMovementData movement) =>

I definitely have many entities with, at the very least, the SatelliteMovementData component (they all also have the dynamic buffer). Given the screenshot and my ForEach query, are there any gotchas I could be missing? Is the system disabled because the entities didn't exist when the system was created?

I've attached a screenshot that shows

 

Does the entity have a Disabled or Prefab component? Those get filtered out.

Just don't use ParallelWriter(). The type returned by CreateCommandBuffer() is that other version.

 

Thanks everybody so much for all of your help! I think my issue actually stemmed from DOTSNET. It requires that you disable auto creation on certain systems and author them via some custom attributes so that they are selectively created. I think I had all that messed up. Movement system is running now. My performance is still not great, but that could just be a combo of my laptop + the fact that I'm running server and client and streaming positions, but at least everything is working and designed quite a bit better.