Storing a jagged aray to .csv or other file in coroutine?

Hi everyone,

I've made a little simulation of an ecosystem of creatures, and I'm trying to write a script to gather data on the creature's attributes. Every 10 seconds or so, each creature's attributes (speed, size, etc) will be written to a file, currently a .csv. Each attribute has it's own file, and since the number of creatures at any given point can vary, each row cannot be guarenteed to be the same size, so the final array will be jagged.

The simulation can be paused, and the speed can be changed, so the rate of data collection should be able to change with that.

Currently, I'm using a coroutine that looks something like this:

Code (CSharp):

1. IEnumerator DataCollectLoop() {
2.        using (StreamWriter writer = new StreamWriter(path){
3.             while (isCollecting) {
4.                 while (TimeManager.isPaused) {
5.                         Debug.Log("Data loop paused");
6.                         yield return null;
7.                 }
8.                     WriteData(writer);
9.  
10.                     Debug.Log("Data point collected");
11.                     yield return new WaitForSeconds(pollingPeriod);
12.                 }
13.     }
14.  
15.         Debug.Log("Data collection complete");
16.     }

The coroutine is activated by some functions that are attached to a play and pause button in the UI.

Code (CSharp):

1.     public void StartCollection() {
2.         isCollecting = true;
3.         StartCoroutine(DataCollectLoop());
4.     }
5.  
6.     public void StopCollection() {
7.         isCollecting = false;
8.     }

Currently I have two problems.

1. Pausing doesn't work as intended. Currently, I am implementing a pause by changing the timeScale to 0. This does pause the coroutine, as the WaitForSeconds becomes blocked until timeScale becomes > 0, but I intend for the file to be safe if the game is exited during a pause. I could use WaitForSecondsRealTime, but that would mean that the data collection rate no longer scales with the simulation speed.
2. If the simulation is paused, the coroutine will not actually stop until it has yielded, and gone back to the first while loop check. This means that if the game is exited before the coroutine finishes, the writer may not finish writing the file safely, as the using statement would be interrupted. I have solved this by moving the using statement around the writeData section, but this impacts perfomance.

Ideally what I would do is open the StreamWriter in the StartCollection method, and close it upon StopCollection (as well as stopping the coroutine), but from what I can tell, this isn't possible with the StreamWriter object.

Is there a way to achieve my requirements, either through scripting, or a different file format? I'm not married to .csv files, if there's a different format that is better for my needs, I will switch.

Thanks in advance!

 

its not recommended to have streams open for such a long time, it leads to the problems you are having.

It would be better if you took your waiting functions out of the using stream and collected your data to a list or array, and only opened the stream when your collection was ready to save.

 

Thanks for the response! So would a good solution be to have two coroutines, one that collects data into a buffer, and a second that periodically saves the buffer into the file?

 

since I dont know exactly what your method WriteData is doing this is something you could maybe do

Code (CSharp):

1.  
2.  
3. IEnumerator DataCollectLoop() {
4.          
5.                 while (isCollecting) {
6.                     while (TimeManager.isPaused) {
7.                             Debug.Log("Data loop paused");
8.                             yield return null;
9.                     }
10.                         //DataList.Add(datacollected);
11.    
12.                         Debug.Log("Data point collected");
13.                         yield return new WaitForSeconds(pollingPeriod);
14.                     }
15.         }
16.    
17.             Debug.Log("Data collection complete");
18.  
19. using (StreamWriter writer = new StreamWriter(path){
20. WriteDataList(writer);
21. }
22.         }
23.  
24.  

of course if the game stops while collecting nothing gets saved, but this way you will never have unsafe files