Question Script is very CPU demanding

I am currently working on a project where I need to log the player's position and FOV (pitch, yawn and rotation) at every frame. The idea is to use that data to create a position heatmap and that sort of stuff. However, I am currently struggling at finding a good way to do this.
I initially decided to go the easy way and simply write to a CSV file the values, in realtime (in the Update() function). But I noticed that, as I start adding more assets and scripts, this approach is demanding a lot of my CPU (making my FPS drop to a miserable 20 FPS).
Then, I decided to just right the values to a list and, when the program ends (with the OnDestroy() function), iterate through this list and write the values to the CSV file. However, the project is supposed to last for 3 minutes which means that the list is HUGE, and doing this "list iteration" approach takes a very long time to complete (I still can't see the script reaching the end, I think Unity just freezes completely at that point).

So, after all this, I just wanted to get some opinions and tips on how to approach this problem the best way possible.

 

For my master thesis we also logged the position and rotation of several objects. In the worst case we had 3 virtual objects with real-object counterparts, up to 5 purely virtual objects, a HMD (plus the virtual camera) and both player hands. For all of this we logged basically most of the transform (minus scale), even including things like localEulerAngles and the forward vector. We also logged some other relevant informations on top of all this. So it was quite a bit of data.

As far as i remember we wrote a custom logger class, internally managing a StreamWriter object:
https://docs.microsoft.com/en-us/dotnet/api/system.io.streamwriter?view=net-5.0
In each scene we had a SceneManager object handling this current experiment / test run. So all the data was available in one spot. This object then handled its own copy of the custom logger and logged all relevant information in LateUpdate. The process of logging itself happened every frame, no buffers used (tho StreamWriter internally uses one), and we simply wrote to a .csv using out own layout. It did have some impact on framrate, but nothing too bad even considering we did the study for VR. If more performance is required you could outsource this to a different thread tho.
If you only log the player position and rotation and crumble down to 20 FPS, or logging this for 3 minutes crashes Unity at the end, then you are doing something wrong. Maybe you re-open the and read the file every time you want to write a new line of information? Hard to tell without seeing what exactly you are doing, but it certainly should be possible to log so much in real-time. That is not a lot of data whatsoever.

Ask yourself whether you even need to log these things to a textfile. Taking the example of a heatmap, you could do a ton of optimizations. For one, you dont have to track the position every frame. Players dont move that fast. Remembering their position once a second, or a couple times a second, should be more than enough. You also shouldnt have to keep a huge list of all the data. Instead you could apply the changes to the current heatmap once a minute or so. There you could remember the amount of samples per position, or area, or whatever it is you want to do, and once you updated these values, you can throw all of the currently kept data away and simply save the new heatmap relevant data. You could also decrease the resolution, ie only saving full integers, or even just the id of the area the player is in (depending on what is needed). In the end, all the relevant data could be reduced to saving something like "area a visited for 278 ticks, area b visited for 50 ticks, ..".
This is of course just an example. It always depends on what you really do. But you usually dont have to remember tons of data for extended periods of times, especially not over several game sessions.

 

The approach I would consider is move csv writing to its own thread. Instead of adding to your list, instead add to a concurrent queue. The separate thread then reads from the concurrent queue and writes to the file. That way you're not holding up the main thread with writing, and you aren't waiting to the end of the run to start writing either.

Note that you might be tempted to output data each frame using Debug.Log, but test performance with those commented out, because Debug.Log sucks up a lot more performance than you'd expect it would.

 

Ok yeah multithreading seems to be a nice option. Any idea where I can find things related to multithreading and Unity?

 

I was talking about general C# multithreading. Unity implements the Jobs system, but I don't know what kind of overhead it would produce for this kind of work.

 

Really?! Even five minutes of 60fps running is only 18,000 entries. If each entry is 100 bytes that isn't even the size of one single 1024x1024 texture! In 100 bytes you should be able to compactly write the position and rotation data for your player as just CSV decimal numbers.

This will not change anything. You're either blasting way more data than you need to or doing something else wrong.

For instance, I just ran for 3 minutes and recorded 10,000+ lines of a single GameObject's position and rotation with this script:

https://pastebin.com/BBVD6x0Z

It wrote a 716kb file in a fraction of a second right when I disabled the player.

 

I was making 1 of 3 assumptions, he/she was on some very performance restrictive hardware or very slow storage, the code for writing the CSV lines is extremely inefficient, or everything written to the CSV is also being output to Debug.Log and it is actually Debug.Log that is causing the performance problem.

Use of the Profiler could help here.

 

For what it's worth, in my main project I do a lot of CSV reading/writing. Writing 100k lines to a CSV takes less than 2 seconds on a decade old rented server using HDD, not even SSD.