Help Optimizing Visual Observations (Camera) performance

Visual observations are great, but computationally intensive, therefore I have been looking for ways to make it less resources intensive. Some good initial ideas are camera sharing /w offset decisions for multiple agents, but much more needs to be done to maximize the number of concurrent cameras that exist in a scene.

There has to be other tricks and best practices that can help increase the performance of individual cameras (capping performance is important so we can use bigger images or more cameras for multi agent envs).

Toward this end, I have been looking at the "Vertex lit" legacy texture rendering mode, which is apparently many times quicker than the new methods. It sacrifices realism, but if we're honest with ourselves, agent's don't need realism too badly.

Lighting, mesh stuff, and all the other elements affecting camera performance probably hold several keys to improvement, but I'm inexperienced when it comes to these things.

Advice is badly needed...


EDIT:
I originally included a potential bug report in this post, but i have since narrowed down the issue to something with different target frame rate settings between training and inferencing (I suspect the old academy variables are causing this). ML-agents 13.0

 

Have you tried making the camera/rendertexture input to the sensor smaller? That should make some difference.
Also you could try making some modifications to the mlagents/trainers/models.py script.
For instance, in my case I could decrease number of filters of the conv layers without impacting its performance too much.

Would actually be great if there was a little more freedom in the architecture choices for future versions of mlAgents

 

Already using 36x36 resolution. (the smallest possible res i think). I might see if i can thin out the CNN in models.py, but ultimately the real bottleneck may always be the rendering expense.

I have made a raycast eyeball which isn't too bad for making a depth based render texture (yes, it's a big grid of raycasts xD), and in some cases it seems to out perform cameras...

I suspect what we need is a seriously modified camera component which can be easily stripped down to the bare minimum.

 

You might want to checkout the Grid Sensor which gets doesn't use rendering at all to get game data into a WxHxC tensor.