Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only.

    Please, do not make any changes to your username or email addresses at id.unity.com during this transition time.

    It's still possible to reply to existing private message conversations during the migration, but any new replies you post will be missing after the main migration is complete. We'll do our best to migrate these messages in a follow-up step.

    On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live.


    Read our full announcement for more information and let us know if you have any questions.

Help Optimizing Visual Observations (Camera) performance

Discussion in 'ML-Agents' started by ChrissCrass, Mar 22, 2020.

  1. ChrissCrass

    ChrissCrass

    Joined:
    Mar 19, 2020
    Posts:
    31
    Visual observations are great, but computationally intensive, therefore I have been looking for ways to make it less resources intensive. Some good initial ideas are camera sharing /w offset decisions for multiple agents, but much more needs to be done to maximize the number of concurrent cameras that exist in a scene.

    There has to be other tricks and best practices that can help increase the performance of individual cameras (capping performance is important so we can use bigger images or more cameras for multi agent envs).

    Toward this end, I have been looking at the "Vertex lit" legacy texture rendering mode, which is apparently many times quicker than the new methods. It sacrifices realism, but if we're honest with ourselves, agent's don't need realism too badly.

    Lighting, mesh stuff, and all the other elements affecting camera performance probably hold several keys to improvement, but I'm inexperienced when it comes to these things.

    Advice is badly needed...


    EDIT:
    I originally included a potential bug report in this post, but i have since narrowed down the issue to something with different target frame rate settings between training and inferencing (I suspect the old academy variables are causing this). ML-agents 13.0
     
    Last edited: Mar 22, 2020
    MarkTension likes this.
  2. MarkTension

    MarkTension

    Joined:
    Aug 17, 2019
    Posts:
    43
    Have you tried making the camera/rendertexture input to the sensor smaller? That should make some difference.
    Also you could try making some modifications to the mlagents/trainers/models.py script.
    For instance, in my case I could decrease number of filters of the conv layers without impacting its performance too much.

    Would actually be great if there was a little more freedom in the architecture choices for future versions of mlAgents
     
    Last edited: Mar 22, 2020
  3. ChrissCrass

    ChrissCrass

    Joined:
    Mar 19, 2020
    Posts:
    31
    Already using 36x36 resolution. (the smallest possible res i think). I might see if i can thin out the CNN in models.py, but ultimately the real bottleneck may always be the rendering expense.

    I have made a raycast eyeball which isn't too bad for making a depth based render texture (yes, it's a big grid of raycasts xD), and in some cases it seems to out perform cameras...

    I suspect what we need is a seriously modified camera component which can be easily stripped down to the bare minimum.
     
  4. SeriousJoke

    SeriousJoke

    Joined:
    Dec 12, 2018
    Posts:
    1
    You might want to checkout the Grid Sensor which gets doesn't use rendering at all to get game data into a WxHxC tensor.
     
    Mehrdad995 likes this.
  5. ervteng_unity

    ervteng_unity

    Unity Technologies

    Joined:
    Dec 6, 2018
    Posts:
    150
    BTW, the minimum resolution depends on the `vis_encode_type`. The default "simple" type has a minimum resolution of 20x20, the "nature" CNN has 36x36, and the "resnet" actually has a smaller minimum resolution of 15x15.

    GridSensor is probably the way forward for this type of application, stay tuned!
     
    Mehrdad995 likes this.