Search Unity

Engine Stripping to reduce number of threads used by Unity?

Discussion in 'ML-Agents' started by MarcoMeter, Dec 8, 2021.

  1. MarcoMeter

    MarcoMeter

    Joined:
    Jun 14, 2013
    Posts:
    69
    Hello everyone,

    I'm using the ml-agents envs package to run Unity environments on a HPC. As a Unity build instance utilizes roughly 80 threads, a scaled up training process with more than 32 instances overloads the CPU. Hence, the Unity build instances become unresponsive.

    Is it possible to reduce the number of threads by stripping the engine?

    CPU: 2x AMD Epyc 7742 (32 cores)
    GPU: 2x nvidia Tesla A100
    RAM: 512GB

    I'm also working on utilizing more than one environment per build. Well, the low level Python API is not a charm to work with when it comes to multiple agents in one build. As the simulation of the agents seems to be asynchronous, each simulation step might not communicate a constant number of experience tuples. I would need to rewrite my training code so that it utilizes list structures instead of arrays of fixed length. That will cost quite some performance, which is likely to negate the effect of reducing threads. Any thoughts on this? Can anybody confirm that decision steps + terminal steps is not always equal to the agent count (let's assume that multiple agents of only one behavior are present)?
     
    Last edited: Dec 8, 2021