Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Question Agent not training in parallel but trains properly when it's just a single instance

Discussion in 'ML-Agents' started by aashwinsnambiar12, Mar 4, 2024.

  1. aashwinsnambiar12

    aashwinsnambiar12

    Joined:
    Feb 7, 2024
    Posts:
    2
    Describe the bug
    Hi there. First off I would like to share my appreciation for the whole team behind this project. I'm kind of new to ml-agents and unity. But, I have tried to learn maximum by going through the source code. I have a custom environment of a robot arm with two camera sensors and vector positions as input to the PPO network.
    My problem is that when I run a single agent in a one environment, the training takes around 8 hours to complete 2M steps but it produces a very good policy. When I try to replicate the same using multiple training areas (num-areas=4) in one single environment, it doesn't train properly at all. It learns some random policy. But, if I train one single agent in multiple environments (num-env=4), it starts training properly again and it gives a good enough policy.
    After going through the source code, it was my understanding that both num-env and num-areas are trying to achieve the same effect. Both are trying to parallelly produce more number of trajectories to fill up the buffer faster. And once the buffer fills up, the policy update steps takes place. Is my understanding correct?

    If so, any ideas on why it works with multiple num-env but not with multiple training areas. Has anyone else faced this issue? Am I supposed to change any particular parameters other than the num-areas when increasing the training areas? I tried out a few trial runs with varying batch size, buffer size, time horizons, beta.

    It would be really great if somebody could point out if I am making some mistake.

    To Reproduce
    Can't really share the custom environment.

    Environment (please complete the following information):

    • Unity Version: Unity 2022.3f1
    • OS + version: Windows 10 and Ubuntu 20.04
    • ML-Agents version: ML-Agents Release 21 branch
    • Torch version: 1.13.1
    • Environment: Custom