Search Unity

Bug Cannot spawn more than 32 envs

Discussion in 'ML-Agents' started by GamerLordMat, Jul 14, 2023.

  1. GamerLordMat

    GamerLordMat

    Joined:
    Oct 10, 2019
    Posts:
    185
    Hello,

    When starting Unity ML training with more than 32 envs it has a time out and weird bugs. Is there a fix or just a hard limit? I have a lot of RAM left so it is a pitty that I can't spawn more (cant have more individual agents in the scene due to some restrictions)

    Thanks in advance!

    Cheers
     
  2. smallg2023

    smallg2023

    Joined:
    Sep 2, 2018
    Posts:
    144
    it's normal based on your yaml settings, the more your try to push the sizes the less environments you'll be able to fit
     
  3. GamerLordMat

    GamerLordMat

    Joined:
    Oct 10, 2019
    Posts:
    185
    hey thanks for answering, but I dont get your point. I should have plenty of space left both in RAM and GPU RAM
     
  4. Luke-Houlihan

    Luke-Houlihan

    Joined:
    Jun 26, 2007
    Posts:
    303
    I've done 35 environments and not had any issues so my guess would be something specific to your environment.

    How many cpu cores do you have? General advice is 2 per environment minimum. One for logic, one for physics. But this can depend on what is heavy within the environment.

    Have you changed the cpu limit for python processes in the mlagents python code? Unity (for some reason) hard coded the number of python process training uses to be 4 or less. I've changed this to use a max of 1/4 available cpu cores which trains much better on my 64 core workstation and 40+ core servers.
     
    Energymover likes this.
  5. GamerLordMat

    GamerLordMat

    Joined:
    Oct 10, 2019
    Posts:
    185
    thanks for the answer. I have an i9 13th gen with 24 cores and 32 threads. Could you please tell me how exactly I change it to enable what you said? My envs are very empty with only one agent (cant have more really)