Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Question Identical movements when running multiple behaviors using default_settings?

Discussion in 'ML-Agents' started by CallieChaos, Dec 1, 2020.

  1. CallieChaos

    CallieChaos

    Joined:
    Oct 3, 2019
    Posts:
    4
    Hi all,

    I'm setting up a system where multiple agents with different characteristics learn simultaneously. However, I want the initial NN settings in the .yaml file to be identical. So I've set up my yaml to have a default_settings category, and set the behavior names of each agent to be different. However, this makes all their movements identical (I.E. they're receiving the same values from the vectorActions in OnActionRecieved).

    This can be reproduced in 3dBall by changing each 3dBall Agent to have different behavior names and adding a default_settings section to the 3DBall.yaml file, copying the behaviors from 3DBall:

    How do I get each agent to receive its own set of vectorActions?

    Thank you!
     
  2. vincentpierre

    vincentpierre

    Joined:
    May 5, 2017
    Posts:
    160
    What versions of ML-Agents are you using?
    I tried to reproduce your bug on latest using the 3DBall example you provided, but each Agent seemed to be doing its own thing. (Taking independent actions)
    How did you make sure the actions were identical?
    Are the actions of the Agents identical even though the observations are different?
     
  3. CallieChaos

    CallieChaos

    Joined:
    Oct 3, 2019
    Posts:
    4
    I appreciate you looking at this. I took the opportunity to update to latest (0.22.0), but I'm still seeing the behavior there. I took a video of what's happening. As you can see, even though the position gets reset if the ball falls off, they're all moving in the exact same way, and this continues indefinitely, so I'm pretty sure it's not a seeding/randomization issue.



    Some other info that may be useful:
    The command that I'm using is "mlagents-learn 3DBall.yaml --run-id=test --tensorflow --force --time-scale 1"

    The only changes that I've made to the 3dBall project/scene is in the Behavior Parameters component
    1.) Change the behavior names of each agent from "3dBall" to "3dBall<num>" where <num> is 0 through 11 (just to give each of them a different behavior name).
    2.) Remove the pre-existing model from the model field.

    My 3dBall.yaml file looks like this:

    Thanks again for your help!
     
  4. vincentpierre

    vincentpierre

    Joined:
    May 5, 2017
    Posts:
    160
    I was able to reproduce and I tried to debug it a bit.
    First thing: This only happens with our TensorFlow trainers, I would recommend to switch to Torch trainers.
    The actions are actually not the exact same, they are only similar. I suspect it has to do with the way the network is initialized or with the way the actions are sampled (we sample Gaussian noise for continuous actions, I suspect the noise is sampled from a random number generator that is initialized the same way for all agents). I wonder if each Agent eventually learns or if this prevents training entirely. Training would be slower than usual since 12 models are being trained but I think it should still eventually converge. I need to do more experimenting.
     
  5. CallieChaos

    CallieChaos

    Joined:
    Oct 3, 2019
    Posts:
    4
    Switching to pytorch did indeed fix it, and that's enough for my purposes at the moment. Once I get my project running I'll be posting here about training multiple similar brains simultaneously. Thanks again for your research and debugging!