Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Curriculum Training not increasing lesson.

Discussion in 'ML-Agents' started by MidnightGameDeveloper, Oct 6, 2020.

  1. MidnightGameDeveloper

    MidnightGameDeveloper

    Joined:
    Apr 26, 2014
    Posts:
    122
    Hello,
    I am trying to use curriculum training in combination with self-play. The problem is that during the whole training process, the lesson does not change. The parameter is shown in Tensorboard but does not change from the value defined in the first lesson. I did some research and it looks like in older versions it was necessary to call Academy.Reset(). Is something similar still necessary in the newer versions? I am using ML Agents Version 1.0.4 from the package manager and my version information from the console is:

    Version information:
    ml-agents: 0.19.0,
    ml-agents-envs: 0.19.0,
    Communicator API: 1.0.0,
    TensorFlow: 2.3.0


    I don't use maxStep (I set it to 0) in the AgentClass, but I guess the threshold is depended on the maxStep set in the configuration file anyways. To access the parameter I only use a single line of code, I don't have a custom Academy Class.
    Code (CSharp):
    1.  
    2. float obstacleScale = Academy.Instance.EnvironmentParameters.GetWithDefault("obstacle_size", 0);
    3.  
    Code (CSharp):
    1. environment_parameters:
    2.   obstacle_size:
    3.     curriculum:
    4.       - name: Lesson0 # The '-' is important as this is a list
    5.         completion_criteria:
    6.           measure: progress
    7.           behavior: CarAgentBehavior
    8.           signal_smoothing: true
    9.           min_lesson_length: 0
    10.           threshold: 0.1
    11.         value: 0.0
    12.       - name: Lesson1 # This is the start of the second lesson
    13.         completion_criteria:
    14.           measure: progress
    15.           behavior: CarAgentBehavior
    16.           signal_smoothing: true
    17.           min_lesson_length: 0
    18.           threshold: 0.3
    19.         value: 5.0
    20.       - name: Lesson2
    21.         completion_criteria:
    22.           measure: progress
    23.           behavior: CarAgentBehavior
    24.           signal_smoothing: true
    25.           min_lesson_length: 100
    26.           threshold: 0.5
    27.         value: 10.0
    28.       - name: Lesson3
    29.         value: 12.5
    30.  
    Bildschirmfoto 2020-10-06 um 13.23.30.png Bildschirmfoto 2020-10-06 um 13.23.30.png


    EDIT:
    So I am just trying to manage my lessons via C# for now, in a short test this worked quite well. The "issue" I have now is that it looks like the Academy.Instance.TotalStepCount are not the same as the steps in the Tensorboard. For example in Tensorboard the steps are already at 1.66 Mio while Academy.Instance.TotalStepCount is still at 1.45 Mio. Would be nice if someone can explain me the difference between them.
     
    Last edited: Oct 6, 2020
  2. henrypeteet

    henrypeteet

    Unity Technologies

    Joined:
    Aug 19, 2020
    Posts:
    37
  3. MidnightGameDeveloper

    MidnightGameDeveloper

    Joined:
    Apr 26, 2014
    Posts:
    122
    Hi, for testing I did set it to 500k. As you can see in the screenshot of Tensorboard the training stopped automatically at 500k as expected. Thanks for the link, that explains why there is a difference.
     
  4. henrypeteet

    henrypeteet

    Unity Technologies

    Joined:
    Aug 19, 2020
    Posts:
    37
    Thanks for the information, would it be possible to include your whole conifg (.yaml) file here? I am trying to figure out if this is a config issue or if this is something on our end. Have you tried running the example environment that uses curriculum learning (WallJump) even if it is with fewer steps?
     
  5. MidnightGameDeveloper

    MidnightGameDeveloper

    Joined:
    Apr 26, 2014
    Posts:
    122
    I attach my config file. No I haven't tried running the example environment yet. But I could give it a try.
    One thing that I have also noticed, is that I can only train in the UnityEditor. When I try to use mlagents-learn with the --env=<env_name> parameter it starts my game/build successfully (.app) but it does not start training. I only get this message after some time: The Unity environment took too long to respond.

    I have the correct scene as only scene in the build settings and it doesn't require any user interaction to start training. I think this must be a problem caused either through my project setup (I am using a Networking Solution for the multiplayer of my game) or my operating system macOS Catalina. Maybe it's the same cause for the problem with curriculum training.
     

    Attached Files:

  6. henrypeteet

    henrypeteet

    Unity Technologies

    Joined:
    Aug 19, 2020
    Posts:
    37
    Thanks so much. Can you try setting max_steps=50000 for testing, just to see if progress increases? I don't expect anything useful to be trained but it will be useful to check if the curriculum is working. The file you sent had max_steps=5M so it never would have hit lesson 1 IIUC since 0.1 progress = 500K steps
     
  7. MidnightGameDeveloper

    MidnightGameDeveloper

    Joined:
    Apr 26, 2014
    Posts:
    122
    Hi, I have changed the max_steps parameter for my testing yesterday, I did set it max_steps=500000 in my config file. Today I had another run with max_steps=5M also no increase in lesson. Should I do a test with the example environment with max_steps=50000?
     
  8. henrypeteet

    henrypeteet

    Unity Technologies

    Joined:
    Aug 19, 2020
    Posts:
    37
    Yes please or some smaller number of steps where we can be certain the 0.1 (10%) is reached. If that doesn't work then I can try to recreate this possible bug with an example environment. Did you have similar trouble with `config/ppo/WallJump_curriculum.yaml` and the walljump environment since it also uses curriculum?