Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice
  2. Ever participated in one our Game Jams? Want pointers on your project? Our Evangelists will be available on Friday to give feedback. Come share your games with us!
    Dismiss Notice

Drop in 'performance' after --resume

Discussion in 'ML-Agents' started by andrzej_, Jun 25, 2020.

  1. andrzej_

    andrzej_

    Joined:
    Dec 2, 2016
    Posts:
    74
    I noticed I get quite often drops in the extrinsic reward after I resume a training session and it takes a bit until the training 'recovers'
    upload_2020-6-25_11-20-43.png
     
  2. ervteng_unity

    ervteng_unity

    Unity Technologies

    Joined:
    Dec 6, 2018
    Posts:
    48
    Are the agents doing anything different/unusual at those beginning periods? Sometimes it's just a matter of the environment doing something slightly different on initialization, and the agents being confused for a bit.
     
  3. andrzej_

    andrzej_

    Joined:
    Dec 2, 2016
    Posts:
    74
    I'm using curriculum, so the environment did change (and actually those drops did trigger a reversal in the current 'lesson'), but those changes should make it easier not harder and the drop in reward caused the changes in curriculum, not the other way round.
    This is the Value loss graph (the sudden changes are even better visible along with stepping up the curriculum): upload_2020-6-25_23-52-30.png
     
  4. ervteng_unity

    ervteng_unity

    Unity Technologies

    Joined:
    Dec 6, 2018
    Posts:
    48
    Hi andrzej_, do you have the plot of lesson number for the curriculum? Resuming (at least prior to Version 0.17.0) resets the lesson back to 0, but during a run lessons cannot go "backwards" even if the reward drops.
     
  5. andrzej_

    andrzej_

    Joined:
    Dec 2, 2016
    Posts:
    74
    Sure ... here is the Lesson plot upload_2020-6-26_14-56-5.png
    ml-agents 0.16.1
    ml-agents-env 0.16.1
    TF 2.2.0
     
  6. ervteng_unity

    ervteng_unity

    Unity Technologies

    Joined:
    Dec 6, 2018
    Posts:
    48
    Ah yeah, it looks like the lesson is being reset when the training is quit/resumed. For ml-agents 0.16 and below, you have to add --lesson=X, where X is the last lesson you got to in the last run, to mlagents-learn to resume training where you left off.
     
  7. andrzej_

    andrzej_

    Joined:
    Dec 2, 2016
    Posts:
    74
    that drop makes sense now. Thanks a lot!
     
unityunity