Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Unite 2023 Registrations are now LIVE! We are thrilled to announce that Unite 2023 Registration is now live and open for all!
    Dismiss Notice
New Forum User Notice Unite 2023 Registrations are now LIVE!

Training for longer period of time shows worse results

Discussion in 'ML-Agents' started by unity_5kcJqPVlb9nWrA, Aug 4, 2021.

  1. unity_5kcJqPVlb9nWrA

    unity_5kcJqPVlb9nWrA

    Joined:
    Apr 15, 2020
    Posts:
    27
    Hello , Unity Members/Team!

    I have a self parking car agent and it does very good the first 15MIL steps. After that everything goes downhill and it doesn't perform even the tasks that it learned. Any ideas how I may fix that ?

    The longer I train him for first gets very good results and then back to worse .
     
    unity_5kcJqPVlb9nWrA, Aug 4, 2021
    #1

  2. vincentpierre

    vincentpierre

    Unity Technologies

    Joined:
    May 5, 2017
    Posts:
    160
    Does you environment reset properly when the agents terminate? It could be that the environment randomly enters a weird state that prevents the training from reaching the previous optimum. If you are using PPO, it is possible that the entropy bonus takes over the extrinsic reward. I would recommend tuning the beta coefficient for entropy (reducing it). If you do not have intrinsic reward signals, you could try using SAC (SAC does a better job at balancing entropy if that is the issue.)
     
    vincentpierre, Aug 4, 2021
    #2

  3. unity_5kcJqPVlb9nWrA

    unity_5kcJqPVlb9nWrA

    Joined:
    Apr 15, 2020
    Posts:
    27
    I do a lot of stuff in the EpisodeBegin method if that is what its meant by properly resetting the env ?
     
    unity_5kcJqPVlb9nWrA, Aug 4, 2021
    #3

  4. vincentpierre

    vincentpierre

    Unity Technologies

    Joined:
    May 5, 2017
    Posts:
    160
    Yes, you need to make sure there will not be a case where the resetting is not behaving like you expect. You also need to make sure the scene resets frequently enough (by setting a max_step value for instance)
     
    vincentpierre, Aug 5, 2021
    #4

  5. unity_5kcJqPVlb9nWrA

    unity_5kcJqPVlb9nWrA

    Joined:
    Apr 15, 2020
    Posts:
    27
    I think the problem was I was underfitting. Fixed the problem by just adding 2*hidden layers. It turned out superb. Thank you vince !
     
    unity_5kcJqPVlb9nWrA, Aug 6, 2021
    #5

  6. GamerLordMat

    GamerLordMat

    Joined:
    Oct 10, 2019
    Posts:
    177
    What exactly have you changed to counter the underfitting problem?
     
    GamerLordMat, Aug 13, 2021
    #6

  7. unity_5kcJqPVlb9nWrA

    unity_5kcJqPVlb9nWrA

    Joined:
    Apr 15, 2020
    Posts:
    27
    @GamerLordMat Sorry for late reply I was very busy lately with the diploma work. I had in the yaml configuration file 128 neurons in hidden layers and 2 hidden layers. I boosted them up to 256 and it worked nicely. I tried with 512 as well but that didn't show such good results :) I hope that helps.
     
    unity_5kcJqPVlb9nWrA, Sep 3, 2021
    #7