Search Unity

Where are auto save model saved?

Discussion in 'ML-Agents' started by n3wb13, Mar 5, 2020.

  1. n3wb13

    n3wb13

    Joined:
    Jul 12, 2017
    Posts:
    24
    When you run a training auto save kicks in every 50k step. Example if I name the training "test1" I expect that the nn model are auto saved in test1 folder. However, when I copy the model inside (while the training is still on going) and rename it e.g. "mlagent100kstep". I notice that when using the supposed auto saved 100k model, the result is only equivalent to 0 step. However the fully trained version of the nn model is fine. So my guess is that the auto save are located somewhere else? Or is this a bug?
     
    n3wb13, Mar 5, 2020
    #1

  2. ascrawfo

    ascrawfo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    4
    The .nn models are actually only saved at the end of the training, or when exited with CTRL-C. What's being saved at these 50k steps are checkpoint (.ckpt) files, but the actual models won't be written until the training has completed/gracefully exited
     
    ascrawfo, Mar 5, 2020
    #2
    n3wb13 likes this.

  3. n3wb13

    n3wb13

    Joined:
    Jul 12, 2017
    Posts:
    24
    Thanks but just a follow up question. How do I save variants of .nn models in 1 training session? The things is I'm trying to have multiple difficulties of the agent and the perfect agent trains in about 5m steps (ppo + curriculum) so it's quite long. I'm planning to get variants every 500k so, I'm looking for a way to do this, aside from running the training multiple times, which is not ideal cause sometimes for some reason the training fails (agent just fails to learn some skills)
     
    n3wb13, Mar 6, 2020
    #3
    unity_-DoCqyPS6-iU3A likes this.

  4. ascrawfo

    ascrawfo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    4
    Currently, for this intended functionality, you would have to run the training multiple times. I would recommend using the `--load` flag to load an existing model when training, working your way up to your target steps. And I would also ask that you submit a feature request for saving .nn models at these autosave points so that it can be surfaced and prioritized!
     
    ascrawfo, Mar 6, 2020
    #4

  5. igsrd3unitytest

    igsrd3unitytest

    Joined:
    Mar 9, 2020
    Posts:
    5
    Hello dev, I'd like to know if the learning rate will be loaded or not? If it resets, the training result will not be the same.:confused:
     
    igsrd3unitytest, Mar 9, 2020
    #5