Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Saved Replay Buffer 13 MB -> 4KB after Training

Discussion in 'ML-Agents' started by BotAcademy, Aug 1, 2020.

  1. BotAcademy

    BotAcademy

    Joined:
    May 15, 2020
    Posts:
    32
    Hey,

    just trained the PushBlock example environment with 'save_replay_buffer' set to 'true'. During training the 'last_replay_buffer.hdf5' filesize was at 13MB. After training it decreased to 4KB.

    I wonder what causes this huge compression after training? I mean how can you store an experience replay buffer with 50k observations in 4KB of storage?
     
  2. celion_unity

    celion_unity

    Joined:
    Jun 12, 2019
    Posts:
    289
    Hi @BotAcademy,
    Can you be more specific about the steps to reproduce this? Are you interrupting training and using --resume?

    One other thing to note is that if you hit Ctrl-C to interrupt training, hitting it again could interrupt the saving, which would lead to a truncated file.
     
  3. BotAcademy

    BotAcademy

    Joined:
    May 15, 2020
    Posts:
    32
    When interrupting training, it works fine. Just experienced it when I finished a training run, without interruption. I'd still expect that the buffer gets correctly saved for the case that I want to improve an already trained agent and some point.

    Can be easily reproduced by training the PushBlock environment with SAC and save_replay_buffer:true
     
  4. celion_unity

    celion_unity

    Joined:
    Jun 12, 2019
    Posts:
    289
    OK, I can reproduce this quickly now (I added a log line after writing the buffer to output the file size. My guess is that we clear the buffer at the end of training before saving it. I'll look more into it tomorrow.
     
    BotAcademy likes this.
  5. celion_unity

    celion_unity

    Joined:
    Jun 12, 2019
    Posts:
    289
  6. BotAcademy

    BotAcademy

    Joined:
    May 15, 2020
    Posts:
    32
    No problems because I'm not using the saved buffer (just tested it for understanding).

    But if I'd use it I would expect a saved buffer at end of training in case I want to resume it with a greater step size at some point. If there are reasons to clear at the end even if 'save_replay_buffer' is set to true it's totally fine, but I don't see them, so its not the expected behaviour for me (but I am by far no expert on this).
     
  7. celion_unity

    celion_unity

    Joined:
    Jun 12, 2019
    Posts:
    289
    Good point. I've got this logged as MLA-1251 in our internal tracker.

    The clearing goes back to this change which prevents a memory leak from happening if some agents are learning and others are doing (python driven) inference. So it's still necessary, but maybe we can do it at a different time (or just not append to the buffer during inference).
     
    BotAcademy likes this.