Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. We have updated the language to the Editor Terms based on feedback from our employees and community. Learn more.
    Dismiss Notice
New Forum User Notice Update to the Unity Editor Software Terms

Saved Replay Buffer 13 MB -> 4KB after Training

Discussion in 'ML-Agents' started by BotAcademy, Aug 1, 2020.

  1. BotAcademy

    BotAcademy

    Joined:
    May 15, 2020
    Posts:
    32
    Hey,

    just trained the PushBlock example environment with 'save_replay_buffer' set to 'true'. During training the 'last_replay_buffer.hdf5' filesize was at 13MB. After training it decreased to 4KB.

    I wonder what causes this huge compression after training? I mean how can you store an experience replay buffer with 50k observations in 4KB of storage?
     
    BotAcademy, Aug 1, 2020
    #1

  2. celion_unity

    celion_unity

    Unity Technologies

    Joined:
    Jun 12, 2019
    Posts:
    289
    Hi @BotAcademy,
    Can you be more specific about the steps to reproduce this? Are you interrupting training and using --resume?

    One other thing to note is that if you hit Ctrl-C to interrupt training, hitting it again could interrupt the saving, which would lead to a truncated file.
     
    celion_unity, Aug 3, 2020
    #2

  3. BotAcademy

    BotAcademy

    Joined:
    May 15, 2020
    Posts:
    32
    When interrupting training, it works fine. Just experienced it when I finished a training run, without interruption. I'd still expect that the buffer gets correctly saved for the case that I want to improve an already trained agent and some point.

    Can be easily reproduced by training the PushBlock environment with SAC and save_replay_buffer:true
     
    BotAcademy, Aug 4, 2020
    #3

  4. celion_unity

    celion_unity

    Unity Technologies

    Joined:
    Jun 12, 2019
    Posts:
    289
    OK, I can reproduce this quickly now (I added a log line after writing the buffer to output the file size. My guess is that we clear the buffer at the end of training before saving it. I'll look more into it tomorrow.
     
    celion_unity, Aug 5, 2020
    #4
    BotAcademy likes this.

  5. celion_unity

    celion_unity

    Unity Technologies

    Joined:
    Jun 12, 2019
    Posts:
    289
    celion_unity, Aug 5, 2020
    #5

  6. BotAcademy

    BotAcademy

    Joined:
    May 15, 2020
    Posts:
    32
    No problems because I'm not using the saved buffer (just tested it for understanding).

    But if I'd use it I would expect a saved buffer at end of training in case I want to resume it with a greater step size at some point. If there are reasons to clear at the end even if 'save_replay_buffer' is set to true it's totally fine, but I don't see them, so its not the expected behaviour for me (but I am by far no expert on this).
     
    BotAcademy, Aug 6, 2020
    #6

  7. celion_unity

    celion_unity

    Unity Technologies

    Joined:
    Jun 12, 2019
    Posts:
    289
    Good point. I've got this logged as MLA-1251 in our internal tracker.

    The clearing goes back to this change which prevents a memory leak from happening if some agents are learning and others are doing (python driven) inference. So it's still necessary, but maybe we can do it at a different time (or just not append to the buffer during inference).
     
    celion_unity, Aug 6, 2020
    #7
    BotAcademy likes this.