Memory leak using imitation learning

I'm training a virtual robot using only visual observations, everything was going as expected in the training phase until I tried to use imitation learning as it explains here: https://blogs.unity3d.com/es/2019/11/11/training-your-agents-7-times-faster-with-ml-agents/

It is true that the robot learns what to do faster than regular learning, but the memory used by the trainer grows linearly. If I let the trainer run 4-5 hours, python reach 13gb of RAM.

Here's the config file :

Code (CSharp):

1. behaviors:
2.   Robot_Imitation:
3.     trainer_type: sac
4.     hyperparameters:
5.       learning_rate: 0.0003
6.       learning_rate_schedule: constant
7.       batch_size: 64
8.       buffer_size: 500000
9.       buffer_init_steps: 1000
10.       tau: 0.01
11.       steps_per_update: 10.0
12.       save_replay_buffer: false
13.       init_entcoef: 0.01
14.       reward_signal_steps_per_update: 10.0
15.     network_settings:
16.       normalize: false
17.       hidden_units: 256
18.       num_layers: 2
19.       vis_encode_type: simple
20.     reward_signals:
21.       extrinsic:
22.         gamma: 0.99
23.         strength: 2.0
24.       gail:
25.         gamma: 0.99
26.         strength: 0.02
27.         encoding_size: 128
28.         learning_rate: 0.0003
29.         use_actions: true
30.         use_vail: false
31.         demo_path: Demos/RobotDemo.demo
32.     keep_checkpoints: 5
33.     max_steps: 10000000
34.     time_horizon: 128
35.     summary_freq: 10000
36.     threaded: true

 

Hello, that was exactly the problem, buffer_size should be 50.000 instead of 500.000.

Anyways, that is a lot of images. Do you know if having a big buffer for a visual observation problem is ok, or should I make it smaller?

Thanks for your time.

 

You have a serious memory leak issue. I've started oneof your examples the 3D ball. Within 30 minutes I've been using 2GB of memory. The issue is in the unity ML library 2.0.0