Difference in SAC training between ml-agents 0.13.0, 0.14.1 and 0.15.0

I've been working in a project in which I used ml-agents from 0.8.0 to 0.13.0 and the agent trained well with PPO and SAC since it was released for 0.10.0. In 0.13.0 my agent learned fast with SAC (in about 300k steps) as it can be seen in the image:

In this case, the reward stabilizes at 300k more or less at 12, and it is ok for the episode length to decrease as the agent needs to learn to do the action as fast as it can, up to 25.

For training in 0.14.1 and 0.15.0 I followed the instructions in the migration guide, and multiplied the max_steps by the number of agents in my environment, used the same parameters as in 0.13.0, but the training didn't converge:

The reward never stabilizes and doesn't reach 4, while the episode length stays between 350 and 500 which is the max number of steps of my agent.

The parameters I used in 0.13.0 were:
RobotLearning:
trainer: sac
batch_size: 128
buffer_size: 500000
buffer_init_steps: 0
hidden_units: 128
init_entcoef: 1.0
learning_rate: 0.0003
learning_rate_schedule: constant
max_steps: 500000
memory_size: 256
normalize: true
num_update: 1
train_interval: 1
num_layers: 2
time_horizon: 1000
sequence_length: 64
summary_freq: 3000
tau: 0.005
use_recurrent: false
vis_encode_type: simple
reward_signals:
extrinsic:
strength: 1.0
gamma: 0.99

And for 0.14.1 and 0.15.0 max_steps: 12500000 and summary_freq: 75000 (I have 25 agents).

I'd like to know if there is another parameter I'm missing.

Thanks in advance for any help.

 

I'm using python 3.6.10 and .NET Framework 4.8.03752 (I assume C# 7.3).
Thank you!

 

Hi, thanks for your response.

Yes, the buffer_size is equal to the max_steps, but that is the value that I had originally for buffer_size in the training configuration that was working in 0.13.0. I'm using the same value for 0.15.0, I multiplied only max_steps and summary_freq.

Anyway, I'm going to try a lower value for buffer_size to see if it makes it work. I'll let you know.

 

I tried with a buffer_size of 50000 and the results were similar in 0.15.0:

The pink line is the new training. I'm going to try changing other parameters to see if there's a change in convergence.

 

Hi ervteng_unity,
Here are the tensorboard graphics for SAC training with 0.13.0:

and SAC training with 0.15.0:

I'm going to do the same for PPO to see if there's any change.
Thanks in advance for any help!

 

Hi ervteng_unity, sorry for the late response.

The first run is repeatable on ml-agents 0.13.0. The environment has rewards every step, so sparse rewards are not a problem. Here are the TB graphics for 3 repetitions of SAC in 0.13.0:

As you can see here, it always learns quickly, but in 0.15.0 it doesn't.

Also, I did some runs for PPO with ml-agents 0.10.1 (red), 0.13.0 (orange) and 0.15.0 (blue):

In this case, the red and orange lines are similar in most of the graphics, but the blue one has some peaks, and the entropy graphic is very different for 0.15.0 too.

If we compare SAC on 0.13.0 with 0.15.0 in entropy, they are different too:

Maybe a change to beta on PPO or ent_coeff on SAC is needed with the version change?

 

Thanks for the response!

Yes, with every version I change the ML-Agents folder and install the package from ml-agents/com.unity.ml-agents with Unity's package manager. I also checked the steps to migrate (https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Migrating.md).

I'm not using raycast sensors, only Vector3 observations.

I'm gonna try with that version and let you know.

 

Hi! I used ml-agents 0.14.1 for SAC (blue) and PPO (pink), and here are the results:

The parameters were:

PPOLearning:

    batch_size: 2024

    buffer_size: 20240

    max_steps: 12500000

    normalize: true

    num_epoch: 3

    time_horizon: 1000

    summary_freq: 75000

    reward_signals:

        extrinsic:

            strength: 1.0

            gamma: 0.995



SACLearning:

    batch_size: 128

    buffer_size: 500000

    max_steps: 12500000

    normalize: true

    time_horizon: 1000

    summary_freq: 75000

With the same params it works for me in versions 0.10.1, 0.13.0 and 0.14.1. The only one I can't get to work is the 0.15.0 one (SAC-orange, PPO-Dark blue):

 

Hi ervteng_unity, here are my state and action spaces (for ml-agents 0.14.1):

As you can see the action space has 5 continuous actions. My decision requester has a decision period of 1 with repeat action enabled and no offset.
I hope this information helps!