SAC training very unstable

Hi all, this is the result of my SAC training. It is very unstable with the rewards jumping so much. Is this supposed to be the case for SAC?

behaviors:
Car:
trainer_type: sac
hyperparameters:
learning_rate: 0.0003
learning_rate_schedule: constant
batch_size: 128
buffer_size: 50000
buffer_init_steps: 10
tau: 0.005
steps_per_update: 10.0
save_replay_buffer: false
init_entcoef: 0.01
reward_signal_steps_per_update: 10.0
network_settings:
normalize: false
hidden_units: 128
num_layers: 1
vis_encode_type: resnet
reward_signals:
extrinsic:
gamma: 0.99
strength: 1.0
keep_checkpoints: 5
max_steps: 750000
checkpoint_interval: 100000
time_horizon: 4
summary_freq: 1000
threaded: true


When i tried using the trained onnx model,
I get a whole lot of errors and warnings


Changing to CPU fixed some of it but I still get the AssertionException: Assertion failure. Values are not equal.

Trained another SAC model to test, but this time round, the vis_encode_type is simple instead of resnet. Inference worked this time around so the issue lies with the IMPALA/Resnet?

 

Hi @MrOCW,
There are many reasons why your training could be unstable.
Could you provide more information about your training environment such as:
1. observation and action space
2. your reward functions
3. Which OS are you are using
4. Version of unity/ml-agents/operating system

It is hard for us to diagnose issues with such sparse information. Please try to help us out by providing as much information as you can upfront. Otherwise, we both lose time by having this back and forth for questions.

 

Could you also check to make sure that your inputs in your project actually match the onnx model you are using? If you update the inputs you will need to retrain your model. You can check this by viewing your model in the inspector and looking at the input shape, or by looking at the agent inspector and making sure that the BehaviorParameters component doesn't show any warnings.

 

1. CameraSensor/Visual Observation + 2 continuous actions
2. -1f and end episode for going off lane + 0.01f for staying in lane (car), max steps 20 000 (episode doesnt seem to end after 20 000 steps though..)
3. Ubuntu 18.04
4. Unity 2020.3.5f1 , ML-Agents 1.9.1

 

yes. i tried inferring the minute the model finished training. No changes to input at all

 

Could you provide the model so the Barracuda team could take a look at it?

 

@christophergoy How do i upload the onnx file?
I'm getting an error: The uploaded file does not have an allowed extension.

 

could you add it to a zip file and upload it?

 

Also, what version of the mlagents pip package are you using?

 

@christophergoy here it is. thanks!

 

I've sent the model off to the barracuda team for them to take a look. You can also post in their forum with questions for them in the future if you run into model loading issues.

 

Hey @alexandreribard_unity,
I think that’s on the ml-agent’s side of things. I can poke the research team to see if we can get that done.

 

Hi @alexandreribard_unity , any updates on the fix? May I know how do I go about changing those layers to a Flatten?