Search Unity

  1. All Pro and Enterprise subscribers: find helpful & inspiring creative, tech, and business know-how in the new Unity Success Hub. Sign in to stay up to date.
    Dismiss Notice
  2. Dismiss Notice

SAC training very unstable

Discussion in 'ML-Agents' started by MrOCW, May 5, 2021.

  1. MrOCW

    MrOCW

    Joined:
    Feb 16, 2021
    Posts:
    21
    upload_2021-5-5_17-48-14.png
    Hi all, this is the result of my SAC training. It is very unstable with the rewards jumping so much. Is this supposed to be the case for SAC?

    behaviors:
    Car:
    trainer_type: sac
    hyperparameters:
    learning_rate: 0.0003
    learning_rate_schedule: constant
    batch_size: 128
    buffer_size: 50000
    buffer_init_steps: 10
    tau: 0.005
    steps_per_update: 10.0
    save_replay_buffer: false
    init_entcoef: 0.01
    reward_signal_steps_per_update: 10.0
    network_settings:
    normalize: false
    hidden_units: 128
    num_layers: 1
    vis_encode_type: resnet
    reward_signals:
    extrinsic:
    gamma: 0.99
    strength: 1.0
    keep_checkpoints: 5
    max_steps: 750000
    checkpoint_interval: 100000
    time_horizon: 4
    summary_freq: 1000
    threaded: true


    When i tried using the trained onnx model,
    I get a whole lot of errors and warnings
    upload_2021-5-5_18-19-3.png

    Changing to CPU fixed some of it but I still get the AssertionException: Assertion failure. Values are not equal.

    Trained another SAC model to test, but this time round, the vis_encode_type is simple instead of resnet. Inference worked this time around so the issue lies with the IMPALA/Resnet?
     
    Last edited: May 5, 2021
  2. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    716
    Hi @MrOCW,
    There are many reasons why your training could be unstable.
    Could you provide more information about your training environment such as:
    1. observation and action space
    2. your reward functions
    3. Which OS are you are using
    4. Version of unity/ml-agents/operating system

    It is hard for us to diagnose issues with such sparse information. Please try to help us out by providing as much information as you can upfront. Otherwise, we both lose time by having this back and forth for questions.
     
  3. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    716
    Could you also check to make sure that your inputs in your project actually match the onnx model you are using? If you update the inputs you will need to retrain your model. You can check this by viewing your model in the inspector and looking at the input shape, or by looking at the agent inspector and making sure that the BehaviorParameters component doesn't show any warnings.
     
  4. MrOCW

    MrOCW

    Joined:
    Feb 16, 2021
    Posts:
    21
    1. CameraSensor/Visual Observation + 2 continuous actions
    2. -1f and end episode for going off lane + 0.01f for staying in lane (car), max steps 20 000 (episode doesnt seem to end after 20 000 steps though..)
    3. Ubuntu 18.04
    4. Unity 2020.3.5f1 , ML-Agents 1.9.1
     
    Last edited: May 6, 2021
  5. MrOCW

    MrOCW

    Joined:
    Feb 16, 2021
    Posts:
    21
    yes. i tried inferring the minute the model finished training. No changes to input at all
     
  6. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    716
    Could you provide the model so the Barracuda team could take a look at it?
     
  7. MrOCW

    MrOCW

    Joined:
    Feb 16, 2021
    Posts:
    21
    @christophergoy How do i upload the onnx file?
    I'm getting an error: The uploaded file does not have an allowed extension.
     
  8. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    716
    could you add it to a zip file and upload it?
     
  9. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    716
    Also, what version of the mlagents pip package are you using?
     
  10. MrOCW

    MrOCW

    Joined:
    Feb 16, 2021
    Posts:
    21

    Attached Files:

    christophergoy likes this.
  11. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    716
    I've sent the model off to the barracuda team for them to take a look. You can also post in their forum with questions for them in the future if you run into model loading issues.
     
    MrOCW likes this.
  12. alexandreribard_unity

    alexandreribard_unity

    Unity Technologies

    Joined:
    Sep 18, 2019
    Posts:
    29
  13. christophergoy

    christophergoy

    Unity Technologies

    Joined:
    Sep 16, 2015
    Posts:
    716
    Hey @alexandreribard_unity,
    I think that’s on the ml-agent’s side of things. I can poke the research team to see if we can get that done.
     
  14. MrOCW

    MrOCW

    Joined:
    Feb 16, 2021
    Posts:
    21
    Hi @alexandreribard_unity , any updates on the fix? May I know how do I go about changing those layers to a Flatten?
     
unityunity