Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Unite 2023 Registrations are now LIVE! We are thrilled to announce that Unite 2023 Registration is now live and open for all!
    Dismiss Notice
New Forum User Notice Unite 2023 Registrations are now LIVE!

Resolved PPO On-Policy or Off-policy

Discussion in 'ML-Agents' started by mateolopezareal, Jun 15, 2020.

  1. mateolopezareal

    mateolopezareal

    Joined:
    Jun 4, 2020
    Posts:
    54
    mateolopezareal, Jun 15, 2020
    #1

  2. vincentgao88

    vincentgao88

    Unity Technologies

    Joined:
    Feb 7, 2018
    Posts:
    21
    PPO is a on-policy algorithm, you can learn about it more by looking at this paper. https://arxiv.org/abs/1707.06347
     
    vincentgao88, Jun 15, 2020
    #2
    mateolopezareal likes this.

  3. TulioMMo

    TulioMMo

    Joined:
    Dec 30, 2020
    Posts:
    29
    Does the "actor" network for the agent in PPO shares its weights with the "critic" network? When estimating the baseline for the advantage function? Or does Unity generates a copy of the neural network with independent weights for actor and critic? Many thanks!
     
    TulioMMo, Dec 30, 2020
    #3

  4. ervteng_unity

    ervteng_unity

    Unity Technologies

    Joined:
    Dec 6, 2018
    Posts:
    150
    In the latest version, ML-Agents generates a copy of the network with independent weights for actor and critic. In some older versions, they were shares but only for discrete action networks.
     
    ervteng_unity, Jan 5, 2021
    #4
    TulioMMo likes this.

  5. TulioMMo

    TulioMMo

    Joined:
    Dec 30, 2020
    Posts:
    29
    Thank you for the reply!
     
    TulioMMo, Jan 8, 2021
    #5