Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Lifecycle

Discussion in 'ML-Agents' started by dhaesedavid, Oct 16, 2020.

  1. dhaesedavid

    dhaesedavid

    Joined:
    Oct 16, 2020
    Posts:
    3
    I spent quite time trying to understand exactly how the life-cycles of the Unity Environment are intertwined with the life cycle of TensorFlow, but I cannot find this piece of evidence in the documentation. For teaching purposes, I would like to create a schema similar to this one:

    upload_2020-10-16_10-6-3.png

    albeit, a correct one!

    Any help or reference would be greatly appreciated, thanks!
     
    layker90524 likes this.
  2. vincentpierre

    vincentpierre

    Joined:
    May 5, 2017
    Posts:
    160
    Hi,

    There is no documentation on this topic unfortunately.
    This is what happens (simplified) in the UnityEnvironment :
    • During each fixed update, the Academy (the orchestrator) calls CollectObservations on all Agents that requested a decision since the last fixed update (Note that RequestDecision can be called anytime from anywhere on the Agent).
    • Once all relevant observations are collected, the data is communicated to Python. The data includes all observations and all rewards that were collected since last fixed update (note that AddReward can be called from anywhere anytime on the Agent). It also includes some other useful information such as wether or not the Agent terminated since the last fixed update.
    • Python selects an action and sends it to the UnityEnvironment.
    • The Academy calls OnActionReceived on all of the Agents that requested a decision since the last fixed update.
    • The rest of the fixed update and other game loop unfolds.
    It is important to note that RequestDecision and AddReward can be called anytime in the game loop. We recommend doing that in the OnActionReceived method because it is easier for the Agents to learn when the rewards are direct consequences of their actions.
    The hook into fixed update is in this script : https://github.com/Unity-Technologi..._7/com.unity.ml-agents/Runtime/Academy.cs#L33 Note that it is possible to disable the automatic stepping in the fixed update and manually call Academy.Instance.EnvironmentStep(); to trigger a RL loop.
     
    layker90524 likes this.