Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Join us on March 30, 2023, between 5 am & 1 pm EST, in the Performance Profiling Dev Blitz Day 2023 - Q&A forum and Discord where you can connect with our teams behind the Memory and CPU Profilers.
    Dismiss Notice

Question Agent not moving between OnEpisodeBegin and first requestDecision call

Discussion in 'ML-Agents' started by unity_AA80832FC112213DE8BF, Dec 15, 2022.

  1. unity_AA80832FC112213DE8BF


    Sep 2, 2022

    I have a strange problem where my Agent seems to not do anything between the time when OnEpisodeBegin() is called and the first time requestDecision() is called.

    I have a robotic arm (Franka Emika Panda) that starts with each angle set to 0 (stands straight).
    At the start of each episode, I want the robotic arm to move to a specific position. To do that I created an OnEpisodeBegin()where I set the angles of the joints how I like them:
    Code (CSharp):
    1. public override void OnEpisodeBegin()
    2. {
    3.     Mujoco.MjActuator[] actuatorComponents = GetComponentsInChildren<Mujoco.MjActuator>();
    4.     for (int i = 0; i < 7; i++)
    5.     {
    6.         actuatorComponents[i].Control = initialAngles[i];
    7.     }
    8. }
    What I want
    When I call reset() from python, I want to ensure that the arm has finished moving to the position set in OnEpisodeBegin() function.
    For that purpose, I step the environment by myself. At first, I wait for a fixed amount of physics steps, then I start stepping:
    Code (CSharp):
    1. void FixedUpdate()
    2. {
    3.     if (!envParams.HasValue)
    4.     {
    5.         ReadEnvironmentParameters();
    6.     }
    7.     else
    8.     {
    9.         StepOrWait();
    10.     }
    11. }
    12. private void StepOrWait()
    13. {
    14.     if (physicsStepCount < envParams.Value.WaitForNPhysicsStepsBeforeStarting)
    15.     {
    16.         physicsStepCount += 1;
    17.     }
    18.     else
    19.     {
    20.         Step();
    21.     }
    22. }
    23. private void Step()
    24. {
    25.     if (Academy.Instance.StepCount % envParams.Value.DecisionPeriod == 0)
    26.     {
    27.         agent.RequestDecision();
    28.     }
    29.     Academy.Instance.EnvironmentStep();
    30. }


    After calling reset() from python and subsequently getting the observations with get_steps(), the robotic arm still has all angles set to 0, i.e. it did not move at all despite t waiting for waitForNPhysicsSteps (tried with large enough values). I also verified that OnEpisodeBegin is actually being called and that the physics simulation is stepping the right amount of time before starting to step the Academy.
    Why is the agent not doing anything despite the angles being set? Note that afterward, setting actions and stepping works as intended. The robot otherwise works as expected and responds appropriately when I change the angles using the actuators, as in the first code snippet.

  2. hughperkins


    Dec 3, 2022
    It's hard to parse what you are doing, but basically, physics essentially only 'run' each time you call 'step'.

    - if you move things immediately, without physics, then you can move them inside OnEpisodeBegin, and they will be *instantly* in the right position
    - if you move things with physics, so they move a bit each physics time step, you basically need to call 'step()' a few times, for the physics to run and move them