Question Agent not moving between OnEpisodeBegin and first requestDecision call

Hello,

I have a strange problem where my Agent seems to not do anything between the time when OnEpisodeBegin() is called and the first time requestDecision() is called.

Setting
I have a robotic arm (Franka Emika Panda) that starts with each angle set to 0 (stands straight).
At the start of each episode, I want the robotic arm to move to a specific position. To do that I created an OnEpisodeBegin()where I set the angles of the joints how I like them:

Code (CSharp):

1. public override void OnEpisodeBegin()
2. {
3.     Mujoco.MjActuator[] actuatorComponents = GetComponentsInChildren<Mujoco.MjActuator>();
4.     for (int i = 0; i < 7; i++)
5.     {
6.         actuatorComponents[i].Control = initialAngles[i];
7.     }
8. }
9.  

What I want
When I call reset() from python, I want to ensure that the arm has finished moving to the position set in OnEpisodeBegin() function.
For that purpose, I step the environment by myself. At first, I wait for a fixed amount of physics steps, then I start stepping:

Code (CSharp):

1. void FixedUpdate()
2. {
3.     if (!envParams.HasValue)
4.     {
5.         ReadEnvironmentParameters();
6.     }
7.     else
8.     {
9.         StepOrWait();
10.     }
11. }
12. private void StepOrWait()
13. {
14.     if (physicsStepCount < envParams.Value.WaitForNPhysicsStepsBeforeStarting)
15.     {
16.         physicsStepCount += 1;
17.     }
18.     else
19.     {
20.         Step();
21.     }
22. }
23. private void Step()
24. {
25.     if (Academy.Instance.StepCount % envParams.Value.DecisionPeriod == 0)
26.     {
27.         agent.RequestDecision();
28.     }
29.     Academy.Instance.EnvironmentStep();
30. }
31.  

Problem
After calling reset() from python and subsequently getting the observations with get_steps(), the robotic arm still has all angles set to 0, i.e. it did not move at all despite t waiting for waitForNPhysicsSteps (tried with large enough values). I also verified that OnEpisodeBegin is actually being called and that the physics simulation is stepping the right amount of time before starting to step the Academy.
Why is the agent not doing anything despite the angles being set? Note that afterward, setting actions and stepping works as intended. The robot otherwise works as expected and responds appropriately when I change the angles using the actuators, as in the first code snippet.

Best,
P

 

It's hard to parse what you are doing, but basically, physics essentially only 'run' each time you call 'step'.

- if you move things immediately, without physics, then you can move them inside OnEpisodeBegin, and they will be *instantly* in the right position
- if you move things with physics, so they move a bit each physics time step, you basically need to call 'step()' a few times, for the physics to run and move them