Resolved Unable to solve problem

Hello everyone,

I want to make a very “simple” thing: A (2D) Rigid object should apply torque so that it’s rotation matches a target rotation. The starting rotation, beginning rotational speed and Target Rotation is randomized.

Basically, I want to make a PID controller, which really isn’t witchcraft.

And it doesn’t work. I tried a lot of reward strategies, and none really converged. The best one was just the dot product between the target and current angle. But the training was unstable. In theory that should lead to an optimal behaviour, bc. the optimal strategy is to get as fast as possible to the goal and stay there.

(Really everything that I would consider useful didn’t work with Machine Learning. )

I also tried all the best practices (normalizing input, Rewards not too high or low, etc.)

I think that there is a really easy answer to that problem, bc I solved it one time while doing another project (but didn’t save the code after I had done some changes later on ☹.)


Has anyone an idea how to solve that primitive problem with mlagents? Maybe some hints on the choice of the hyperparameters?

This problem is important for all kinds of physically driven A.I. (lets say a drone that tries to stabilize itself, a turret that tries to follow a target, or in my case a leg for a funny roboter)

It seems like a challenge to me, please help . I am doing that for 3 days straight now without getting a good solution.

Thank you in advance!

Ps: What I also tried:

- Penalizing for velocity (-Abs(Velocity.magnitude))| Result where ok but could lead to slow movement when not

- Comparing the current distance to the goal and the last one, and giving Reward = (lastDistance – Currentdistance). Worked fine but worse than just dot

- Giving Reward only when it hits the goal +/- some delta (didn’t work at all)

- Using sigmoid function on dot product (didn’t work)

- Init every Agent with the same target or init every Agent individually (also didn’t work)

The main code:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using Unity.MLAgents;
5. using Unity.MLAgents.Actuators;
6. using Unity.MLAgents.Sensors;
7.  
8. public class LegAgent : Agent
9. {
10.     public GameObject visual;
11.     Rigidbody rb;
12.     HingeJoint hJoint;
13.     public Quaternion targetRotation;
14.    
15.     public override void OnActionReceived(ActionBuffers actions)
16.     {
17.        
18.         float f = actions.ContinuousActions[0];
19.         rb.AddTorque(Vector3.right * manager.legStrenght * f);
20.     }
21.  
22.     public override void CollectObservations(VectorSensor sensor)
23.     {
24.         //because I rotate only on the x-axis, only the x and w Quaternion components // are changing
25.         Quaternion q1 = Quaternion.Euler((hJoint.angle), 0, 0).normalized;
26.         sensor.AddObservation(new Vector2(q1.x, q1.w));
27.         Quaternion q2 = targetRotation.normalized;
28.         sensor.AddObservation(new Vector2(q2.x, q2.w);
29.         sensor.AddObservation((rb.angularVelocity.x)/rb.maxAngularVelocity);
30. Quaternion q3 = Quaternion.FromToRotation(targetRotation.ToEuler(), transform.rotation.ToEuler()).normalized;
31.         sensor.AddObservation(new Vector2(q3.x, q3.w));
32.        
33.        
34.  
35.  
36.     }
37.    
38.     public Mananger manager;
39.    
40.     private void FixedUpdate()
41.     {
42.        
43.         float dot = (Quaternion.Dot(targetRotation, transform.rotation) + 1) * 0.5f;
44.        
45.         AddReward(dot);
46.     }
47.    
48.     public override void OnEpisodeBegin()
49.     {
50.        
51.  
52.         int i = (int)Academy.Instance.EnvironmentParameters.GetWithDefault("lecture", 0.1f);
53.         switch (i)
54.         {
55.             case 0:
56.                 //getting same Random for all Agents
57.                 rb.velocity = Vector3.zero;
58.                 rb.angularVelocity = new Vector3(manager.randomFloat, 0, 0);
59.                 delta = manager.delta;
60.                 rb.rotation.SetEulerAngles(manager.randomFloat2, 0, 0);
61.                 targetRotation = Quaternion.Euler(manager.randomFloat3, 0, 0);
62.                 break;
63.             case 1:
64.                 //Random input for each Agent
65.                 rb.velocity = Vector3.zero;
66.                 rb.angularVelocity = new Vector3(Random.Range(-100, 100), 0, 0);
67.                 delta = 2;
68.                 rb.rotation.SetEulerAngles(Random.Range(-180, 180), 0, 0);
69.                 targetRotation = Quaternion.Euler(Random.Range(-180, 180), 0, 0);
70.                 lastRotation = Quaternion.FromToRotation(transform.rotation.eulerAngles, targetRotation.eulerAngles);
71.  
72.  
73.                 break;
74.             case 2:
75.                 rb.velocity = Vector3.zero;
76.                 rb.angularVelocity = new Vector3(Random.Range(0, 0), 0, 0);
77.                 delta = 2;
78.                 rb.rotation.SetEulerAngles(Random.Range(-180, 180), 0, 0);
79.                 targetRotation = Quaternion.Euler(Random.Range(-180, 180), 0, 0);
80.                 break;
81.             case 3:
82.                 rb.velocity = Vector3.zero;
83.                 rb.angularVelocity = new Vector3(Random.Range(0, 0), 0, 0);
84.                 delta = 2;
85.                 rb.rotation.SetEulerAngles(Random.Range(-180, 180), 0, 0);
86.                 targetRotation = Quaternion.Euler(Random.Range(-180, 180), 0, 0);
87.                 break;
88.         }
89.        
90.         //targetRotation = -160;
91.     }
92.     // Start is called before the first frame update
93.     void Start()
94.     {
95.         hJoint = GetComponent<HingeJoint>();
96.         rb = GetComponent<Rigidbody>();
97.         rb.maxAngularVelocity = 300;
98.         manager = FindObjectOfType<Mananger>();
99.  
100.  
101. }
102.  

The manager code:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4.  
5. public class Mananger : MonoBehaviour
6. {
7.     public float delta;
8.     public float legStrenght;
9.     public Quaternion randomQuaternion1;
10.     public Quaternion randomQuaternion2;
11.     public float randomFloat;
12.     public float randomFloat2;
13.     public float randomFloat3;
14.     public float min;
15.     public float max;
16.     public bool locked;
17.     // Start is called before the first frame update
18.     void Start()
19.     {
20.        
21.     }
22.  
23.     // Update is called once per frame
24.     void FixedUpdate()
25.     {
26.         if (!locked)
27.         {
28.             randomQuaternion1 = Random.rotation;
29.             randomQuaternion2 = Random.rotation;
30.             randomFloat = Random.Range(min, max);
31.             randomFloat2 = Random.Range(-180, 180);
32.             randomFloat3 = Random.Range(-180, 180);
33.         }
34.     }
35. }
36.  

 

So, I solved it!

To sum up my project, I wanted a pointer/object to point in the direction of target, or said differently, to match the rotation of my object to a target object's rotation.

The problem lies in the changing of the angular velocity through applying torque, what is a pretty difficult problem to solve (bc of gravity (acts unevenly on the object depending on the position in the circle) and I added a starting random velocity in each iteration.

The solution:

Firstly a crucial basic thing that I missed was that my Input in Observation was not normalized and the according Hyperparameter was not set to true. Normalizing the input solved it already.

Secondly the best results I could get were with the following Reward, Observations:

Observations:

targetVector (so basically the rotation regarding to a unit circle)

current pointing direction vector (transform.up)

(targetVector – transform.up).normalized (works bc you can interpret the vectors as positions lying on a unit circle)

Vector3.Angle(targetVector, transform.up) * (Mathf.degToRad * lengthOfVectors)

(Radians are basically the distance on a circle between two points on a circle. You could leave out the whole second part if you normalize it with 2 * Pi, what leaves you with 360 (degree). But it shows that you can think of distances even though it is about angles)

rigidbody.angularVelocity.x/rigidbody.maxAngularVelocity;

Reward:

Delta = 0.03f;

If(Vector3.Angle(targetVector, transform.up) * (Mathf.degToRad * lengthOfVectors) < delta)

{

Addreward(1f);

}


So, I apologize if that was explained in too much detail, but I was frustrated not being able to solve that. I hope that helps someone.

If you have any suggestions for improvement/ critic
please make a comment!