Help Wanted Movement with Velocity

Discussion in 'ML-Agents' started by Hsgngr, Aug 1, 2020.

1. Hsgngr

Joined:
Dec 28, 2015
Posts:
53
In most of the example agents are using similar movement style -discrete rotation and forward with adding force. Even though it is a great way to move, it is not the best solution for my environment. My agent needs to be agile and cancelling forces sometimes result with death. How can I directly use velocities ? Also how can I use them as continuous actions ? (not only 0 and 1)

Here is the code that I'm using below What changes can be made ?

Code (CSharp):
1.    public void MoveAgent(float[] act)
2.     {
3.         var dirToGo = Vector3.zero;
4.         var rotateDir = Vector3.zero;
5.
6.         var rotateAxis = (int)act[0];
7.
8.         var direction = (int)act[1];
9.
10.         switch (rotateAxis)
11.         {
12.             case 0:
13.                 rotateDir = -transform.up;
14.                 break;
15.             case 1:
16.                 rotateDir = transform.up;
17.                 break;
18.             case 2:
19.                 rotateDir = Vector3.zero;
20.                 break;
21.         }
22.         switch (direction)
23.         {
24.             case 0:
25.                 dirToGo = transform.right;
26.                 break;
27.             case 1:
28.                 dirToGo = -transform.right;
29.                 break;
30.         }
31.
33.         transform.Rotate(rotateDir, Time.fixedDeltaTime * turnSpeed);
34.
35.         if (rb.velocity.sqrMagnitude > 25f) // slow it down
36.         {
37.             rb.velocity *= 0.95f;
38.         }
39.     }

2. m4l4

Joined:
Jul 28, 2020
Posts:
26
To use switch from discrete to continuous action, you just have to go in Behavior Parameters, vector action, space type, select Continuous. With a space size of 3 you could try something like that:

Rb.velocity = new Vector3(vectorAction[0] * speed, 0f, vectorAction[1] * speed);
transform.Rotate(Vector3.up, vectorAction[2] * rotationSpeed);

if you want the agent to be agile, than i suggest you to increase the movement possibility. My agents has 3 discrete actions with 3 branches each.
0 = [do nothing, move fwd, move backward]
1 = [do nothing, rotate left, rotate right]
2 = [do nothing, strife left, strife right]

by combining forward movement, strife and rotation, the agent can navigate very efficiently, but i'm moving it with math, not with physics, maybe you want a different solution

3. Hsgngr

Joined:
Dec 28, 2015
Posts:
53
Can you share your knowledge about mathematically calculating these, maybe it could be better option. And how to code strive left,right ?

4. m4l4

Joined:
Jul 28, 2020
Posts:
26
sure man, here you go.

that's my heuristic function and OnActionReceived for a 3x3 discrete action size

Code (CSharp):
1. public override void Heuristic(float[] actionsOut)
2.     {
3.
4.         if(Input.GetKey("a")){
5.             actionsOut[0] = 1;
6.         }else if(Input.GetKey("d")){
7.             actionsOut[0] = 2;
8.         }else{
9.             actionsOut[0] = 0;
10.         }
11. if(Input.GetKey("w")){
12.             actionsOut[1] = 1;
13.         }else if(Input.GetKey("s")){
14.             actionsOut[1] = 2;
15.         }else{
16.             actionsOut[1] = 0;
17.         }
18.
19.         if(Input.GetKey("q")){
20.             actionsOut[2] = 1;
21.         }else if(Input.GetKey("e")){
22.             actionsOut[2] = 2;
23.         }else{
24.             actionsOut[2] = 0;
25.         }
26.     }
27.
28. public override void OnActionReceived(float[] vectorAction)
29.     {
30.
31.         var rotation = (int)vectorAction[0];
32.         var fwdAxes = (int)vectorAction[1];
33.         var lateralAxes = (int)vectorAction[2];
34.
35.
36.             // vectorAction[0] 0 = no rotation, 1 = rot sx, 2 = rot rx
37.             switch(rotation)
38.             {
39.                 case 1:
40.                     rb.transform.Rotate(0, -rotSpeed*Time.deltaTime, 0);
41.                     break;
42.                 case 2:
43.                     rb.transform.Rotate(0, rotSpeed*Time.deltaTime, 0);
44.                     break;
45.             }
46.
47.             // vectorAction[1] 0= no movement, 1= mov fwd, 2= mov bkw,
48.             switch(fwdAxes)
49.             {
50.                 case 1:
51.                     rb.transform.Translate(0, 0, speed*Time.deltaTime, Space.Self);
52.                     break;
53.                 case 2:
54.                     rb.transform.Translate(0, 0, -speed*Time.deltaTime, Space.Self);
55.                     break;
56.             }
57.
58.             // vectorAction[2] 0= no movement, 1= strife left, 2= strife right
59.             switch(lateralAxes)
60.             {
61.                 case 1:
62.                     rb.transform.Translate(-speed*Time.deltaTime, 0, 0, Space.Self);
63.                     break;
64.                 case 2:
65.                     rb.transform.Translate(speed*Time.deltaTime, 0, 0, Space.Self);
66.                     break;
67.             }
68.     }

Hsgngr likes this.
5. Hsgngr

Joined:
Dec 28, 2015
Posts:
53
Thanks for the info, I will try and come back

6. Hsgngr

Joined:
Dec 28, 2015
Posts:
53
This works like charm but one question: Shouldn't we also need to define case 0: for doing nothing ?

7. m4l4

Joined:
Jul 28, 2020
Posts:
26
as far as i understand, if the case is not listed, it just doesn't perform any of the listed action.
But i've noticed that if you don't put case 0 in the heuristic function, you cannot properly control the agent (try for yourself, it's a mess).
As usual, i might be wrong, can anyone deny/confirm we don't need to define "do nothing" in the OnAcionReceived() method ?

8. Hsgngr

Joined:
Dec 28, 2015
Posts:
53
Yeah that would be great if someone can confirm this, otherwise the actions will increase +1 for every branch and its definetly slows the process.Thanks for the movement code again.

9. m4l4

Joined:
Jul 28, 2020
Posts:
26
well, i'm starting to think that since we mostly use "take actions between decisions", if we do not define a case 0, the "last action taken" might not be changed to "do nothing".
The agent goes left, keeps going left till the next decision, next decision is 0, case is not defined, agent keeps going left till next decision and so on.

actually it would be pretty easy to fix.

case 0:
rb.transform.Rotate(0,0,0);

also, by just looking at the training process, if they now willingly stop, their movements should look a lot more jittery at the start.

10. Hsgngr

Joined:
Dec 28, 2015
Posts:
53
Alright I was thinking the same. I tried the movement code and at some point they learned to go inside the outer walls of the environment and escape the area. Since we are not using physics that is a bit of a problem. Do you have a solution for this ? I increased the wall thickness a little bit but it didn't help. Maybe some combination of using velocity can solve this. If I solve I will paste the code.

File size:
29.5 KB
Views:
4
11. m4l4

Joined:
Jul 28, 2020
Posts:
26
wow that's weird, i haven't had any escape problem. no idea what's causing it.

Hsgngr likes this.
12. m4l4

Joined:
Jul 28, 2020
Posts:
26
i forgot to mention, if you are interest in physic based movements, search for "the nature of code" by Daniel Shiffman.
it goes from explaining the basics of using velocity, acceleration etc. to coding behaviors like seek, intercept, flocking, and so on. Really interesting stuff

13. Hsgngr

Joined:
Dec 28, 2015
Posts:
53
Great source, I skimmed it, I think it is essential for these kind of simulation

m4l4 likes this.
unityunity