Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Movement with Velocity

Discussion in 'ML-Agents' started by Hsgngr, Aug 1, 2020.

  1. Hsgngr

    Hsgngr

    Joined:
    Dec 28, 2015
    Posts:
    61
    In most of the example agents are using similar movement style -discrete rotation and forward with adding force. Even though it is a great way to move, it is not the best solution for my environment. My agent needs to be agile and cancelling forces sometimes result with death. How can I directly use velocities ? Also how can I use them as continuous actions ? (not only 0 and 1)

    Here is the code that I'm using below What changes can be made ?


    Code (CSharp):
    1.    public void MoveAgent(float[] act)
    2.     {
    3.         var dirToGo = Vector3.zero;
    4.         var rotateDir = Vector3.zero;
    5.  
    6.         var rotateAxis = (int)act[0];
    7.  
    8.         var direction = (int)act[1];
    9.  
    10.         switch (rotateAxis)
    11.         {
    12.             case 0:
    13.                 rotateDir = -transform.up;
    14.                 break;
    15.             case 1:
    16.                 rotateDir = transform.up;
    17.                 break;
    18.             case 2:
    19.                 rotateDir = Vector3.zero;
    20.                 break;
    21.         }
    22.         switch (direction)
    23.         {
    24.             case 0:
    25.                 dirToGo = transform.right;
    26.                 break;
    27.             case 1:
    28.                 dirToGo = -transform.right;
    29.                 break;
    30.         }
    31.  
    32.         rb.AddForce(dirToGo * moveSpeed, ForceMode.VelocityChange);
    33.         transform.Rotate(rotateDir, Time.fixedDeltaTime * turnSpeed);
    34.  
    35.         if (rb.velocity.sqrMagnitude > 25f) // slow it down
    36.         {
    37.             rb.velocity *= 0.95f;
    38.         }
    39.     }
     
  2. m4l4

    m4l4

    Joined:
    Jul 28, 2020
    Posts:
    81
    To use switch from discrete to continuous action, you just have to go in Behavior Parameters, vector action, space type, select Continuous. With a space size of 3 you could try something like that:

    Rb.velocity = new Vector3(vectorAction[0] * speed, 0f, vectorAction[1] * speed);
    transform.Rotate(Vector3.up, vectorAction[2] * rotationSpeed);

    if you want the agent to be agile, than i suggest you to increase the movement possibility. My agents has 3 discrete actions with 3 branches each.
    0 = [do nothing, move fwd, move backward]
    1 = [do nothing, rotate left, rotate right]
    2 = [do nothing, strife left, strife right]

    by combining forward movement, strife and rotation, the agent can navigate very efficiently, but i'm moving it with math, not with physics, maybe you want a different solution
     
  3. Hsgngr

    Hsgngr

    Joined:
    Dec 28, 2015
    Posts:
    61
    Can you share your knowledge about mathematically calculating these, maybe it could be better option. And how to code strive left,right ?
     
  4. m4l4

    m4l4

    Joined:
    Jul 28, 2020
    Posts:
    81
    sure man, here you go.

    that's my heuristic function and OnActionReceived for a 3x3 discrete action size

    Code (CSharp):
    1. public override void Heuristic(float[] actionsOut)
    2.     {
    3.  
    4.         if(Input.GetKey("a")){
    5.             actionsOut[0] = 1;
    6.         }else if(Input.GetKey("d")){
    7.             actionsOut[0] = 2;
    8.         }else{
    9.             actionsOut[0] = 0;
    10.         }
    11. if(Input.GetKey("w")){
    12.             actionsOut[1] = 1;
    13.         }else if(Input.GetKey("s")){
    14.             actionsOut[1] = 2;
    15.         }else{
    16.             actionsOut[1] = 0;
    17.         }
    18.  
    19.         if(Input.GetKey("q")){
    20.             actionsOut[2] = 1;
    21.         }else if(Input.GetKey("e")){
    22.             actionsOut[2] = 2;
    23.         }else{
    24.             actionsOut[2] = 0;
    25.         }
    26.     }
    27.  
    28. public override void OnActionReceived(float[] vectorAction)
    29.     {
    30.  
    31.         var rotation = (int)vectorAction[0];
    32.         var fwdAxes = (int)vectorAction[1];
    33.         var lateralAxes = (int)vectorAction[2];
    34.  
    35.  
    36.             // vectorAction[0] 0 = no rotation, 1 = rot sx, 2 = rot rx
    37.             switch(rotation)
    38.             {
    39.                 case 1:
    40.                     rb.transform.Rotate(0, -rotSpeed*Time.deltaTime, 0);
    41.                     break;
    42.                 case 2:
    43.                     rb.transform.Rotate(0, rotSpeed*Time.deltaTime, 0);
    44.                     break;
    45.             }
    46.  
    47.             // vectorAction[1] 0= no movement, 1= mov fwd, 2= mov bkw,
    48.             switch(fwdAxes)
    49.             {
    50.                 case 1:
    51.                     rb.transform.Translate(0, 0, speed*Time.deltaTime, Space.Self);
    52.                     break;
    53.                 case 2:
    54.                     rb.transform.Translate(0, 0, -speed*Time.deltaTime, Space.Self);
    55.                     break;
    56.             }
    57.  
    58.             // vectorAction[2] 0= no movement, 1= strife left, 2= strife right
    59.             switch(lateralAxes)
    60.             {
    61.                 case 1:
    62.                     rb.transform.Translate(-speed*Time.deltaTime, 0, 0, Space.Self);
    63.                     break;
    64.                 case 2:
    65.                     rb.transform.Translate(speed*Time.deltaTime, 0, 0, Space.Self);
    66.                     break;
    67.             }
    68.     }
     
    Hsgngr likes this.
  5. Hsgngr

    Hsgngr

    Joined:
    Dec 28, 2015
    Posts:
    61
    Thanks for the info, I will try and come back
     
  6. Hsgngr

    Hsgngr

    Joined:
    Dec 28, 2015
    Posts:
    61
    This works like charm but one question: Shouldn't we also need to define case 0: for doing nothing ?
     
  7. m4l4

    m4l4

    Joined:
    Jul 28, 2020
    Posts:
    81
    Glad it helped :)
    as far as i understand, if the case is not listed, it just doesn't perform any of the listed action.
    But i've noticed that if you don't put case 0 in the heuristic function, you cannot properly control the agent (try for yourself, it's a mess).
    As usual, i might be wrong, can anyone deny/confirm we don't need to define "do nothing" in the OnAcionReceived() method ?
     
  8. Hsgngr

    Hsgngr

    Joined:
    Dec 28, 2015
    Posts:
    61
    Yeah that would be great if someone can confirm this, otherwise the actions will increase +1 for every branch and its definetly slows the process.Thanks for the movement code again.
     
  9. m4l4

    m4l4

    Joined:
    Jul 28, 2020
    Posts:
    81
    well, i'm starting to think that since we mostly use "take actions between decisions", if we do not define a case 0, the "last action taken" might not be changed to "do nothing".
    The agent goes left, keeps going left till the next decision, next decision is 0, case is not defined, agent keeps going left till next decision and so on.

    actually it would be pretty easy to fix.
    just add

    case 0:
    rb.transform.Rotate(0,0,0);

    also, by just looking at the training process, if they now willingly stop, their movements should look a lot more jittery at the start.
     
  10. Hsgngr

    Hsgngr

    Joined:
    Dec 28, 2015
    Posts:
    61
    Alright I was thinking the same. I tried the movement code and at some point they learned to go inside the outer walls of the environment and escape the area. Since we are not using physics that is a bit of a problem. Do you have a solution for this ? I increased the wall thickness a little bit but it didn't help. Maybe some combination of using velocity can solve this. If I solve I will paste the code.

    upload_2020-8-3_19-32-48.png
     

    Attached Files:

  11. m4l4

    m4l4

    Joined:
    Jul 28, 2020
    Posts:
    81
    wow that's weird, i haven't had any escape problem. no idea what's causing it.

    btw, you asked for agility... :D
     
    Hsgngr likes this.
  12. m4l4

    m4l4

    Joined:
    Jul 28, 2020
    Posts:
    81
    i forgot to mention, if you are interest in physic based movements, search for "the nature of code" by Daniel Shiffman.
    it goes from explaining the basics of using velocity, acceleration etc. to coding behaviors like seek, intercept, flocking, and so on. Really interesting stuff
     
  13. Hsgngr

    Hsgngr

    Joined:
    Dec 28, 2015
    Posts:
    61
    Great source, I skimmed it, I think it is essential for these kind of simulation
     
    m4l4 likes this.