Search Unity

  1. The 2022.1 beta is now available for testing. To find out what's new, have a look at our 2022.1 beta blog post.
    Dismiss Notice

How to make fleeing agent avoid getting stuck in corners?

Discussion in 'Navigation' started by SomeHumbleOnion, Nov 25, 2021.

  1. SomeHumbleOnion

    SomeHumbleOnion

    Joined:
    Jan 18, 2021
    Posts:
    18
    Hello forum!

    I'm working on a navmesh agent that flees from the player. I have this implemented using the code snippet below, but the problem is you can easily catch this fleeing agent by chasing it into a wall where it will get stuck. Here is the code I'm using for the basic fleeing logic (ran in the Update method):

    Code (CSharp):
    1.         // calculate the distance between the agent and the player
    2.         float distance = Vector3.Distance(transform.position, player.position);
    3.  
    4.         // Run from player
    5.         if (distance < agentDistanceRun)
    6.         {
    7.             // Calculate the vector pointing from Player to the Agent
    8.             Vector3 dirToPlayer = transform.position - player.position;
    9.  
    10.             // Calculate the vector from the Enemy to the direction away from the Player the new point
    11.             Vector3 newPos = transform.position + dirToPlayer.normalized;
    12.  
    13.             // Shoot raycast forward to check for wall
    14.             //CheckForWall();
    15.  
    16.             //newPos = Quaternion.Euler(0, vRotation, 0) * newPos;
    17.  
    18.             agent.SetDestination(newPos);
    19.         }
    You'll notice I have two lines commented out. This was my attempt at making making the agent steer away from walls while still avoiding the player. To do this, I call the CheckForWall method which shoots a raycast forward to check for walls. If it does hit a wall, I start updating the float vRotation in increments of 5 until you no longer detect a wall. Then I try applying that new rotation to the destination vector 'newPos', but that's where the agent starts bugging out and the destination variable becomes incorrect. I've pasted my raycasting method CheckForWall() below:

    Code (CSharp):
    1.     private void CheckForWall()
    2.     {
    3.         Vector3 origin = transform.position;
    4.         origin.y += rayCastOffset;
    5.         RaycastHit hit;
    6.  
    7.         // shoot raycast forward to detect wall. If wall detected, update rotation value so we can apply it to agent's destination vector
    8.         // until we no longer detect a wall.
    9.         if (Physics.Raycast(origin, transform.forward, out hit, rayDist, layerMask))
    10.         {
    11.             // check if agent is currently moving right or left and increase rotation value based on that direction.
    12.             if (rotationDirection == RotationDirection.Rightward)
    13.             {
    14.                 vRotation += 5f;
    15.             } else
    16.             {
    17.                 vRotation -= 5f;
    18.             }
    19.         }
    20.     }
    I feel like my logic is right, but maybe the way I'm calculating the agents destination vector based on this rotation values is incorrect? I'd love some advice on this as I'm not the best with Vectors or Quaternions. Thanks!
     
  2. SomeHumbleOnion

    SomeHumbleOnion

    Joined:
    Jan 18, 2021
    Posts:
    18
    UPDATE:

    I've added a new method that will take the agent's destination vector and create a new destination vector based on a rotation value by rotating around the agent. It's not a perfect solution and the agent can still get stuck in corners, but for the most part it's working pretty smoothly. I've pasted it below in case anyone is having a similar issue and hopefully this helps.

    Code (CSharp):
    1.     private void Update()
    2.     {
    3.         // Check which direction the agent is turning
    4.         //GetRotationDirection();
    5.  
    6.         // Move agent
    7.         Flee2();
    8.     }
    9.  
    10.     private void Flee2()
    11.     {
    12.         // calculate the distance between the agent and the player
    13.         float distance = Vector3.Distance(transform.position, player.position);
    14.  
    15.         vRotation = 0f;
    16.  
    17.         // Run from player
    18.         if (distance < agentDistanceRun)
    19.         {
    20.             // Calculate the vector pointing from Player to the Enemy
    21.             Vector3 dirToPlayer = transform.position - player.position;
    22.  
    23.             // Calculate the vector from the Enemy to the direction away from the Player the new point
    24.             Vector3 newPos = transform.position + dirToPlayer;
    25.  
    26.             // Shoot raycast forward to check for wall
    27.             CheckForWall();
    28.  
    29.             // Apply rotation to destination vector
    30.             Vector3 newPos2 = RotateAroundPivot(newPos, transform.position);
    31.  
    32.             agent.SetDestination(newPos2);
    33.         }
    34.     }
    35.  
    36.     // calculates a vector based on direction and rotation.
    37.     private Vector3 RotateAroundPivot(Vector3 point, Vector3 pivot)
    38.     {
    39.         // Get point direction relative to pivot (the agent)
    40.         Vector3 dir = point - pivot;
    41.  
    42.         // Apply rotation
    43.         dir = Quaternion.Euler(0, vRotation, 0) * dir;
    44.  
    45.         //calculate the rotated point
    46.         point = dir + pivot;
    47.  
    48.         return point;
    49.     }
    50.  
    51.     private void CheckForWall()
    52.     {
    53.         Vector3 origin = transform.position;
    54.         origin.y += rayCastOffset;
    55.         RaycastHit hit;
    56.  
    57.         // shoot raycast forward to detect wall. If wall detected, update rotation value so we can apply it to agent's destination vector
    58.         // until we no longer detect a wall.
    59.         if (Physics.Raycast(origin, transform.forward, out hit, rayDist, layerMask))
    60.         {
    61.             // check if agent is currently moving right or left and increase rotation value based on that direction.
    62.             if (rotationDirection == RotationDirection.Rightward)
    63.             {
    64.                 vRotation += 45f;
    65.             } else
    66.             {
    67.                 vRotation -= 45f;
    68.             }
    69.         }
    70.     }
     
    way3edgy likes this.
  3. danielboldroff

    danielboldroff

    Joined:
    Sep 27, 2021
    Posts:
    7
    Is there a reason you withheld your variables?
     
    way3edgy likes this.
  4. SomeHumbleOnion

    SomeHumbleOnion

    Joined:
    Jan 18, 2021
    Posts:
    18
    Not really, this is mostly a prototype script right now.
     
unityunity