How to make fleeing agent avoid getting stuck in corners?

Hello forum!

I'm working on a navmesh agent that flees from the player. I have this implemented using the code snippet below, but the problem is you can easily catch this fleeing agent by chasing it into a wall where it will get stuck. Here is the code I'm using for the basic fleeing logic (ran in the Update method):

Code (CSharp):

1.         // calculate the distance between the agent and the player
2.         float distance = Vector3.Distance(transform.position, player.position);
3.  
4.         // Run from player
5.         if (distance < agentDistanceRun)
6.         {
7.             // Calculate the vector pointing from Player to the Agent
8.             Vector3 dirToPlayer = transform.position - player.position;
9.  
10.             // Calculate the vector from the Enemy to the direction away from the Player the new point
11.             Vector3 newPos = transform.position + dirToPlayer.normalized;
12.  
13.             // Shoot raycast forward to check for wall
14.             //CheckForWall();
15.  
16.             //newPos = Quaternion.Euler(0, vRotation, 0) * newPos;
17.  
18.             agent.SetDestination(newPos);
19.         }

You'll notice I have two lines commented out. This was my attempt at making making the agent steer away from walls while still avoiding the player. To do this, I call the CheckForWall method which shoots a raycast forward to check for walls. If it does hit a wall, I start updating the float vRotation in increments of 5 until you no longer detect a wall. Then I try applying that new rotation to the destination vector 'newPos', but that's where the agent starts bugging out and the destination variable becomes incorrect. I've pasted my raycasting method CheckForWall() below:

Code (CSharp):

1.     private void CheckForWall()
2.     {
3.         Vector3 origin = transform.position;
4.         origin.y += rayCastOffset;
5.         RaycastHit hit;
6.  
7.         // shoot raycast forward to detect wall. If wall detected, update rotation value so we can apply it to agent's destination vector
8.         // until we no longer detect a wall.
9.         if (Physics.Raycast(origin, transform.forward, out hit, rayDist, layerMask))
10.         {
11.             // check if agent is currently moving right or left and increase rotation value based on that direction.
12.             if (rotationDirection == RotationDirection.Rightward)
13.             {
14.                 vRotation += 5f;
15.             } else
16.             {
17.                 vRotation -= 5f;
18.             }
19.         }
20.     }

I feel like my logic is right, but maybe the way I'm calculating the agents destination vector based on this rotation values is incorrect? I'd love some advice on this as I'm not the best with Vectors or Quaternions. Thanks!

 

UPDATE:

I've added a new method that will take the agent's destination vector and create a new destination vector based on a rotation value by rotating around the agent. It's not a perfect solution and the agent can still get stuck in corners, but for the most part it's working pretty smoothly. I've pasted it below in case anyone is having a similar issue and hopefully this helps.

Code (CSharp):

1.     private void Update()
2.     {
3.         // Check which direction the agent is turning
4.         //GetRotationDirection();
5.  
6.         // Move agent
7.         Flee2();
8.     }
9.  
10.     private void Flee2()
11.     {
12.         // calculate the distance between the agent and the player
13.         float distance = Vector3.Distance(transform.position, player.position);
14.  
15.         vRotation = 0f;
16.  
17.         // Run from player
18.         if (distance < agentDistanceRun)
19.         {
20.             // Calculate the vector pointing from Player to the Enemy
21.             Vector3 dirToPlayer = transform.position - player.position;
22.  
23.             // Calculate the vector from the Enemy to the direction away from the Player the new point
24.             Vector3 newPos = transform.position + dirToPlayer;
25.  
26.             // Shoot raycast forward to check for wall
27.             CheckForWall();
28.  
29.             // Apply rotation to destination vector
30.             Vector3 newPos2 = RotateAroundPivot(newPos, transform.position);
31.  
32.             agent.SetDestination(newPos2);
33.         }
34.     }
35.  
36.     // calculates a vector based on direction and rotation.
37.     private Vector3 RotateAroundPivot(Vector3 point, Vector3 pivot)
38.     {
39.         // Get point direction relative to pivot (the agent)
40.         Vector3 dir = point - pivot;
41.  
42.         // Apply rotation
43.         dir = Quaternion.Euler(0, vRotation, 0) * dir;
44.  
45.         //calculate the rotated point
46.         point = dir + pivot;
47.  
48.         return point;
49.     }
50.  
51.     private void CheckForWall()
52.     {
53.         Vector3 origin = transform.position;
54.         origin.y += rayCastOffset;
55.         RaycastHit hit;
56.  
57.         // shoot raycast forward to detect wall. If wall detected, update rotation value so we can apply it to agent's destination vector
58.         // until we no longer detect a wall.
59.         if (Physics.Raycast(origin, transform.forward, out hit, rayDist, layerMask))
60.         {
61.             // check if agent is currently moving right or left and increase rotation value based on that direction.
62.             if (rotationDirection == RotationDirection.Rightward)
63.             {
64.                 vRotation += 45f;
65.             } else
66.             {
67.                 vRotation -= 45f;
68.             }
69.         }
70.     }

 

Is there a reason you withheld your variables?

 

Not really, this is mostly a prototype script right now.