Problem with agents trying to reach their target through walls

Here is an issue I guess everyone's having but I haven't figured out yet a satisfying (simple, reliable) way of solving it. It involves an agent and its target in a room. The target is on an unreachable area (not on the navmesh).

In the first image, the agent goes towards its target because it can reach it (its stopping distance is its attach reach distance). But in the second picture the target has moved away and the agent can't reach it anymore. So the agent decides to get out of the room to try and attack its target from the other side of the wall. Because in computer-logic the distance would be technically shorter. Of course it doesn't make any practical sense.

So is there a way to keep this behavior from happening? In other words is there a concept of "walls" in the navmesh system?

 

I assume that the target is your character. You may consider to have triggers in the areas that are not part of the navigation mesh. In the trigger game object, you may have another component that has a reference to a game object which can be used to redirect the agents. Whenever the character enters the trigger, the agents won't go directly to its position, but instead use the position of that special game object.

Another possibility would be to keep track of Tilo's last position on the navigation mesh. You may check in every frame what the closest point on the navigation mesh is and if it is close enough, keep that position. The enemies may now completely ignore Tilo's actual position when they are far away, but instead walk to the last known position on the navigation mesh.

 

Thanks Andy! That's in so many words what I'm forced to do right now. But I wish it would be simpler to tackle. I mean this is a basic issue and having to find hacks around the system is far from ideal.

There should be a proper, native way of dealing with this situation. Like marking some areas as "walls" and getting the agents to NOT choose a destination that would put them behind those when they're trying to reach their target.

 

When you set the destination of an agent outside of an actual navigation mesh, it will calculate the closets position on a navigation mesh. In your case, it is unfortunately on the other side of the wall. Agents can't deal with anything that is not on the navigation mesh.
The information that there is a wall and the agent can't pass through it needs to be stored in the navigation mesh. If you want the correct behavior, you need to add the area where the character is to the navigation mesh as well. Like that, you store the information how to reach those positions and with that indirectly also where the walls are.
In order to prevent your enemies to reach those areas, you may have to setup additional navigation areas or you may not allow them to pass certain off mesh links.