Navmesh agent strangeness

Ive got some issues with the navmesh agent and orientation.

Take the following 2D top down scenario - my agent is moving in x and z only.



(For now ignore the red arrows)
Scenario 1 : I start at A, tell the agent to go to B, it goes there via the green arrow (As expected)
Scenario 2 : I start at B, and tell the agent to go back to A. I EXPECT it to go back the way it came from
Scenario 3 : BUT instead it goes this way (the long way around), which looks a bit silly.

Now this is made more complicated if I allow the agent to control it's own rotations. Here the rotations im starting the agent at in red.

Now I've tried many different ways to get the agent to end up in the right orientation, but I ended up allowing the agent to control the orientation then I rotate the agent at the end the way I want him to actually face (that way the agent faces the way he's walking and curves around nicely, etc).

The problem with doing that is, because in Scenario 3 he's decided to take the long way around, he reaches his destination A - but is only half way around rotating towards A when he gets there. So now, If I lerp the quaternion, he will now start rotating back the other way (as it's closer). So he walks the long way around, turning in that direction, gets to his destination, the rotates in the other direction to get to where he's supposed to be facing (which looks even more stupid).

So my questions are :

1. Is there any way to get the agent to realize that the best way from B to A is the reverse of the route he took from A to B? (there are zero obstacles, etc)

2. If not, does anyone have a "nice" solution for handling rotations so that the agent ends rotates nicely along the path as he's walking to destination, ends up facing in a specified direction at the end (but does so, without suddenly start rotating in the opposite direction to do so?)

There must be a way, but no matter what i do - doesnt look right. Anyone got a nice solution?

Or am i just doing it wrong?