Best method for getting the agent to shoot its gun

How would i create a vector action for the agent to shoot? I am creating an FPS level using an agent and i currently have it set up with a field of view raycast script so that if the agent has an enemy within its view radius it will fire a bullet. I was wanting to change this so that the agent decides whether to shoot or not because at the moment it does not control this, it only has to move towards an enemy and keep it in its field of view. The problem i would have when it comes to creating a new vector action is the fact that the agent would constantly want to fire its gun no matter what its doing. Does anyone have an idea on how i could stop the agent from wanting to fire randomly and only when it sees an enemy? I have tried adding a negative reward for every time it fires and doesnt hit an enemy but this results in the agent not firing at all because its losing reward for doing so. Any ideas would be appreciated.

Thanks

 

I've struggeled with this myself a couple of times. You could write a target lock heuristic that only shoots if an enemy is within the line of fire. For that to work, the bullet would have to move fast enough (or the enemy slow enough) to ensure the bullet will hit while a target is locked.
If on the other hand things are more dynamic and involve movement delays, it would make sense for the agent to figure out when to fire. In that case, I think setting a small penalty for each missed shot is the way to go. That penalty would have to be small enough though compared to the hit reward, so that the agent isn't discouraged from shooting at all.
Another idea - haven't tested this myself yet - would be to limit the amount of ammo. The agent wouldn't receive penalties, but observe a normalized value for the number of bullets left, e.g. between 0 (no bullets) and 1 (100% bullets).

 

Thank you for the suggestions, do you possibly have an example for writing a target lock heuristic?

 

I encounter such a problem all the time. I have a different scenario, but in general the problem is the same.
But it all happens by itself when I cancel the “shot” penalty. You just need to give the reward correctly. So that the agent would understand that it is not profitable for him to shoot just like that and make x1,000,000 more simulations.
Well, or as suggested above, just block the shot if the enemy is not on target.
And it also depends on the settings in the hyperparameters, which I myself have not really figured out yet. to the identity in such cases it is better not to use SAC for me.

 

Something like this

Code (CSharp):

1. bool CanShoot(Transform agent, Transform target, float maxDistance, float maxAngle, out Vector3 direction)
2. {
3.     Vector3 delta = target.position - agent.position;
4.     direction = delta.normalized;
5.  
6.     return delta.magnitude <= maxDistance
7.         && Vector3.Angle(agent.forward, direction) <= maxAngle
8.         && !Physics.Raycast(agent.position, direction, maxDistance);
9. }

EDIT: The raycast is supposed to look for obstacles in the line of fire, so you'd of course have to set a layer mask excluding the target from detection.

 

You want to do action masking. When the gun cannot be fired, the action should be masked. In situations where you want the agent to fire (such as when it is aiming), then you can mask actions like reloading and jumping at that time.

You can also boot strap your learning in a curriculum that starts with just aiming at targets. Once it accidentally fires and starts hitting targets, remove the aiming reward and let it go from there with target-hit rewards only.

Of course, it all depends on the specific environment and the agent:

Code (CSharp):

1. if(gunsAreTakenToTownIsTrue)
2.     AddReward(gotShotFactor)