Question Encourage faster driving for racing agents?

Hello,

I'm currently making a racing game using ML agents. I have it set up where the AI car has to go through checkpoints that are placed throughout the track. If the car goes through a checkpoint, they get a reward.

Here are the additional rewards/penalties for my agents:

• Penalty when initially colliding with a barrier (-1).
• Penalty while colliding with a barrier (-1).
• Reward for going through a checkpoint (+1).
• Penalty for facing in the wrong direction (-1).

My issue is that I cant come up with a reward/penalty to encourage the agents to go faster. I tried doing these 3 things so far:

1. Adding a penalty of -0.001 at each time step.
2. Give a reward at each time step based on the speed: AddReward(Mathf.Clamp01(carController.speed / 175f) * 0.05f);
3. Give a reward or penalty at each time step based on the speed by scaling the speed into a different range:
   movement = Scale(-0.005f, 0.0025f, 0.0f, 175.0f, carController.speed); AddReward(Mathf.Clamp(movement, -0.005f, 0.0025f));

However, the best that I can do is get the agent to drive around the track at a relaxed speed. This is not suitable for an intense racing game. Does anybody have any tips on how I could effectively encourage the agent to achieve high speeds so they can compete with the player? Any help is appreciated.

Thank you.

 

Have you tried reducing the collision penalties? Maybe just penalize collision enter, but not collision stay events. The latter can fire very frequently and result in quite large penalties altogether. My guess is that your penalties outweigh the rewards to an extent which causes the agent to be super careful not to bump into anything. Besides, rewarding for speed already implies that getting stuck at a barrier will reduce overall rewards.
You could also try combining speed rewards and wrong direction penalties by setting rewards proportial to the dot product of the car's velocity and the direction towards the next waypoint.

 

Thanks for the insight.

Ah ok. You make a good point. I will try to take out the penalty for OnCollision stay and see what happens. I don't think it would be a good idea for me to use velocity in my case because that would lead to penalizing the agent for going in reverse. Whenever the agent clashes head-on into a wall, they need to learn that they need to go in reverse in order to get unstuck. I forgot to mention this in the starter thread, but I also have a penalty/reward for this reason:

Code (CSharp):

1. if (frontCol)
2.  {
3.       AddReward((-AIThrottle + AIBrake) * 0.5f);
4.  }

... where AIThrottle and AIBrake are in the range of [0, 1]

The brake and reverse are mapped to the same command. So I don't want to always discourage the action of going in reverse.