Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Question Kart racing with inter-agent collisions

Discussion in 'ML-Agents' started by victornor, Sep 10, 2023.

  1. victornor

    victornor

    Joined:
    Jan 17, 2014
    Posts:
    88
    Hey, I'm looing at the Kat Racing project created by unity.
    It trains each agent in completing races by avoiding walls.
    But will a model trained like this also behave correctly if i run them in a game together with collisions turned on?

    What's the best approach to train an agent that behaves like the AI in mario kart, where players can collide with each other. Do i just train each agent in isolation (with no inter-player collision during training) and expect it to behave properly when they collide?

    Thanks
     
    heartingNinja likes this.
  2. victornor

    victornor

    Joined:
    Jan 17, 2014
    Posts:
    88
    Side question: Can i expect to have a working agent if i only train it on one general map with all obstacle features?
    Or am i better off training a separate model for each level?
     
  3. Energymover

    Energymover

    Joined:
    Mar 28, 2023
    Posts:
    33
    I've been working on this exact task for nearly 4 months now. Basically think NASCAR. It is WAY harder than you imagine, but I have learned so much, and it is actually getting close now days. Some day I will post a video, but here are some things.

    Iterative training!!!
    Use the --initialize-from for each iteration, could also use the academy lessons I suppose, though I didn't.

    Start with a single car, use demonstration recordings to get it going. Don't add any negative rewards, only positive for crossing a checkpoint along the track and directional velocity towards their target checkpoint. Once it can get around the track most of the time, then move on to next iteration of training and don't use the demonstrations any more. Next I added in the negative reward for colliding with a wall and trained a single car a lot more. Then I moved onto 3 cars for an iteration. Then I went to 3 cars using Team Training, however I put them all on the same team (hoping it would make them a little friendly with each other). Then I did an iteration with 3 cars and using negative reward for collisions. Then I started using 6 cars, then 12 cars, then 16 cars.

    Setting up the 3d Ray Perception Sensors takes a lot of consideration. I use a checkpoint system for the track, with the next checkpoint being the target. I ended up using multiple sensors, each detects something different. If you try to use a single sensor for multiple detections you end up with cars hiding behind checkpoints, checkpoints hiding behind cars, or cars hiding the wall collisions since the sensor stops at the first object it detects.

    I used 30 checkpoints around the oval track. But each checkpoint has two pieces, the checkpoint target (gets the reward and only spans the area of the road I want them on) and the checkpoint which spans across the entire track area, has no reward and only increments the target to the next checkpoint.

    Train multiple laps around the track, not just one. If you train a single lap I found the cars get wonky crossing the finish when I used inference to run a multiple lap race. This was probably because the entry lane is after the finish line so it never could train on that area. I now train 3 lap races.

    I use the normal starting position for all cars when a new race starts, but respawn them to the pit spot when they hit max steps. I disable each car as it crosses it's 3rd lap and start a new race when they have all crossed. I also found resetting their Reward to 0 when they hit max steps helped some, otherwise they can technically get a higher reward for resetting multiple times because they get to run the same piece of track multiple times.

    It's no small task. I spent first month or two just trying to find all the hyperparameters that seem to work best and this mostly has to be done by trial & error. I've spent the last two months tuning the rewards, they make a huge difference in behaviors'. These little guys are tricky and I've seen some cheating that even Dale would be proud of. Hundreds of millions of training steps, maybe even more than a billion all together.

    So, the short answer is yes, you can train a car racing agent that will work with collisions. Including attempting to avoid other cars. I even have them drafting each other!
     
    heartingNinja and victornor like this.
  4. victornor

    victornor

    Joined:
    Jan 17, 2014
    Posts:
    88
    Cool, thank you for the very detailed response.
    4 months was not exactly what i was hoping to hear lol.
    I will go through your suggestions and see if i can't get this done in less than 4 months haha.

    Thanks!
    I think a video would be really cool!
     
  5. victornor

    victornor

    Joined:
    Jan 17, 2014
    Posts:
    88
    Are you retraining the AI for every course you have? Or do you have like a generic model that you then "specialize" on each course?
     
  6. Energymover

    Energymover

    Joined:
    Mar 28, 2023
    Posts:
    33
    I am focusing on a single track and trying to get the racing to be as "realistic" as I can get it. I figured I would train on any new tracks if I can get good results on a single track. Since it is oval track racing I think it should be able to train on new tracks fairly easy. I observe a "track number" variable, its just always set to zero for now. I also observe a "car type" variable incase I decide to have different cars. I do plan to try different power and gear ratios on the cars if I achieve results I can accept. I want to compare an agent trained on one car power setting and then change the car power settings, versus training different cars with different power settings.
     
  7. victornor

    victornor

    Joined:
    Jan 17, 2014
    Posts:
    88
    I have gotten my AI to a really great point, even handling dynamic obstacles and a ton of inter-agent collisions.

     
    Energymover and smallg2023 like this.
  8. Energymover

    Energymover

    Joined:
    Mar 28, 2023
    Posts:
    33
    Sorry, took a break and worked on some other projects. Here is the latest model I got cooking. They started liking a wall for some reason but I am sure they will grow out of it. I recorded two consecutive races with 8 cars; can be hard to tell with changing the camera. You even can see a failed drafting pass at the end of the first lap on the first race, in the second race they SUCCEED! :D
     
    Last edited: Dec 31, 2023
    tgdivy77 likes this.
  9. Zetazed

    Zetazed

    Joined:
    May 16, 2019
    Posts:
    6
    i am trying to do something similar, may i ask you a few questions? Are you using PPO? Any tips for setting the hyperparameters? My biggest problem right now is that when cars crash into walls, they remain blocked and don't figure out how to drive backwards. Did you also encounter this problem?