n00b questions relating to the karting demo

spent my first day with ML agents, and finally got it running (not a nice experience, if i'm honest, i nearly gave up several times) based off the karting demo.

i'd really really appreciate it if any of you nice peoples could bring me up to speed!

* how comes the latest karting demo doesn't need an academy? the tutorial suggests there should be one.
* is it right to just add loads of clones of a prefabbed agent in order to speed up the process, at what point does this become negative? my framerate says 60 when i run but the screen updates at around 2fps.
* is it normal if i have 40 agents, that a significant number of them drive the wrong way and seem to keep doing it?
* my mean reward after 3500 seconds and 3m steps still jolts around between 1200 and 300, does this seem right to you?
* how long should it take roughly to train a car to go round a track, i mean should it be hours, days, weeks? just wondering!
* should i be looking for the highest mean reward alongside the lowest std of reward?
* do we HAVE to go through all that python, pip, pytorch, venv stuff to get this working, is there not a simpler way, at least in windows?
* do the various ANN asset store things work the same way or do the same job as the karting demo, are they different ways of doing it altogether?

i do hang out in the discord but...i dunno it seems kinda dead, i suspect i'm too n00bish to engage with right now, which i can understand.

 

thanks for the replies! i left the learning running overnight, and they pretty much all go the right way around now. but the variance is still massive so i think it must be down to the rewards =)

 

okay sorry i have another question:

i see vids of generations learning from the best of previous generations. it seems to yield quite effective results quickly. however this unity one seems to just keep resetting and trying again getting better over time. is this just because the generation based vids are not using the unity ml-agents, they are a totally different thing?

 

all i changed in the rewards from the karting demo was the checkpoint from 1 to 5 as the little dudes decided to just spin around in an open space for the constant small speed reward rather than go for a checkpoint. maybe 5 was a little high.

I'm pretty happy with the reward system in as much as its making sense how i'd try and influence then agents and i do kinda love the kerbal space program aspect of setting it up and watching what they do! (i did reward them for not steering but they decided to just not move in the end and reap the reward per frame without incurring any penalty, which did raise a smile)

it does feel if it takes 12 hours or more for them to learn to go the right way around, that i just need to make it more obvious to the agents whats expected of them, and perhaps end an episode after one lap.

 

@christophergoy I think he's referring to a type of Evolutionary Algorithm. Probably a Genetic Algorithm.

@RR7_ Genetic algorithms are often used in youtube videos because they are fairly easy to implement and produce nice explainable results in simple simulations. They are not often used outside of toy problems or research though because they are less sample-efficient (they require more iterations) than reinforcement learning algorithms and produce a lot of random dead ends much like in real life when species go extinct (at least through natural process, aka not cause by humans). Check out the limitations for other reasons as well.

If you're interested in both approaches you can actually use a pretty simple evolutionary computational approach to RL by training a few policies, keeping the best performing policies, then using something like behavior cloning on a new policy using the top performers (just resuming training works too). For example - train 5 policies, grab the best 2, train 5 more from each of the 2, grab the best 2, rinse and repeat until you're happy with the outcome. Interesting stuff.

 

Hey there, I'm also struggling with my Agents going the wrong way on the tracks sometimes, I have noticed this happens more when they have some sort of a bump on the road or if the road goes up. Did you manage to find a way to learn them not to do that? I remember the problem occured even in the demo version when I checked it.

Also, did you train your agents by onlu adjusting their rewards or did you change any scripts or added some new ones?