Curriculum Learning - not engaging

Hi all,

I've been following a tutorial for a low poly racing game, and thought I'd extend it by using ML agents to train an AI to drive round the track. I've attached a screenshot of the game so you can see the style of the game if you're interested.



It's going pretty well and the agent is able to learn to drive very well around the track and through checkpoints etc. I do have to have the checkpoints quite large, and so I thought I'd use curriculum learning to reduce their size as it progresses so that it can't cut as many corners - "cheating".

It seems like when I add curriculum learning nothing is changing, and not only is it not progressing through the levels, it seems to not even engage with any part of the curriculum. It does still train though, as it did before I added curriculum learning, just ignoring the curriculum.

I say this because in the Agent Initialize I have:

m_ResetParams = Academy.Instance.EnvironmentParameters;

and then in OnEpisodeBegin() I have:

        var checkpointSize = m_ResetParams.GetWithDefault("checkpoint_size", 20f);

        area.currentCheckpointSize = checkpointSize;

        area.ResizeCheckpoints();

My curriculum setup is:

Code (CSharp):

1. default:
2.     trainer: ppo
3.     batch_size: 32
4.     beta: 5.0e-3
5.     buffer_size: 960
6.     epsilon: 0.2
7.     hidden_units: 128
8.     lambd: 0.95
9.     learning_rate: 3.0e-4
10.     learning_rate_schedule: linear
11.     max_steps: 5.0e6
12.     memory_size: 256
13.     normalize: false
14.     num_epoch: 3
15.     num_layers: 2
16.     time_horizon: 256 #64
17.     sequence_length: 64
18.     summary_freq: 1000
19.     use_recurrent: false
20.     vis_encode_type: simple
21.     reward_signals:
22.         extrinsic:
23.             strength: 1.0
24.             gamma: 0.99
25.  
26. CarDriveBehaviour:
27.     summary_freq: 4000
28.     time_horizon: 128
29.     batch_size: 512 #128
30.     buffer_size: 4096 #2048
31.     hidden_units: 128 #512
32.     num_layers: 2
33.     beta: 1.0e-2
34.     max_steps: 5.0e6 #5.0e6
35.     num_epoch:  3 #6
36.     reward_signals:
37.         extrinsic:
38.             strength: 1.0
39.             gamma: 0.99
40.     curriculum:
41.       measure: reward
42.       thresholds: [2, 5, 10, 25, 50]
43.       min_lesson_length: 2
44.       signal_smoothing: true
45.       parameters:
46.         checkpoint_size: [70.0, 15.0, 10.0, 5.0, 2.5, 1.0]

It seems that the checkpoint size of 70 is not used, even initially, the default of 20 is being used. I have also tried having float checkpointSize = ... but that didn't work either.

Another note, my tensorboard doesn't have the Environment/Lesson plot as it should display as per: https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Using-Tensorboard.md.
This is another reason to suspect it is not engaging with the curriculum.

In case it is relevant, I don't have an AgentAcademy in my scene, the only reference to an Academy is in

m_ResetParams = Academy.Instance.EnvironmentParameters;

I realise this is a very lengthy post, but if anyone can give me any pointers as to why the curriculum is not being engaged, I would really appreciate it.

 

I found the cause of the issue - I was using Release 1 of MLAgents, but when googling the help for curriculum learning the links coming up took me to the help for the latest (master) version.

It looks like curriculum learning has been combined into 1 file for master, but for Release 1 (and 2) it needs to be a separate file.

Anyway posting this in case it helps anyone else out, and I'll be very careful in future that the help file I look at exactly matches the version I am using! (sounds obvious right?)