ML Agent not advancing to next lesson

I am trying to train a 2d agent to move towards an opponent and perform an attack that is suppose to hit said opponent.
When the agent hits the opponent, it is given a reward and the episode ends.
The way the curriculum works is similar to that coin collector video where the agent pushes a box to get across a gap. The first lesson the opponent is at a close range, and each following lesson the opponent is placed further and further away.

I am using a reward based measure in each lesson.
The agent makes it passed the first lesson with a threshold of 0.99
The second lesson has a threshold of 1.0 and the agent never progresses to the third lesson.

I figured this would be okay since the coin collector shows thresholds higher than 1.0

Lastly, my agent does appear to be getting the rewards but the mean reward is 1.000 and the std of reward is zero.

I tried adding a negative reward for performing an attack that does not hit, however, that resulted in the agent never performing an attack.

I am currently running the agent for 1 million steps. I don't think more would help b/c of the mean reward feedback.