Self-Play resulting in Negative ELO with mlagents 1.0.0

Hey!

I kindly need some assistance.
I am training a simple soccer agent AI with PPO and Self Play.
The orange curve in picture (although almost flat) is an example of a typical ELO trend of my project with 0.14.1: it starts at 1200, it can decrease at the beginning, but then grows.
As I switched to Unity 2019.3.12 and mlagents 1.0.0 I noticed that I can't find a way to make ELO grow anymore. I even wondered if it is normal for ELO to go negative... I guess it would require each model iteration to be steadily worse than previous iterations, and that explanation is not easy to support, as the training works and the NN learns as expected (well... kind of ).
At the beginning I thought I had done some mistake when migrating from 0.14.1 to 0.16, but after many attempts I still can't make it grow.
Please advice

 

m_BallTouch
--curriculum=config/curricula/soccer.yaml
It's done

 

@andrewcoh_unity ,

I am still trying many different reward functions, but the result is still the same: unless I make sure that agents get a positive reward, the ELO decreases. Thanks to your hint I am now testing only positive rewards to try and see if that actually makes a difference.
I am no expert (at all) but if avg reward increases (although still in the negative space), ELO should increase, shouldn't it?

@bobben I was thinking about curriculum, too! I will try that, thanks! The point is that NN learns despite the ELO value, so it gets a little confusing.

 

Does "final reward" mean the very last reward the agent receives or the final cumulative reward over the entire episode?

 

No it's not problem I was just clarifying. Thanks.

 

In an environment where multiple agents fight for limited resources, accumulated reward would better represent their skill.
Also it'd be easier to integrate competitive training into existing training environments that are already using accumulated rewards.

 

Hi, I am making a 4 player turn based card game team of 2 vs 2. While training what is happening is my ElO increases with +ve mean group reward but at 200000 step, when there is team swap, it starts decreasing rapidly with -ve mean group reward. Is this normal?