ELO decreasing with positive mean reward

Hello,

I have setup a competitive environment with two teams of agents and as the mean reward has increased overtime, reaching satisfactory results, the ELO has been (almost linearly) decreasing.
The past mean rewards of the agent are consistently positive (although the STD is high too), the ELO score of the agent keeps decreasing.
Is it common to see the ELO decrease with positive mean rewards?
Is it bad that the ELO is decreasing? or can it be ignored as the agent has learned to act in the said environment?

As for the environment and the reward function, there are two teams of agents in a fight, with the following rewards:

• Reward depending on the distance between the agent and its closest enemy (negative if its far away and positive if its close)
  
• Positive reward for useful actions, such as: hit enemy or defended an attack
• Negative reward when taking damage
• Setting of the episode reward depending on the result of the battle: -1 if the team lost, 0 if it was a draw and +1 if the team won

Although the mean reward surpasses the value of 1, it has learned to a satisfactory degree, but as mentioned above, the ELO keeps decreasing.

 

are you sure your last reward is -1/0/1 depend on loose/draw/win?

 

The mean reward does go above the value of 1. In any case it is set to the specific values at the end of the episode.
Does it strictly have to be +1 to be considered as win and -1 for loss? Or just being >=1 can be considered as win and <=-1 for loss?

 

Thinking about this, most likely that's the issue.

During an episode, when an agent dies I do not set its reward to negative and instead try to set it when all the agents of the team die (when they lose). As I am disabling agents that have died, I try to set their reward to negative during their re-activation, which most likely messes things up (reward set for wrong episode).
Nevertheless, I will have to test it and come back to you.
Thank you both for your replies.

On that note though, how should one design the reward function in this kind of situation? The agent dies during an episode, but may win due to the actions of another agent.
For example, it's a 2 versus 2 situation, where one agent from the first team dies, but its remaining teammate manages to defeat the other two agents. Thus, the result for the first team is a win, but the agent's episode reward is still negative.
Do you just reward agents that win and survived and simply punish agents for dying?

 

That's great!
I will check this out as well.
Thank you for you help.

 

sorry to hijack this thread, but i need clarification on what you wrote here.
during the game we assign rewards&penalties and on the conclusion i added reward as -1/0/1 based on loose/draw/win, from what i understand here its wrong?

i did it based on this documnetation
https://github.com/Unity-Technologi...-Configuration-File.md#note-on-reward-signals

should i do it like the following:
1. give penalties/rewards saving the accumulated thing asside
2. on game conclusion - use the "SetReward" assigning -1 / 0 based on loose/draw and SetReward(1-(rewards-penalties)) to the winning team?

 

I have a related question about this (not sure if that warrants its own thread) - In most situations, one agent wins and the other loses, so I assign +1/-1 rewards accordingly. But then there are also cases where an agent can be disqualified for performing some prohibited action unrelated to win/lose criteria. If that happens, the agent receives a -1 penalty and the episode ends. However, the opponent hasn't really won, it didn't contribute anything to ending the episode, so it isn't rewarded. As a result, cumulative rewards are slightly below zero on average, and ELO keeps decreasing.
How should I change my reward logic to be more in line with self-play zero sums, but still be informative as to whether an episode ended in win/lose or not?

 

Given this statement and the results that mbaske is referring to (the dropping elo): is it safe to conclude that with some self play problems, it's ok/normal to observe a drop in ELO initially?

 

Hi! I'm making turn-based battler game, and I need a bit clarification about your reply:
So you talking about Agent's reward? I mean in end of episode I need to SetReward for Groups -1/0/1-penalty and in addition I need to SetReward for each agent in loosing group as -1?

For now my reward system is:
-existentional penalty to force agent making less actions to reach result
+0.1 reward to agent when he attack another team agen
-0.05 penalty if agent was attacked.
+0.5 reward when agent destroy another agent
-0.25 penaltyif agent was destroyed

And for groups I use this reward at end of episode:
SetReward(-1/0/1-penalty) (based on loose/draft/win

Penalty is currStep/maxSteps

 

Hey, I am making a 4 player turn based card game, played as teams of 2 vs 2.
Both teams are in different groups but share same behaviour.

While training what is happening is my ElO increases with +ve mean group reward but at 200000 step, when there is team swap, it starts decreasing rapidly with -ve mean group reward.