Reward self-play, how can an agent understand for what actions?

Hi @LexVolkov,
From the self-play documentation:

The reward signal should still be used as described in the documentation for the other trainers and reward signals. However, we encourage users to be a bit more conservative when shaping reward functions due to the instability and non-stationarity of learning in adversarial games.
​

Since agents on the same "team" are executing the same policy, you can assume that rewards are working as they normally would for the current policy execution.

 

Hi @LexVolkov ,

What you're identifying is a very hard problem faced by multi-agent scenarios (and also just sparse reward reinforcement learning in general) known as credit assignment. Consider the scenario where two soccer agents are on the same team, and one scores while the other is doing something nonsensical that does not impact the game play. You are correct, the agent might learn to associate this nonsense with goal scoring/reward which would be incorrect. This makes it a hard learning problem, and to combat this, we train the agents for many timesteps and use many samples per update. The idea is that if we use many, many training samples, we can in some sense "average out" nonsense actions like this.

TLDR; Team level rewards can create credit assignment issues for the individuals, so we use very large batches to "average out" these issues.

Hope this helps. Let us know if you have any more questions.