When learning Match3, the award in the built-in Unity ML-Agents algorithm is different from the awar

Hello! Faced the following problem: when learning Match3, the reward in the built-in PPO Unity ML-Agents algorithm (about 35) is different from the reward of the self algorithm in Python (about 3). That is, different awards are shown on the graphs, and I cannot figure out how to fix this. Implemented using stable-baselines as in the manual: https://github.com/Unity-Technologi...d77b9e39f06074adaa04a8a2d/gym-unity/README.md
https://ltdfoto.ru/image/oxYAA
https://ltdfoto.ru/image/oxqsU

 

Thank you! Waiting for an answer.

 

Hi,
Are you using the example scene here? I think what's happening is

1. gym has no concept of "masked" discrete actions. We use these a lot for the match3 integration, because many moves are not valid.
2. The example scene ignores any moves that are not valid. The code for this is here. Since it ignores the move, no points are awarded.

If you needed this to work in gym, you'd need to convert the action mask to an observation, and probably need to apply a penalty when trying to make an invalid move; the agent should eventually learn what the action mask observation means and start avoiding those moves, but it will be harder for it to learn.

 

I didn't quite understand where exactly it is necessary to change the action mask.
Should I just remove the return from the code and give the agent a negative reward instead?

 

You don't need to change the action mask, but I don't think there's any way for gym to use it, so you should provide it as an observation instead.

Giving a negative reward instead will still be very hard to learn from, since the agent has to basically learn the rules of what's a valid move, instead of just being told what's valid or not.

 

Good day! I added an action mask to the observation as follows in the Write method of the Match3Sensor class. I also added a negative reward if the player makes the wrong move, but it didn't work. What's my mistake? Am I teaching too little? My timesteps = 300000. Thank you very much for your help!
Image: https://ibb.co/bv0xBy4

The code is available here:
https://pastebin.com/A4WnW7cc