Entropy Up and Down

I am working in an environment where an agent is places in random locations and must find a target, also placed in random locations, in a square environment. The agent has up-down, left-right movement movement (combining them it can move diagonally) and rays to detect objects. The environment is deliberately made large and curiosity is thus used.

No matter hat value I set beta to, Entropy always is as above: up and down, hovering about exactly the same value.

Please advise as to why this may be the case. It is also very weird that the value is 1.95 only.

 

I have even simplified the task drastically to have a small floor size and the goal positions and agent start potions fixed BUT entropy is still behaving weirdly. I want to encourage my agent to take different actions hence a high entropy at the start is important.

I am rewarding +1 if it finds target and -1/MaxSteps (10000) every timestep.

Please advise. This has been immensely frustrating for a while

 

For completeness of information, I am using curriculum learning. Maybe there is a problem with the other hyperparameters. Let me know if any further info is required.

 

Setting normalize to true seems to have improved results.
Entropy is still weird though.

Please advise on normalization.

https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Training-PPO.md
It says: Normalization can be helpful in cases with complex continuous control problems, but may be harmful with simpler discrete control problems.

My action space is discrete.

 

I upgraded mlagents to the latest version. With EXACTLY the same hyperparam, entropy responds correctly. I was on 0.15.0 now i am on 0.15.1.

Think this was the issue https://github.com/Unity-Technologies/ml-agents/pull/3684

 

With regards to buffer and batch size: is it related to the length of a successful episode?. I've looked at the PPO doc to for help but what does one base the value on? I have tuned manually but this is messy and random