Curiosity Value Estimate keeps increasing

Hello,

I've been training an agent to do a physic-related task in Unity using mlagents-learn, and as the task I'm interested about has a sparse-reward setup, I tried using curiosity.
What I observed during the training using curiosity is that the quantity Curiosity Value Estimate keeps increasing (goes from 0 at the beginning of training to 4e4 at about 1M steps). The curiosity reward slowly drops to 0, and the agent doesn't seem to explore the environment.

Any suggestions about what I'm doing wrong ?

 

Ok so after some digging, I found out that the observation the agent sees is very incomplete relative to the task it needs to perfom. I will try adding more information in the observation and see if this was the cause of the problem observed. (It wouldn't surprise me if it is as an incomplete observation can't determine the futur, thus that's why the agent seemed forever curious)