Question Question about negative rewards

Lets say there is a episode and add bunch for + rewards, but later -1 and end the episode. So if the overall reward score in `time_horizon` is positive, this -1 will probably have no much impact, right?

 

This has similar questions, not sure I understand the answers https://github.com/Unity-Technologies/ml-agents/issues/242 about the "future" and stuff =D

 

It depends a bit on the situation.

Basically there are 2 parameters that are the most relevant here, how large the negative reward is relative to the previous positive reward and the discount factor. Where the main question for the discount factor is whether it is planning short-term or long-term.

For the actions that receive the positive rewards (and presumably should change to prevent the negative reward) the discounted negative reward should be of "sufficient" size compared to the discounted sum of positive rewards. The "sufficient" size here depends on the task/scenario. If evading the negative reward is relatively simple to do by accident (during exploration) and doesn't necessarily result in the loss of positive rewards a relatively modest discounted negative reward should suffice.

If evading the negative reward can't be done without losing positive rewards it will require a larger discounted negative reward to make preventing the negative reward "profitable".

Finally if evading the negative reward is hard to do by accident and therefor occurs rarely during training or on the way to evading this negative reward there is less potential positive reward resulting the policy might not learn to prevent the negative reward at all. In this situation, from the point of view of the learning algorithm, the negative reward may as well be a constant since it never observes a state in which it didn't receive the negative reward.

 

And as I discovered there is the "gamma" hyperparameter, that actually sets how much future rewards will effect the current actions. (clap)