Question Smoother Continuous Actions

I have trained a hovercraft driver with curriculum learning, it flies really well, but its a bit jittery and twitchy,
is there a "Best Practice" way of training it to output smoother actions?

was thinking of stacking the previous actions as observations and rewarding for "smoothness".
or should I just use a PID on the actions to smooth them?
if so should I train the agent with the PID or add it later only for inference?

any suggestions will be welcome!

 

One of the Unity blog posts mentions 'soft actor critic', https://bair.berkeley.edu/blog/2018/12/14/sac/ , which apparently gives less jittery outputs. Here's the blog post that references this paper https://blog.unity.com/technology/training-your-agents-7-times-faster-with-ml-agents , and the relevant screenshot from the SAC blogpost:

 

Thanks!

 

for anyone in the future looking for a solution.
I've ended up using ppo with a custom gaussian smoothing filter, that I apply after training when the model is running in game, works great for now

 

Great info. Thank you for sharing!

 

Hi, I am only familiar with Gaussian smoothing of images, where all values are already known, since we have the entire picture. May I ask how you smooth when you only have the past actions, but don't know yet what the future actions will be? Thanks for your help!

 

I record the inputs, and sample the middle point after smoothing, I have a variable to control how many samples to record and where to sample