LSTM usage logic

use_recurrent: true
sequence_length: 64
memory_size: 256

• use_recurrent is a flag that notifies the trainer that you want to use a Recurrent Neural Network.
• sequence_length defines how long the sequences of experiences must be while training. In order to use a LSTM, training requires a sequence of experiences instead of single experiences.
• memory_size corresponds to the size of the memory the agent must keep. Note that if this number is too small, the agent will not be able to remember a lot of things. If this number is too large, the neural network will take longer to train.

How all those parameters are connected to vector of observations?
Does "sequence_length" exactly mean that 64 x observation vectors will be used?

What is the best practice to calculate "memory_size" based on the size of observation vector?

 

I had exactly these same questions and came to the forum looking for an answer, so I'm sad not to see them here. Does anyone know the answer to AndrewGri's questions?

 

As I understand it, sequence_length means indeed that 64 observations are used during training to learn the temporal relationships.

I haven't found any definite ways to calculate the memory_size (or any model architectural choices). This is generally quite difficult. Good practice would be to start by using something equal to another network with similar input, like one of the ml-agents examples. In the first place see if adding recurrence makes a positive difference to your agent's performance first by establishing a baseline.

Then start experimenting:
Reduce memory size by half, and see if this decreases (or even increases, because faster to train) your agent's performance. If performance decreased, you can maybe double your initial baseline, and see how that affects performance. If this also doesn't increase performance you know you're in the right ballpark, and can optionally tune further in between what worked best.

 

Is there any further progress and results to the posts provided here? I am working on an LSTM for a Unity agent. But I need to decide to create the cells inside or outside the Unity framework.