Question Is memory increase when player join and leave is normal?

I made game multiplayer and it has a hub like City that will contains 100 player. I use client server method with Photon Fusion and Multiplay to deploy my dedicated game server (Linux).
After tested it with 15 players, I noticed my Memory Usage is always increased for each player joined and also when player leave.
If 1 player joined the memory usage is 300mb, when 15 players joined it becomes 1gb. If each player leave one by one, it also shows memory increase. When all player leave memory usage becomes 1.5gb.
I think it will be a problem because at City players will always be joining and leaving then memory usage will always increasing. Is the memory increase when player join and leave is normal?