Memory Usage when game has a lot of data.

Hello I have been working on a ARPG game for a few years now.
I have quite a bit of data in the game at this point.

Currently I am using XML files to store the data.
I have a DataManager class that loads the xml data into lists, dictionary's etc.
The game runs great no problems the but I noticed the memory usage is quite high.

I'm guessing that is due to loading all the data into the data manager class.
This is my first game so I'm not sure what is best practice for storing/retrieving game data.

I started to move some of the data into ScriptableObjects.
I saw little or no difference changing over(but it is cumbersome and a huge pain due to needing a reference).

I thought about limiting the amount of data loaded but most of it is needed.
If an item drops it could any type with randomized data.
So most of it is needed at runtime.

Does anyone have tips for controlling memory usage when the game has a lot of data.
Think games with the amount of data like Grim Dawn, Diablo, Path of Exile, etc.
Any help is appreciated.

 

I found what I think I need right here:
https://forum.unity3d.com/threads/referenced-prefabs-and-memory-usage-is-there-a-better-way.103904/