holding references inside ScriptableObjects or identifiers (int)

Let's imagine this scenario (Invented):

I have a class that is Level. This class contains information for the levels, like difficulty, enemy spawn ratio or whatever.

then I have a ScriptableObject that has a list of Sprite-Level relation, as in which sprites will appear in the level. this scriptable object is just an array of LevelSpriteRelation, which holds a level reference and a textureid or gameobject instance ID.

Now my question is... should it be better to have unique ID's for Level and hold only integers inside the scriptable object? Then I guess a Manager would have an array of levels to find by id

Holding a reference is easier, but I would like to know if there is any cons on doing that, like garbage, leakage or more.