ScriptableObject referencering GameObject inside the prefab

Is there anyway to save ScriptableObject inside a prefab that references GameObjects that are inside the said prefab ?

The ScriptableObjects is saved via AssetDatabase.AddObjectToAsset as recommended in some other post from Unity I found.
But it seems the serializer is unable to understand reference from GameObject inside the prefabs.

This is a needed behavior of a Tutorial plugin we use - that was not meant to work with prefab but scenes
I could change all my ScriptableObject to MonoBehaviours but I feel I will lose in perfomance.

Any idea on how to solve this ?