Editor Performance on Windows IOS platform

It seems like this has gotten worse. Probably due to optimizing the editor to be better about memory.
The editor seems to be doing some work when on the iOS platform on Windows. It feels like its converting assets from iOS formats to PC compatible when playing.

This can take 10-60s sometimes. Once its playing its fine.
If I keep a scene open in the editor with most of the assets needed during play this seems to speed up the start time. I am assuming the editor has already converted these assets in memory since they are being rendered in the scene view.

Now though when exiting playmode the delay happens. Where I think it has basically released all these assets and now is having to reload them to show them in the scene view again. This is a little better because I can do something else while this is resetting.

Its like the editor is not caching anything in memory. I am hopping there is a setting somewhere that might tell it to not do this. I have 64gb of ram and feel like the editor should not be needing to wipe its internal memory.

When running PC on windows this issue doesn't happen.
Wondering if anyone else noticing this or has any workarounds.

Unity 2021.1.25f1, 64gb ram, rtx3080.