Need help debugging long Unity compiles (MonoManager ReloadAssembly)

Using latest LTS release v2018.4.21f1 on Mac (10.14.6)

Been trying to debug why Unity compiles are taking much long than I would like. I have read up and implemented all the common knowledge of separating assemblies with special folders & asmdefs.
However the Reload Assembly step is taking a very long time:

Code (logs):

1. Reloading assemblies after finishing script compilation.
2. Begin MonoManager ReloadAssembly
3. Initializing Unity.PackageManager (PackageManager) v2018.4.21 for Unity v2018.4.21f1
4. Registering platform support modules:
5. Registered platform support modules in: 0.0819534s.
6. Native extension for Android target not found
7. Native extension for iOS target not found
8. Native extension for OSXStandalone target not found
9. Refreshing native plugins compatible for Editor in 2.54 ms, found 3 plugins.
10. Preloading 1 native plugins for Editor in 1.29 ms.
11. UnityIAP: [InitializeOnLoad] Facebook Check
12. UnityIAP: Runtime [2018.4.21f1]
13. UnityIAP: Current Build is iPhone:iOS
14. UnityIAP: stub FacebookStore.dll enabled
15. Mono: successfully reloaded assembly
16. Refreshing native plugins compatible for Editor in 2.22 ms, found 3 plugins.
17. Preloading 1 native plugins for Editor in 1.35 ms.
18. Compilation Report: 35.48 seconds
19. 1.32s Assembly-CSharp.dll
20. 0.88s Assembly-CSharp-Editor.dll
21. compilation total: 2.20s
22. Assembly Reload Time: 33.279296s

When using the Profiler on the Editor during compile I see this:

Where BackupScriptedObjects takes the brute force of the ReloadAssembly step with many many GC Allocs and some GCs.Collect's during.

I cannot find anything on this 'BackupScriptedObjects'

Any ideas?

 

There's a big tradeoff with asmdef files - they speed up compile times, but they incur a cost per assembly in the assembly getting reloaded.

Backup Scripted Objects is a part of that. When Unity reloads an assembly, it takes all objects that belong to that assembly, serializes them, reloads the assembly, and then deserializes those objects - I'm guessing that's what's Backup Scripted Objects in the profiler.

The big way to speed this up is to use the "fast enter playmode" feature, which skips assembly reloading entirely. The downside with that is that it's only available in Unity 2019, and won't get backported.

If that's not an option, then you could try deep profiling (use the hierarchy, not the timeline view, deep profiled recompiles takes a lot of memory already). I'm guessing that you have scripts that take a long time to serialize, which might be due to ISerializationCallbackReceiver shenanigans taking a long time.

 

Still attempting to uncover ways to work around this. I tried reducing any added asmdefs to a minimal count (<5), but it had no effect on overall compile time.

Seeing >5million calls to GC.Alloc and over 400MB in total being allocated during the ReloadAssembly step, is this expected?

Here is a snapshot from a deep compile:

 

Hmm, seems like FormerlySerializedAs is taking up a lot of time.

You could use AssetDatabase.ForceReserializeAssets to reserialize your entire project, and then purge all of the FormerlySerializedAs instances from your codebase.

 

But, yeah, you're going to see a lot of GC, since the editor's essentially serializing every single UnityEngine.Object, and then recreating all of those.