UnloadUnusedAssets causes GC.MarkDependencies to spike

We're loading out levels asynchronously in small chunks (using SECTR), and unloading the parts that get too far away. UnloadUnusedAssets is called when parts are unloaded in order to clear away orphaned assets. Unfortunately, this seems to cause massive spikes (up to 450ms), attributed by the profiler to GC.MarkDependencies (UnloadUnusedAssets is not in the call stack, but I know from experimentation that calling it is the trigger). This spike is smaller if there are fewer objects in the main scene (size of the small additive scenes seems to have little effect).

My interpretation is that when UnloadUnusedAssets is called, it needs to check every possible reference in the main scene, and any reference in assets they reference, to see which assets are unused. This seems like a really inefficient method, but that's all I can think of. I'd appreciate any alternative theories, or advice in avoiding this issue. I could remove the call to UnloadUnusedAssets, but that would mean we'd have to manually manage the asset loading and unloading.