Resolved Imprecise "Write asset changes to disk" takes 80+ seconds during scene save, how to profile?

I'm debugging a bottleneck in our game's editor performances and it boiled down to EditorSceneManager.SaveOpenScenes, which underneath it has a profiler entry for "Write asset changes to disk" that often takes well over a minute on my machine:



However, there isn't any more information available and I'm not sure how to debug this. I'm already using Deep Profile.

Here's another profile with more depth, but all the time is spent in Self MS, so I have no idea what's causing this:



What could cause this? Can I get more precise debugging?

 

Similar reports with no solution:

• https://forum.unity.com/threads/play-mode-taking-insanely-long.1078502/
  
• https://forum.unity.com/threads/first-save-in-editor-causes-extremely-large-hang.1129319/
  
• https://forum.unity.com/threads/project-takes-over-10-minutes-to-open.1083935/
  
• https://forum.unity.com/threads/saving-almost-empty-scene-takes-ages.1159247/
  
• https://forum.unity.com/threads/ent...-sudden-and-last-until-unity-restart.1122292/
  
• https://www.reddit.com/r/Unity3D/comments/o826wt/editor_being_slow_in_2020_couple_of_improvements/

Like suggested in some of these threads, I:

• Rebuilt my project library, didn't fix
• Disabled sprite packing, didn't fix

 

EDIT: This was not the issue, Wwise was.

I think I fixed it.

This seems suspiciously like some sort of file lock / mutex not getting released, and I remember a lot of those happening ever since Unity has had a deeper integration with Plastic SCM after acquiring them. There were multiple recent patch notes about fixing these mutex locks, but apparently this one is another remaining one.

The fix, in my case, seems to have been to put PlasticSCM in Offline Mode from Edit > Project Settings > Version Control > Plastic SCM Settings:

 

Hi Javier, thanks a lot for these instructions!

I will re-enable Plastic SCM and test in the near future.

However, in the mean time, even after disabling, I do regularly get smaller/shorter hangs still (20-30s). Here is the native stack for them:

Code (CSharp):

1. ntdll.dll!NtWaitForSingleObject+0x14
2. KERNELBASE.dll!WaitForSingleObjectEx+0x8e
3. Unity.exe!UnityClassic::Baselib_SystemSemaphore_TryTimedAcquire+0x1a
4. Unity.exe!AutoBackgroundProgressBar::ThreadFunction+0x6c
5. Unity.exe!Thread::RunThreadWrapper+0x7b6
6. KERNEL32.DLL!BaseThreadInitThunk+0x14
7. ntdll.dll!RtlUserThreadStart+0x21
8.  

As suspected, it seems to be a semaphore wait that's taking longer than expected, but I'm not sure what for from that stack.

Again, this is with Plastic SCM in offline mode already, so (presumably?) unrelated to Plastic. I'll post here again with repros of the Plastic hang later.

Edit: here's another suspicious stack that caused a 60s+ hang:

 

Hi Javier,

I'll be on the lookout, this second one was harder to reproduce, but the first "not very interesting one" happens a lot more often.

What "suspicious" thing should I be looking for in the worker log files?

 

Hi Javier,

I've setup my project on a brand new machine with frankly ridiculous specs (16-core Intel Core i7 13th gen), and the first hang still occurs frequently, locking the editor for ~30s at a time on various operations (scene load, playmode entry, asset database refresh, etc. -- it's hard to tell what exactly causes it). I get exactly the same stack from the debugging tool every time. If that stack is "not interesting" by itself, do I have any other way of profiling what might be going wrong?

 

Thanks to Javier's help in DMs, we fixed the issue!

From looking at the asset import worker logs, it turns out the issue was that Wwise (which we use in our project) was trying to initialize on the asset worker thread and failed to do so because asset database operations are not available there, and thus tried to rebuild its entire project data asset from scratch every time. The rebuild took ~60s in our case (big production project) and caused a significant hang on the asset worker thread, which then the main thread waited on in some circumstances (scene save, play mode entry, etc.). This seems related to an issue reported on AudioKinetic's QA website here: https://www.audiokinetic.com/qa/10506/wwise-unity-unable-to-load-wwise-data

I'll report the issue to AudioKinetic, but in the mean time, the hotfix for us was to add this check:

In AkWwiseWWUBuilder.cs, in the static initializer, prefix the method with this short circuit:

Code (csharp):

1.     static AkWwiseWWUBuilder()
2.     {
3.         // Fix for hang caused by Wwise trying to load its asset data in the worker process
4.         if (UnityEditor.AssetDatabase.IsAssetImportWorkerProcess())
5.         {
6.             UnityEngine.Debug.Log("Skipping Wwise project data initialization in asset import worker process");
7.             return;
8.         }
9.        
10.         // This method gets called from InitializeOnLoad and uses the AkWwiseProjectInfo later on so it needs to check if it can run right now
11.         InitializeWwiseProjectData();
12.  
13.         UnityEditor.EditorApplication.playModeStateChanged += (UnityEditor.PlayModeStateChange playMode) =>
14.         {
15.             if (playMode == UnityEditor.PlayModeStateChange.EnteredEditMode)
16.             {
17.                 RestartWWUWatcher();
18.             }
19.         };
20.     }