Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Speeding up "MonoManager ReloadAssembly "

Discussion in 'Scripting' started by MuseGames, Oct 15, 2014.

  1. MuseGames

    MuseGames

    Joined:
    Mar 19, 2008
    Posts:
    98
    We have a rather large project, and it's gotten to the point where Unity freezes for 30+ seconds every time it recompiles or enters playmode. Based on the logs (see below), most of the time by far is spent reloading the compiled assemblies.

    Is there anything we can do with the structure of our code or scene to speed this process up?

    And/or, what exactly is going on during that ReloadAssembly block? Profiling the editor does not appear to provide any useful information about what goes on in that time.

    Through some trial and error, I've learned that deleting certain chunks of the scene speeds this process up considerably (and recompiling in an empty scene is quite fast.) So it seems to me like there's something more going on here than just literally loading compiled assemblies into memory. The two possibilities that come to mind are Unity's deserialization (the chunks mentioned involve lots of deeply-nested GameObjects with attached scripts) or static constructors (for types that are referenced by the scripts attached to said GameObjects; I haven't found any obviously bad ones yet in reviewing the code involved, but there's a lot) However, these are only educated guesses, and I'd love to get them confirmed or denied by someone in the know.

    A relevant chunk of the editor log, demonstrating the 30-second ReloadAssembly:

    Code (csharp):
    1.  
    2. Reloading assemblies after finishing script compilation.
    3. Begin MonoManager ReloadAssembly
    4. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Managed\UnityEngine.dll (this message is harmless)
    5. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Managed\UnityEditor.dll (this message is harmless)
    6. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Managed\Unity.Locator.dll (this message is harmless)
    7. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Mono\lib\mono\2.0\I18N.dll (this message is harmless)
    8. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Mono\lib\mono\2.0\I18N.West.dll (this message is harmless)
    9. Registering custom dll's ...
    10. Non platform assembly: data-1CA41A98 (this message is harmless)
    11. Non platform assembly: data-1AFC6758 (this message is harmless)
    12. Non platform assembly: data-1CF05E28 (this message is harmless)
    13. Non platform assembly: data-1A905148 (this message is harmless)
    14. Non platform assembly: data-1AED0008 (this message is harmless)
    15. Non platform assembly: data-1A80BC18 (this message is harmless)
    16. Non platform assembly: data-1CB15200 (this message is harmless)
    17. Non platform assembly: data-1B12AE18 (this message is harmless)
    18. Non platform assembly: data-1A986138 (this message is harmless)
    19. Non platform assembly: data-1A80BC18 (this message is harmless)
    20. Non platform assembly: data-833F1BC0 (this message is harmless)
    21. Non platform assembly: data-1A80BC18 (this message is harmless)
    22. Non platform assembly: data-3FFA1340 (this message is harmless)
    23. Non platform assembly: data-3FAB1610 (this message is harmless)
    24. Non platform assembly: data-1B4C66F0 (this message is harmless)
    25. Non platform assembly: data-1B228008 (this message is harmless)
    26. Non platform assembly: data-1CBD2860 (this message is harmless)
    27. Non platform assembly: data-3F7CF368 (this message is harmless)
    28. Non platform assembly: data-1AED0008 (this message is harmless)
    29. Non platform assembly: data-3F2EE2A0 (this message is harmless)
    30. Register platform support module: C:/Program Files (x86)/Unity 4.5/Editor/Data/PlaybackEngines/AndroidPlayer/UnityEditor.Android.Extensions.dll
    31. Register platform support module: C:/Program Files (x86)/Unity 4.5/Editor/Data/PlaybackEngines/iOSSupport/UnityEditor.iOS.Extensions.dll
    32. Register platform support module: C:/Program Files (x86)/Unity 4.5/Editor/Data/PlaybackEngines/WP8Support/UnityEditor.WP8.Extensions.dll
    33. Register platform support module: C:/Program Files (x86)/Unity 4.5/Editor/Data/PlaybackEngines/MetroSupport/UnityEditor.Metro.Extensions.dll
    34. Register platform support module: C:/Program Files (x86)/Unity 4.5/Editor/Data/PlaybackEngines/BlackBerryPlayer/UnityEditor.BB10.Extensions.dll
    35. Registered in 0.329366 seconds.
    36. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Library\ScriptAssemblies\Assembly-CSharp-firstpass.dll (this message is harmless)
    37. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Library\ScriptAssemblies\Assembly-CSharp-firstpass.dll into Unity Child Domain
    38. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Library\ScriptAssemblies\Assembly-CSharp.dll (this message is harmless)
    39. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Library\ScriptAssemblies\Assembly-CSharp.dll into Unity Child Domain
    40. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Library\ScriptAssemblies\Assembly-CSharp-Editor-firstpass.dll (this message is harmless)
    41. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Library\ScriptAssemblies\Assembly-CSharp-Editor-firstpass.dll into Unity Child Domain
    42. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Library\ScriptAssemblies\Assembly-CSharp-Editor.dll (this message is harmless)
    43. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Library\ScriptAssemblies\Assembly-CSharp-Editor.dll into Unity Child Domain
    44. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Library\ScriptAssemblies\Assembly-UnityScript-firstpass.dll (this message is harmless)
    45. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Library\ScriptAssemblies\Assembly-UnityScript-firstpass.dll into Unity Child Domain
    46. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Library\ScriptAssemblies\Assembly-UnityScript-Editor.dll (this message is harmless)
    47. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Library\ScriptAssemblies\Assembly-UnityScript-Editor.dll into Unity Child Domain
    48. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\PackageManager\Unity\PackageManager\4.5.4\Unity.PackageManager.dll (this message is harmless)
    49. Loading C:\Program Files (x86)\Unity 4.5\Editor\Data\PackageManager\Unity\PackageManager\4.5.4\Unity.PackageManager.dll into Unity Child Domain
    50. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Behave\Behave.Unity.Assets.dll (this message is harmless)
    51. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Behave\Behave.Unity.Assets.dll into Unity Child Domain
    52. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Behave\Behave.Unity.Runtime.dll (this message is harmless)
    53. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Behave\Behave.Unity.Runtime.dll into Unity Child Domain
    54. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Behave\Editor\Behave.Unity.Compilers.dll (this message is harmless)
    55. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Behave\Editor\Behave.Unity.Compilers.dll into Unity Child Domain
    56. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Behave\Editor\Behave.Unity.Editor.dll (this message is harmless)
    57. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Behave\Editor\Behave.Unity.Editor.dll into Unity Child Domain
    58. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\MuseAPI\GUIScripts\MuseJayrockJson.dll (this message is harmless)
    59. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\MuseAPI\GUIScripts\MuseJayrockJson.dll into Unity Child Domain
    60. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Lib\Retlang.dll (this message is harmless)
    61. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Lib\Retlang.dll into Unity Child Domain
    62. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Lib\Steam\UnitySteam.dll (this message is harmless)
    63. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Lib\Steam\UnitySteam.dll into Unity Child Domain
    64. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\MuseAPI\Core\MuseBase.dll (this message is harmless)
    65. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\MuseAPI\Core\MuseBase.dll into Unity Child Domain
    66. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\MuseAPI\Core\MuseKey.dll (this message is harmless)
    67. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\MuseAPI\Core\MuseKey.dll into Unity Child Domain
    68. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Common.dll (this message is harmless)
    69. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Common.dll into Unity Child Domain
    70. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Goi2.Entity.dll (this message is harmless)
    71. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Goi2.Entity.dll into Unity Child Domain
    72. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\HTTP.dll (this message is harmless)
    73. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\HTTP.dll into Unity Child Domain
    74. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Icarus.Common.dll (this message is harmless)
    75. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Icarus.Common.dll into Unity Child Domain
    76. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Ionic.Zlib.dll (this message is harmless)
    77. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Ionic.Zlib.dll into Unity Child Domain
    78. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Math.dll (this message is harmless)
    79. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Math.dll into Unity Child Domain
    80. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Netcode.dll (this message is harmless)
    81. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Netcode.dll into Unity Child Domain
    82. Non platform assembly: E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Photon3Unity3D.dll (this message is harmless)
    83. Loading E:\Muse Projects\Guns Online\trunk\GunsOfIcarusMMO\Assets\Source\Networking\Lib\Photon3Unity3D.dll into Unity Child Domain
    84. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Managed\UnityEditor.Graphs.dll (this message is harmless)
    85. Loading C:\Program Files (x86)\Unity 4.5\Editor\Data\Managed\UnityEditor.Graphs.dll into Unity Child Domain
    86. Non platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\PlaybackEngines\AndroidPlayer\UnityEditor.Android.Extensions.dll (this message is harmless)
    87. Loading C:\Program Files (x86)\Unity 4.5\Editor\Data\PlaybackEngines\AndroidPlayer\UnityEditor.Android.Extensions.dll into Unity Child Domain
    88. Non platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\PlaybackEngines\iOSSupport\UnityEditor.iOS.Extensions.dll (this message is harmless)
    89. Loading C:\Program Files (x86)\Unity 4.5\Editor\Data\PlaybackEngines\iOSSupport\UnityEditor.iOS.Extensions.dll into Unity Child Domain
    90. Non platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\PlaybackEngines\WP8Support\UnityEditor.WP8.Extensions.dll (this message is harmless)
    91. Loading C:\Program Files (x86)\Unity 4.5\Editor\Data\PlaybackEngines\WP8Support\UnityEditor.WP8.Extensions.dll into Unity Child Domain
    92. Non platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\PlaybackEngines\MetroSupport\UnityEditor.Metro.Extensions.dll (this message is harmless)
    93. Loading C:\Program Files (x86)\Unity 4.5\Editor\Data\PlaybackEngines\MetroSupport\UnityEditor.Metro.Extensions.dll into Unity Child Domain
    94. Non platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\PlaybackEngines\BlackBerryPlayer\UnityEditor.BB10.Extensions.dll (this message is harmless)
    95. Loading C:\Program Files (x86)\Unity 4.5\Editor\Data\PlaybackEngines\BlackBerryPlayer\UnityEditor.BB10.Extensions.dll into Unity Child Domain
    96. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Managed\Unity.DataContract.dll (this message is harmless)
    97. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Mono\lib\mono\2.0\System.Core.dll (this message is harmless)
    98. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Managed\Unity.IvyParser.dll (this message is harmless)
    99. Initializing Unity.PackageManager (PackageManager) v4.5.4 for Unity v4.5.4
    100.  
    101. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Managed\nunit.framework.dll (this message is harmless)
    102. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Managed\Mono.Cecil.dll (this message is harmless)
    103. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Mono\lib\mono\2.0\UnityScript.dll (this message is harmless)
    104. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Managed\ICSharpCode.NRefactory.dll (this message is harmless)
    105. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Mono\lib\mono\2.0\WindowsBase.dll (this message is harmless)
    106.  
    107. ----- Total AssetImport time: 0.059074s, AssetImport time: 0.000000s, Asset hashing: 0.000000s [0 B, 0.000000 mb/s]
    108.  
    109. Platform assembly: C:\Program Files (x86)\Unity 4.5\Editor\Data\Mono\lib\mono\2.0\System.Xml.Linq.dll (this message is harmless)
    110. Mono: successfully reloaded assembly
    111. - Completed reload, in 29.941 seconds
    112.  
     
  2. Graham-Dunnett

    Graham-Dunnett

    Administrator

    Joined:
    Jun 2, 2009
    Posts:
    4,287
    It's tempting to think that the Game window is just part of the editor. What it is is a completely standalone Mono instance, so pressing play means "get all the scripts compiled, fire up a new Mono instance, and copy across all the game objects/assets/etc into that Mono." That takes time.
     
  3. MuseGames

    MuseGames

    Joined:
    Mar 19, 2008
    Posts:
    98
    Yes, I noticed. About 30 seconds in this case.

    Still hoping for a little more detail on what is going on during that time and if there's anything we can do to minimize it.
     
  4. MuseGames

    MuseGames

    Joined:
    Mar 19, 2008
    Posts:
    98
    Okay, sorry for the double post, but I'm going to keep this thread updated with my experiments in the hope that useful insights will appear (either for me or a future searcher.)

    Graham, you seem to be suggesting that the time is spent primarily loading scene data (assets, serialized fields) off the disk and into memory. If this is true, I should see faster "compile" (actually ReloadAssembly) times on a workstation with a faster drive. I would also be able to defer some of the cost by moving the complicated (deeply nested, lots of SerializedFields) chunks of the scene to another and loading that scene additively, but it would still ultimately take about the same amount of time at scene startup.

    A quick test seems to show that recompiling on a machine with an SSD does not show any significant difference in time taken, which suggests to me that disk I/O is not the bottleneck here. Breaking up the scene is a bigger project but it's on my list of things to try.

    I should also note that if I watch Task Manager while the ReloadAssembly process is going on, I can see the Unity process allocate about an extra 1GB of memory, which is then returned to the system over a minute or so after the process completes.
     
    io-games, nvnmog and Aka_ToolBuddy like this.
  5. MuseGames

    MuseGames

    Joined:
    Mar 19, 2008
    Posts:
    98
    Alright, so after testing the "load everything complex later" version, I discovered some mildly disturbing things. Here's the sequence of events:

    1) Open Unity.
    2) Run the scene the first time. ReloadAssembly takes ~2 seconds, plus another <5 seconds before the LoadLevelAdditiveAsync call finishes.
    3) Stop the scene. The complicated bits vanish, as expected.
    4) Hit play again. This time, ReloadAssembly takes ~17 seconds!
    5) Subsequent stop/play cycles continue to take about 17 seconds in ReloadAssembly

    So I see two things about this that appear to point to some kind of bug:
    - Why is the LoadLevelAdditive approach so much faster than having everything in the scene at startup? At best, I would have expected it to be no slower, but it's considerably faster.
    - Why does the second entry into play mode take so much longer? It feels almost like something is being left behind in the scene...

    Is there a more innocent explanation for this? I can attempt to file a bug report, but I'm not quite sure how to demonstrate the issue without my ~16GB project, which would be a bear to upload.
     
    Last edited: Oct 16, 2014
  6. MuseGames

    MuseGames

    Joined:
    Mar 19, 2008
    Posts:
    98
    This took an unexpected turn: in trying to reproduce the issue with LoadLevelAdditive for a bug report, I discovered an issue with HideFlags.HideAndDontSave that is probably the underlying cause of the strange behavior I mentioned. The case number is 640509, in short: objects created at runtime but marked HideAndDontSave are not fully cleaned up when the editor exits playmode (they are not visible in the scene hierarchy, but leave traces elsewhere in memory as the reproduction case shows.)
     
  7. Graham-Dunnett

    Graham-Dunnett

    Administrator

    Joined:
    Jun 2, 2009
    Posts:
    4,287
    Have notified QA.
     
  8. MuseGames

    MuseGames

    Joined:
    Mar 19, 2008
    Posts:
    98
    Well, while the issue with HideAndDontSave is real, it is not the cause of my extra ReloadAssembly time on the second and subsequent runs. Or at least, I'm not leaking anything that appears in FindObjectsOfTypeAll like it did in my test case. The hunt continues.
     
  9. MuseGames

    MuseGames

    Joined:
    Mar 19, 2008
    Posts:
    98
    More fun with Resources.FindObjectsOfTypeAll:

    I've found a correlation between the slow ReloadAssembly and a large number of assets (textures etc) remaining "loaded" according to this function; i.e., when I open the scene the first time, Resources.FindObjectsOfTypeAll( typeof(Texture) ) return around 300 items, but after starting and stopping it returns 1100. A great many of these extras are obviously textures that are not referenced by any scene object - they are part of objects that are loaded through Application.LoadLevelAdditiveAsync at runtime.

    I've been attempting to trace this further by debugging what objects are still referencing these, but haven't found an editor function or combination thereof to actually do this. Is there a way through scripting to find out why these are still in memory? The memory profiler seems like it should be able to help, but in practice has so far simply frozen up when I try to take a sample...
     
  10. MuseGames

    MuseGames

    Joined:
    Mar 19, 2008
    Posts:
    98
    Turns out "memory profiler freezing up" was actually "memory profiler taking quite a few minutes to process some circular references." Progress.
     
  11. MuseGames

    MuseGames

    Joined:
    Mar 19, 2008
    Posts:
    98
    Alright, so I've solved most of the immediate issue now. Cleaning up some circular references made the memory profiler behave much more reasonably, and allowed me to actually track down what was keeping those textures around. The chief cause of the problem was static references to various objects, which I guess meant that "transferring assets to the new instance of mono" included a lot more assets than one would have reasonably expected.

    The one weird part I still don't understand: why is loading these objects through a LoadLevelAdditive call so much faster than having them in the active scene? This is still the case (~2 seconds vs ~17 seconds), even after the static reference issue was resolved.
     
    inZania, Harinezumi and Andy-Korth like this.