Search Unity

Question Editor routinely uses >25GB memory and we can't find out why with Memory Profiler?

Discussion in 'Editor & General Support' started by strich, Feb 2, 2023.

  1. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    374
    Hi,

    Members of our team who have 32GB RAM are starting to seriously struggle to use the editor at all for our project now, to the point that is has become a serious issue. Whilst some of us have upgraded to 64GB RAM to resolve this, we cannot use this workaround forever as we have external people use the project to perform things like trailer recording etc that we cannot force hardware requirements on.

    Project Info:
    • We currently use Unity 2022.1.24f1 - though I have upgraded locally to 2022.2.5f1 for some Memory Profiler tests below.
    • Our project has an assets folder approximately 30GB in size.
    • When launching the editor: 12GB
      • Consists of 8GB ArtifactDB and 6GB "Native Memory" usage
    • When launching the game in the editor: 24GB+
    Data:
    2022.1.24f1 - Instantly using 11GB, with a comparison between 2 fresh loads into the editor. I wanted to see if maybe user scripts were doing/loading something or creating trash, so Capture A is the project with all *.cs scripts deleted. Capture B is the project as standard.
    Outcome - Scripts seem to be loading at least 4.5GB of "Single Large Allocation Memory Block"? I don't know how to investigate this further.
    upload_2023-2-2_15-22-32.png

    2022.2.5f1 - Fresh load into the editor, instantly using 12+GB.
    Per the Memory Profiler, it seems the editor loads the entire ArtifactDB into memory? This is huge!
    I thought maybe my local ArtifactDB might have a lot of old data, so I deleted it and regenerated it. No help.
    It seems there is a ton used in the ArtifactDB on its own, and then another 6GB in "Native Memory" - I can't seem to inspect the native memory any further still.
    upload_2023-2-2_15-40-27.png
    upload_2023-2-2_15-41-17.png

    2022.2.5f1 - Did another comparison with and without any script files. Definitely something in our scripts is chewing the hell out of "Native Memory"? How can we figure out what?
    Outcome - Same as above. Scripts somewhere loading a 5GB of native memory, and the ArtifactDB eating 8GB.
    upload_2023-2-2_16-5-6.png

    2022.2.5f1 - Took a capture after launching the editor and loading into an in-game level with a reasonable amount of content. The memory footprint balloons to over 24GB+.


    ---

    Any help here would be greatly appreciated! (@MartinTilo ?)
     

    Attached Files:

    Last edited: Feb 2, 2023
    strich, Feb 2, 2023
    #1
    SateyrOfficial likes this.

  2. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,456
    Executables and Mapped may have a lot of committed memory but consist to a large degree not of actually resident memory so that may not be as much of a concern for keeping the app alive and running well on the system. 1.1.0-exp.1, which we hope to release within the next 4 weeks, will make it easier to allow differentiating that.

    According to the All Off Memory table there, a lot of that seems to be reserved memory, possibly a high water mark left by some initial loading (maybe [InitializeOnLoad] functionality in these scripts?), which looks to be reused when going into Play mode? Maybe some divide and conquer (binary search by removing one half or the other of the scripts aonnd keep halving) the scripts can narrow down what causes the high water mark? Might not be worth the effort if the reserved space is used in playmode anyways though.

    Have you looked at the All Of Memory table for that? The list of things in Native and Managed memory may provide further direction of where to investigate.

    If you opened a memory snapshot before taking that snapshot, it may have contributed to the managed memory usage because 1.0.0 loads the entire managed heap into managed Byte arrays and may not release those until a new snapshot is opened or the window closed. That will be fixed in 1.1.0-exp.1.
     
    MartinTilo, Feb 15, 2023
    #2

  3. strich

    strich

    Joined:
    Aug 14, 2012
    Posts:
    374
    Thanks @MartinTilo. I forgot to update this thread - We believe we've found the main culprit in the ScriptableImporter API. For a reason that seems very odd to us, a custom importer we've written is storing and loading 300+ textures into memory just simply loading up the editor. We've raised a bug about it: IN-30856.

    I look forward to being able to dive a bit deeper with future versions of the Memory Profiler thanks.
     
    strich, Feb 15, 2023
    #3
    MartinTilo likes this.