Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Bug Memory profiler 1.1.0 not starting / crashing

Discussion in 'Editor & General Support' started by Qleenie, Nov 15, 2023.

  1. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    835
    Hi,

    just tried to profile memory on 2023.1.20 for the first time. I installed the memory profiler as suggested by profiler.
    Now whenever I want to start the profiler, the Editor waits endlessly, while simultaneously writing A LOT of data to disk, basically until it's full (>100GB), and also filling the CPU memory completely (in my case 64GB).

    Downgrading from 1.1.0 to 1.0.0 solves the issue, so I guess that's a very bad bug in new version.
     
  2. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    835
    As an update: 1.0.0 open fine, but once I try to analyze a snapshot, same behavior occurs, disk is being written until I stopped it at 100GB, and main memory at 100% occupied by Unity Editor. Has this tested with bigger RAM sizes at all (assuming 64GB counts as "bigger")?
     
  3. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    835
    Second update: I looked in task manger what is happening; I made a snapshot, which says it's using 15.9 out of 63.9 GB of memory, which is reasonable. Once I try to open the snapshot, main memory consumption goes up to 100%, and after a few minutes it seems to start using disk cache until disk is full.

    So there seems to be a memory leak in the memory profiler, which would be kind of funny if I'd not really need it to work right now urgently, and it seems the old memory profiler, which always worked fine for me, has been removed....
     
  4. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    835
    Third update: I waited a bit longer (around 10 minutes), and it opened the snapshot, while occupying 2x the amount of my RAM on disk (around 120GB).
    So this system might work fine with 16GB of RAM, but in my case of 64GB it's pretty much unusable.
     
  5. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,407
    Hi there,

    Between 1.0.and 1.1 we changed how managed memory bytes are loaded from the byte dump contained in the snapshot into memory for analysis by the memory Profiler. In 1.0 that would all be loaded as nested managed byte arrays, in 1.1 it is loaded as native allocations using UnsafeUtility.Malloc. In 1.0, taking a snapshot of the editor after opening a snapshot can therefore bloat the managed memory usage incredibly fast.

    Also, I just (10mins ago) started looking at a similar report of issues opening on 1.1.

    For context: Does your snapshot happen to be very big or contain a lot of managed memory?

    Also:
    Was this with 1.0 or 1.1? And how does this amount relate to the snapshot size and Manager Memory amount captured by the snapshot?

    (For context, the snapshot size is mostly informed by managed heap bytes as native memory is not reported byte by byte like the managed heap dump is, but as a summary of objects, allocations and their types and sizes and some meta data.)
     
  6. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    835
    A concrete example (with 1.0.0, did not try again with 1.1.0):
    A snapshot of size 10GB used memory (out of 64 GB) took 9 minutes to open, occupying 59GB of main memory and 100GB of disk space.
     
  7. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,407
    That's the size of the capture file?! Or paging memory after opening it?
    Resident or Allocated? Or put a different way where are you taking this number from?
     
  8. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    835
    100GB is paging memory, not sure where the actual file is on disk to check it's size.
    the 10GB is what's reported in the snapshot; there are two numbers, the first says 10GB, marked with a grey bar, I guess this is the total used memory of Unity, and the second number is around 64GB, which seems the available main memory..
     
  9. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,407
    If you right click it in the memory profiler, you can click "Open Folder"

    Yeah, that's the allocated vs available. Thanks for confirming that.

    In the instance where you opened it. Do you roughly remember what it said on the Summary page for the Managed Memory summary?
     
  10. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    835
    I dared to reopen the snapshot, took again 10 minutes with same behavior.
    I opened the folder (which again took like 2 minutes, and during that time, Unity used all main memory up to 100% again!).
    The .snap file is 1.5 GB big.
    Here is a screen of the summary:

    profiler.PNG
     
  11. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,407
    Thanks for bearing with me here. I've now also concluded the other investigation and at first glance I'd say that there is a good chance that these two are unrelated. Could you please file a bug report via the Editor menu
    Help > Report a Bug
    ? It should be sufficient to just attach the snapshot file to it and (next to the repro steps of roughly: open snapshot and observe how long it takes) drop a link to this message in the description. And once you have an email reply with the issue ID, please post that ID (not the link to the ticket!) in here so that I can have a look at this and get started on a fix :)
     
  12. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,407
    Depending on what needs fixing, I could possibly get you a quick stop-gap patch pretty quickly that way, so that you can continue your investigations.
     
  13. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    835
    I remember seeing a message to not share the snapshots as it it might contain personal data ;)
     
  14. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,407
    Fair, if that is a concern for you, then we might have to go a different route. Otherwise, us employees are all under a general NDA that includes our handling of what we might see in a bug report (as well as the obligation to not willfully, or through neglect, hurt our customers ;)).

    But yeah, data in the snapshot could e.g. contain your OS user profile name via file paths, project names, and if those are held in managed memory somewhere in your project, login information to services you use via your code or 3rd party packages...

    We'll hopefully eventually get to the point where we can think about adding an option to scrub the actual byte by byte memory (or at least string memory) from the snapshot file without thereby loosing the ability to show you the analysis. And while the string data is likely unimportant in this case, the managed bytes as raw data is very likely crucial.
     
    Last edited: Nov 15, 2023
  15. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    835
    We indeed have some credentials in code.
    I'll try to do some tests on fresh projects to see if it might be in any way project or framework (HDRP, OpenXR, ...) or hardware related and will write back once I find anything valuable. For now I was able to get all the info I needed from memory profiling, albeit it took longer than planned ;)
     
    MartinTilo likes this.
  16. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,407
    Fair, let me know what xou find :)
    Also if you can use a native platform profiler like Pix or Superluminal (*) to analyze roughly where it is spending too long, I might be able to extrapolate from that info on where I could focus for optimization efforts. Though that might not be enough to figure out why it is using that much memory, which likely is the issue in the first place or very deeply connected.
    I also just remembered that our native allocator used by UnsafeUtility.Malloc had a bug in some versions on, at least, 2023, where it would fail to allocate chunks bigger than 2GB and cause some trouble that way. Given that your Managed Memory looks to be less than that, it might not be the issue, but just to mention that updating your Unity Editor version might also affect this behavior.

    (* I'm deeply aware of the irony but I have tried to use our own CPU Profiler to profile the Memory Profiler while opening large captures and in Deep Profiling the resulting frame data, together with the opened capture,
    was using so much memory that
    the relevant frames were immediately expunged from memory again to keep Unity alive on my 16GB laptop...:(:oops: One way around this is to profile to a log file in .raw format instead, which is also the only Unity CPU Profiler format that can store frame data for frames bigger than 2 GB, but native tooling might be smoother.)
     
    Last edited: Nov 15, 2023
    Qleenie likes this.
  17. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    835
    I did run some tests on a fresh project. Everything works okay here, using HDRP and OpenXR, just as in our production project. No memory leaks, it takes a few seconds to create the snapshot, and snapshot opens immediately. This time I used 1.1.0, which also fails on our production project.
    So I guess we can at least rule out the hardware / amount of main memory as source of the issue.
     
  18. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    835
    More observations: On production project in empty scene same bad behavior, exactly same memory consumption and disk space consumption, same sized .snap file, although the total resident memory is only around 5GB.

    We also have a simplified fork of project running on URP and Unity 2022.3.13 LTS. On the dedicated machine with 32GB RAM the memory profiler is crashing Windows completely during opening (memory profiler 1.1.0).
     
  19. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,407
    Some more detail to help you maybe reason through what might be causing the issue here:

    Something else (beyond and additional to what I mentioned above) that changed for 1.1 is that we now more properly go through all fields on value type objects (and nested value type fields, particularly those held in static fields were ignored in 1.0) to see if they reference any managed objects. We still don't mimic the Boehm GC completely and interpret any value type field that is of pointer size as a potential pointer, but that still adds a good deal more work. I had deemed that harmless but in a different case I've seen snapshot opening times escalate dramatically when managed value-type containing arrays with billions of entries had to be parsed by the memory profiler.

    You could try adding
    Code (CSharp):
    1. if (i > 5000)
    2.     return false;
    right after this line to see if that hotfixes your issue.
     
  20. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    835
    I get the issue in both 1.0.0 (here when I open the snapshot), and on 1.1.0 (here when I open the memory profiler, or better I did not manage to open the memory profiler in 1.1.0 at all in our project because I killed it after a few minutes).
    If it still makes sense to try the fix, I am happy to do!
     
  21. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,407
    It might be worth a shot. In 1.0 it might fail to load the managed bytes, in 1.1 it might instead take ages on parsing the fields of value types in managed arrays.
     
  22. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,407
    Did you give this a try?