Search Unity

Need some help understanding data

Discussion in 'Profiler Previews' started by QAG_Alex, Jul 22, 2020.

  1. QAG_Alex

    QAG_Alex

    Joined:
    Mar 5, 2020
    Posts:
    61
    We're using Sectr to stream scenes at runtime, however, some assets seem to be sticking around in memory. We're not sure if this is a bug with Sectr or an issue on our part so I've been trying to dig into it to figure out whats going on.

    For example using the native profiler on a player (not the editor) the object "scooter_1" is showing up when should it be unloaded:
    Screenshot_692.png
    I've confirmed that when run in the editor, the scene that this object belongs to does not exist/is not loaded. Since the native profiler wasn't giving me any more info, I went digging for additional tools and found the memory profiler!

    Using the memory profiler, I recreated the same circumstances and I see:
    Screenshot_694.png
    Why does everything but the GameObject have a RefCount of 0? As shown in the previous image, the native profiler shows these same objects referencing each other.

    Clicking the RefCount number on the "scooter_1" GameObject I'm immediately confused:
    Screenshot_695.png
    Why does RefCount now say 2? Shouldn't this still be 1 as it was in the previous image?

    I assume the two sub-entries represents the objects that are referencing the "scooter_1" GameObject, but what do I do with this information? It's not clear to me how these map back to objects/code in my game. Any help understanding this data would be greatly appreciated!

    We're using Unity 2019.4.0f1 and Memory Profiler 0.2.5 Preview 1.

     
  2. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,586
    Hi there,

    That first screenshot is showing a Material that was used in a scene that has been unloaded I guess? Provided there are no other things holding any references to that material, which is what the 2nd screenshot suggests as well, I guess you just didn't trigger Resources.UnloadUnusedAssets() (Unity's Asset-GC) yet, either directly or through a destructive (i.e. non-additive, non-async or with UnloadSceneOptions.UnloadAllEmbeddedSceneObjects) scene (un)load.

    Unity retains assets in memory after a scene has been unloaded until the next destructive load action triggers Resources.UnloadUnusedAssets (or you do so manually), since there is a high likelihood that parts of these assets are used in the next scene as well and unnecessarily un- and reloading them would be quite wasteful.

    Regarding that GameObject from screenshot 2 and 3:
    Screenshot 2 shows the Native Object of the GameObject. There is 1 reference to it and as you can see in screenshot 3, that reference is coming from the Managed Wrapper object (i.e. the thing your c# scripts can use to interface with it and hold a (kinda weak) reference to it ("kinda weak reference" because you could call Destroy on it and end up with just the managed wrapper).

    Now, the two things you can see when unfolding that wrapper is the field data held by that wrapper. As you can see, there isn't much to it, only a cached IntPtr to the native object (in screenshot 2) and a static offset for the instance id. Now, you're half way right that that m_CachedPtr is technically what really holds the reference to the Native Object but the Memory profiler knows about the relationship between the Managed and Native objects and has attributed the reference to the Managed object that has holds that field for you.

    Now why does it suddenly say RefCount 2?
    The Managed Object itself has 2 references to it specifically. Since Unity will only ever give you one and the same (i.e. cached) Managed wrapper per Native Object, no matter how often you request a reference to that Native Object, you'll actually be getting a reference to the Managed wrapper instead. So there are two Managed things that are holding on to that GO and you can see those by clicking on the ref count again.

    I hope that helps to understand it? Please don't hesitate if anything was left unclear. I'm writing up a full on guide on this all and any such questions and feedback will help in making it better :)

    That all said, we know that this hunting down the reference one level at a time is tedious and confusing in the current UI and are actively and currently working on an improved way to display these chains of references, all the way down to their roots (i.e. where this chain is "anchored" to keep it all from being GarbageCollected, wether that's the Asset- or the Scripting-GC)
     
  3. QAG_Alex

    QAG_Alex

    Joined:
    Mar 5, 2020
    Posts:
    61
    Hi @MartinTilo, thanks for the reply!

    That certainly helped a lot! I think what tripped me up was not understanding that the elements that are folded under another are owned by the parent and not references to it. Clicking the 2 in screenshot 3 from the first post I see this:
    Screenshot_697.png
    Which is certainly more helpful!

    I have a follow up question now. There are a number of objects that have a RefCount of 0. Even the "scooter_1" object in some snapshots will show up like this:
    Screenshot_698.png
    I'm not sure how to interpret this. I did some searching and found this thread where you said:
    1) I assume "root" means that the object is a root object in the scene? If that's true, "scooter_1" is certainly not a root object.
    2) Screenshot 2 shows a snapshot after I updated the unload code so that Resources.UnloadUnusedAssets is called immediately (it was being called after a delay previously) and updated the SceneManager.UnloadAsync call to use UnloadSceneOptions.UnloadAllEmbeddedSceneObjects, so I'm pretty sure the GC is not collecting these because they still are referenced somewhere.
    3) Can you elaborate on this a bit? My understanding of memory leaks is that they occur when something is allocated every frame or a reference to an object is never cleaned up causing it to stick around in memory. The former certainly isn't happening since these are assets in question and while the latter seems likely, the snapshot is showing 0 references.
     
    Last edited: Jul 23, 2020
  4. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,586
    Yeap, other roots include e.g.:
    • A static field referencing stuff.
    • A managed object that someone called UnsafeUtility.PinGCObjectAndGetAddress on, pinning it down as a root. Unless you released it again, obviously.
    • Some Manager objects in native Unity code can be roots.
    • ...
    Not necessarily a complete list, but I will compile one for that guide.

    Re 2. Could you list out the order of events here? It should be:
    1. Scene is fully unloaded (not still asynchronously unloading), ideally destructively / with that option
    2. GC.Collect *
    3. GC.WaitForPendingFinalizers *
    4. Resources.UnloadUnusedAssets
    * Steps 2-3 are something to test, but that could be included in Unload unused assets (away from my PC so I can't check in the source code just now)

    Re point 3, there are managed "leaks" which are more like "whoops, forgot a reference/pin/static root somewhere so now it's still around even though it shouldn't be" the GC still knows where it is, it just can't do anything about it until you clean up what is keeping it. Native leaks can occured if you forgot to call delete on something you malloced before (in C/C++ code).

    Leaked Native Objects without a ref, that aren't getting collected by the Asset GC shouldn't really happen (not sure why I wrote that... Should correct that and also put that into the guide). So please check the order of events again.

    If that persists, it's likely either a bug in the engine or a bug (or oversight) in the Memory snapshot backend, Or, most likely scenario, a bug in the Memory Profiler package's crawler... Would be awesome if you could file a bug report, for starters we should only need a snapshot file to check the most likely scenario, but if it's one of the other two, we'd need a repro project then.

    If you file a bug report, please ping me the issue ID once you got the email reply with it, so I can take it from there :)
     
  5. QAG_Alex

    QAG_Alex

    Joined:
    Mar 5, 2020
    Posts:
    61
    The game is currently doing just:
    1. UnloadAsync with UnloadAllEmbeddedSceneObjects
    2. Resources.UnloadUnusedAssets

    I will try this with the non-async method for the sake of testing this and report back, but we're not going to actually be able to ship with that. Since we're loading in chunks of the game with Sectr, doing a non-async load causes the game to hitch as it loads in chunks.

    Also while double checking the order of events, I realized that Resources.UnloadUnusedAssets is being called immediately after UnloadAsync which I assume may not clean up everything as the unload method is being called asynchronously. I will set this up to be called instead when SceneManager.sceneUnloaded is fired.

    Oddly I had done a barebones test of Sectr in another project and everything was unloaded as intended....
     
  6. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,586
    I strongly suspect that this was the issue and that async unloading might be fine as long as it happens with UnloadAllEmbeddedSceneObjects and if you wait until it's actually done.
     
  7. QAG_Alex

    QAG_Alex

    Joined:
    Mar 5, 2020
    Posts:
    61
    Unfortunately even after waiting until SceneManager.sceneUnloaded is fired before calling Resources.UnloadUnusedAssets, objects in unloaded scenes are still hanging around in memory :(

    Edit: also the non async method of unload (SceneManager.UnloadScene) has been deprecated. Is there an alternative method? Or did you mean I should use
    SceneManager.UnloadSceneAsync(ScenePath, UnloadSceneOptions.UnloadAllEmbeddedSceneObjects);
    (in which case I'm already doing this)?
     
    Last edited: Jul 23, 2020
  8. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,586
    Ah no, I meant SceneManager.LoadScene, but I could see how that might not fit your scenario. Still, would be interesting to see if adding 2 different empty scenes to the build and loading first the one, then the other, then taking a snapshot. At that point the GameObject should definitely be gone... If not, you got some uncleared references that the Memory Profiler is missing to capture or crawl correctly
     
  9. QAG_Alex

    QAG_Alex

    Joined:
    Mar 5, 2020
    Posts:
    61
    Ahhhh that makes more sense.

    I did a test this morning (as per your suggestion) where I loaded an empty scene via
    SceneManager.LoadScene
    then called
    Resources.UnloadUnusedAssets
    . I repeated (load scene & unload unused assets) three times. I took a snapshot each time, but I really dug into the third one. While some things seem to have been cleaned up (no longer seeing "scooter_1"!) there are still a number of things hanging around in memory, and many of them have a RefCount of 0.

    Screenshot_704.png
     
  10. QAG_Alex

    QAG_Alex

    Joined:
    Mar 5, 2020
    Posts:
    61
    I was able to locate an object that showed RefCount 0 in the native memory profiler, which was quite difficult without being able to sort objects alphabetically :(

    Here is the object in the snapshot showing 0 RefCount:
    Screenshot_708.png

    And here is the native memory profiler, where if I drill down I eventually find an element called "ManagedStaticReferences" which I believe means there is a static reference somewhere that is referencing this object:
    Screenshot_707.png
     
  11. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,586
    Uh, yeah, that'd be a bug in the crawler. Can you please file a bug report with that snapshot and ping me with the issue ID?
     
  12. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    1,586
    I mean, one is a material and the other a GameObject but I'm guessing the one is using the other and should therefore have a reference to it? Just going by the name here
     
  13. QAG_Alex

    QAG_Alex

    Joined:
    Mar 5, 2020
    Posts:
    61
    @MartinTilo 1265706

    I didn't capture it in the image, but in the second image in the left pane, I had expanded the "Material" fold out. So the object selected in the left pane is the "energyDrink" material. The object I highlighted in the right pane was arbitrary and wasn't meant to call out that particular reference. Sorry for any confusion!
     
    MartinTilo likes this.
unityunity