Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice
  2. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  3. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Performance/Usability

Discussion in 'Profiler Previews' started by LaireonGames, Dec 19, 2021.

  1. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Just downloaded this to have a play and WOW is it slow. I have a 11GB snapshot (big one for sure) but I can do nothing with it.

    I clicked on one section for strings taking up 200mb (I think it was this size, could have been bigger) which seemed interesting enough to look into. It took, literally, 10 minutes to process the mouse click.

    Once it loaded then I could look at thousands of strings in this weird layout but it felt useless since clicking on a string only told me its length, the first character and I'm assuming a list of references.

    Since that first click took so long, I'm afraid to use any other functions now for fear of hitting another massive loading screen!
     
  2. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    Hi there,
    We are aware that the current table and tree map UI isnt as efficient as it could be. I'd avoid the Overview page and try just the All Managed Objects table. Though that might still not be that much faster. We've recently started rewriting the table UI to something more scalable, which will likely still take us through spring. The Tree Map will also be overhauled.

    Having a more varied set of snapshots that where previously a pain to work with would be good to have something to measure against. So if you're up for it, it would be great if you could zip that capture up and attach it to a bug report & ping me the issue ID so I can pick it up. Or if it's still too big when zipped, upload it elsewhere, like Google drive or something and pm me the link.
     
  3. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Dang thanks for the fast reply!

    Ah I deleted the snapshot cause I assumed it would be pretty huge but I'll try and remember to look into taking another one tonight and upload it somewhere.

    Unfortunately the All Managed Objects table is probably going to be useless to me as well, my project is using burst compiled jobs for the heaviest lifting so everything is (understandably) lumped into unmanaged memory. I was already fairly sure this tool wouldn't help much in my situation I just wanted to use it to sanity check its not something else eating memory (its not).

    As a side note, do you think there would be a mode in the future to swap jobs memory from unmanaged to managed so it can be analysed?

    I fully appreciate this is likely no small feat and to elaborate what I am hoping for is an option (editor only is fine) as a part of packages like Collections and Jobs that would then run them using managed arrays. Of course it means performance gains would be gone but if it meant I could get some memory insight it would be really helpful.

    Overal, I guess I am asking do you have an advice/plans to make it easier to profile memory issues in a large project using lots of jobs/unmanaged memory? Cause my alternative is tediously editing each job and/or manually logging native array lengths!
     
  4. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    I know the DOTS teams are working to improve debugging and profiling capabilities via DOTS Profiler Modules that can be added to the main Profiler Window, as well as the Entity & Systems debuggers and there's some backend work happening towards better DOTS & Temp memory profiling. Eventually that will make the package more useful for DOTS profiling too but yes, right now the focus is on getting a good base covering for the non-DOTS usecases. However, I'm at least not aware of a Native collection -> Managed Array swap. Would the content of the collections be necessary/helpful for your profiling needs?

    Or rather, what kind of problems are you running into and what extra info do you think would help you in solving these?
     
  5. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Well at the moment I'm having unpredictable crashes and the only info I get from the dump is that an external tool (FMOD for audio) is trying to access memory that it doesn't have access to.

    When looking at a build I see about 3GB out of 16GB in use but in the editor I'm seeing much higher usage closer to 12GB at times. I'm fully aware that the memory between editor and builds will be different but since its a vast difference I'm not confident that things like windows task manager is fully tracking unmanaged memory as a part of the application.

    So to start, all I was trying to do was a sanity check in the editor of where my memory is and which jobs etc where allocating/processing the most to make sure things lined up.

    As a note, this is quite likely to be a crash caused by an error in a job since I do have a few of those that appear rarely in the editor I've not managed to track down yet. I just wanted to sanity check that its not actually running out of memory first.

    What would really help is seeing which classes etc the unmanaged memory currently belongs to. In my case I'm making a game similar to Minecraft and I keep a Lot of data around for things like chunks, lighting, caves, animals, biomes, insects, decorations, GPU instancing etc etc. Being able to see that the caves manager class is holding onto 1gb of memory in total would help narrow down the search.
     
    alexeyzakharov likes this.
  6. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Oh sorry as for the contents of the collections. No I don't care about the contents of the collections. Sure it would be nice to have but I'm more interested in seeing where that collection is held.

    The only reason I was interested in the unmanaged to managed swap is because it seemed an easier way to work. Rather than trying to make the profiler work with unmanaged memory I could run the build/editor in a managed mode and use the same tools to see more info.
     
  7. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    Right, that's why I was asking. In terms of ease of implementation, there might be something somewhat easier that could be done in the package, at least for some non temp collections that aren't just referenced from the stack. Gotta prototype something in January to verify that thought though.
     
    LaireonGames likes this.
  8. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Sounds great!
     
  9. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    So, I'm going to be a horrible tease but: yeah something is possible.
    For the next version tough, it'll just be in the details for Managed Objects:
    Unity_UJQUZiU9io.png devenv_vZIYMInVVZ.png

    Do you have an idea if that would already help at least somewhat or still be pretty useless for your use case?
     
    LaireonGames likes this.
  10. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Oh very nice!! :D

    Yeah I think this will be perfect because I allocate and hold onto the memory references ready to go into jobs so this will point those out.

    Come to think, this is the recommended best practise anyway (sort of):
    https://docs.unity3d.com/Manual/JobSystemTroubleshooting.html

    So it should help most uses cases and only leaves memory generated in jobs as a question mark but honestly this would be a drastic improvement because that becomes a much more niche use case to track down.

    Thanks again for looking into this!
     
    Last edited: Jan 21, 2022
  11. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    The only other scenario I have a question around is would this work for detecting the memory within the job so long as we have a reference to the job itself? E.G:

    Screenshot_1.png
     
    Last edited: Jan 21, 2022
  12. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    Happy to hear that :)
    A follow up release will then hopefully provide even better clarify on the NativeArray allocations in the Allocations Table and Memory map, based on the same approach.
    Yes. As long as something is still referencing the job, and the struct with the NativeArray isn't an unsafe struct, the thing that is referencing the job struct will show all fields of the struct, including the Native Array.
     
  13. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Amazing, looking forward to seeing this in a future update! Do you have any rough ETA on a first release?
     
  14. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    Yep, same ETA as the release mentioned in this other thread ;)
     
    LaireonGames likes this.
  15. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
  16. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Looks great, thanks for letting me know! I've got some stuff to wrap up but will check it out over the weekend
     
    MartinTilo likes this.
  17. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    I found time to give this a try but didn't make much progress :( Crashed on me a few times and got stuck on loading screens for too long, one was over 2 hours long whilst trying to quit but it seems most likely it got stuck in an infinite loop.

    I am seeing one interesting thing though that I've not been able to confirm yet due to the loading screens. First I see 3.69gb of strings in use and I can't think of an immediate reason to have any large number of strings in the project. E.G I'm not assigning random/unique names to things for example.

    However I do use this class extensively:

    Screenshot_3.png

    and I see it coming up a Lot in the profiler:

    Screenshot_4.png

    Screenshot_5.png

    I get that this class isn't really ideal since I'm making and throwing away a class with each call, each frame. Its a shame I can't make structs implement a pseudo IDisposable! But that aside, do you think its generating a string each call?

    In my head, that string is a const so it shouldn't be generating each frame but given the info above do you think that is happening as a bug/unintended consequence? I realise as I'm writing this I might need to play with passing it by ref but surely I shouldn't need to for a const.

    Anyway this weekend I'm demoing the project to the public so I don't have time to play more but after that I will try and play properly and make some example projects to try and isolate if this even a cause. Thought I would share what I have played with so far though.
     
  18. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    Well, that string is send to the native backend each time, which isn't ideal, and we have ProfilerMarker.Auto for that, which
    A) doesn't send a string to the backend on each Begin/End but only on creation and
    B) Has its AutoScope struct's Constructor and Dispose calls ignored for DeepProfiling so that it won't trash the recorded sample structure when enabling deep profiling

    But I'm unsure what kind of string mess could be caused by your solution
     
  19. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    The long opening times (which is where I presume you're hit with the long wait times?) might be down to the mesh generation for the Tree Map, which is terribly slow with lots of objects and needs to get improved/replaced. We already have that planned. I might still move it out of the Summary page for 0.5.1 to speed up initial opening at least.
     
  20. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    Alternatively 0.5 also got some fixes on being able to read strings from the snapshot more reliably and it added those to the name column... I'd hope that wouldn't impact performance too much on opening though (*), but it could affect Filtering and sorting on the Name Column...

    (* If it can now correctly read more strings that would've otherwise been "Unable Too Read Object" entries... Yeah that might impact things as there'd be more string objects that the rest of the tool would need to deal with, e.g. for that System.String group in the Tree Map and calculating their sizes and such)
     
    Last edited: Feb 7, 2022
    LaireonGames likes this.
  21. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Hmm.. so to be clear are the strings I'm using acting as a const but Profiler.BeginSample is then making a copy of them internally with the call?

    If so it makes sense now that I think about it since I'm guessing it just adds the name onto a list of some sort. I've always intended to rip out this code (well comment out actually sending the samples) once I was happy with performance so I've never tried to optimise it but your solution sounds like a best of both worlds approach that I could keep in there.
     
    Last edited: Feb 7, 2022
  22. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    It does take a while to load and yeah its one of the big waits but it does seem to trigger a long load for me with most functions that I press. E.G sorting the memory size by ascending still took a few minutes to complete. I don't think I got any further than that before it froze on me again, I think I tried changing view type and gave up on it after a 20 minute wait.

    It looks like you've pointed me towards an easy win to reduce some of my issues inspecting memory so hopefully reducing that side of things will make the tree/map etc far less complex and let me get into the rest of it.

    After my demo I can look into sending you another snapshot if you think it would be helpful
     
    MartinTilo likes this.
  23. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    yep, together with the 0.5.1 update and your changes I'd hope it gets more manageable. I do fear that there is a part to this that will need to be dealt with via the new table code we're cooking up and... some string hackery magic I literally can't wait any longer to ship... that should help long term I hope.
     
    LaireonGames likes this.
  24. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    and then, having another snapshot to benchmark against would be very helpful
     
    LaireonGames likes this.
  25. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    actually, those strings look more like callstacks, could those be from exceptions getting logged?

    if you select them, the Selection Details would show them properly with multiple lines and select- and copy-able and all that.
     
  26. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Hmm.. good point that 219B (assuming Bytes rather than Bits) is too big for the strings I thought they where. I don't think its a call stack from a log though, I'm pretty careful around monitoring them and not seeing any per frame errors currently however I wasn't looking for them when I took that sample so I can't rule it out entirely. Feels pretty unlikely though.

    I'll endeavour to do a proper investigation next week rather than guessing left, right and centre and let you know if I get to the bottom of anything.
     
    MartinTilo likes this.
  27. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    could be the Profiler Window attached in the background, switched on Deep Profiling in the build and in turn that might've created Mismatching Begin/ End Profiler Marker errors, but I'd expect that first line to look differently then so, I'm just guessing...
     
    LaireonGames likes this.
  28. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    I think you are onto something with strings being the issue. I've still not done proper tests but whilst tracking down a crash it seems I do have a memory issue so I opened up the profiler again after entirely removing that profile code I mentioned above.

    Massive difference now. Its barely any time to load a snapshot and sort/view all the data.

    Its gone from unusable to totally fine!
     
    MartinTilo likes this.
  29. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Screenshot_1.png

    What would be nice is if in this scenario that data listed a summary of its native data as well. E.G it says data is 8B which I'm assuming is the size of the pointers etc to the memory. Would be good if it was something like 8B (1MB total) where it counts all the data within it.

    Cause I know for a fact that 'data' is a big field and chunks are a big source of my memory. It lists them as 18MB in that snapshot but they will be at least 5GB, heck I wouldn't be surprised if they account for 10GB.

    Basically if I can have a way to view the total memory all chunks are taking up that would be amazing. Maybe something like a setting to include child size but I get that communicating this will be confusing.
     
  30. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    Yes, figuring out a clear way to communicate "owned/solely referenced" memory or worse, "shared memory referred to" is on issue we need to tackle before we do so. The other is just processing that info in a performant way. It's on our radar though.
     
    LaireonGames likes this.
  31. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Ah yeah totally get it, lots of crossing checking! Even if its not the default behaviour and I need to click on a group to count a total of all referenced memory would be really helpful. Wouldn't account for shared references as you mention but it would give a good starting point.

    Anyway, really glad its on the radar!

    P.S check your PM's. I sent you a snapshot as well
     
  32. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Just updated to the latest package btw but had to revert because it had a compiler error! I can re-install and get details if your not already aware (was something to do with the search package and Unity 2021)
     
  33. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    @MartinTilo version 0.6.0 sound great but still has that error:

    "Library\PackageCache\com.unity.memoryprofiler@0.6.0-preview.1\Editor\Utilities\EditorAssetFinderUtility.cs(538,29): error CS0246: The type or namespace name 'SearchViewState' could not be found"

    I'm on Unity 2021.1.24
     
  34. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    I'll check it out. It might not have been tested as 2021.1 is no longer supported.
     
  35. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    ah, what is its intended support versions? I've been finding the versions later than this one really unstable so I'm trying to hold of until 2021 LTS lands
     
  36. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    As soon as 2021.2 came out of beta, the previous Tech release dropped off from support (there may be a short period of overlap. Also since then 2021.2 starts only getting bugfixes and will eventually just be relabeled to 2021.3/LTS, but that's essentially the same branch with just more time and bugfixes passed
     
    LaireonGames likes this.
  37. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    I've put a fix in for 0.6.1 anyways.
     
  38. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    sweet thanks for that! Greatly appreciated
     
    MartinTilo likes this.
  39. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Hey again. As a bit of feedback it would be great if I could use the Up and Down arrows on my keyboard to navigate lists like this:

    Screenshot_3.png
     
  40. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Another thing that would really help would be to be able to multi select fields and show me a summary. E.G this is what I see:

    Screenshot_4.png

    Even being able to select them all and see its say 200 items select means I can figure out if this number is too high, ideally though I would like to select a group of objects like this and see the total memory of it all.

    Edit:
    Ah just noticed once you filter that the total size and count gets updated. Still a bit more convenient to be able multi select but this does the job fine
     
    Last edited: Apr 25, 2022
  41. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,431
    Yeah, that entire table UI there is getting replaced in the next big update. The new one works similarly with the summary updating According to the filters but doesn't yet have multi selection either. I'm also not sure what the side panel would be supposed to be showing in that case...

    Better keyboard navigation is one reason to be moving away from this old IMGUI table UI and unifying towards UI TK tables, but I need to check if that actually works out of the box or needs some further setup. It's a good point to bring up though!
     
    LaireonGames likes this.
  42. LaireonGames

    LaireonGames

    Joined:
    Nov 16, 2013
    Posts:
    705
    Well I was thinking it would be the summary for a multi selection. E.G I filter by mesh and can see the total is X. However I noticed a lot of the meshes are for Text Mesh Pro which I'm not interested in so if I select them I can see they are Y MB worth of my mesh total.

    This was my first chain of thought at the time since I wasn't sure how to exclude from search results
     
    MartinTilo likes this.