Search Unity

Feedback Incremental GC feedback thread

Discussion in 'Experimental Scripting Previews' started by jonas-echterhoff, Nov 26, 2018.

  1. Ivan-Pestrikov

    Ivan-Pestrikov

    Joined:
    Aug 8, 2014
    Posts:
    12
    100% the standalone build, did it many time with various setups. Probably, the wrong screenshot.
     
  2. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    Thanks. This is a good test case. I have reproduced the issue and will investigate to find out what is going on.
     
    Ivan-Pestrikov likes this.
  3. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    Alright. I understand the issue now, and have a proposed fix.

    Basically, Boehm GC has a code path where when certain internal tables reach a size where they needed to grow, it would trigger a full GC and wait on that, to see if it can avoid growing the table that way to save some memory. I believe that this optimization is probably a bad idea in incremental mode, and should be disabled in that case.

    I will implement such a fix. As this has to go from Boehm into mono/il2cpp and then into Unity, it will likely take a few weeks to propagate. I will let you know when a fix is available. Thanks for your help in providing the repro project!
     
  4. Ludiq

    Ludiq

    Joined:
    Mar 6, 2015
    Posts:
    594
    Hi Jonas! Thanks for letting us experiment with this early.

    Is there any timeline for it being available for the editor?

    My current spike problem is part of my editor extension (Bolt, a visual scripting tool). I load a huge database of node options (~50K) in memory to get optimal search speeds. The problem is that this allocates a baseline of 100-200MB of managed memory that never goes out of scope. Because Boehm is non-generational, it means each GC collect causes a massive spike as it needs to traverse the full massive object list in the mark phase.

    The only solutions I see to this would be:
    • Incremental GC splitting the mark phase over multiple frames (is that possible?)
    • GC becoming generational and eventually pushing my stale root to higher gens (could that ever happen?)
    • Somehow telling the GC to ignore a specific managed root pointer (is there any way to do that?)
    Do you have any advice on solving a use case like that?
     
  5. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,607
    Last I saw was some notes on 2019.2
     
  6. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    Yes, 19.2 will have incremental GC on most platforms, including the editor.
     
    rizu likes this.
  7. Huacanacha

    Huacanacha

    Joined:
    Aug 16, 2013
    Posts:
    40
    Thanks. I believe I may have experienced the same problem in a large project with heavy memory use. On initial testing after turning on incremental GC it removed the GC spikes. Now it seems that I can't get incremental GC to work once data is loaded (in fact I just looked again and even on the initial menu screen incremental isn't called). I confirmed GarbageCollector.GCMode is enabled and isIncremental is true and it is running in a build. As a workaround I added a script to manually call CollectIncremental every frame and that does remove the GC spike again.

    I will test again with your fix when it's out.
     
  8. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    FYI, This fix will be in 2019.1.0b6 (for Mono - IL2CPP will follow later).
     
    Huacanacha and Ivan-Pestrikov like this.
  9. RichSiegel

    RichSiegel

    Joined:
    Jun 13, 2014
    Posts:
    2
    Does that include PS4?
     
  10. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    Yes.
     
  11. Aiursrage2k

    Aiursrage2k

    Joined:
    Nov 1, 2009
    Posts:
    4,807
    Cant we manually call the garbage collector. We need this for our project using webgl, we are loading models in our runtime webGL AR editor and eventually it runs out of memory because we cant free up the garbage. Cant you add it as an experimental build. Which would solve our memory problem. Or add an option to turn on/off garbage collection.
     
    Last edited: Mar 13, 2019
  12. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    777
    Wondering if you could answer @Ludiq s question? We also have large data sets that we load and that never go out of scope. If we could speed up the GC process by telling it to ignore some stuff, that would be great.
     
  13. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    No. The problem with WebGL is that the platform does not allow introspection of the stack. So we cannot really know which objects are used by local variables. So we might end up collecting objects which are still used, causing your content to crash. So the only place we can safely run GC is when we are outside of the player loop and we know that the stack cannot contain any references to managed objects.

    The only way to work around this (which I can think of) would be to create an artificial "second stack" for GC purposes, which we could then inspect. That would require building a custom compiler (a significant engineering effort), and carry a significant performance overhead.
     
    Aiursrage2k likes this.
  14. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    But the problem is: just because your data sets never go out of scope, that does not imply that the GC could ignore them - because those data sets could still contain references to other objects which need to be preserved?
     
  15. Aiursrage2k

    Aiursrage2k

    Joined:
    Nov 1, 2009
    Posts:
    4,807
    Eventually it will always crash, and require too much ram to scale up (imagine 100's or 1000's of instances on the server).
    https://forum.unity.com/threads/webgl-never-releasing-memory.520766/#post-4318303
    https://forum.unity.com/threads/webgl-excessive-memory-consumption-1-5gb.622720/
    https://forum.unity.com/threads/sin...y-webgl-heap-is-always-bound-to-crash.644305/
     
  16. Frooxius

    Frooxius

    Joined:
    Aug 15, 2012
    Posts:
    57
    Hello!

    First thanks for working on the GC, it's been a long needed change, especially with my project that was suffering from severe freezes, even when I reduced allocations as much as I could (it has lots of objects in memory).

    I've been experimenting with it for a while and since Unity 2019.1b6 I've got big improvements, with the hiccups being completely gone in normal use. It seems that with heavier and prolonged use (for hours) the overall performance starts degrading and I start getting some hiccups again, but I'll need to do more testing.

    One major issue right now are random crashes. What is the best way to report these? They are very unpredictable, so I don't know how to reliably reproduce them. I've collected some crash dumps from our users and submitted here, but not sure how much that helps: https://fogbugz.unity3d.com/default.asp?1137207_brieqrvqiubi155r

    Another issue is that this doesn't seem to help with memory fragmentation, so the memory usage seems to increase more than it should, which is particularly noticeable on mobile builds, which eventually crash due to out of memory exception, through repeated allocations and deallocations of larger objects. Having a more modern GC to deal with this would be great in the future, but even the incremental mode already helps tremendously now.
     
  17. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    Thanks for the feedback!

    As for the crashes, this is the kind of risk we're trying to minimize for this feature, so I am very curious to learn more.

    The crash dumps by itself don't help us much, but a repro project would. Even if you don't have exact repro steps which can reproduce the crash, a project which shows the crashes helps us a lot. We have internal tools to analyze the correct incremental GC usage in running projects which might produce the information on what is causing the crashes (even without reproducing them).

    (For everyone, if you have such projects in bug reports, please notify me here so I can take a look!)
     
  18. Frooxius

    Frooxius

    Joined:
    Aug 15, 2012
    Posts:
    57
    Thanks for the reply! I'll send the project over then and let you know. Is it necessary to include all the sources? I compile majority of the codebase in Visual Studio and copy over the assemblies into the Unity project, but I'm not sure if that would interfere with the analysis tools or not.
     
  19. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    Assemblies are fine - as long as I have a Unity project which lets me build a player which has the problem, I should have all I need.
     
  20. Aiursrage2k

    Aiursrage2k

    Joined:
    Nov 1, 2009
    Posts:
    4,807
    Okay any word on webgl GC?
     
  21. Frooxius

    Frooxius

    Joined:
    Aug 15, 2012
    Posts:
    57
    Oh sweet, I cleaned up the project a bit and submitted, it's an issue 1139956. I hope it will help find the issues. Let me know if you need anything extra to help.

    Thanks for your time!
     
    GloriaVictis likes this.
  22. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    I have nothing to add to what I already wrote. You linked some threads about WebGL memory issues, but I don't believe that most of those are actually specific to _managed memory usage_ which is the only thing affected by our GC - there are many different levels on which WebGL uses browser memory, and there may be problems with those, but that is not the scope of this thread.
     
  23. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    Thanks! I'll try to find time to take a look this week!
     
  24. Huacanacha

    Huacanacha

    Joined:
    Aug 16, 2013
    Posts:
    40
    Automatic incremental garbage collection is indeed working for my project again in 2019.1b8 (likely since b6 but I just got around to confirming). I’ve been able to remove the manual GC calls I had added as a workaround. Thanks for the fix!
     
    jonas-echterhoff likes this.
  25. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    I replied to your bug report, but never got a reply from that, so trying here:

     
  26. Frooxius

    Frooxius

    Joined:
    Aug 15, 2012
    Posts:
    57
    Oops, sorry it slipped through! Do you need to actually replicate the crash or just execute enough code to find the issue? Most people play in VR, so I'm not certain how often the bug should happen in screen, but most of the code is same. One thing that you might need to do is to run it on two computers and join from one to another so networking is used (joining on LAN is sufficient).

    This is possible on desktop as well. The buttons should have labels though. Try downgrading TextMesh Pro package to version 1.3.0 before building, the newer versions don't work properly yet with my project, but Unity sometimes auto-upgrades it.

    However I think the issue might actually be mostly gone. Some of our users who had most issues with GC crashes (usually during heavier sessions with 10 headsets within a session on LAN) don't seem to be crashing anymore with newer versions of the beta (b10 is when I first noticed it being better, but might've been earlier), so whatever fixes were done probably fixed this too!

    I'm not 100 % sure yet, because sometimes they would run for hours with no issues and sometimes it would be crashing quickly, but so far it seems solid with the latest version. A few users still seem to crash sometimes, but I'm not sure if those are related to the GC (it's in the PAL_Memory_Free function, I can send whole crash dump).

    Hope this helps, thanks for all your hard work on this!
     
  27. sathya

    sathya

    Joined:
    Jul 30, 2012
    Posts:
    234
    @jonas-echterhoff

    I have enabled incremental GC in player settings (Android), but when I check isIncremental it returns false.
    How to fix this. I am using 2019.1.0f2.

    Code (CSharp):
    1. internal static void GCCollect()
    2.     {
    3.         Log($"GCCollect isIncremental? {UnityEngine.Scripting.GarbageCollector.isIncremental}");
    4.         if (UnityEngine.Scripting.GarbageCollector.isIncremental)
    5.             UnityEngine.Scripting.GarbageCollector.CollectIncremental(3000000);
    6.         else
    7.             GC.Collect();
    8.     }
     
  28. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    I guess executing enough code is fine, but replicating the crash would be a good proof of executing enough code.

    will try that.

    I don't think there were any incremental gc specific fixes. But it might of course be the case that the bug was not caused by the incremental GC itself, but simply triggered by the difference in GC behavior somehow.

    Crash dump would definitely be useful!
     
  29. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633

    Are you checking this when running on the device, or in the editor?
    Also, are you building mono or il2cpp? Old or new scripting runtime?
     
  30. sathya

    sathya

    Joined:
    Jul 30, 2012
    Posts:
    234
    In the editor. Well, Editor runs in mono, isn't it?. Tried on the phone and it works as expected. Thanks.
     
  31. sathya

    sathya

    Joined:
    Jul 30, 2012
    Posts:
    234
    So If incremental GC is enabled in the settings, do we need to replace good old
    Code (CSharp):
    1. System.GC.Collect();
    with
    Code (CSharp):
    1. GarbageCollector.CollectIncremental()
    ? or does it get handled internally?
     
  32. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,607
    Firstly why are you even calling System.GC.Collect()? Pretty much no games should ever require it. I ask because you labelled it "good old" as if it's regular thing for you. If it's a regular thing for you then please make it less regular, since it's actually bad practise to use it.

    You would only want it for very niche situations... such as perhaps calling it on a menu screen or after some intensive setup and want a clean slate for the next step to avoid a hitch.
     
    dadude123 likes this.
  33. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    777
    We do it at the end of every loading screen, because why not? Do we use the new CollectIncremental method in these situations, or do we use same method we've always used - (good old) GC.Collect? :)
     
  34. Ziflin

    Ziflin

    Joined:
    Mar 12, 2013
    Posts:
    35
    This should be the reason for getting a generational garbage collector working in Unity. Most assets should end up in an older generation with relatively few references to young-generation objects. Assets without old-to-young references don't need to be visited/marked when collecting younger generations. Even with just two generations, this made a huge difference in collection times in the GC I worked on.

    Adding the incremental GC is nice to break up the large spikes, but it doesn't actually reduce the time spent in the GC. It's still wasting time marking a ton of long-lived objects (assets) in most games that never change. So it would be nice to see some improvements that reduce the total GC time as well as Unity games continue to get bigger and have more assets.
     
    Prodigga and dadude123 like this.
  35. sathya

    sathya

    Joined:
    Jul 30, 2012
    Posts:
    234
    Correct. We are aware of it and we use it only when necessary. My concern is do we need to upgrade the API or let unity handle it.
     
  36. Sebsc

    Sebsc

    Joined:
    Oct 18, 2016
    Posts:
    16
    Hi all, I've just upgraded to 2019.1 and noticed the GC kicks in every 5Mb of newly allocated memory in the editor. Is that intended behavior or did I hit on a bug? I've tested on 2 separate installs/machines and I get the same thing.

    Really looking forward to the new GC features to help in the build. Unfortunately neither the incremental GC or disabling the GC don't work in the editor. Getting hit with a full on GC every 5Mb while trying to debug is a buzz kill :)
     
  37. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    The Editor does not support incremental GC in 19.1 yet. In 19.1, incremental GC is only supported for Mac, Windows, Linux, iOS, UWP or Android player builds. Editor and console support is coming in 19.2.
     
  38. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    It's not unreasonable to call GC.Collect in places where you can afford a blocking collection killing your frame rate (such as in loading screens). If you use incremental GC, you can still call GC.Collect to perform a full GC (blocking as long as it takes). GarbageCollector.CollectIncremental will only spend as much time as you tell it (maximum). Which one you want to use really depends on your situation - also you might want to reassess if you still see a need to manually trigger any collections at all with incremental GC.
     
    Prodigga and hippocoder like this.
  39. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    Yes, a generational GC could potentially handle this particular situation better. But we don't have such a GC today (nor do we have a compacting GC, which could also be useful). We might be working on one in the future - or we might find that the move to ECS generally means less game data in managed memory to deal with. We'll watch this and continue to evaluate options. But at this point our goal (and the goal of this thread) is to stabilize the incremental GC, so we can remove the preview flag and promote it to the default option in Unity. I'm still curious in getting more feedback to that regard (ie, are you seeing incremental GC in 19.1 have the expected effects? Is it stable?).
     
  40. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    Incremental GC will come to the editor in 19.2. (Disabling GC will not come to the editor, there's too many ways how that will let you shoot yourself in the foot, as it can cause editor internals to lock up in too many ways for us to reliably document).

    As for GC collecting every 5MB, intended behavior or bug? Generally Boehm GC sets a margin of memory when it will try a collection. You could try bumping that margin by temporarily allocating a very large array.
     
  41. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    25,607
    Just reporting that I've not had any issue at all so far using it, and I get a fair bit of GC from some Unity systems (unfortunately my own code doesn't pressure the GC and I will not create artificial situations). I'm just reporting in for 2019.1 standalone builds.

    Unity's allocations from HDRP and other sources do not really pressure it enough to test but I thought I would add that at least, never seen a hitch from this end. I expect this is something of a thankless task because you only really become aware of a problem when incremental is not running ;)
     
    jonas-echterhoff likes this.
  42. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    Thanks. "I'm actually using it and things are not on fire" is very useful feedback :)
     
    Prodigga and hippocoder like this.
  43. Sebsc

    Sebsc

    Joined:
    Oct 18, 2016
    Posts:
    16
    So let's say allocate 1Gb, let it clean it up and then maybe it will be more... "relaxed"? Definitively going to give that a go.
     
  44. Doug-Wolanick

    Doug-Wolanick

    Joined:
    Mar 8, 2008
    Posts:
    406
    I just upgraded to 2019.1 today, was excited to try out incremental GC, and the in-game results are impressive :D

     
    jonas-echterhoff likes this.
  45. Huacanacha

    Huacanacha

    Joined:
    Aug 16, 2013
    Posts:
    40
    We may be releasing an update to our app shortly with incremental GC enabled. It's currently in QA. I'll report results here if we do.

    I haven't seen any issues except the bug I posted here earlier (incremental not activating automatically - since fixed). From what we can see it seems stable on Windows and Mac standalone builds on all hardware we have tested with.

    We have used incremental GC for production (but not in the hands of end-users) since earlier in the Beta. In fact it was a critical enabler of using our app for producing live broadcast TV graphics output, so you can add us to the "I'm actually using it and things are not on fire" list!
     
    jonas-echterhoff likes this.
  46. Oshigawa

    Oshigawa

    Joined:
    Jan 26, 2016
    Posts:
    252
    Wow, that looks awesome, can't wait to try it out. I hope it will lower the requirements for pooling most of the things in faster paced games.
     
  47. Frooxius

    Frooxius

    Joined:
    Aug 15, 2012
    Posts:
    57
    Oh that's interesting, although not too surprising given the random nature of these kinds of bugs. In any case, things seem to be working really well now. One of our users who who was most affected by the crashes did another heavy stress test (8 hours of continuous run with 8 clients within the same networked session) in a build with 2019.1f2 and reported no crashes, so I think things are all good now!

    Sending it over as part of that same issue if you want to have a look. The user who provided this still keeps crashing now and then, but I think it's most likely unrelated to this and seems to be quite specific to their machine.

    The incremental GC is a great help with my project, the GC freezes were previously very jarring (especially with a VR project) and difficult to work around (I had to memory pool things that were never meant to be pooled, making the code difficult to work with and some I couldn't even find a reasonable approach to pool - like libraries for working with REST API's).

    Personally I would still like to see a modern generational and compacting GC after this one is stabilized. The incremental mode takes a good chunk of precious CPU time while doing its work (my project has a fair amount of objects live in memory), reducing framerate in some cases. On mobile VR I still can run into memory fragmentation issues causing out of memory crashes and the CPU time is even more precious there.

    But even this is a great step to making things run better.
     
    jonas-echterhoff likes this.
  48. Oshigawa

    Oshigawa

    Joined:
    Jan 26, 2016
    Posts:
    252
    I finally found some time to toy around with this. I'm making a shoot em up which means a lot of things being instantiated and destroyed. I'm not yet at the performance optimization stage so a lot of stuff is not using pooling yet.

    I have one huge spike from time to time, but when using incremental GC that spike appears at least 3-4 times less often then with old garbage collector. Between them there's no noticeable frame rate drop.

    Good job, i like it.
     
    jonas-echterhoff likes this.
  49. Garainis

    Garainis

    Joined:
    Oct 14, 2013
    Posts:
    14
    @jonas-echterhoff Would you please kindly answer Ludiq's question? Thousands of Bolt users including me are affected.

    The fix Unity attempted had no impact on the issue (which didn't exist before 2018.3) which is why this question is now more relevant than ever:
    https://issuetracker.unity3d.com/is...g-noticeable-lag-spikes-when-using-bolt-asset

    https://fogbugz.unity3d.com/default.asp?1130308_u42rbj11cojvd2i6
     
    Last edited: May 31, 2019
  50. jonas-echterhoff

    jonas-echterhoff

    Unity Technologies

    Joined:
    Aug 18, 2005
    Posts:
    1,633
    Incremental GC actually does what @Ludiq recommends as the first solution:
    • Incremental GC splitting the mark phase over multiple frames (is that possible?)
    That is what incremental GC is all about. So it could help with this problem.