Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice

Bug Severe bug: Build crashes if "High Quality Lines" is being used

Discussion in 'High Definition Render Pipeline' started by Qleenie, Jul 5, 2023.

  1. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    Hey,

    I am a bit worried that this issue or the severity of this issue of the topic might be overlooked, as the main thread is in a now archived forum:
    https://forum.unity.com/threads/in-...ite-with-high-quality-line-rendering.1447150/
    and the description of the issue by QA seems to be rather cryptic:
    https://issuetracker.unity3d.com/is...ommand-arg-release-when-it-runs-out-of-memory

    The main issue is that builds with the new (2023.1+) feature "High Quality Lines" will crash after a certain amount of time, which can be less than a minute, depending on the hardware and the amount of lines on screen.
     
    schema_unity likes this.
  2. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    114
    The nature of the crash seems mostly random and will only happen on release builds (not on development builds).

    Because of this bug, it's impossible to release any builds that use the new high definition lines feature. Furthermore, it will cost devs a lot of time trying to debug this bug, because there is no error or anything they can type into google. Initially, I only found the cause by painstakingly deactivating feature by feature. Thanks to Qleenie, we were able to confirm that it is reproducible on a lot of (but not all) hardware configurations.
     
  3. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    I was able to reproduce on all of my hardware configs (4 different ones, including AMD + NVIDIA GPUs and AMD + Intel CPUs). Only thing was that on one configuration (3080 + Ryzen) it takes much longer until the crash occurs, for whatever reason this might be. On all other configs builds crashed in less than one minute.
     
  4. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    114
    I just got a crash on a dev-build. I let it run over night to see if at least dev-builds were stable. According to the last access to the log, it took around 6 hours for the crash to happen.

    Here is the stack trace (the most common trace on tlsf_free):

    Code (CSharp):
    1.  
    2. ========== OUTPUTTING STACK TRACE ==================
    3. 0x00007FFCB344D4CB (UnityPlayer) tlsf_free
    4. 0x00007FFCB1AC7CFD (UnityPlayer) DynamicHeapAllocator::Deallocate
    5. 0x00007FFCB1AC8306 (UnityPlayer) MemoryManager::Deallocate
    6. 0x00007FFCB1AD012E (UnityPlayer) free_alloc_internal
    7. 0x00007FFCB2118923 (UnityPlayer) profiling::ProfilerManager::StartNewFrame
    8. 0x00007FFCB21408B8 (UnityPlayer) profiler_start_new_frame
    9. 0x00007FFCB20DE5C4 (UnityPlayer) `InitPlayerLoopCallbacks'::`2'::InitializationProfilerStartFrameRegistrator::Forward
    10. 0x00007FFCB20CC63A (UnityPlayer) ExecutePlayerLoop
    11. 0x00007FFCB20CC70E (UnityPlayer) ExecutePlayerLoop
    12. 0x00007FFCB20D0A5A (UnityPlayer) PlayerLoop
    13. 0x00007FFCB2665CE9 (UnityPlayer) PerformMainLoop
    14. 0x00007FFCB26648CB (UnityPlayer) MainMessageLoop
    15. 0x00007FFCB266AD59 (UnityPlayer) UnityMainImpl
    16. 0x00007FFCB266CB6B (UnityPlayer) UnityMain
    17. 0x00007FF7220111F2 (D-Sim) __scrt_common_main_seh
    18. 0x00007FFE014B7614 (KERNEL32) BaseThreadInitThunk
    19. 0x00007FFE02A426F1 (ntdll) RtlUserThreadStart
    20. ========== END OF STACKTRACE ===========
    21.  
     
  5. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    114
    Just crashed again in a dev build. This time after only a few minutes. It seems like my initial notion of it not happening on dev builds is definitely not true, even though it does happen more rarely.
     
  6. MariaKhomenko

    MariaKhomenko

    Joined:
    Dec 13, 2019
    Posts:
    43
    Hey, @schema_unity !

    We are seeing very similar stacktraces and random crashes in our build, but we do not use High Quality Lines feature, we use URP and out Unity version is 2022.2.2, and the same random crash happens in 2022.3.2 version as well. Are you sure that the crash completely disappears if you disable the feature or use a previous Unity version? Sometimes we got a crash after 2 minutes running, and sometimes its 2+ hours.

    I'm attaching some callstacks we got in our crashes: 2ODTX0v.png image.png uiKcady.png

    We looked at you description of the problem, it does look identical, but only our crashes are not related to High Quality Lines feature.
     
  7. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    @schema_unity and myself did many in depth tests, and the crash we see is clearly related to the HDRP and 2023.1+ only feature "High Quality Lines". If feature is turned off = no crash. Also some of the stack traces we collected in the other thread directly point to code of High Quality Lines. There is a slim chance that it's an interaction between this feature and another Unity wide bug, but this sounds a bit unlikely.
     
  8. MariaKhomenko

    MariaKhomenko

    Joined:
    Dec 13, 2019
    Posts:
    43
    Thank you for your swift reply! It so weird that we have the same stack traces and behavior of the problem, but different crashes reason. We will continue to follow your progress with the issue just in case the root of the problem is similar.
     
  9. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    114
    Those stack traces are generally an indication of memory allocation failing in some way, as if memory was allocated once and then the same memory is being freed twice. When I first researched the trace, I found issues of this happening in different unity versions on different features. It's possible that multiple features might introduce similar issues though the root causes are not necessarily related (there is still a small chance as @Qleenie said, like it might be some new memory utility used for multiple features or something).

    If you can get it to crash reliably, you could try disabling different features one by one until the crashes stop to narrow down the problem. If you find something, I'd be happy to test it on my machines to confirm.
     
  10. MariaKhomenko

    MariaKhomenko

    Joined:
    Dec 13, 2019
    Posts:
    43
    I have some updates @schema_unity @Qleenie

    Even though we had random memory-free stack traces when crashing release builds, we tried the thing you did with development build and debugallcator arguments ang got exactly the same result as you wrote in a previous thread and the same stack https://i.imgur.com/KVrmaK4.png

    Then we narrowed the issue down to our custom render features, the code that was causing the problem was the "new ProfilingSampler" call we did every frame. Caching the call or removing the profiler calls from the code fixed the issue.

    We managed to reproduce the crash in an empty project with a test code that does exactly this. I'm attaching the code we've used to crash the builds.

    Also, we noticed that this code can even crash the editor after some time with the same stack, especially if the editor runs with attributes -debugallocator -systemallocator.

    We will be reporting this bug today with the repo project. I will update the thread with the issue number, just in case those issues are related.
     

    Attached Files:

    schema_unity and Qleenie like this.
  11. MariaKhomenko

    MariaKhomenko

    Joined:
    Dec 13, 2019
    Posts:
    43
    PS. I forgot to add that it is also a necessary step to reproduce the crash in the empty project - constantly allocate any garbage somewhere and invoke GC.Collect() and Resources.UnloadUnusedAssets.
     
    schema_unity and Qleenie like this.
  12. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    that's indeed very interesting! Seems that in the code pointed at by the stacktrace the HQ line renderer is doing the following:
    Code (CSharp):
    1. using (new ProfilingScope(cmd, new ProfilingSampler("Vertex Setup")))
    I hope Unity will fix the issue fast, if not I'll probably do a branch of HDRP and remove the calls.

    PS: That's the code which probably causes the crash:
    https://github.com/Unity-Technologi...ore/RenderPass/LineRendering.Pass.Geometry.cs
     
    Last edited: Jul 21, 2023
    schema_unity and MariaKhomenko like this.
  13. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    114
    Very nice find @MariaKhomenko. And great work finding the possible cause in HDLines @Qleenie
    It gives me quite a big relief knowing that I can now probably create a fix should the error not be fixed in time for release builds.
     
    Qleenie likes this.
  14. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    Hey @MariaKhomenko, did you report the bug, and if yes, do you have a link to the issuetracker yet?
     
  15. MariaKhomenko

    MariaKhomenko

    Joined:
    Dec 13, 2019
    Posts:
    43
    Hey, yeah! We did report it right that day I wrote you, issue number IN-48616 but I think it is not open to public yet? Not sure the link will work. The status is still "in review", but it was reproduced by unity internally in the editor.
     
    schema_unity likes this.
  16. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    Ah, thanks! You’ll get a public link as soon as it gets accepted. Might take a few more days, things are going slow due to summer probably.
     
    MariaKhomenko likes this.
  17. impheris

    impheris

    Joined:
    Dec 30, 2009
    Posts:
    1,652
    almost one month and not a word from unity?
     
  18. MariaKhomenko

    MariaKhomenko

    Joined:
    Dec 13, 2019
    Posts:
    43
    Hey, @Qleenie and @impheris , today we got a reply from Unity, stating "Your bug report has been confirmed and transferred to the appropriate internal development team at Unity. Your bug report has the following internal ID: UUM-44946"

    There is still no public link so I will keep you posted.
     
    Qleenie likes this.
  19. MariaKhomenko

    MariaKhomenko

    Joined:
    Dec 13, 2019
    Posts:
    43
  20. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    According to issuetracker it should be fixed with this weeks update of 2023.1.11.
     
    schema_unity likes this.
  21. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    114
    This is great news. I was about to ask if anyone tried fixing it by removing the Profiling scope lines in the HDRP package. I implemented it, and I haven't had a crash so far, but I didn't test it for long and not with the test project.

    I'll try the test project once the fix is out. Would be really nice if this was indeed the cause for the crash with HDLines.
     
    Qleenie likes this.
  22. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    Did you manage to test it? We released the feature as a kind of alpha version to our users, and get some reports about crashes as soon as it’s activated. So not 100% convinced it is fixed, but the cause for crash could also be something else, as we need to change some additional settings to make it work.
     
  23. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    114
    Not yet, unfortunately. I'm adding a lot of content, so I haven't made a build in a while. Sorry to hear that there are still crashes.
    I'll check out the new unity release with the test project.
     
  24. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    114
    Tested with 2023.1.11f1. Same crash after a few seconds.

    Code (CSharp):
    1. ========== OUTPUTTING STACK TRACE ==================
    2.  
    3. 0x00007FFF57638642 (UnityPlayer) block_remove
    4. 0x00007FFF57638D9A (UnityPlayer) tlsf_free
    5. 0x00007FFF56719973 (UnityPlayer) DynamicHeapAllocator::Deallocate
    6. 0x00007FFF5672104B (UnityPlayer) DualThreadAllocator<DynamicHeapAllocator>::TryDeallocate
    7. 0x00007FFF5671D410 (UnityPlayer) MemoryManager::Deallocate
    8. 0x00007FFF569A0C9D (UnityPlayer) profiler_start_new_frame
    9. 0x00007FFF56988A2E (UnityPlayer) `InitPlayerLoopCallbacks'::`2'::InitializationProfilerStartFrameRegistrator::Forward
    10. 0x00007FFF5697ECA7 (UnityPlayer) ExecutePlayerLoop
    11. 0x00007FFF5697EE34 (UnityPlayer) ExecutePlayerLoop
    12. 0x00007FFF5697F203 (UnityPlayer) PlayerLoop
    13. 0x00007FFF56BD257B (UnityPlayer) PerformMainLoop
    14. 0x00007FFF56BD48DB (UnityPlayer) MainMessageLoop
    15. 0x00007FFF56BD7D10 (UnityPlayer) UnityMainImpl
    16. 0x00007FFF56BD7F2B (UnityPlayer) UnityMain
    17. 0x00007FF7904A11F2 (TestHairHDRP) __scrt_common_main_seh
    18. 0x00007FF87C597614 (KERNEL32) BaseThreadInitThunk
    19. 0x00007FF87C8626F1 (ntdll) RtlUserThreadStart
    20.  
    21. ========== END OF STACKTRACE ===========

    Removing all ProfilingScope lines from the HDRP package above does unfortunately also not work. Same crash within seconds.
     
    Last edited: Sep 6, 2023
    Qleenie likes this.
  25. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    ok, that's what I feared. Also the ticket https://issuetracker.unity3d.com/is...ommand-arg-release-when-it-runs-out-of-memory still seems not to be in progress. That's bad. So basically the feature is broken and cannot be used.
     
  26. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    @MariaKhomenko can you confirm that your reported issue about ProfilingScope was fixed in 2023.1.11 , so that we know these are two separate issues we are facing?
     
  27. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    114
    Good news. I have found the reason for the crash and made a fix.

    I painstakingly pulled apart every piece of the HairRenderer passes. The first clue was, when commenting out the "binSort/GpuSort" section, the build would crash after MUCH longer time (minutes vs seconds).

    After pulling apart that Utility as well, I found that it was indeed the ProfilingScope that is responsible for the crashes. The more it was used per frame, the higher the chance for a crash in a release build.
    Removing all occurrences of it fixed the issue for good.

    I then took a look at the ProfilingScope class, and it became somewhat clear what was happening.

    ProfilingScope has a preprocessor directive with (#IF UNITY_EDITOR || DEVELOPMENT_BUILD), which turns the struct into a stub in release builds.

    However, every time 'using(new ProfilingScope...)' gets called in the LineRenderer it is constructed using a 'new ProfilingSampler()' instance.

    This is a full class, and it gets instantiated multiple times per frame in the HairRenderer. This also explains why there is so much garbage from running the LineRenderer to begin with. This happens on every platform (dev/release/editor).

    BUT, in the editor and development builds, ProfilingScope does Begin/End and Dispose over its lifetime. This does NOT happen in release builds, which I think is the reason it is crashing. The ProfilingSampler class allocates a marker, but it is never consumed or cleaned up properly. I don't know what it allocates exactly because those functions are internal, but it's creation uses something called "ProfilerUnsafeUtility.Create", but it's never properly consumed and cleaned, which I think is highly likely the cause for the crashes.

    There are two ways to fix this:
    1. Remove all ProfilingScope lines from the HairRenderer and GpuSort. This also gets rid of the garbage, but it requires you to edit multiple classes in two core packages. (HDRP and RenderPipeline.Core)

    2. (what I did for now): Edit ProfilingScope and add a stub for the ProfilingSampler class the same way it was done for the ProfilingScope struct. The static function can return null in release builds. However, there has to be a null check in RenderGraph.cs for this to fully work (or also branch because the profiling pass is technically not needed), but both classes are in the same package, and the change in RenderGraph is minimal. The downside is that this is still allocating the memory for the instantiations, but it's a lot easier to revert and implement.

    I would upload the changed files, but I'm not sure if that is allowed.

    I submitted another report stating my findings, and added an attached project that can turn the fix on/off via setting a scripting directive in the player settings. I hope this will help speed things up for an actual fix.

    I think it boils down to the ProfilingScope not being used correctly in the LineRenderer. Other occurrences I've seen seem to use some sort of pool (using that static function). Imo, it should not be allowed to instantiate the class freely like this. All the profiling stuff probably should also not exist in release builds in the first place (as I think was the original intention making it a stub struct that doesn't cost any overhead)
     
    Last edited: Sep 6, 2023
    Qleenie likes this.
  28. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    This sounds good. But this also means that Unity did not fix the bug properly, if I understood correctly. The ticket should be reopened in that case:

    https://issuetracker.unity3d.com/is...eprofilerrecorder-when-editor-is-in-play-mode
     
  29. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    114
    I think these are actually separate issues, since that one can happen in the editor. They might be the same in a way that both probably use the unsafe methods in the wrong way.

    They definitely mislabeled the original issue above, though.
     
  30. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    Is there any news on this issue? Did the ticket get accepted by QA? Should probably be high priority, as we still don't have a working High Quality Rendering feature without changing Unity source code....
     
  31. schema_unity

    schema_unity

    Joined:
    Jun 13, 2018
    Posts:
    114
    It has been marked as "In Review" for ~13 days now (pretty much the same day as the original pricing changes dropped). For the last bugs I submitted I usually have gotten an accepted or rejected message within a few hours. I'm not sure what happened.

    I did finally get emails for the other (rebuild hair) bug you already told me was fixed...

    At this rate I wouldn't be surprised if this bug was going to make it to LTS.
     
  32. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    Yeah, I am very worried about the pace in QA and bug fixing lately. It seems that there are not enough resources available at Unity. I have multiple pending bug reports in review, and even more waiting since long time for a fix...
     
  33. johnpa_

    johnpa_

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    3
    Hi everyone. The fix for this issue has landed and I'll be setting up the backports shortly. Just as you discovered, the issue was indeed due to misuse of profiling markers.
     
    schema_unity, Kabinet13 and Qleenie like this.
  34. Qleenie

    Qleenie

    Joined:
    Jan 27, 2019
    Posts:
    861
    Hey @johnpa_ is there a ETA for backports to happen? The official ticket still says "under consideration".
     
  35. johnpa_

    johnpa_

    Unity Technologies

    Joined:
    May 23, 2017
    Posts:
    3
    Hi @Qleenie, the backport for 23.2 has landed internally, and will be available in the next provisioned release or two -- I'll update the ticket.

    Regrettably for 23.1, the backport cutoff was missed here. I can ask our release management if we can make an exception for it if the need is dire, however since it is a tech stream, I hope that upgrading to 23.2 or the LTS is an acceptable alternative.