Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Bug Crash in ProfilerRecorder finalizer

Discussion in 'Editor & General Support' started by sixolar, Jul 13, 2023.

  1. sixolar

    sixolar

    Joined:
    Jan 8, 2014
    Posts:
    37
    Hello,

    I'm working on an online coop game, everything was mostly tested in editor until we started beta testing about 2 months ago. At that points testers started reporting crashes, upon inspection of the crash dumps, every crash comes from internal Unity code, seemingly on totally unrelated fonctions : animator, gameobject enable/disable, physx, URP, etc. All are memory access violation (I guess it's the most common cause of crash)

    I tried updating Unity, all packages, switching from mono to il2cpp, disabled code stripping as much as possible, no luck.. I guess I'm doing something I shouldn't in user code but I can't see where I can start seaching for the cause.

    There are also a lot of times where testing sessions went without crash at all for very long, then the next day the same user would crash every 30 minutes or so... Also worth noting the game never crashes in the editor during gameplay

    We're using Mirror and the steam API, here's some of the dumps from the latest version, usable with the unity symbol server (http://symbolserver.unity3d.com)

    I have absolutely no way of reproducing those crashes, and they all happened at seemingly very different state of the game. Any help welcome.
     

    Attached Files:

  2. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,628
    Crashes like that are usually indicative of memory corruption, especially if they happen after running the game for a while. Did you try running the game using the "-debugallocator" command line argument?
     
  3. sixolar

    sixolar

    Joined:
    Jan 8, 2014
    Posts:
    37
    Thanks for the reply. Well it doesn't happen specifically after running the game for a long time, it's actually somewhat the opposite : sometimes it crashes very frequently, but most of the time everything is fine for hours. And it seem to be totally random, not associated with a particular computer or build version...

    I didn't know about -debugallocator, I can't fine much information about it (it doesn't seem to be listed anywhere here https://docs.unity3d.com/Manual/CommandLineArguments.html). I'll give it a try.
     
  4. sixolar

    sixolar

    Joined:
    Jan 8, 2014
    Posts:
    37
    Here are two dumps with -debugallocator argument, I'm not sure what to make of it, it seems to have crashed at the same line (something in the GC finalizer?) but I have no clue how to identify the cause in my code from that. If you have any insight, that would be great !
    https://drive.google.com/file/d/1S9qb17oUn8Uo6jv4syZyn15oT6eqnXFu/view?usp=drive_link

    I'm using a lot of statics, delegates and events in my code, I've read those can cause memory leak if not handled properly, but I find very few information about memory corruption, could that be a possible cause?
     
  5. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,628
    Can you share GameAssembly.dll and also GameAssembly.pdb from the "<GAMENAME>_BackUpThisFolder_ButDontShipItWithYourGame" folder? You can send them to me privately if you wish.

    I am not able to read the dump files properly without them - but your analysis about the finalizers may be correct. Given the callstack, you should be able to find the class name that is being finalized.
     
  6. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,628
    Do you use the profiler recorder APIs at all in your project?
     
  7. sixolar

    sixolar

    Joined:
    Jan 8, 2014
    Posts:
    37
    Yes, I use Profile.BeginSample and EndSample, that's it.
    I thought those were not compiled when building the game, is that not the case?
    I'll do some testing after removing them
     
  8. sixolar

    sixolar

    Joined:
    Jan 8, 2014
    Posts:
    37
    No luck, crashes still happen with the same callstack...
    Also, crashes seem to happen consistently after 10-15 minutes of gameplay with the -debugallogator argument, but without it, it crashes a lot less, sometimes not at all for hours. Is that expected behaviour? Could it be because debugallocator drastically increase memory usage (from 1Gb without to 5Gb with it)? Then could it be the debugger that is actually crashing?..
     
  9. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,628
    DebugAllocator works by making all out of bounds and freed memory accesses induce a crash, rather than creating a ticking time bomb that will crash at some point in the future (when the evidence of what went wrong is gone). It does increase memory usage and make memory allocation slower. So that is expected.

    One thing to check: what happens if you run a non-development build with -debugallocator? Does it still crash in the same place?

    I find it surprising you're not using profiler recorder APIs (those are separate from Begin/end sample) and that this is what is causing a crash in your project. Let me talk with some folks internally.
     
  10. sixolar

    sixolar

    Joined:
    Jan 8, 2014
    Posts:
    37
    I'm not sure how to identify if profiler recorder api is used in the project, I use several plugins that could use it. I'm thinking specifically of backtrace, a plugin that detects crashes and upload them. However, I installed this plugin because crashes were happening, so that's not the main cause anyway, but maybe it's doing weird things that triggers the debugallocator before the actual issue?

    Edit : I tried a build without backtrace and it still crashes at the profilerrecorder finalizer, so I guess it's not that particular plugin, I could not find any reference to ProfilerRecorder in the entire solution in visual studio, although I'm not sure that includes code in packages, and I guess it could still be used in a dll?
     
    Last edited: Jul 21, 2023
  11. sixolar

    sixolar

    Joined:
    Jan 8, 2014
    Posts:
    37
    Actually it's already the case, development build is not checked in the build window, and c++ compiler configuration is set to release. Is that a mistake? Should I try changing either / both of those?
     
  12. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,628
    I doubt it. I suspect this could be a bug in Unity itself at this point. But let me discuss with my peers first.

    It could very well be, I just assumed development build and I can't easily confirm through the dump file. If the development build watermark is not there, it's not a development build.
     
  13. sixolar

    sixolar

    Joined:
    Jan 8, 2014
    Posts:
    37
    I've tested in a dev build, with the c++ compiler config still on release, and I can confirm it crashes at the same place.
    Thanks a lot for looking into it, let me know if I can test some other things to try to pin down the issue.
     
  14. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,628
    There is some usages of profiler recorder within URP: com.unity.render-pipelines.core\Runtime\Debugging\ProfilingScope.cs

    Are you able to embed the package into your project for the time being and trying to comment out the code that constructs and operatates on the these recorder objects? https://docs.unity3d.com/Manual/upm-embed.html

    It would also be good to search the source code of all packages you're using to figure out if any of them use the recorder too.
     
  15. sixolar

    sixolar

    Joined:
    Jan 8, 2014
    Posts:
    37
    Alright, I just commented the #define UNITY_USE_RECORDER (lines 6 to 8) of ProfilingScore.cs, and no crash happened since.
    Many thanks for helping to find a fix on the user side, before a global fix can be deployed !
     
  16. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,628
    Could you by any chance report a bug on this? We are unable to figure just from the dump what exactly goes wrong in order to fix it, unfortunately.
     
  17. sixolar

    sixolar

    Joined:
    Jan 8, 2014
    Posts:
    37
    I won't be able to do that right now as I'm taking a 2 week vacation, also I can't really attach the whole project or give step by step reprod, so I'm not sure what additionnal information I could give... I'll try to setup a minimal project with the issue when I get back, but I have no idea if that will work.
     
  18. leehangyu

    leehangyu

    Joined:
    Oct 26, 2023
    Posts:
    4
    happen the same crash on ios,android,windows platform, can i do any thing to stop crash without change package code?