Search Unity

Question Debug Builds on Android Busting Memory Limits, Crashing

Discussion in 'Editor & General Support' started by CPlusSharp22, Jul 25, 2020.

  1. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    I have a fairly massive game that uses AssetBundles. Recently hit an issue where the debug mono builds are crashing during gameplay when I load some of my larger assets. They were unchanged recently, so I must've pushed memory over somewhere else and it was the tipping point. But it lead me down a rabbit hole observing Unity behavior in my project.

    For reference, I'm on Unity 2019.4.3f1

    My initial scene when booting up the game that leads to a UI menu shows this in the profiler. Now, this profile is the *first* frame that shows up in the Unity Profiler after waiting to attach the debugger on Android:
    Code (CSharp):
    1.  
    2. Used Total: 377.1 MB   Unity: 134.0 MB   Mono: 58.3 MB   GfxDriver: 145.9 MB   Audio: 3.5 MB   Video: 0 B   Profiler: 35.3 MB  
    3. Reserved Total: 429.3 MB   Unity: 171.0 MB   Mono: 73.4 MB   GfxDriver: 145.9 MB   Audio: 3.5 MB   Video: 0 B   Profiler: 35.5 MB  
    4. Total System Memory Usage: 3.46 GB
    5.  
    6. Textures: 251 / 133.8 MB
    7. Meshes: 57 / 20.4 MB
    8. Materials: 163 / 477.0 KB
    9. AnimationClips: 48 / 0.8 MB
    10. AudioClips: 8 / 2.7 MB
    11. Assets: 52014
    12. GameObjects in Scene: 134
    13. Total Objects in Scene: 570
    14. Total Object Count: 52584
    15. GC Allocations per Frame: 2040 / 197.0 KB
    Here is another profile but this one is Standalone windows, again, first frame that shows up:
    Code (CSharp):
    1. Used Total: 423.0 MB   Unity: 181.7 MB   Mono: 76.0 MB   GfxDriver: 154.9 MB   Audio: 3.9 MB   Video: 0 B   Profiler: 6.4 MB
    2. Reserved Total: 507.1 MB   Unity: 258.0 MB   Mono: 82.3 MB   GfxDriver: 154.9 MB   Audio: 3.9 MB   Video: 0 B   Profiler: 8.0 MB
    3. Total System Memory Usage: 0.76 GB
    4.  
    5.  
    6. Textures: 246 / 142.6 MB
    7. Meshes: 65 / 20.5 MB
    8. Materials: 153 / 0.6 MB
    9. AnimationClips: 56 / 0.8 MB
    10. AudioClips: 8 / 2.7 MB
    11. Assets: 52819
    12. GameObjects in Scene: 146
    13. Total Objects in Scene: 648
    14. Total Object Count: 53467
    15. GC Allocations per Frame: 11 / 0 B
    The *Reserved Total* is concerning me the most. It seems quite high especially on the `GfxDriver` no?

    So what is it showing when I crash? Well, the profiler is showing much much worse:

    Code (CSharp):
    1. Used Total: 1.14 GB   Unity: 0.60 GB   Mono: 182.4 MB   GfxDriver: 347.1 MB   Audio: 4.0 MB   Video: 0 B   Profiler: 18.8 MB
    2. Reserved Total: 2.00 GB   Unity: 1.44 GB   Mono: 204.8 MB   GfxDriver: 347.1 MB   Audio: 4.0 MB   Video: 0 B   Profiler: 21.0 MB
    3. Total System Memory Usage: 3.46 GB
    4.  
    5.  
    6. Textures: 848 / 367.8 MB
    7. Meshes: 71 / 25.3 MB
    8. Materials: 301 / 503.0 KB
    9. AnimationClips: 50 / 4.1 MB
    10. AudioClips: 8 / 2.8 MB
    11. Assets: 58073
    12. GameObjects in Scene: 125
    13. Total Objects in Scene: 867
    14. Total Object Count: 58940
    15. GC Allocations per Frame: 491327 / 33.5 MB
    Am I crazy? Why are the Unity numbers so pumped at launch and super high during my gameplay? I presume the "unity" number is encompassing the textures and other numbers. I've tried to take memory snapshots using the Memory Profiler and the tree really shows me that during gameplay right before crashing:

    Code (CSharp):
    1. Used Total: 3.43 GB   Unity: 1.50 GB   Mono: 321.3 MB   GfxDriver: 1.03 GB   Audio: 4.0 MB   Video: 0 B   Profiler: 0.58 GB  
    2. Reserved Total: 3.83 GB   Unity: 1.78 GB   Mono: 424.4 MB   GfxDriver: 1.03 GB   Audio: 4.0 MB   Video: 0 B   Profiler: 0.60 GB  
    3. Total System Memory Usage: 5.77 GB  
    4.  
    5. Memory Profiler Tree view:
    6. Textures: 1GB
    7. Animation: 260MB
    8. Mesh: 190MB
    9. RenderTexture: 100MB
    10. Font: 60MB
    11. Shader: 60MB
    12. MonoScript: 66MB
    13. Particle System: 38MB
    Everything else is magnitudes smaller. The above is 1.7 to 1.8GB at most.

    Let's say my game is targeting 4GB RAM devices like a Pixel 2.
    1. I've observed the OS can sometimes take up to 1GB of memory! Let's assume that's worst case. A fresh reboot of a Pixel 2 knocks it down to like 200-300MB and this sometimes has me crashing less.
    2. Debug Mono & Unity are taking up to 500MB on boot, but as I progress into game that reserved number runs up to about 1GB+. At crash, I saw Reserved Total: 2.00GB!!
    3. That leaves my game only 1-2GB~ RAM for textures and other stuff which apparently is insufficient
    What should I be doing? Obviously, I need to knock down texture memory in general to support a healthy build. But, even if I eliminate textures entirely I'm going to keep hitting crashes if I can't figure out how to wrangle the Unity player memory in.
     
    Last edited: Jul 25, 2020
    CPlusSharp22, Jul 25, 2020
    #1

  2. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    I noticed that even if I'm on a device with more than 4GB of ram, if I hit 4GB of used memory, it will crash. 32bit problem?
     
    CPlusSharp22, Jul 27, 2020
    #2

  3. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    Bump
     
    CPlusSharp22, Jul 29, 2020
    #3

  4. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    Bump
     
    CPlusSharp22, Jul 31, 2020
    #4

  5. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,453
    Hi
    First some questions:
    • Are you using any Native Plugins?
    • Are you using reflection code during runtime?
    • Are you using any Unity or Asset Store packages that deal with post effects, textures, video, camera, etc?
    • Have you checked the Memory Profiler Package's Memory Map (also in a diff from start to end or between level loads)?
      Pay attention to the blue sections there (Managed Memory). You could be hitting fragmentation issues. This post of mine should help getting some clarity on that view while I'm fleshing out the documentation write-up. Also generally that Profiler Previews subforum might contain some more helpful information for you.
    • QUOTE="OmegaNemesis28, post: 6129632, member: 160569"]GC Allocations per Frame: 491327 / 33.5 MB[/QUOTE]
      Is this happening because of a LevelLoad or could this be the cause of your crash? because 33.5MB allocs in any normal frame sounds pretty mad to me.
    • If you are observing the Profiler Window's Memory Profiler Module while playing: does the memory usage steadily grow or are there particular moments in which it bumps up? What are those and have you taken Memory Snapshots with the Memory Profiler Package to diff and compare before and after?
    • Is this a 32 bit or 64 bit build
     
    Last edited: Aug 18, 2020
    MartinTilo, Aug 18, 2020
    #5

  6. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,453
    Also, what does your Player log say after the crash? could be those last allocs are a huge bunch of Debug Logs?
     
    MartinTilo, Aug 18, 2020
    #6

  7. CPlusSharp22

    CPlusSharp22

    Joined:
    Dec 1, 2012
    Posts:
    111
    • Native plugins? I use a wide array of stuff from Unity.Mathematics to Entities to the preview transport layer if that's what you mean. I can get a list of those packages.

    • Reflection code: Not during gameplay no.

    • Packages with post effects and textures or camera? I think the only thing used during gameplay for UI is DOTween. Project is URP though worth mentioning.

    • Yes, the memory map is what I was referring to when talking about the memory profiler (sorry). At most it was just displaying what the tree was - 1GB of textures, and the rest nothing bigger than 260MB. The profiler doesn't give insight to Unity's reserved memory or Mono as far as I can see.

    • This is long after LevelLoad. Specifically, I'm spawning characters through assetbundles when the crash was happening which is probably why the allocations are so large that frame. But it's just that frame. I intend to make async loading soon so that 33MB is spread across multiple frames.

    • It bumps up at places you'd expect it to, but app launch does it the most. Memory Profiler does not reveal what it is, I can't take a profile before app launch. Post launch, it only shows a few textures but nothing massive. So I have no insight to what Unity is reserving or doing.

    • Android is on Mono .Net 4.x ARM v7 targeting Lolipop 5.1. Not sure how to specify 64 bit.
    Player log didn't show anything, it would just end. It's just a tombstone callstack. Sometimes it would change to something not exactly the same but similar mentioning GfxDeviceWorker. Here's one example:

    Code (CSharp):
    1. 07-23 02:29:29.375  5876  5876 E CRASH   : other thread is trapped; signum = 1107-
    2. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: java.lang.Error: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
    3. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: Version '2019.4.3f1 (f880dceab6fe)', Build type 'Development', Scripting Backend 'mono', CPU 'armeabi-v7a'
    4. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: Build fingerprint: 'samsung/star2qltecs/star2qltecs:10/QP1A.190711.020/G965WVLS7DTE1:user/release-keys'
    5. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: Revision: '14'
    6. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: ABI: 'arm'
    7. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: Timestamp: 2020-07-23 02:27:49-0400
    8. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: pid: 5876, tid: 6481, name: UnityGfxDeviceW  >>> com.game.game <<<
    9. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: uid: 10473
    10. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
    11. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: Cause: null pointer dereference
    12. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     r0  00000000  r1  00000000  r2  0004c2b8  r3  00000000
    13. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     r4  ee40dd20  r5  000014c9  r6  00000002  r7  00008f37
    14. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     r8  be4d38b0  r9  00000000  r10 c01e1a38  r11 0004c338
    15. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     ip  00000000  sp  b80e9ed0  lr  bf4d40a8  pc  ef7abdf8
    16. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:
    17. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI: backtrace:
    18. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #00 pc 0005ddf8  /apex/com.android.runtime/lib/bionic/libc.so (memset_a7+48) (BuildId: f7fcf1e3a2e994c617c9b9b888e91865)
    19. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #01 pc 009760a4  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (ApiGLES::ClearBufferSubData(unsigned int, gl::BufferTarget, long, long)+148) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
    20. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #02 pc 009518dc  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (DataBufferGLES::DataBufferGLES(BufferManagerGLES&, unsigned int, DataBufferGLES::BufferUsage, bool)+288) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
    21. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #03 pc 009523f8  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (BufferManagerGLES::AcquireBuffer(unsigned int, DataBufferGLES::BufferUsage, bool)+276) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
    22. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #04 pc 00951748  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (BufferGLES::EnsureBuffer(DataBufferGLES*&, unsigned int, DataBufferGLES::BufferUsage)+116) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
    23. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #05 pc 00951294  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (BufferGLES::Initialize(void const*)+32) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
    24. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #06 pc 0095fa90  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (GfxDeviceGLES::InitializeBufferInternal(GfxBuffer*, void const*, GfxUpdateBufferFlags)+28) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
    25. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #07 pc 003715ff  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (GfxDeviceWorker::RunCommand(ThreadedStreamBuffer&)+25830) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
    26. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #08 pc 00371ccb  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (GfxDeviceWorker::RunExt(ThreadedStreamBuffer&)+26) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
    27. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #09 pc 00371c91  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (GfxDeviceWorker::Run()+96) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
    28. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #10 pc 0036aed3  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (GfxDeviceWorker::RunGfxDeviceWorker(void*)+2) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
    29. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #11 pc 007f2e57  /data/app/com.game.game-_byFZYbTU3o6znQmpTvgJw==/lib/arm/libunity.so (Thread::RunThreadWrapper(void*)+342) (BuildId: 6bb88d5fd94d81c332ebfbe5a37085a2c7a40ad0)
    30. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #12 pc 000abcbb  /apex/com.android.runtime/lib/bionic/libc.so (__pthread_start(void*)+20) (BuildId: f7fcf1e3a2e994c617c9b9b888e91865)
    31. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:       #13 pc 00062dc3  /apex/com.android.runtime/lib/bionic/libc.so (__start_thread+30) (BuildId: f7fcf1e3a2e994c617c9b9b888e91865)
    32. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:
    33. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     at libc.memset_a7(memset_a7:48)
    34. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     at libunity.ApiGLES::ClearBufferSubData(unsigned int, gl::BufferTarget, long, long)(ClearBufferSubData:148)
    35. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     at libunity.DataBufferGLES::DataBufferGLES(BufferManagerGLES&, unsigned int, DataBufferGLES::BufferUsage, bool)(DataBufferGLES:288)
    36. 07-23 02:29:29.391  5876  5876 E MessageQueue-JNI:     at libunity.BufferManagerGLES::AcquireBuffer(unsigned int, DataBufferGLES::BufferUsage, bool)(AcquireBuffe
    37. 07-23 02:29:29.392  5876  5876 D AndroidRuntime: Shutting down VM
    38. 07-23 02:29:29.398  5876  5876 E AndroidRuntime: FATAL EXCEPTION: main
    39. 07-23 02:29:29.398  5876  5876 E AndroidRuntime: Process: com.game.game, PID: 5876
    40. 07-23 02:29:29.398  5876  5876 E AndroidRuntime: java.lang.Error: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
    41. 07-23 02:29:29.398  5876  5876 E AndroidRuntime: Version '2019.4.3f1 (f880dceab6fe)', Build type 'Development', Scripting Backend 'mono', CPU 'armeabi-v7a'
    42. 07-23 02:29:29.398  5876  5876 E AndroidRuntime: Build fingerprint: 'samsung/star2qltecs/star2qltecs:10/QP1A.190711.020/G965WVLS7DTE1:user/release-keys'
    43. 07-23 02:29:29.398  5876  5876 E AndroidRuntime: Revision: '14'
    44. 07-23 02:29:29.398  5876  5876 E AndroidRuntime: ABI: 'arm'
    45. 07-23 02:29:29.398  5876  5876 E AndroidRuntime: Timestamp: 2020-07-23 02:27:49-0400
    46. 07-23 02:29:29.398  5876  5876 E AndroidRuntime: pid: 5876, tid: 6481, name: UnityGfxDeviceW  >>> com.game.game <<<
    47. 07-23 02:29:29.398  5876  5876 E AndroidRuntime: uid: 10473
    48. 07-23 02:29:29.398  5876  5876 E AndroidRuntime: signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
    49. 07-23 02:29:29.398  5876  5876 E AndroidRuntime: Cause: null pointer dereference
    50. 07-23 02:29:29.398  5876  5876 E AndroidRuntime:     r0  00000000  r1  00000000  r2  0004c2b8  r3  00000000
    51. 07-23 02:29:29.398  5876  5876 E AndroidRuntime:     r4  ee40dd20  r5  000014c9  r6  00000002  r7  00008f37
    52. 07-23 02:29:29.398  5876  5876 E AndroidRuntime:     r8  be4d38b0  r9  00000000  r10 c01e1a38  r11 0004c338
    53. 07-23 02:29:29.398  5876  5876 E AndroidRuntime:     ip  00000000  sp  b80e9ed0  lr  bf4d40a8  pc  ef7abdf8
     
    CPlusSharp22, Aug 18, 2020
    #7

  8. MartinTilo

    MartinTilo

    Unity Technologies

    Joined:
    Aug 16, 2017
    Posts:
    2,453
    Once again, sorry for the delay in my answer. I might've tried to juggle to many ongoing threads again
    No, I meant native libraries in your 
    Plugins
    folder or the 
    Plugins
    folder of any package. I assume most of these packages would be purely C# code and therefore all their memory usage is fully captured in the snapshot.

    Even using Reflection during startup can bloat memory because type meta data needs to be loaded in. That shouldn't lead to memory growth over time though. Still if you use it in between of scenes or something, it could still grow that Mono memory footprint. Beyond that, it might be down to fragmentation in Mono.
    Not as a summary but all the dark blue regions in the Memory Map are Mono Reserved memory regions, aka reserved memory.

    Also, yeah that crash looks like an OutOfMemory crash...

    It could be something to do with your Asset bundle setup?

    Also, FYI, that 
    Total System Memory Usage
    value is currently bugged on Android and showing overall memory used on the device, not just the current app. That fix is in the process of landing/shipping and we'll adjust the Manual for the Memory Profiler Module to explain that on Android it will show Residual Shared memory from that fix onward.

    Textures yes but not Audio, Video and Profiler. The Manual update should clarify this too.
     
    MartinTilo, Sep 8, 2020
    #8