Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Bug Android Chromium - Unable To Grow Allocated Memory Above 256MB [Confirmed]

Discussion in 'WebGL' started by DerrickBarra, Nov 30, 2020.

  1. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    Hey everyone,

    Our team put together a WebGL memory stress test. You can enter the number of textures to load and press the button to load them. It displays the available memory information we have access to for the platform.
    https://brandxr-discovery.s3.amazonaws.com/WebGL/MemoryStressTestWebGL/index.html

    Here are my findings...

    Windows 10 PC
    • Chrome - 1.96GB
    • Edge - 1.95GB
    • Firefox - 1.96GB
    Android [Galaxy S20+]
    • Chrome Beta - 219MB
    • Chrome - 219MB
    • Ecosia - 219MB
    • Firefox - 1.96GB
    • Samsung Internet - 219MB
    • Opera - 219MB
    • Edge - 1.93GB
    Android [ Galaxy S9]
    • Chrome - 219mb
    iOS [iPhone SE 2020]
    • Safari - 426mb
    • Chrome - 426mb
    • Edge - 426mb
    • XRViewer -- 426mb
    iOS [iPhone 11 Pro Max]
    • Safari - 442mb
    iOS [iPhone 8]
    • Safari - 210mb
    iOS [iPhone 6]
    • Safari - Fails to open (out of memory)

    You can see in my findings that on mobile Android Chrome-based web browsers are unable to increase the size of the memory used by WebGL.

    For the iPhone SE 2020, that device only has 2GB of RAM to start with, so it's no surprise we can't get to the 2GB limit. Weird that we're only able to get to around 426mb, I wonder how that was determined (maybe we can't go larger than 480mb of allocated memory in the heap?).

    The iPhone 6 only has 1GB of memory to start with. I'm going to try making a build with 128mb initial memory and see if WebGL runs on that device. [Update: nope!]

    This build was made with brotli compression, high code stripping, and WebGL 1.0 via Unity 2020.1.4f1.

    I've read that by default WebGL will increase the size of the heap in 32mb blocks, with 256mb allocated at the start.

    Anyone got any ideas on things I should try? Can anyone else share the results of the test from their devices?
     
    Last edited: Dec 4, 2020
  2. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    So I made a new build with 128mb memory to start, and used Chrome dev tools to view the console to see what kind of error feedback I could get on Android Chrome. But the actual error message isn't any different then what I would expect [Could not allocate memory].

    AndroidMemoryCrash.PNG

    Weirdly, the crash still happens when growing the memory above 256mb, it gets past 128mb perfectly fine.

    For my next test, I'm going to try to make a build that starts at 512mb allocated, and see what happens on Android Chrome in that scenario.
     
    Last edited: Dec 1, 2020
  3. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    Well, I made a build that defaults to 512mb, but it still crashes when getting to the same 256mb limit.

    This tells me that the PlayerSettings.WebGL.memorySize doesn't do what I thought it does. And yep, looking it up this value has been deprecated and doesn't do anything on Web Assembly builds. Doh!

    I'm going to continue researching what else can be done about this, having a hard limit of 256mb on mobile Chrome is pretty bad!
     
  4. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    So I made some new builds using PlayerSettings.WebGL.emscriptenArgs to set the initial memory.

    128MB Initial
    512MB Initial

    So on Android Chrome, the 128MB build works fine until the memory reaches 256MB, and then gives the [Could Not Allocate] error in the console.

    On the 512MB build, I'm able to add images until I get near 512MB, and then the app crashes with the [Could Not Allocate] error in the console.

    So, this does show an actual bug! At least I know I'm not crazy. Next step is to figure out the best developers to report this to from the Unity team and file a bug report.
     
  5. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    Submitted a bug report to the Unity tracker.
    #1296850_jo7dcmobll7c762m
     
    Last edited: Dec 3, 2020
  6. De-Panther

    De-Panther

    Joined:
    Dec 27, 2009
    Posts:
    541
    DerrickBarra likes this.
  7. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    So for anyone wondering how to work around this issue, what our team is planning on doing is making multiple WebGL builds with different initial heap values { 64, 128, 256, 512, 1024, 2048 }, and then loading only that version and not allowing our application to go above the initial value (by using assets with different levels of detail and memory required).

    Then when a memory crash occurs, we will log how much memory we initialized with, and then the next time that device loads our app, we'll choose the next lowest initial memory version. So at least there's a limited amount of crashes possible before we automatically start knowing how much memory can be requested.

    We'll probably have to use a device detection API to figure out what device is actually attempting to load our app since there isn't an easy way to know this out of the box.

    Yes, this is a major bug and annoying. But at least with a lot of elbow grease, we can work around it!
     
    Last edited: Dec 7, 2020
  8. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    I've added the test project to a public Github for the Unity QA team and received word from them that they've at least seen the issue, no word on the timing for when this will be looked into yet, but I'm hoping sooner rather than later!
     
    De-Panther likes this.
  9. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    This bug is now officially listed in the Unity Issue Tracker. Make sure to vote for it to be worked on!
     
  10. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    827
    This bug landed on my work desk, and it is unfortunately an issue with Chrome browser.

    The root problem is that the Chrome browser is still 32-bit on Android, and it causes memory address space fragmentation issues that limit the creation of "large" WebAssembly memorys. 256MB in this context does not sound particularly large, but that is the extent that address space is fragmented on Chrome. See https://www.reddit.com/r/chrome/comments/iwyb4g/why_is_chrome_85stable_for_android_still_32_bit/

    Several years ago I worked with King on WebAssembly Candy Crush for mobile Android browsers, where the issue was the same - iirc 120MB memory sizes were still 100% reliable to allocate, but after that, there was a smooth degradation, and iirc by 384 MB heap size, allocation reliability had dropped by several dozens of percents.

    This is an issue that we are tracking with Chrome to resolve. Based on https://www.ghacks.net/2020/07/04/how-to-check-whether-your-chrome-on-android-is-32-bit-or-64-bit/ it looks like they are doing small % trials already, so there is hope.
     
    ROBYER1, De-Panther and DerrickBarra like this.
  11. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    @jukka_j That's super-valuable to know. I'm going to have our dev team check if the browser is 32-bit, and then if it is we'll load a WebGL build that is initialized to 256mb, and only use assets that fit within that limitation with a buffer to prevent memory from growing. Hopefully, Google gets around to transitioning chromium-based browsers to 64-bit this year!
     
  12. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    827
    I have created a test application that can be used to gauge the maximum WebAssembly application memory size. You can find it here: http://clb.confined.space/dump/mem_growth.html

    When testing, tap the button on the page until memory heap size no longer increases.

    I would be curious to know what kind of results people get on different mobile devices, especially whether the results differ if one has a few tabs open, or when there is only one tab open; and whether mobile browsers crash with Chrome's "Aww, snap." dialog or similar.
     
    gtk2k likes this.
  13. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    827
    @DerrickBarra one thing to test I wonder is whether enabling or disabling Data Caching affects the max size that can be acquired from an Unity project? Also make sure that Decompression Fallback is disabled.
     
  14. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    827
  15. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    827
    @DerrickBarra Profiling in Firefox, initial memory usage (without any textures created using the UI) is:

    UI says "In-Use Memory: 16.85 MB"

    Code (JavaScript):
    1. 264.50 MB (100.0%) -- explicit
    2. ├──244.76 MB (92.54%) -- window-objects/top(https://brandxr-discovery.s3.amazonaws.com/WebGL/MemoryStressTestWebGL/index.html, id=34)
    3. │  ├──243.41 MB (92.02%) -- active/window(https://brandxr-discovery.s3.amazonaws.com/WebGL/MemoryStressTestWebGL/index.html)
    4. │  │  ├──243.12 MB (91.91%) -- js-realm(https://brandxr-discovery.s3.amazonaws.com/WebGL/MemoryStressTestWebGL/index.html)
    5. │  │  │  ├──242.66 MB (91.74%) -- classes
    6. │  │  │  │  ├──169.13 MB (63.94%) -- class(WebAssembly.Instance)/objects
    7. │  │  │  │  │  ├───99.38 MB (37.57%) ── non-heap/code/wasm
    8. │  │  │  │  │  ├───69.76 MB (26.37%) ── malloc-heap/misc
    9. │  │  │  │  │  └────0.00 MB (00.00%) ── gc-heap
    10. │  │  │  │  ├───72.49 MB (27.41%) -- class(ArrayBuffer)/objects
    11. │  │  │  │  │   ├──64.00 MB (24.20%) ── non-heap/elements/wasm
    12. │  │  │  │  │   ├───8.49 MB (03.21%) ── malloc-heap/elements/normal
    13. │  │  │  │  │   └───0.00 MB (00.00%) ── gc-heap
    14. │  │  │  │  └────1.03 MB (00.39%) ++ (6 tiny)
    15. │  │  │  └────0.46 MB (00.17%) ++ (7 tiny)
    16. │  │  └────0.29 MB (00.11%) ++ (3 tiny)
    17. │  └────1.35 MB (00.51%) ++ js-zone(0x2ae7588a000)
    18. ├────9.14 MB (03.45%) ++ js-non-window
    19. ├────5.00 MB (01.89%) ++ heap-overhead
    20. ├────2.91 MB (01.10%) ++ (24 tiny)
    21. └────2.70 MB (01.02%) ++ threads
    Then, after creating 50 textures, UI says

    "In-Use Memory: 227.61 MB" with

    Code (JavaScript):
    1. 712.54 MB (100.0%) -- explicit
    2. ├──692.77 MB (97.23%) -- window-objects/top(https://brandxr-discovery.s3.amazonaws.com/WebGL/MemoryStressTestWebGL/index.html, id=34)
    3. │  ├──691.41 MB (97.04%) -- active/window(https://brandxr-discovery.s3.amazonaws.com/WebGL/MemoryStressTestWebGL/index.html)
    4. │  │  ├──691.14 MB (97.00%) -- js-realm(https://brandxr-discovery.s3.amazonaws.com/WebGL/MemoryStressTestWebGL/index.html)
    5. │  │  │  ├──690.67 MB (96.93%) -- classes
    6. │  │  │  │  ├──520.49 MB (73.05%) -- class(ArrayBuffer)/objects
    7. │  │  │  │  │  ├──512.00 MB (71.86%) ── non-heap/elements/wasm
    8. │  │  │  │  │  ├────8.49 MB (01.19%) ── malloc-heap/elements/normal
    9. │  │  │  │  │  └────0.00 MB (00.00%) ── gc-heap
    10. │  │  │  │  ├──169.13 MB (23.74%) -- class(WebAssembly.Instance)/objects
    11. │  │  │  │  │  ├───99.38 MB (13.95%) ── non-heap/code/wasm
    12. │  │  │  │  │  ├───69.76 MB (09.79%) ── malloc-heap/misc
    13. │  │  │  │  │  └────0.00 MB (00.00%) ── gc-heap
    14. │  │  │  │  └────1.04 MB (00.15%) ++ (6 tiny)
    15. │  │  │  └────0.48 MB (00.07%) ++ (7 tiny)
    16. │  │  └────0.27 MB (00.04%) ++ (3 tiny)
    17. │  └────1.35 MB (00.19%) ++ js-zone(0x2ae7588a000)
    18. ├───10.62 MB (01.49%) ++ (26 tiny)
    19. └────9.15 MB (01.28%) ++ js-non-window
    and after creating another 50 textures for a 100 total, UI says

    "In-Use Memory: 438.31 MB" with

    Code (JavaScript):
    1. 712.12 MB (100.0%) -- explicit
    2. ├──692.69 MB (97.27%) -- window-objects/top(https://brandxr-discovery.s3.amazonaws.com/WebGL/MemoryStressTestWebGL/index.html, id=34)
    3. │  ├──691.36 MB (97.08%) -- active/window(https://brandxr-discovery.s3.amazonaws.com/WebGL/MemoryStressTestWebGL/index.html)
    4. │  │  ├──691.08 MB (97.04%) -- js-realm(https://brandxr-discovery.s3.amazonaws.com/WebGL/MemoryStressTestWebGL/index.html)
    5. │  │  │  ├──690.67 MB (96.99%) -- classes
    6. │  │  │  │  ├──520.49 MB (73.09%) -- class(ArrayBuffer)/objects
    7. │  │  │  │  │  ├──512.00 MB (71.90%) ── non-heap/elements/wasm
    8. │  │  │  │  │  ├────8.49 MB (01.19%) ── malloc-heap/elements/normal
    9. │  │  │  │  │  └────0.00 MB (00.00%) ── gc-heap
    10. │  │  │  │  ├──169.13 MB (23.75%) -- class(WebAssembly.Instance)/objects
    11. │  │  │  │  │  ├───99.38 MB (13.95%) ── non-heap/code/wasm
    12. │  │  │  │  │  ├───69.76 MB (09.80%) ── malloc-heap/misc
    13. │  │  │  │  │  └────0.00 MB (00.00%) ── gc-heap
    14. │  │  │  │  └────1.05 MB (00.15%) ++ (6 tiny)
    15. │  │  │  └────0.40 MB (00.06%) ++ (5 tiny)
    16. │  │  └────0.28 MB (00.04%) ++ (3 tiny)
    17. │  └────1.33 MB (00.19%) ++ js-zone(0x2ae7588a000)
    18. ├───10.78 MB (01.51%) ++ (26 tiny)
    19. └────8.65 MB (01.21%) ++ js-non-window

    so there is a very precise linear growth of 210.76MB for both steps (16.85 MB -> 227.61 MB -> 438.31 MB), which gives a memory usage of 4.2MB per texture inside the WebAssembly heap.

    Outside the WebAssembly heap, things look nominal. There is 169.13 MB of memory used for compiled WebAssembly code, and 8.49 MB used for the virtual filesystem. The heap size overreserves using geometric growth, so 512MB at this point is expected, though a tad aggressive. (I have rewritten the geometric growth logic in Emscripten to be more modest, but that is not yet in effect here - that might save 50MB-100MB).

    Something you can also try is to use the new Unity "Optimize for Size" build option, if that would shrink some of that 169MB - not sure though if there will be a noticeable correlation to runtime memory usage.
     
  16. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    @jukka_j Just saw your Safari & Chrome bug reports, hopefully, we this gets the attention of the developers.

    I tested your web assembly app on my Galaxy S20+, and it fails to grow the heap size beyond 501mb (unable to allocate 512mb). But no crash, so that's nice! Would it be possible to let Unity devs be aware of this memory growth failure via a callback?
     
  17. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    827
    The fact that there is no crash is an effect that the test page is a naive/simplistic allocation loop. The test page does not actually use any of the memory it allocates.

    In practice if a Unity application does need the memory, there are few options to gracefully falling back to doing other things. Although if the failing allocation is one that comes directly from user C# code, maybe the new operator could throw a System.OutOfMemoryException instead. (That would require enabling exception handling in project build settings, which does increase shipped code size though) Added a task to our board to investigate if that's possible.
     
    DerrickBarra likes this.
  18. ninipols

    ninipols

    Joined:
    May 28, 2016
    Posts:
    2
    Hi @jukka_j , I'm working with @DerrickBarra on this problem and was hoping you could share any insights on the following possible fixes:
    1. We're trying to detect whether the user's browser is 32 or 64 bit but there doesn't seem to be a reliable way to do this. 32/64 bit Chrome can be detected via the user agent string, but otherwise the closest I've got is identifying whether the OS is 32/64 bit from the user agent string, using these strings: https://stackoverflow.com/a/13709431 but this doesn't help detect 32 bit browsers on 64 bit OS's.
      I was wondering if you know of a more reliable/foolproof way of detecting the browser type.

    2. Specifically, the third solution proposed in the last comment in this thread:
      https://bugs.chromium.org/p/chromium/issues/detail?id=1175564

      Which suggests attempting to allocate WebAssembly.Memory in a loop, starting with the maximum memory and decreasing until it succeeds. If something like this could work, how can it be implemented? For instance, does it have to be part of the framework.js or could it be part of the template files somehow? Is this potentially a feature that we might see for WebGL in the future?
    Thanks!
     
  19. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    827
    1. That is unfortunately my understanding as well. There is no good way to detect 32bit vs 64bit. At present time, I would rather try to detect mobile vs desktop, and treat mobile=32-bit, and desktop=64bit. That matches quite well what browsers are currently doing. (With the exception of Safari on iOS, but it imposes memory constraints that make it effectively look like the other 32bit browsers are behaving)

    2. Posted a followup in that thread. That will unfortunately not help much, it will crash Safari, and on 32-bit Chrome, it will likely lead to the browser itself running out of memory later. We are discussing these issues further in WebAssembly CG at https://github.com/WebAssembly/design/issues/1397
     
    ninipols and De-Panther like this.
  20. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    @jukka_j Just read your write up on the WebAssembly github, that was insightful to read, thanks!

    I'm going to instruct our team to treat all mobile devices as having an upper limit of 256mb, and follow your progress on this issue via this thread. When I see other developers talking about this type of issue, I'll point them here so we can collectively follow the progress.
     
    De-Panther likes this.
  21. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    827
    It looks like Chrome authors found a quick fix to incorporate a better growing success rate for 32-bit Android browser. That should alleviate memory issues in the short term, hopefully Wasm CG improvements will help the remaining issues on the long term.
     
    De-Panther likes this.
  22. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    @jukka_j
    Any idea when this fix will make it into the latest chromium/chrome web browsers on Android? I didn't see anywhere that mentioned what Android Chromium version will receive the patch or when.

    From what I read, we should be able to rely on 32-bit Chromium Android browsers in the future being able to reserve up to 1gb of RAM.

    Once this change rolls out, we'll change our memory logic to the following...

    ---------
    Before Unity WebGL loads, store a variable in javascript called maxAvailableMemory.

    - If [memory api] is not available, set our max available memory to 256mb.
    - Else if [memory api] is available, and we detect [32-bit browser], ensure that the variable does not exceed 1024mb.
    - Else if [memory api] is available, and we detect [64-bit browser] set to 2048mb (Unity WebGL's current limit).

    Our Unity builds will use the default initial heap size and allow the Grow() command to work its magic. Then in our C#, we will make sure the assets we load for our project stay within the maxAvailableMemory we found earlier.
    ----------

    Firefox and Safari desktop will be limited to 256mb max memory, because they lack the performance API to check memory availability. But I can't think of another way to prevent OOM crashes.

    I saw your message to the Firefox team back in 2018 about adding the window.performance.memory to Firefox, shame it was never implemented. I saw they have the mozMemory, but according to this bugzilla thread, it's hidden behind a browser flag. I also didn't spot a Safari equivalent or polyfill for chrome's runtime memory api.

    Alternatively, these types of issues could be solved by being able to respond to memory allocation errors at runtime.

    Overall, this patch means we won't have to worry about building multiple Unity WebGL builds with different initial heap sizes. So that will be nice.
     
    Last edited: Feb 22, 2021
  23. jukka_j

    jukka_j

    Unity Technologies

    Joined:
    May 4, 2018
    Posts:
    827
    Chrome releases a new version every ~6-7 weeks. Once the patch is confirmed to land (looks like currently in review), you can follow up in the bug tracker to ask for the specific version that it should appear in.

    The memory logic looks sane, although I would expect on mobile Firefox the memory allocation issues to be rare. At least in the test that I was doing, Firefox was best at growing Wasm heap, all the way up to 2GB. (although not sure how "old browser address space" scenarios will fare)
     
  24. chibinhdang

    chibinhdang

    Joined:
    Oct 23, 2017
    Posts:
    2
    First of all, I just want to say how happy I am to have found this thread. Thank you for your contributions.

    > Our Unity builds will use the default initial heap size and allow the Grow() command to work its magic. Then in our C#, we will make sure the assets we load for our project stay within the maxAvailableMemory we found earlier.

    The C# workaround is the part I'm trying to develop now. Would you mind sharing what your strategy is? What kind of assets are you loading, and do you know their heap size ahead of time?

    My workplace is developing a WebXR application with runtime model import, and it also hits this 256mb ceiling on Oculus Quest's browser, which is 32-bit Chromium Android. We are loading GLBs with GLTFast. The used & free managed heap size (GetMonoHeapSizeLong) grows wildly during the package's loading function ,but the used managed heap size (GetMonoUsedSizeLong) grows predictably during file load. This leads to the managed heap doubling at a time that I'm pretty sure we can't predict without modifying GLTFast's code. I was wondering if that's a dead-end for us for a solution similar to yours, or if you are measuring asset sizes on-the-fly.

    I am willing to make another thread so we can stay on-topic on this one. (Again, I can't stress how much it's a relief to see this bug getting tracked here.)
     
    De-Panther likes this.
  25. DerrickBarra

    DerrickBarra

    Joined:
    Nov 19, 2013
    Posts:
    210
    @chibinhdang : For our project, we have all of our assets created with multiple LOD's, using Draco + KTX2. And all of our UI textures are in .basis

    I wrote up a guide a little while back for my team to help our overall understanding of heap size requirements, good enough for estimates anyways. I'll make a new WebGL forum thread for the community.

    Edit: Made the thread.
     
    Last edited: Feb 26, 2021
    De-Panther likes this.