Search Unity

Bug il2cpp phase uses up 64G memory！

Discussion in 'Linux' started by mariozhou, Oct 9, 2021.

  1. mariozhou

    mariozhou

    Joined:
    Oct 18, 2019
    Posts:
    23
    Using docker build(ubuntu-2020.3.19f1-ios-0.15.0), during the il2cpp compilation phase, occasional out of memory exceptions occur.
    The machine is 32 core 64G memory exclusive.
    Some of our c# files may have 100000+ lines (generated protocol files) and it looks like the il2cpp stage uses multi-core parallelism for acceleration, but does not take into account the overall memory footprint.
    I upgraded the memory from 32G to 64G when this problem appeared before.

    what should I do now? Upgrade the memory but lower the CPU core count?

    Exception
    企业微信截图_9472c457-a9bc-46fa-8dc9-0a3725107b48.png

    Memory Usage 企业微信截图_5ea8ecc7-9895-4d55-a4a2-9093d27fa4a7.png
     
    Last edited: Oct 9, 2021
    mariozhou, Oct 9, 2021
    #1

  2. mariozhou

    mariozhou

    Joined:
    Oct 18, 2019
    Posts:
    23
    mariozhou, Oct 9, 2021
    #2

  3. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    I suspect there are two things you can do to help here. The best option to improve both build times and memory usage is to make sure those large C# files do not have large methods. It is really large methods, not large files, that causes problems for the generated C++ code at compile time. So if that C# code can be split up into smaller methods, that will help tremendously.

    You can also limit the parallelize that IL2CPP will use during code conversion (which is where this exception happens) by passing the --jobs= argument to IL2CPP. You can use an editor script that call this API to set that:

    https://docs.unity3d.com/ScriptReference/PlayerSettings.SetAdditionalIl2CppArgs.html
     
    JoshPeterson, Oct 11, 2021
    #3

  4. mariozhou

    mariozhou

    Joined:
    Oct 18, 2019
    Posts:
    23
    Thank you very much for your help. @JoshPeterson
    I will try to troubleshoot which function bodies are larger and see if there is a way to optimize them.
    And it's glad to know that jobs can be limited.
     
    mariozhou, Oct 11, 2021
    #4

  5. mariozhou

    mariozhou

    Joined:
    Oct 18, 2019
    Posts:
    23
    finally solved this problem by adding the heap limit to il2cpp.runtimeconfig.json, for example 10G,
    or change "System.GC.Server" to false to use "workstation GC" which will do gc more frequently and take less memory.

    Details:
    We run unity in a docker container with 32core cpu / 64G momory limit，and the host machine is 48core cpu /128G momory。

    But .netcore consider 75% of physical momory (128 * 0.75 = 96G > 64G) as the default max heap limit , and tend to keep the memory for performance. so finally oom



    @JoshPeterson

    Code (CSharp):
    1. {
    2.   "runtimeOptions": {
    3.     "tfm": "netcoreapp3.1",
    4.     "includedFrameworks": [
    5.       {
    6.         "name": "Microsoft.NETCore.App",
    7.         "version": "3.1.7"
    8.       }
    9.     ],
    10.     "configProperties": {
    11.       "System.GC.Server": true,
    12.       "System.Runtime.TieredCompilation.QuickJit": false,
    13.       "System.GC.HeapHardLimit": 10737418240
    14.     }
    15.   }
    16. }
     
    Last edited: Jun 2, 2022
    mariozhou, Jun 2, 2022
    #5
    JoshPeterson likes this.

  6. JoshPeterson

    JoshPeterson

    Unity Technologies

    Joined:
    Jul 21, 2014
    Posts:
    6,938
    Interesting - we use the "server" GC mode by default, as that gives IL2CPP better run time performance. But in this case "workstation" GC makes sense.
     
    JoshPeterson, Jun 2, 2022
    #6

  7. unity_g2sBAPq7KXJ9zQ

    unity_g2sBAPq7KXJ9zQ

    Joined:
    May 4, 2022
    Posts:
    2
    @mariozhou @JoshPeterson could you please provide path to il2cpp.runtimeconfig.json? or which exactly argument should be passed into SetAdditionalIl2CppArgs to change this parameter?
     
    Last edited: Aug 26, 2022
    unity_g2sBAPq7KXJ9zQ, Aug 26, 2022
    #7

  8. mariozhou

    mariozhou

    Joined:
    Oct 18, 2019
    Posts:
    23
    OSX: /Applications/Unity/Hub/Editor/2020.3.25f1/Unity.app/Contents/il2cpp/build/deploy/netcoreapp3.1/
    Linux: /opt/unity/Editor/Data/il2cpp/build/deploy/netcoreapp3.1/

    set "System.GC.Server" to false

    no extra arguments to be passed
     
    mariozhou, Aug 26, 2022
    #8

  9. unity_g2sBAPq7KXJ9zQ

    unity_g2sBAPq7KXJ9zQ

    Joined:
    May 4, 2022
    Posts:
    2
    great. thanks!

    and what about arguments for SetAdditionalIl2CppArgs to change System.GC.Server and System.GC.HeapHardLimit?
     
    unity_g2sBAPq7KXJ9zQ, Aug 26, 2022
    #9

  10. mariozhou

    mariozhou

    Joined:
    Oct 18, 2019
    Posts:
    23
    I think there is no such thing
     
    mariozhou, Aug 26, 2022
    #10