Search Unity

Create Store Package error: STARTPROCESSTASK Out of Memory

Discussion in 'Windows' started by Roywise, Aug 12, 2018.

  1. Roywise

    Roywise

    Joined:
    Jun 1, 2017
    Posts:
    68
    A couple of weeks back we noticed that our builds, made on our workstations, started to report a error when creating Store Packages which included a 86x Master package in Visual Studio. Luckily our Build-PC wasn't affected but sadly it has now.



    Creating a 86x Debug package works fine but we won't be able to push that to the Windows Store. A 64x Master package works fine as well but we have to support x86 systems as well.

    Most "fixes" that I could find didn't work, see the list below.

    1. Adding <Assembly Name="Assembly-CSharp" Dynamic="Required All" DoNotOptimize="true"/> to the Default.rd.xml file. (https://issuetracker.unity3d.com/is...-out-of-memory-when-building-master-x86-build)

    2. Putting EnableLargeAddresses on Yes (https://forum.unity.com/threads/uwp...-700mb-in-heap-in-a-machine-with-16gb.522404/) This value isn't available in the Solution properties.

    3. Using NuGet to update some of the required packages.

    4. Reinstalling Visual Studio.

    I also found the following entry in the Issue Tracker that reads "Won't Fix". https://issuetracker.unity3d.com/is...-out-of-memory-when-building-master-x86-build

    Does anyone know where I should start to figure out what is causing this issue? Has anyone had this issue before?
     
    Last edited: Aug 12, 2018
  2. Roywise

    Roywise

    Joined:
    Jun 1, 2017
    Posts:
    68
  3. Roywise

    Roywise

    Joined:
    Jun 1, 2017
    Posts:
    68
    This is part of the build ouput where it goes wrong.


    Code (CSharp):
    1. 1>  Taak StartProcessTask (TaskId:453)
    2. 1>  Launching 'C:\Program Files (x86)\MSBuild\15.0\.Net\.NetNative\15.0.24211\x86\ilc\Tools\nutc_driver.exe @"D:\Builds\testing\ProjectName\obj\x86\Master\ilc\intermediate\MDIL\ProjectName.rsp"' (TaskId:453)
    3. 1>  Microsoft (R) MDIL Compiler - Version 1.4.24211.00 (TaskId:453)
    4. 1>  Copyright (c) Microsoft Corporation.  All rights reserved. (TaskId:453)
    5. 1>  Scanning input types (TaskId:453)
    6. 1>  Scanning 230381 methods. (TaskId:453)
    7. 1>  Scanning 33580 methods. (TaskId:453)
    8. 1>  Scanning 11478 methods. (TaskId:453)
    9. 1>  Scanning 4383 methods. (TaskId:453)
    10. 1>  Scanning 2359 methods. (TaskId:453)
    11. 1>  Scanning 1508 methods. (TaskId:453)
    12. 1>  Scanning 1322 methods. (TaskId:453)
    13. 1>  Scanning 1098 methods. (TaskId:453)
    14. 1>  Scanning 1222 methods. (TaskId:453)
    15. 1>  Scanning 1089 methods. (TaskId:453)
    16. 1>  Scanning 902 methods. (TaskId:453)
    17. 1>  Scanning 508 methods. (TaskId:453)
    18. 1>  Scanning 427 methods. (TaskId:453)
    19. 1>  Scanning 175 methods. (TaskId:453)
    20. 1>  Scanning 151 methods. (TaskId:453)
    21. 1>  Scanning 178 methods. (TaskId:453)
    22. 1>  Scanning 180 methods. (TaskId:453)
    23. 1>  Scanning 166 methods. (TaskId:453)
    24. 1>  Scanning 90 methods. (TaskId:453)
    25. 1>  Scanning 29 methods. (TaskId:453)
    26. 1>  Scanning 40 methods. (TaskId:453)
    27. 1>  Scanning 48 methods. (TaskId:453)
    28. 1>  Scanning 73 methods. (TaskId:453)
    29. 1>  Scanning 32 methods. (TaskId:453)
    30. 1>  Scanning 12 methods. (TaskId:453)
    31. 1>  Scanning 7 methods. (TaskId:453)
    32. 1>  Scanning 3 methods. (TaskId:453)
    33. 1>  Scanning 3 methods. (TaskId:453)
    34. 1>  Scanning 6 methods. (TaskId:453)
    35. 1>  Scanning 3 methods. (TaskId:453)
    36. 1>  Eager code generation (TaskId:453)
    37. 1>  Compiling 291453 methods. (TaskId:453)
    38. 1>STARTPROCESSTASK : error : Out of Memory
    39. 1>C:\Program Files (x86)\MSBuild\15.0\.Net\.NetNative\15.0.24211\x86\ilc\IlcInternals.targets(1192,5): error : ILT0005: 'C:\Program Files (x86)\MSBuild\15.0\.Net\.NetNative\15.0.24211\x86\ilc\Tools\nutc_driver.exe @"D:\Builds\testing\ProjectName\obj\x86\Master\ilc\intermediate\MDIL\ProjectName.rsp"' returned exit code 1
    40. 1>  Uitvoeren van taak StartProcessTask voltooid -- FAILED. (TaskId:453)
    41. 1>  Opbouwen van doel GenerateMDILGlobalAnalysis in project Unnamed_1 voltooid -- FAILED. (TaskId:453)
    42. 1>  Opbouwen van project Unnamed_1 voltooid -- FAILED. (TaskId:453)
    43. 1>  Opbouwen FAILED. (TaskId:453)
    44. 1>   (TaskId:453)
    45. 1>  The command exited with code 1201. (TaskId:453)
    46. 1>  Output Property: _IlcExitCode=1201 (TaskId:453)
    47. 1>Done executing task "LoggerBasedExecTask" -- FAILED. (TaskId:453)
    48. 1>Done building target "BuildNativePackage" in project "ProjectName.csproj" -- FAILED.: (TargetId:139)
     
  4. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,675
    This usually indicates that Microsoft's .NET native toolchain runs out of memory. Unfortunately there could be multiple reasons for that, including you having too much code or you hitting some pathological path in the compiler. Since it started happening recently, do you know what change you made that had it start happening?
     
  5. Roywise

    Roywise

    Joined:
    Jun 1, 2017
    Posts:
    68
    It's really hard to say, the build PC which we'd usually make the builds with didn't change at, maybe a sneaky Windows update happened but I'm not sure. About the project, it changes constantly and it contains quite a bit of classes, folders, assets and code but what would be "too much"?

    I'm not sure what a pathological path would be, not a native English speaker. Is there a way to track this down or maybe some best practices to avoid issues like that?

    Something I figured out yesterday that I'm able to build with .NET Native enabled but only when I disable the "Optimize code" option in Visual Studio. I tried to figure out what this option does and I sounds like disabling that would be a bad idea. I'm not even sure if the app would be able to go through the validation process.

    Are there any things we'd be able to try? We tried all of these things below but no acceptable results.

    * Force 64 bit compiler.
    * Enable ShortcutGenericAnalysis.
    * Updating/Downgrading the various NuGet packages.
    * Removing the default directive from the .rd.xml file.
    * Adding custom directives to stop optimization or inlining of some assemblies.
    * Using Visual Studio 2015.
    * Building the Unity project with different settings, target PC only, target older SDK, etc.
    * Disable Anti-virus during building. (Why did I even try this?)
    * Stripped unused scenes and assets from the project. (Didn't strip any code though)
    * Tried compiling the Unity project using IL2CPP backend. (Sadly this causes a lot of other issues to pop up so we cancelled trying this)
    * Disabling .NET Native tool chain. (Works, but the app won't be accepted in the store)
    * Disabling Optimize code. (Works, but we have no idea what the side effects are)


    Our previous packages are already on the Windows Store, we planned to release an update this thursday but it seems we'll only be able to push a 64x packages this time. Hopefully we don't have much users that use the 86x application because that's probably going to cause issues.
     
  6. Roywise

    Roywise

    Joined:
    Jun 1, 2017
    Posts:
    68
    Would using Assembly Definition Files to split our codebase into a couple of Assemblies be a way to ease it on the .NET Native compiler? That's something we're trying now but requires quite some fixes because it does not support circular references.
     
  7. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,675
    I do not know whether it will work. It's worth a try.

    Last time I talked with Microsoft, they said that the compiler can handle around 250-300K methods without going out of memory. Any more than that and you'll be pushing it.

    "Pathological path" would be a small piece of code that causes other code (in this case the compiler) to behave incorrectly (crash by going out of memory, in this case).

    Are you using source control for your project? If you are, you could try bisecting which change introduced the failure. It's definitely something in your code or plugins that you use - it won't be a change in toolchain or windows update.

    We very recently saw a similar thing reported to us and it turned out to be caused by the compiler choking on v.4.1.2.0 of Photon3Unity3D.dll. Are you by any chance using that plugin? If so, the workaround is to use a slightly different version (we found v.4.1.1.17 works).
     
  8. Roywise

    Roywise

    Joined:
    Jun 1, 2017
    Posts:
    68
    We're using source control so I'll try some earlier versions. I should be able to compare the method count as well. According to the build output we're already on 291k+ methods.

    We don't use the Photon plugin but I'll rip out some plugins and hopefully I'll be able to figure out what's causing our problem.

    Thanks for the information, I'll keep this thread updated.
     
  9. Bramverhoeven2

    Bramverhoeven2

    Joined:
    Dec 5, 2017
    Posts:
    6
    I am a colleague of Roy and working on the same issue now. As far as I know the issues happened after the big windows april update (1803).
    I even tested with an older version from 3 months back. Which worked perfect at that time. But gives the same error now.
    I will do some more tests to see if I can find a solution for this problem.
     
    Last edited: Aug 21, 2018
  10. Bramverhoeven2

    Bramverhoeven2

    Joined:
    Dec 5, 2017
    Posts:
    6
    It seems that changing the backend from .NET to IL2CPP solves the problem! We are now able to build 86x builds again
     
  11. Tautvydas-Zilys

    Tautvydas-Zilys

    Unity Technologies

    Joined:
    Jul 25, 2013
    Posts:
    10,675
    Yeah, this problem is specific to .NET scripting backend. And we cannot really fix it since it's on Microsoft side.
     
  12. Roywise

    Roywise

    Joined:
    Jun 1, 2017
    Posts:
    68