Search Unity

Help debugging "Internal: JobTempAlloc" warning

Discussion in 'Data Oriented Technology Stack' started by Muse_Watchmaker, Aug 8, 2018.

  1. Muse_Watchmaker

    Muse_Watchmaker

    Joined:
    Aug 8, 2018
    Posts:
    10
    I have a project currently making use of the job system alone (without ECS), that is plagued by warnings of the form:

    Code (csharp):
    1. Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak
    All NativeArrays in this project, by the way, are created with Allocator.Persistent. And yet I can't quite write these off as harmless because this project also has a tendency to freeze on quit that only appears after a long while running, which seems like it could very well mean there is some actual leak at work.

    This occurs in Unity 2018.1.9 and 2018.2.2 both. 2018.2.2 adds an additional warning paired with the first:

    Code (csharp):
    1. To Debug, enable the define: TLA_DEBUG_STACK_LEAK in ThreadsafeLinearAllocator.cpp. This will output the callstacks of the leaked allocations
    ...which isn't particularly helpful as I don't have access to the source of ThreadsafeLinearAllocator.cpp.

    Trial, error, and searching led me to fix some of these warnings by removing static references left by accident in my IJob structs. However, now I'm at a standstill as I can't find any more obvious causes, and trying to narrow down which parts of my job code lead to the warning is getting inconsistent results (I suspect the length of time the job is running is having an effect.)

    Anyone know of other possible causes of this warning, or ways to narrow down the cause?
     
  2. LennartJohansen

    LennartJohansen

    Joined:
    Dec 1, 2014
    Posts:
    2,296
    Do you complete the jobs every frame? Or is does the jobs work for more than 4 frames?
     
  3. Muse_Watchmaker

    Muse_Watchmaker

    Joined:
    Aug 8, 2018
    Posts:
    10
    The job is definitely doing work for more than 4 frames. But as I'm not using Allocator.Temp or Allocator.TempJob, my understanding is that this shouldn't be a problem?
     
  4. LennartJohansen

    LennartJohansen

    Joined:
    Dec 1, 2014
    Posts:
    2,296
    I think this is a known issue. There will be a fix later, how the allocations are done does not matter.
     
  5. Muse_Watchmaker

    Muse_Watchmaker

    Joined:
    Aug 8, 2018
    Posts:
    10
    Anyone (preferably from Unity) know when "later" is (that is, for supporting jobs that run for more than 4 frames without issue?) If I'm just waiting for an upcoming patch/release to fix this that's one thing, but if there's no timeline I need to consider alternatives.
     
  6. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,417
    Can you paste code, so someone can look at?
     
  7. Muse_Watchmaker

    Muse_Watchmaker

    Joined:
    Aug 8, 2018
    Posts:
    10
    So in your experience, is it possible to have a job that runs for more than 4 frames without this warning, Antypodish?

    Full complexity is too much to paste, and so far in a simplified version the problem goes away. If this is indeed a known issue with jobs that run for more than 4 frames (LennartJohansen's theory, and consistent with what I've since found in some other threads on this forum) trying to go through my code seems tangential.
     
  8. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,417
    Well my exp is not that great ;)
    I think I maybe had once the issue. But but somehow resolved. Perhaps Disposed array.
    What you should do perhaps split system into multiple smaller systems.
    That would be first step.
    Of course if this is applicable. For example is not continuous generation of some relational data.
    As far I am concerned, Jobs can run perhaps forever, if you "loop them"?
    My target is to keep all my jobs far below single frame.
    If you have NativeArryay, you may need set allocation to persistent, if you want keep data. Or there is other option of temp allocation.
    Code (CSharp):
    1. Allocator.Temp
    2. Allocator.TempJob
    Which both keep data in array for longer than 4 frames.
    At least what suppose to do. Or I am wrong?
     
  9. LennartJohansen

    LennartJohansen

    Joined:
    Dec 1, 2014
    Posts:
    2,296
    Problem is not keepting the data in an array longer. That works great with a persistent allocation.

    The problem shows up when the job itself runs for more than 4 frames in the background.
     
  10. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,417
    Oh.
    Then I haven't experienced such case.
    While this is valid issue, can the procedure be shortened, split or whatever optimized?
     
  11. Muse_Watchmaker

    Muse_Watchmaker

    Joined:
    Aug 8, 2018
    Posts:
    10
    Perhaps possible, but not the direction I'd choose to take - the whole point of these particular jobs is that this is background simulation work that I can show in the game when it's ready. Using the job system is nice for load balancing purposes, but if it isn't going to support this sort of thing in the near future, I'll just go back to using the normal C# threading constructs for these.

    Tying asynchronous jobs like this to the framerate just seems like asking for future trouble, since now I have a memory leak that will appear more often on hardware that is otherwise better/faster.
     
  12. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,417
    May I ask, what type of script you execute? Is it AI. Or data processing of some sort? For example procedural mesh generation.
     
  13. Muse_Watchmaker

    Muse_Watchmaker

    Joined:
    Aug 8, 2018
    Posts:
    10
    A (suitably simplified and gamified) set of soil/water simulations, on a 2D grid of data. Ultimately they feed into textures and meshes/heightmaps for rendering, but processing the raw data is the most time consuming part right now. So far I've been updating them about once per 5 seconds and found that sufficient for gameplay feedback purposes; faster would be nicer, of course, but isn't strictly necessary.

    Because it's the obvious optimization: breaking the simulation up into smaller chunks is possible, but it's a balancing act and doesn't actually change the total amount of time spent that much (and adds overhead for dealing with chunk edges.)
     
  14. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,417
    Sounds intriguing ;)

    I let you figure out that one. But I think division could help, if done appropriately. However, since you dealing with kind of fluid behavior, I see some challenges, of talking about parallel running.
     
  15. mijagourlay

    mijagourlay

    Joined:
    Apr 18, 2017
    Posts:
    3
    FWIW I see this warning a lot also and my project does not spawn any jobs at all. I mean, maybe indirectly as a consequence of using some Unity feature but not in any C# I wrote for this project.
     
  16. KhynLegend

    KhynLegend

    Joined:
    Aug 8, 2018
    Posts:
    1
    I have the same problem too when I play my game that I'm following on brackeys' tutorial on youtube
     
  17. Muse_Watchmaker

    Muse_Watchmaker

    Joined:
    Aug 8, 2018
    Posts:
    10
    A small update: I sort of worked around this by breaking up jobs into smaller chunks of work, which turned out to be something I needed for other reasons as well. As a side effect this eliminated the warnings, at least most of the time (I am keenly aware that my current solution hasn't guaranteed the 4-frame timing on any kind of fundamental level) - but it did not eliminate the tendency to freeze on quit that I originally thought was associated with them.
     
  18. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,667
    The issue might be something like if you have these giants jobs that last more than 4 frames they're taking all the workers and blocking internal unity jobs that are temp allocated from executing in time.

    When you break into smaller chunks it gives other jobs a chance to schedule some time.
     
    peteyhayman likes this.
  19. Muse_Watchmaker

    Muse_Watchmaker

    Joined:
    Aug 8, 2018
    Posts:
    10
    My understanding from other threads on the topic (gained after I started this one) is that the job structs themselves are internally temp allocated, regardless of what you do. But I have indeed seen user jobs blocking internal Unity jobs mentioned as one of the reasons support for long-running jobs is difficult.
     
  20. QuantumMagDev

    QuantumMagDev

    Joined:
    May 30, 2017
    Posts:
    2
    Hey guys, I had this issue come up.
    I had a 2D Array in the form of

    Code (csharp):
    1.  
    2. List<2DStruct> column = new List<2DStruct>();
    3.  
    4. [System.serializable]
    5. struct 2DStruct {
    6. public List<GameObject> Row;
    7. }
    8.  
    and on Scene Load the warnings:
    -JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak
    -To Debug, enable the define: TLA_DEBUG_STACK_LEAK in ThreadsafeLinearAllocator.cpp. This will output the callstacks of the leaked allocations

    I was not able to enable this or find any info on how, but when i switched from struct to class, suddenly the warning is gone (Example fix below)
    Code (csharp):
    1.  
    2. List<2DClass> column = new List<2DClass>();
    3.  
    4. [System.serializable]
    5. struct 2DClass {
    6. public List<GameObject> Row = new List<GameObject>();
    7. }
    8.  
    Struct seems to throw an allocation warning in Native Arrays, If possible switch to Class for Native Arrays!
    Hope this solves your issue,
    Phil Ferland President/CEO of QME Inc.

    *edit: thanks Antypodish, didn't know how
     
    Last edited: Jan 13, 2019
  21. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,417
  22. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    4,671
    This is a known issue. Unity internally uses the TempJob label for the job type struct itself. We have a bug logged on it.
    The warning is not fatal. It just holds on to the memory for an unexpected amount of time.
     
    psuong and pib78 like this.
  23. llJIMBOBll

    llJIMBOBll

    Joined:
    Aug 23, 2014
    Posts:
    553
  24. knickerbocker

    knickerbocker

    Joined:
    May 8, 2014
    Posts:
    29

    Does it ever stop holding on to the memory? Im building a unity server, so quite persistent, will this cause problems after say 24-48h of use?
     
  25. Lixingjun

    Lixingjun

    Joined:
    Nov 3, 2016
    Posts:
    3
    I encountered the same problem when using the win32 function ShowWindow
     
  26. shandan

    shandan

    Joined:
    Aug 29, 2017
    Posts:
    1
    Hello everyone, has this problem been solved by someone?
     
  27. james141151

    james141151

    Joined:
    Mar 9, 2019
    Posts:
    2
    I have the same problem too! Has this problem been solved? :(:(
     
  28. LabGecko

    LabGecko

    Joined:
    May 22, 2014
    Posts:
    2
    4th hand up, and all I have in the scene is 2 particle systems and a camera... None of the particles loads more than 5 at a time, and nothing is set to last longer than 3 sec.
     
  29. KittyAnn

    KittyAnn

    Joined:
    Dec 25, 2018
    Posts:
    10
    Following, as I too, have this issue & can not finish generating the lighting. It hangs on 7/11 Lighting for days.
     
  30. Lixingjun

    Lixingjun

    Joined:
    Nov 3, 2016
    Posts:
    3
    I solved this problem by using the ColseWindwo() function to minimize the window before ShowWindow()
     
  31. james141151

    james141151

    Joined:
    Mar 9, 2019
    Posts:
    2
    Great job!

    I also solved this problem.

    Thanks you! :):):)
     
  32. bsgbryan

    bsgbryan

    Joined:
    Nov 27, 2015
    Posts:
    15
    Just raising my hand that I'm seeing this problem too.

    My example code is below. I'm running Unity 2019.2.0a13 on macOS 10.14.4.

    Code (CSharp):
    1. using UnityEngine;
    2.  
    3. using Unity.Jobs;
    4. using Unity.Entities;
    5. using Unity.Burst;
    6. using Unity.Collections;
    7.  
    8. public struct Vertex {
    9.   public float X;
    10.   public float Y;
    11.   public float Z;
    12. }
    13.  
    14. public struct Triangle : IComponentData {
    15.   public Vertex Zero;
    16.   public Vertex One;
    17.   public Vertex Two;
    18. }
    19.  
    20. public struct TestJob : IJob {
    21.   public NativeArray<Triangle> Triangles;
    22.  
    23.   public void Execute() {
    24.     Triangles[0] = new Triangle {
    25.       Zero = new Vertex {
    26.         X = 0f,
    27.         Y = 0f,
    28.         Z = 0f
    29.       },
    30.       One = new Vertex {
    31.         X = 1f,
    32.         Y = 1f,
    33.         Z = 1f
    34.       },
    35.       Two = new Vertex {
    36.         X = 2f,
    37.         Y = 2f,
    38.         Z = 2f
    39.       }
    40.     };
    41.   }
    42. }
    43.  
    44. public class ExampleSystem : ComponentSystem {
    45.  
    46.   protected override void OnCreate() {
    47.     Debug.Log("Creating");
    48.   }
    49.  
    50.   protected override void OnUpdate() {
    51.     NativeArray<Triangle> TriangleArray = new NativeArray<Triangle>(1, Allocator.TempJob);
    52.    
    53.     TestJob testJob = new TestJob {
    54.       Triangles = TriangleArray
    55.     };
    56.    
    57.     JobHandle handle = testJob.Schedule();
    58.  
    59.     JobHandle.ScheduleBatchedJobs();
    60.  
    61.     handle.Complete();
    62.  
    63.     Debug.Log("Triangle Zero.Z " + TriangleArray[0].Zero.X);
    64.     Debug.Log("Triangle One.X  " + TriangleArray[0].One.X);
    65.     Debug.Log("Triangle Two.X  " + TriangleArray[0].Two.X);
    66.  
    67.     TriangleArray.Dispose();
    68.   }
    69.  
    70.   protected override void OnDestroy() {
    71.     Debug.Log("Detroying");
    72.   }
    73. }
    Update

    I just upgraded to Unity 2019.2.0a14 and running the above code no longer generates any warnings.
     
    siddharth3322 likes this.
  33. gadgetfan

    gadgetfan

    Joined:
    Oct 1, 2018
    Posts:
    6
    I discovered, that in case of this warning, just restarting of Unity and Rider helps ) .
     
  34. Abbrew

    Abbrew

    Joined:
    Jan 1, 2018
    Posts:
    149
    I tried this and the warnings always eventually crop up after some runs.
     
  35. giggioz

    giggioz

    Joined:
    May 11, 2017
    Posts:
    18
    Same issue here (Unity 2019.1.2f1)

    Is there a way to hide the continuous warning in the console?
     
  36. Guochu

    Guochu

    Joined:
    Apr 2, 2018
    Posts:
    1
    same issue
     
  37. giggioz

    giggioz

    Joined:
    May 11, 2017
    Posts:
    18
    I restarted Unity and/or Unity Hub several times but no luck.

    Yesterday I restarted my macbook and now the warning has gone!
     
  38. jdtec

    jdtec

    Joined:
    Oct 25, 2017
    Posts:
    87
    Do the latest DOTS packages or Unity version fix this issue?
     
  39. siddharth3322

    siddharth3322

    Joined:
    Nov 29, 2013
    Posts:
    705
    Yes again this issue started coming, I am also having the same issue.

    Unity Version 2018.4.2f1
     
  40. m3rt32

    m3rt32

    Joined:
    Mar 5, 2013
    Posts:
    60
    I also have this issue when I allocate a native collection. Though, I dispose it on OnDestroy function. If I dispose the collection at the end of my method, it throws an exception about "This collection is disposed" and doesn't run the thing. Is there a solution for this ?
     
  41. mnarimani

    mnarimani

    Joined:
    Mar 27, 2017
    Posts:
    211
    This problem still exists in 2019.2f1
     
  42. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,667
    need code
     
  43. Jbs_GameZone

    Jbs_GameZone

    Joined:
    Dec 14, 2018
    Posts:
    105
    Having the same issue with 2019.2.f1, lots of warnings, any way to debug or workaround ?
     
  44. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,417
    try disable all systems and enable one by one, to see, which one causes. As @tertle mentioned is most likely, you have not disposed Native Arrays correctly, either at end of program, or in job.
     
    Last edited: Aug 21, 2019
  45. Jbs_GameZone

    Jbs_GameZone

    Joined:
    Dec 14, 2018
    Posts:
    105
    A bit more complex than that. In my game logic scripts I don't allocate arrays or work with jobs, but I use many assets (even ones that are shipped without the source code) in my simulator such as : car physics, traffic AI, etc. So I'd probably need to disable them one by one and see what asset is the culprit and then contact the author, this will require at least several days of debugging.

    Nonetheless, I've found this thread https://developer.vuforia.com/forum/unity/known-issues-unity-20191 (although I don't use vuforia) and checked the render over native UI which apparently seems to eliminate that warning without any drawbacks from a visual pov. For the moment I'll leave that flag checked.

    Thanks
     
  46. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,417
    Does mentioned assets, use DOTS? If not, I wouldn't be expecting, causing discussed warnings.
     
  47. Jbs_GameZone

    Jbs_GameZone

    Joined:
    Dec 14, 2018
    Posts:
    105
    Thanks for your reply. No, it doesn't use DOTS.

    This is by far the hardest debug and a pain in the neck. I've managed to isolate it to a method call (that handles particle generation and sound management), unfortunately that has tens of other method calls in it that spans across hundreds of lines of code. And every new attempt for checking if the issue persists or not implies a new build and a new deploy and then letting the game play for 5-10 minutes, horror.

    Isn't there a way (changing/hacking/injecting a new cpp file) that will display a stack trace to see where exactly that issue is coming from ?

    Otherwise I need to think of a more clever approach to debug this (add tens of buttons on the screen that set certain booleans on and off, and based on them parts of the methods execute or not, in this way I could change what actually is being executed from that method at runtime), but still would totally prefer to see a stack trace.
     
  48. Antypodish

    Antypodish

    Joined:
    Apr 29, 2014
    Posts:
    5,417
    Just on the side note, I have noticed, If you think you may have resolved warning messages cause, make sure you restart Unity. Often Unity keeps in cache last warning state, and keep producing such warnings, even code is healthy, until Unity restart. At least how I works for me in 2019.2, if I forget to dispose arrays.
     
  49. Jbs_GameZone

    Jbs_GameZone

    Joined:
    Dec 14, 2018
    Posts:
    105
    Messages are not displayed in Unity during gameplay. Thet appear only in Xcode 11 (Beta 5) during gameplay.
     
  50. Muse_Watchmaker

    Muse_Watchmaker

    Joined:
    Aug 8, 2018
    Posts:
    10
    A little behind, but since I started this, I figured I should post an update:

    Upgrading to 2019.1 fixed the issue for my own job code.

    I still, however, got the warning when doing runtime navmesh generation with Unity's built-in system. I ended up switching to a 3rd-party pathfinding package for a number of reasons, which avoided the issue and resolved the last of my crashes/freezes.

    For others seeing this warning still, my one possibly useful suggestion: if you're writing your own jobs, make sure the IJob classes do not have static references to non-value types. This caused a lot of issues for me back at the start of this process.