Search Unity

[Jobs][Lags] JobTempAlloc has allocations that are more than 4 frames old

Discussion in 'Data Oriented Technology Stack' started by dyox, Jan 18, 2018.

  1. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    208
    Welp, please insert a meme of me eating my hat here.

    I had this in a system.

    Code (CSharp):
    1. protected override void OnCreate()
    2. {
    3.  
    4.    Verts = new NativeList<Vector3>(prevVertCapacity, Allocator.TempJob);
    5.    Tris = new NativeList<int>(prevTrisCapacity, Allocator.TempJob);
    6.    Normals = new NativeList<Vector3>(prevUVsCapacity, Allocator.TempJob);
    7.    UVs = new NativeList<Vector2>(prevNormalsCapacity, Allocator.TempJob);
    8.  
    9.    CreateCrossSections();
    10. }
    Which would work find if the system was being updated. But if the system wasn't being updated every round because it had no queries to do on, it wouldn't dispose these temp jobs and then get a leak.

    Now I've learned about OnStartRunning() and OnStopRunning() method calls in ECS.

    Thanks for the support and sorry for being a dick about insisting this is an internal Unity problem. Lucky 2019.2 gave me one more stack trace once which helped me track this down...

    Again, I will show myself out.
     
    PaulUsul and xVergilx like this.
  2. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    208
    I spoke too soon, about an hour after doing the above, it came back. Restarted Unity and it went away... I am going to go crazy.
     
    xVergilx likes this.
  3. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    2,171
    Your post got me an idea of checking in every TempJob usage.

    I doubt it will help, but at least it may lead to what's leaking inside hybrid render.
     
  4. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,716
    If you create containers in OnCreate and want them to live untill the system destroyed you should use Persistent allocator instead, and dispose them in OnDestroy.
     
  5. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    208
    Yeah, I didn't realize OnUpdate may not fire if there is nothing going on in the system. I will probably move that code to test on the first time it is created to initialize it.
     
  6. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,984
    Happy you figured something out goose (but yeah I don't think you'd want to do anything but Persistant allocations in OnCreate or OnStartRunning - with the exception being a temp allocation for initialization within the same method.)

    I'm 99% certain there are no leaks inside hybrid render.

    Have you tried turning on full stack traces?
     
    Last edited: Aug 2, 2019
  7. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    2,171
    Yep, it still doesn't say what's leaking. That's why I'm confused.
    It happens mostly when entities that are rendered moved, or rotated.
     
    siddharth3322 likes this.
  8. siddharth3322

    siddharth3322

    Joined:
    Nov 29, 2013
    Posts:
    760
    Even I am working on a small game then also I am getting this kind of log messages within the XCode.

    Example of a log:

    xcode_project_error.png
     
    Ignacii likes this.
  9. Lynxed

    Lynxed

    Joined:
    Dec 9, 2012
    Posts:
    70
    started getting those after 2019.2 migration.
    And i have no TempJob allocations.

    upload_2019-8-2_12-38-50.png

    Is there a particular place, where we have to allocate Persistent NativeCollections or can i allocate them anywhere as long as i deallocate them later?
     
  10. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    2,171
    It doesn't matter where. As long as you deallocate them at some point.

    TempJob allocs expected to live less than 4 frames, Temp - only one frame.
    So you have to deallocate them in that ranges.

    Not sure about the rest.
     
  11. dyox

    dyox

    Joined:
    Aug 19, 2011
    Posts:
    566
    Still on Unity 2019.3.0a11.

    Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak
    Internal: deleting an allocation that is older than its permitted lifetime of 4 frames (age = 15)
     
    Lynxed likes this.
  12. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    208
    Full Stack Trace won't track those warnings down unfortunately. Since I changed some code around tempjobs I have had less errors so we'll see if they pop up today.
     
    Antypodish and xVergilx like this.
  13. Lynxed

    Lynxed

    Joined:
    Dec 9, 2012
    Posts:
    70
    Can confirm having bunch of this in Unity 2019.3.0a11.
    My project uses only Allocations.Persistent
     
  14. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,984
    Have you tried disabling all your systems then enabling them 1 at a time till it appears?
     
  15. Lynxed

    Lynxed

    Joined:
    Dec 9, 2012
    Posts:
    70
    Funny thing: if you have at least one leak with your Persistent collections, you may or may not get this error randomly.
     
  16. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    208
    Oh that is wonderful :p. Must be if I kill a process without disposing properly. Since I fixed those temp problems, I haven't been getting this. It is just unfortunate that the stack trace is so cryptic.
     
  17. lilbevo

    lilbevo

    Joined:
    Aug 5, 2019
    Posts:
    3
    Welp... also getting this error but I have absolutely no mention of jobs in my single C# script in the project I just started. The error seems to show up when I preform ANY action in the editor (moving an object, placing an object, etc) besides literally clicking around folders in the heirarchy. And then it spams a bunch when I click play. Totally lost here— no idea what’s going on.
     
  18. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    208
    If you restart Unity does it go away for a bit?
     
  19. lilbevo

    lilbevo

    Joined:
    Aug 5, 2019
    Posts:
    3
    Nope :(
     
  20. lilbevo

    lilbevo

    Joined:
    Aug 5, 2019
    Posts:
    3
    Well... noticed my browser was running weirdly slow, and I realized I haven't restarted my computer in a while. It's just been on sleep mode. Restarted it and now everything's fixed! Not very knowledgeable on computer stuff but I'm assuming that not restarting for extended periods of time leads to a large decrease in usable RAM
     
  21. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,984
    Only if you're leaking memory - which you are if you get this warning!

    Yep, if you leak once you need to restart editor to get rid of it. Though I suspect most of the time this issue is caused by not completing jobs rather than allocating containers.
     
  22. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    208
    If you get a null error then that dispose call isn't called and those just build up over a lot of time.
     
  23. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    2,171
    So this one is weird.

    I was getting these warnings on an old cpu that had 2 cores / 2 max threads available.
    Now that I've upgraded to a better PC I no longer get them.

    On top of that QA was not able to reproduce this one as well.

    Got no idea what would be causing this, but perhaps there was simply not enough time to execute all jobs in time.
     
  24. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    208
    I am thinking that at some point what happens is the editor doesn't complete it's shut down cycle so things aren't disposed properly in Unity's code or your own code. You then get that temp job error till you restart Unity and the problem goes away. This is why it is so hard to track down.
     
  25. Emery-Monzerol

    Emery-Monzerol

    Joined:
    Nov 1, 2012
    Posts:
    4
    Has anyone else noticed crashes/freezes that seemed linked to that warning? My team and I have noticed that our project has started freezing occasionally, maybe once every 10 runs, and that every time it does, the freeze was preceded by a large amount of this JobTempAlloc warning (something like 30 warnings approximately).

    Happens both in build and in editor, and when it happens in editor, the whole editor freezes.

    I tried attaching Visual Studio to debug and pause, but when I pause there's not even a main thread present in the Threads window. I don't believe that's supposed to even be possible? What am I missing here? Is it possible for the main thread to die off, but still have the process running, with the editor frozen in place?

    We're not even using Jobs knowingly (unless it's done by one of the packages we got from the asset store), so it's pretty darn hard to debug this issue.

    Any help would be appreciated, we're starting to run out of options here.
     
    laeusnb and Ignacii like this.
  26. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    843
    Check RAM usage. Could be a memory leak by something you are using.
     
    dyox likes this.
  27. PaulUsul

    PaulUsul

    Joined:
    Nov 20, 2012
    Posts:
    29
    I know exactly how you feel, it's like it terrarizes you! I've gone over to just using persistent in active development as it's much better at telling when it's leaky and it runs faster than 1 million warnings :p
     
  28. Steve-Tou

    Steve-Tou

    Joined:
    Aug 4, 2018
    Posts:
    8
    I got a run of about 40 of these warnings. Some with a slight variation of ( x=5) or (x=6) at the end. Sorry I don't remember what x was- frames maybe? And I haven't been able to reproduce since I cleared the console. Anyways, I am running no scripts- haven't even brought one into the project-- I was just starting to test out 2019.2 Personal after having been using 2017.1 for years. I have 3 textures, 4 materials, and about a dozen of my own models and a few primitives in the scene. I was messing around with clicking things on/off in the inspector window while in play mode. Don't know if this helps anyone, but maybe it'll help someone not do too deep of a dive. Gotta be something with the editor.
     
    Morbeavus likes this.
  29. Ignacii

    Ignacii

    Joined:
    May 23, 2013
    Posts:
    20
    Still in 2019.2.3 :(
     
    Immu and dyox like this.
  30. dyox

    dyox

    Joined:
    Aug 19, 2011
    Posts:
    566
    I've found a workaround that is a little bit tricky and cost few ms (for person using jobs, not unity internal bug)

    I'm using InvokeRepeating with small update time 0.1s, i've divided my ParralelJob in small batches of xx (batchsize) schedules with a start index added to Execute index.

    Disabled index check with attribute on native container and access by :
    nativecontainer[index + startindex].

    I adapt the batch size in real time by recording the frame count at job start and end.

    I'm able to generate 8000 voxels chunks in 0.6s and 4096×4096 noises heightmap without warning.
     
  31. Immu

    Immu

    Joined:
    Jun 18, 2013
    Posts:
    162
    Still getting them in 2019.2, kinda same as Steve-Tou. It's hard to tell when they started to popup, but it was probably around 2018 came out. The moment on which they popup is a hell to understand. But I felt the particleSystem might be the reason sometimes because I was selecting it or spawing one.

    Of course, can't repro. My project is like 4 + years old and 12gb of data. The warning spawn happens out of nowhere understandable that I could make a report project from.
    And I'm using wide range of computer types.

    Can Unity do something to let us have kinda like the 'full' stack trace of that issue when it popups ?
    So we can at least have more clues of the origin.

    I've 64G of RAM by the way. If anyone's tempted to say I haven't enough XD
     
    Last edited: Sep 2, 2019
  32. Simon-O

    Simon-O

    Joined:
    Jan 22, 2014
    Posts:
    34
    "Copy around huge chunks of memory" is your preferred solution? All to prevent people taking advantage of the real benefits of multi-threading.

    The real underlying question is why?

    "We can't guarantee there won't be a race condition" is not a compelling answer. I can write code with race conditions in it right now and you couldn't detect that either... That didn't result in you pulling Unity, did it?

    Yes, if you hand someone a tool, they might stab themselves in the leg. I've yet to hear anyone suggest we ban all tools.
     
    dyox and Immu like this.
  33. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    208
    I also got this error in the player because my game was running a job in fixed update and the game was running TOO fast. You should be sure to set your application frame rate to 120 so that in the future as computers get faster and faster this error doesn't pop up.

    Unity should definitely suppress this warning (or have a way to suppress it) in the player. I've seen this warning pop up on Steam forums because it is happening in random games.
     
    Immu and dyox like this.
  34. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,250
    No, you'd just store it laid out that way at all times. No need to copy back and forth.

    What benefits do you think we are preventing people from taking advantage of?
     
    dyox likes this.
  35. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,250
    @dyox what are you trying to say with that video?
     
  36. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    3,137
    "This" issue started for me today. I have been using Unity 2019.1.14f1 for almost as long as it's available, and I'm not aware of any changes in my system. What I did do was disconnect the Oculus Rift and re-connect the Vive Pro - but I had done this many times before.

    I had also opened the Interaction System demo scene of the SteamVR Unity Plugin. That's where I saw it the first time, so I assumed something was wrong in that scene. Changed to my project's default scene. Issue still happened. Restarted Unity. Rebooted Windows 10. Opened another project. Created a new empty project.

    The only difference that I'm seeing is that in my main project, this issue spams the log as soon as I open Unity. In the other project, and in the new project, it starts when I hit play, and it stops when I exit play. These two projects are not VR projects.

    The warnings I'm getting:
    Code (CSharp):
    1. Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak
    2. To Debug, enable the define: TLA_DEBUG_STACK_LEAK in ThreadsafeLinearAllocator.cpp. This will output the callstacks of the leaked allocations
    3.  
    That's all I'm getting. No stacktrace or anything. I have also tried Stacktrace Logging / All / Full - still only one line for each entry.
     
  37. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    3,137
    I have now also filed a bug-report for this:

    (Case 1186264) Warnings spamming the console in empty project: JobTempAlloc has allocations that / enable: TLA DEBUG STACK LEAK

    As this hasn't happened on my system until about an hour ago, and it's now happening in an empty project that I have just created, it looks like this might be something on my system. I'm not using ECS or the job-system.
     
  38. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    3,137
    ... and ... after another reboot, it's gone. In all of the projects. I'm really glad now that I had filed a bug report while it was there. Hopefully the logs in the bug report have useful information that helps tracking this down. What a strange issue.
     
  39. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    208
    @jashan What probably happened was Unity stopped at a bad time or you had a null error which prevented one of Unity's internal jobs from properly disposing itself. Once you reset everything it cleared that warning and it was gone.
     
  40. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    3,137
    The strange thing was that the warning spam persisted over a full reboot, and over closing Unity and creating a new, empty project. And now, I can't reproduce it at all.

    I won't complain - this was spooky and I prefer spending my time working on my project over chasing down mysterious warnings. But I would still love to know how something like this can happen. Going by your explanation, if I understand it correctly, it would mean that Unity does something to persistently store jobs so that they could remain and cause trouble even after closing / re-opening Unity, and even after a reboot. I'm not saying this is impossible - I'm aware that Unity does launch separate processes and apparently doesn't always properly close them when closing itself (that's the reason why Steam will often show the Unity game as "playing" even after shutting down Unity).

    I believe I first opened the project where the issue occurred the first time after the reboot, so something may have been stuck there (making it persist over the reboot), and then it could theoretically still be stuck when closing Unity to open a new project (due to Unity launching separate processes and not closing them properly). But then, the first time I saw it it kept spamming the log while not in play mode but in the new project, it only spammed the log while in play mode.

    The next thing I would have tried would have been another Unity version ... but I'm glad it's gone now and hope it will never return (if it does, I'll make sure to file a bug report for each time I run Unity where it occurs).
     
  41. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    3,137
    I probably shouldn't have said that. So here's the full list of bug reports I filed on this issue now (the first should include the list, I replied to the original bug report autoresponder):

    (Case 1186264) <--- original report
    (Case 1186448)
    (Case 1186450)
    (Case 1186451)
    (Case 1186454)
    (Case 1186456)

    The only thing I know now is that this didn't happen when I created a new project in Unity 2019.2.5, so either the issue I'm seeing is fixed in Unity 2019.2.5, OR whatever gets stuck is version specific, so the 2019.2.5 installation is not effected when it happens on 2019.1.14. This time, Unity crashed, then I rebooted, then I saw those warnings. Then I tried several things and as promised, filed a bug report each time.

    I really hope the logfiles will contain useful information.

    I also know that while it seems that first starting Steam, SteamVR, JetBrains Rider and Visual Studio before starting Unity makes this happen after the reboot, I can also start all of these without the issue happening. And starting Unity first, and then starting all of these before starting playing, I once saw those warnings start during my initialization. But the first time I saw this, it happened in a completely different scene that's not even from my project (the SteamVR Interaction System demo scene). I haven't seen it in that scene again (and as mentioned earlier, I have also seen it in completely empty new projects).
     
  42. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    2,574

    That general pattern can and has been caused by a lot of things going back several years.

    I'm seeing it now linked to the alloc warnings and terrain lightmapping when going into play mode. Editor log shows gi::BakeBackendSwitch then hundreds of alloc warnings before freezing. When it hits it freezes the editor and then after 10-15 seconds Unity goes from normal cpu usage to zero cpu usage.

    Consistency seems to be tied to the overall profile. Ie it reliably freezes in our main game scene entering play mode where there are a bunch of heavy jobs that run for 2-3 seconds. In a simpler test scene it's completely infrequent. But the same editor log patterns so same core problem

    In the current scenario if I disable the terrain it doesn't trigger, and then it doesn't trigger afterwards either even with the terrain enabled. So it seems to be logic that's a first time the scene is loaded thing.
     
    Immu likes this.
  43. kingstone426

    kingstone426

    Joined:
    Jun 21, 2013
    Posts:
    36
    Just a little heads-up if you are scheduling jobs in FixedUpdate. You might get TempAlloc warnings if you don't Complete the job in both LateUpdate and FixedUpdated, as explained here in Unity's netcode sample project code:


    Code (CSharp):
    1. void LateUpdate()
    2.     {
    3.         // On fast clients we can get more than 4 frames per fixed update, this call prevents warnings about TempJob
    4.         // allocation longer than 4 frames in those cases
    5.         m_updateHandle.Complete();
    6.     }
    7.  
    8. void FixedUpdate()
    9.     {
    10.         // Wait for the previous frames ping to complete before starting a new one, the Complete in LateUpdate is not
    11.         // enough since we can get multiple FixedUpdate per frame on slow clients
    12.         m_updateHandle.Complete();
    13.  
    14.         // Update the ping client UI with the ping statistics computed by teh job scheduled previous frame since that
    15.         // is now guaranteed to have completed
    16.         PingClientUIBehaviour.UpdateStats(m_pingStats[0], m_pingStats[1]);
    17.         var pingJob = new PingJob
    18.         {
    19.             driver = m_ClientDriver,
    20.             connection = m_clientToServerConnection,
    21.             serverEP = PingClientUIBehaviour.ServerEndPoint,
    22.             pendingPings = m_pendingPings,
    23.             pingStats = m_pingStats,
    24.             fixedTime = Time.fixedTime
    25.         };
    26.         // Schedule a chain with the driver update followed by the ping job
    27.         m_updateHandle = m_ClientDriver.ScheduleUpdate();
    28.         m_updateHandle = pingJob.Schedule(m_updateHandle);
    29.     }
     
  44. echo_vr

    echo_vr

    Joined:
    May 17, 2016
    Posts:
    3
    Getting this error on Unity 2019.2.4f1 - 'Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak'.

    It only happens when I click on 'Console' in Unity when in 'Play' mode. I'm viewing a 16MB texture as a Skybox material on the Oculus Rift. Skybox works without issue until I click on the Unity 'Console' tab. Result is massive glitching of the output to the Oculus Rift, just from one click.
     
  45. echo_vr

    echo_vr

    Joined:
    May 17, 2016
    Posts:
    3
    Found a solution to my leak... the 'Console' tab and 'Game' tab were on the same window. Splitting them apart solves the issue... I can now be in 'Play' mode and click 'Console' with no leak.
     
  46. Nyanpas

    Nyanpas

    Joined:
    Dec 29, 2016
    Posts:
    145
    I also got this error in Unity 2019.2.5f1 for some unknown reason, but I restarted Unity and it went away.
     
  47. GuardHei

    GuardHei

    Joined:
    Feb 10, 2018
    Posts:
    55
    I am writing a scriptable render pipeline, and use NativeArray with Allocator.Temp (I'm sure there's no JobTemp allocator). The Jobs package is not installed, however, my console is also spammed by these warnings. I restarted Unity for over ten times and was no use.
     
  48. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,984
    Without source we can't really see what is going wrong.
     
  49. GuardHei

    GuardHei

    Joined:
    Feb 10, 2018
    Posts:
    55
    I'm using 2019.2.8f1, you can find the script here https://github.com/GuardHei/SRP/blob/master/Assets/Scripts/SRPAsset.cs
    The reason I didn't post it was simply because it was too complex.
    The code snippets that are related to nativearrays are the following:
    Code (CSharp):
    1. var allLights = cull.visibleLights;
    2. var lightIndexMap = cull.GetLightIndexMap(Allocator.Temp);
    3. var pointLights = new NativeArray<PointLight>(pointLightCount, Allocator.Temp);
    4. var spotLights = new NativeArray<SpotLight>(spotLightCount, Allocator.Temp);
    5.  
    And these are the codes I disposed them within the same method.
    Code (CSharp):
    1. // allLights.Dispose();
    2. lightIndexMap.Dispose();
    3. pointLights.Dispose();
    4. spotLights.Dispose();
    You can see I didn't dispose allLights, which is because it is a NativeArray returned by the struct CullingResult (an api for SRP), and if I dispose it, the error "Invalid Operation" gonna appear. But for all the other native collections, I dispose them all, however still receiving the "Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak" warnings.

    Thanks for helping!
     
  50. GuardHei

    GuardHei

    Joined:
    Feb 10, 2018
    Posts:
    55
    After testing multiple times, I realize it is possibly a bug of frame debugger in my case. Everything works fine after restarting the editor. However, if you enable frame debugger and click in whatever drawcall to check the info, the warnings gonna spam out your console. Then even if you exit the frame debugger, you'll still get warnings after entering the play mode, which won't happen if you haven't enabled frame debugger before
     
unityunity