Search Unity

[Jobs][Lags] JobTempAlloc has allocations that are more than 4 frames old

Discussion in 'Data Oriented Technology Stack' started by dyox, Jan 18, 2018.

  1. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    193
    Welp, please insert a meme of me eating my hat here.

    I had this in a system.

    Code (CSharp):
    1. protected override void OnCreate()
    2. {
    3.  
    4.    Verts = new NativeList<Vector3>(prevVertCapacity, Allocator.TempJob);
    5.    Tris = new NativeList<int>(prevTrisCapacity, Allocator.TempJob);
    6.    Normals = new NativeList<Vector3>(prevUVsCapacity, Allocator.TempJob);
    7.    UVs = new NativeList<Vector2>(prevNormalsCapacity, Allocator.TempJob);
    8.  
    9.    CreateCrossSections();
    10. }
    Which would work find if the system was being updated. But if the system wasn't being updated every round because it had no queries to do on, it wouldn't dispose these temp jobs and then get a leak.

    Now I've learned about OnStartRunning() and OnStopRunning() method calls in ECS.

    Thanks for the support and sorry for being a dick about insisting this is an internal Unity problem. Lucky 2019.2 gave me one more stack trace once which helped me track this down...

    Again, I will show myself out.
     
    xVergilx likes this.
  2. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    193
    I spoke too soon, about an hour after doing the above, it came back. Restarted Unity and it went away... I am going to go crazy.
     
    xVergilx likes this.
  3. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    1,906
    Your post got me an idea of checking in every TempJob usage.

    I doubt it will help, but at least it may lead to what's leaking inside hybrid render.
     
  4. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    1,432
    If you create containers in OnCreate and want them to live untill the system destroyed you should use Persistent allocator instead, and dispose them in OnDestroy.
     
  5. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    193
    Yeah, I didn't realize OnUpdate may not fire if there is nothing going on in the system. I will probably move that code to test on the first time it is created to initialize it.
     
  6. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,666
    Happy you figured something out goose (but yeah I don't think you'd want to do anything but Persistant allocations in OnCreate or OnStartRunning - with the exception being a temp allocation for initialization within the same method.)

    I'm 99% certain there are no leaks inside hybrid render.

    Have you tried turning on full stack traces?
     
    Last edited: Aug 2, 2019
  7. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    1,906
    Yep, it still doesn't say what's leaking. That's why I'm confused.
    It happens mostly when entities that are rendered moved, or rotated.
     
    siddharth3322 likes this.
  8. siddharth3322

    siddharth3322

    Joined:
    Nov 29, 2013
    Posts:
    705
    Even I am working on a small game then also I am getting this kind of log messages within the XCode.

    Example of a log:

    xcode_project_error.png
     
    Ignacii likes this.
  9. Lynxed

    Lynxed

    Joined:
    Dec 9, 2012
    Posts:
    50
    started getting those after 2019.2 migration.
    And i have no TempJob allocations.

    upload_2019-8-2_12-38-50.png

    Is there a particular place, where we have to allocate Persistent NativeCollections or can i allocate them anywhere as long as i deallocate them later?
     
  10. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    1,906
    It doesn't matter where. As long as you deallocate them at some point.

    TempJob allocs expected to live less than 4 frames, Temp - only one frame.
    So you have to deallocate them in that ranges.

    Not sure about the rest.
     
  11. dyox

    dyox

    Joined:
    Aug 19, 2011
    Posts:
    537
    Still on Unity 2019.3.0a11.

    Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak
    Internal: deleting an allocation that is older than its permitted lifetime of 4 frames (age = 15)
     
    Lynxed likes this.
  12. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    193
    Full Stack Trace won't track those warnings down unfortunately. Since I changed some code around tempjobs I have had less errors so we'll see if they pop up today.
     
    Antypodish and xVergilx like this.
  13. Lynxed

    Lynxed

    Joined:
    Dec 9, 2012
    Posts:
    50
    Can confirm having bunch of this in Unity 2019.3.0a11.
    My project uses only Allocations.Persistent
     
  14. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,666
    Have you tried disabling all your systems then enabling them 1 at a time till it appears?
     
  15. Lynxed

    Lynxed

    Joined:
    Dec 9, 2012
    Posts:
    50
    Funny thing: if you have at least one leak with your Persistent collections, you may or may not get this error randomly.
     
  16. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    193
    Oh that is wonderful :p. Must be if I kill a process without disposing properly. Since I fixed those temp problems, I haven't been getting this. It is just unfortunate that the stack trace is so cryptic.
     
  17. lilbevo

    lilbevo

    Joined:
    Aug 5, 2019
    Posts:
    3
    Welp... also getting this error but I have absolutely no mention of jobs in my single C# script in the project I just started. The error seems to show up when I preform ANY action in the editor (moving an object, placing an object, etc) besides literally clicking around folders in the heirarchy. And then it spams a bunch when I click play. Totally lost here— no idea what’s going on.
     
  18. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    193
    If you restart Unity does it go away for a bit?
     
  19. lilbevo

    lilbevo

    Joined:
    Aug 5, 2019
    Posts:
    3
    Nope :(
     
  20. lilbevo

    lilbevo

    Joined:
    Aug 5, 2019
    Posts:
    3
    Well... noticed my browser was running weirdly slow, and I realized I haven't restarted my computer in a while. It's just been on sleep mode. Restarted it and now everything's fixed! Not very knowledgeable on computer stuff but I'm assuming that not restarting for extended periods of time leads to a large decrease in usable RAM
     
  21. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,666
    Only if you're leaking memory - which you are if you get this warning!

    Yep, if you leak once you need to restart editor to get rid of it. Though I suspect most of the time this issue is caused by not completing jobs rather than allocating containers.
     
  22. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    193
    If you get a null error then that dispose call isn't called and those just build up over a lot of time.
     
  23. xVergilx

    xVergilx

    Joined:
    Dec 22, 2014
    Posts:
    1,906
    So this one is weird.

    I was getting these warnings on an old cpu that had 2 cores / 2 max threads available.
    Now that I've upgraded to a better PC I no longer get them.

    On top of that QA was not able to reproduce this one as well.

    Got no idea what would be causing this, but perhaps there was simply not enough time to execute all jobs in time.
     
  24. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    193
    I am thinking that at some point what happens is the editor doesn't complete it's shut down cycle so things aren't disposed properly in Unity's code or your own code. You then get that temp job error till you restart Unity and the problem goes away. This is why it is so hard to track down.
     
  25. Emery-Monzerol

    Emery-Monzerol

    Joined:
    Nov 1, 2012
    Posts:
    4
    Has anyone else noticed crashes/freezes that seemed linked to that warning? My team and I have noticed that our project has started freezing occasionally, maybe once every 10 runs, and that every time it does, the freeze was preceded by a large amount of this JobTempAlloc warning (something like 30 warnings approximately).

    Happens both in build and in editor, and when it happens in editor, the whole editor freezes.

    I tried attaching Visual Studio to debug and pause, but when I pause there's not even a main thread present in the Threads window. I don't believe that's supposed to even be possible? What am I missing here? Is it possible for the main thread to die off, but still have the process running, with the editor frozen in place?

    We're not even using Jobs knowingly (unless it's done by one of the packages we got from the asset store), so it's pretty darn hard to debug this issue.

    Any help would be appreciated, we're starting to run out of options here.
     
    Ignacii likes this.
  26. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    436
    Check RAM usage. Could be a memory leak by something you are using.
     
    dyox likes this.
  27. PaulUsul

    PaulUsul

    Joined:
    Nov 20, 2012
    Posts:
    11
    I know exactly how you feel, it's like it terrarizes you! I've gone over to just using persistent in active development as it's much better at telling when it's leaky and it runs faster than 1 million warnings :p
     
  28. Steve-Tou

    Steve-Tou

    Joined:
    Aug 4, 2018
    Posts:
    7
    I got a run of about 40 of these warnings. Some with a slight variation of ( x=5) or (x=6) at the end. Sorry I don't remember what x was- frames maybe? And I haven't been able to reproduce since I cleared the console. Anyways, I am running no scripts- haven't even brought one into the project-- I was just starting to test out 2019.2 Personal after having been using 2017.1 for years. I have 3 textures, 4 materials, and about a dozen of my own models and a few primitives in the scene. I was messing around with clicking things on/off in the inspector window while in play mode. Don't know if this helps anyone, but maybe it'll help someone not do too deep of a dive. Gotta be something with the editor.
     
    Morbeavus likes this.
  29. Ignacii

    Ignacii

    Joined:
    May 23, 2013
    Posts:
    16
    Still in 2019.2.3 :(
     
    Immu and dyox like this.
  30. dyox

    dyox

    Joined:
    Aug 19, 2011
    Posts:
    537
    I've found a workaround that is a little bit tricky and cost few ms (for person using jobs, not unity internal bug)

    I'm using InvokeRepeating with small update time 0.1s, i've divided my ParralelJob in small batches of xx (batchsize) schedules with a start index added to Execute index.

    Disabled index check with attribute on native container and access by :
    nativecontainer[index + startindex].

    I adapt the batch size in real time by recording the frame count at job start and end.

    I'm able to generate 8000 voxels chunks in 0.6s and 4096×4096 noises heightmap without warning.
     
  31. Immu

    Immu

    Joined:
    Jun 18, 2013
    Posts:
    144
    Still getting them in 2019.2, kinda same as Steve-Tou. It's hard to tell when they started to popup, but it was probably around 2018 came out. The moment on which they popup is a hell to understand. But I felt the particleSystem might be the reason sometimes because I was selecting it or spawing one.

    Of course, can't repro. My project is like 4 + years old and 12gb of data. The warning spawn happens out of nowhere understandable that I could make a report project from.
    And I'm using wide range of computer types.

    Can Unity do something to let us have kinda like the 'full' stack trace of that issue when it popups ?
    So we can at least have more clues of the origin.

    I've 64G of RAM by the way. If anyone's tempted to say I haven't enough XD
     
    Last edited: Sep 2, 2019
  32. Simon-O

    Simon-O

    Joined:
    Jan 22, 2014
    Posts:
    25
    "Copy around huge chunks of memory" is your preferred solution? All to prevent people taking advantage of the real benefits of multi-threading.

    The real underlying question is why?

    "We can't guarantee there won't be a race condition" is not a compelling answer. I can write code with race conditions in it right now and you couldn't detect that either... That didn't result in you pulling Unity, did it?

    Yes, if you hand someone a tool, they might stab themselves in the leg. I've yet to hear anyone suggest we ban all tools.
     
    dyox and Immu like this.
  33. 8bitgoose

    8bitgoose

    Joined:
    Dec 28, 2014
    Posts:
    193
    I also got this error in the player because my game was running a job in fixed update and the game was running TOO fast. You should be sure to set your application frame rate to 120 so that in the future as computers get faster and faster this error doesn't pop up.

    Unity should definitely suppress this warning (or have a way to suppress it) in the player. I've seen this warning pop up on Steam forums because it is happening in random games.
     
    Immu and dyox like this.
  34. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,186
    No, you'd just store it laid out that way at all times. No need to copy back and forth.

    What benefits do you think we are preventing people from taking advantage of?
     
    dyox likes this.
  35. dyox

    dyox

    Joined:
    Aug 19, 2011
    Posts:
    537
     
    Immu likes this.
  36. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,186
    @dyox what are you trying to say with that video?