[Jobs][Lags] JobTempAlloc has allocations that are more than 4 frames old

Welp, please insert a meme of me eating my hat here.

I had this in a system.

Code (CSharp):

1. protected override void OnCreate()
2. {
3.  
4.    Verts = new NativeList<Vector3>(prevVertCapacity, Allocator.TempJob);
5.    Tris = new NativeList<int>(prevTrisCapacity, Allocator.TempJob);
6.    Normals = new NativeList<Vector3>(prevUVsCapacity, Allocator.TempJob);
7.    UVs = new NativeList<Vector2>(prevNormalsCapacity, Allocator.TempJob);
8.  
9.    CreateCrossSections();
10. }

Which would work find if the system was being updated. But if the system wasn't being updated every round because it had no queries to do on, it wouldn't dispose these temp jobs and then get a leak.

Now I've learned about OnStartRunning() and OnStopRunning() method calls in ECS.

Thanks for the support and sorry for being a dick about insisting this is an internal Unity problem. Lucky 2019.2 gave me one more stack trace once which helped me track this down...

Again, I will show myself out.

 

I spoke too soon, about an hour after doing the above, it came back. Restarted Unity and it went away... I am going to go crazy.

 

Your post got me an idea of checking in every TempJob usage.

I doubt it will help, but at least it may lead to what's leaking inside hybrid render.

 

If you create containers in OnCreate and want them to live untill the system destroyed you should use Persistent allocator instead, and dispose them in OnDestroy.

 

Yeah, I didn't realize OnUpdate may not fire if there is nothing going on in the system. I will probably move that code to test on the first time it is created to initialize it.

 

Happy you figured something out goose (but yeah I don't think you'd want to do anything but Persistant allocations in OnCreate or OnStartRunning - with the exception being a temp allocation for initialization within the same method.)

I'm 99% certain there are no leaks inside hybrid render.

Have you tried turning on full stack traces?

 

Yep, it still doesn't say what's leaking. That's why I'm confused.
It happens mostly when entities that are rendered moved, or rotated.

 

Even I am working on a small game then also I am getting this kind of log messages within the XCode.

Example of a log:

 

started getting those after 2019.2 migration.
And i have no TempJob allocations.



Is there a particular place, where we have to allocate Persistent NativeCollections or can i allocate them anywhere as long as i deallocate them later?

 

It doesn't matter where. As long as you deallocate them at some point.

TempJob allocs expected to live less than 4 frames, Temp - only one frame.
So you have to deallocate them in that ranges.

Not sure about the rest.

 

Still on Unity 2019.3.0a11.

Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak
Internal: deleting an allocation that is older than its permitted lifetime of 4 frames (age = 15)

 

Full Stack Trace won't track those warnings down unfortunately. Since I changed some code around tempjobs I have had less errors so we'll see if they pop up today.

 

Can confirm having bunch of this in Unity 2019.3.0a11.
My project uses only Allocations.Persistent

 

Have you tried disabling all your systems then enabling them 1 at a time till it appears?

 

Funny thing: if you have at least one leak with your Persistent collections, you may or may not get this error randomly.

 

Oh that is wonderful . Must be if I kill a process without disposing properly. Since I fixed those temp problems, I haven't been getting this. It is just unfortunate that the stack trace is so cryptic.

 

Welp... also getting this error but I have absolutely no mention of jobs in my single C# script in the project I just started. The error seems to show up when I preform ANY action in the editor (moving an object, placing an object, etc) besides literally clicking around folders in the heirarchy. And then it spams a bunch when I click play. Totally lost here— no idea what’s going on.

 

If you restart Unity does it go away for a bit?

 

Nope

 

Well... noticed my browser was running weirdly slow, and I realized I haven't restarted my computer in a while. It's just been on sleep mode. Restarted it and now everything's fixed! Not very knowledgeable on computer stuff but I'm assuming that not restarting for extended periods of time leads to a large decrease in usable RAM

 

Only if you're leaking memory - which you are if you get this warning!

Yep, if you leak once you need to restart editor to get rid of it. Though I suspect most of the time this issue is caused by not completing jobs rather than allocating containers.

 

If you get a null error then that dispose call isn't called and those just build up over a lot of time.

 

So this one is weird.

I was getting these warnings on an old cpu that had 2 cores / 2 max threads available.
Now that I've upgraded to a better PC I no longer get them.

On top of that QA was not able to reproduce this one as well.

Got no idea what would be causing this, but perhaps there was simply not enough time to execute all jobs in time.

 

I am thinking that at some point what happens is the editor doesn't complete it's shut down cycle so things aren't disposed properly in Unity's code or your own code. You then get that temp job error till you restart Unity and the problem goes away. This is why it is so hard to track down.

 

Has anyone else noticed crashes/freezes that seemed linked to that warning? My team and I have noticed that our project has started freezing occasionally, maybe once every 10 runs, and that every time it does, the freeze was preceded by a large amount of this JobTempAlloc warning (something like 30 warnings approximately).

Happens both in build and in editor, and when it happens in editor, the whole editor freezes.

I tried attaching Visual Studio to debug and pause, but when I pause there's not even a main thread present in the Threads window. I don't believe that's supposed to even be possible? What am I missing here? Is it possible for the main thread to die off, but still have the process running, with the editor frozen in place?

We're not even using Jobs knowingly (unless it's done by one of the packages we got from the asset store), so it's pretty darn hard to debug this issue.

Any help would be appreciated, we're starting to run out of options here.

 

Check RAM usage. Could be a memory leak by something you are using.

 

I know exactly how you feel, it's like it terrarizes you! I've gone over to just using persistent in active development as it's much better at telling when it's leaky and it runs faster than 1 million warnings

 

I got a run of about 40 of these warnings. Some with a slight variation of ( x=5) or (x=6) at the end. Sorry I don't remember what x was- frames maybe? And I haven't been able to reproduce since I cleared the console. Anyways, I am running no scripts- haven't even brought one into the project-- I was just starting to test out 2019.2 Personal after having been using 2017.1 for years. I have 3 textures, 4 materials, and about a dozen of my own models and a few primitives in the scene. I was messing around with clicking things on/off in the inspector window while in play mode. Don't know if this helps anyone, but maybe it'll help someone not do too deep of a dive. Gotta be something with the editor.

 

Still in 2019.2.3

 

I've found a workaround that is a little bit tricky and cost few ms (for person using jobs, not unity internal bug)

I'm using InvokeRepeating with small update time 0.1s, i've divided my ParralelJob in small batches of xx (batchsize) schedules with a start index added to Execute index.

Disabled index check with attribute on native container and access by :
nativecontainer[index + startindex].

I adapt the batch size in real time by recording the frame count at job start and end.

I'm able to generate 8000 voxels chunks in 0.6s and 4096×4096 noises heightmap without warning.

 

Still getting them in 2019.2, kinda same as Steve-Tou. It's hard to tell when they started to popup, but it was probably around 2018 came out. The moment on which they popup is a hell to understand. But I felt the particleSystem might be the reason sometimes because I was selecting it or spawing one.

Of course, can't repro. My project is like 4 + years old and 12gb of data. The warning spawn happens out of nowhere understandable that I could make a report project from.
And I'm using wide range of computer types.

Can Unity do something to let us have kinda like the 'full' stack trace of that issue when it popups ? So we can at least have more clues of the origin.

I've 64G of RAM by the way. If anyone's tempted to say I haven't enough XD

 

"Copy around huge chunks of memory" is your preferred solution? All to prevent people taking advantage of the real benefits of multi-threading.

The real underlying question is why?

"We can't guarantee there won't be a race condition" is not a compelling answer. I can write code with race conditions in it right now and you couldn't detect that either... That didn't result in you pulling Unity, did it?

Yes, if you hand someone a tool, they might stab themselves in the leg. I've yet to hear anyone suggest we ban all tools.

 

I also got this error in the player because my game was running a job in fixed update and the game was running TOO fast. You should be sure to set your application frame rate to 120 so that in the future as computers get faster and faster this error doesn't pop up.

Unity should definitely suppress this warning (or have a way to suppress it) in the player. I've seen this warning pop up on Steam forums because it is happening in random games.

 

"This" issue started for me today. I have been using Unity 2019.1.14f1 for almost as long as it's available, and I'm not aware of any changes in my system. What I did do was disconnect the Oculus Rift and re-connect the Vive Pro - but I had done this many times before.

I had also opened the Interaction System demo scene of the SteamVR Unity Plugin. That's where I saw it the first time, so I assumed something was wrong in that scene. Changed to my project's default scene. Issue still happened. Restarted Unity. Rebooted Windows 10. Opened another project. Created a new empty project.

The only difference that I'm seeing is that in my main project, this issue spams the log as soon as I open Unity. In the other project, and in the new project, it starts when I hit play, and it stops when I exit play. These two projects are not VR projects.

The warnings I'm getting:

Code (CSharp):

1. Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak
2. To Debug, enable the define: TLA_DEBUG_STACK_LEAK in ThreadsafeLinearAllocator.cpp. This will output the callstacks of the leaked allocations
3.  

That's all I'm getting. No stacktrace or anything. I have also tried Stacktrace Logging / All / Full - still only one line for each entry.

 

I have now also filed a bug-report for this:

(Case 1186264) Warnings spamming the console in empty project: JobTempAlloc has allocations that / enable: TLA DEBUG STACK LEAK

As this hasn't happened on my system until about an hour ago, and it's now happening in an empty project that I have just created, it looks like this might be something on my system. I'm not using ECS or the job-system.

 

... and ... after another reboot, it's gone. In all of the projects. I'm really glad now that I had filed a bug report while it was there. Hopefully the logs in the bug report have useful information that helps tracking this down. What a strange issue.

 

@jashan What probably happened was Unity stopped at a bad time or you had a null error which prevented one of Unity's internal jobs from properly disposing itself. Once you reset everything it cleared that warning and it was gone.

 

The strange thing was that the warning spam persisted over a full reboot, and over closing Unity and creating a new, empty project. And now, I can't reproduce it at all.

I won't complain - this was spooky and I prefer spending my time working on my project over chasing down mysterious warnings. But I would still love to know how something like this can happen. Going by your explanation, if I understand it correctly, it would mean that Unity does something to persistently store jobs so that they could remain and cause trouble even after closing / re-opening Unity, and even after a reboot. I'm not saying this is impossible - I'm aware that Unity does launch separate processes and apparently doesn't always properly close them when closing itself (that's the reason why Steam will often show the Unity game as "playing" even after shutting down Unity).

I believe I first opened the project where the issue occurred the first time after the reboot, so something may have been stuck there (making it persist over the reboot), and then it could theoretically still be stuck when closing Unity to open a new project (due to Unity launching separate processes and not closing them properly). But then, the first time I saw it it kept spamming the log while not in play mode but in the new project, it only spammed the log while in play mode.

The next thing I would have tried would have been another Unity version ... but I'm glad it's gone now and hope it will never return (if it does, I'll make sure to file a bug report for each time I run Unity where it occurs).

 

I probably shouldn't have said that. So here's the full list of bug reports I filed on this issue now (the first should include the list, I replied to the original bug report autoresponder):

(Case 1186264) <--- original report
(Case 1186448)
(Case 1186450)
(Case 1186451)
(Case 1186454)
(Case 1186456)

The only thing I know now is that this didn't happen when I created a new project in Unity 2019.2.5, so either the issue I'm seeing is fixed in Unity 2019.2.5, OR whatever gets stuck is version specific, so the 2019.2.5 installation is not effected when it happens on 2019.1.14. This time, Unity crashed, then I rebooted, then I saw those warnings. Then I tried several things and as promised, filed a bug report each time.

I really hope the logfiles will contain useful information.

I also know that while it seems that first starting Steam, SteamVR, JetBrains Rider and Visual Studio before starting Unity makes this happen after the reboot, I can also start all of these without the issue happening. And starting Unity first, and then starting all of these before starting playing, I once saw those warnings start during my initialization. But the first time I saw this, it happened in a completely different scene that's not even from my project (the SteamVR Interaction System demo scene). I haven't seen it in that scene again (and as mentioned earlier, I have also seen it in completely empty new projects).

 

That general pattern can and has been caused by a lot of things going back several years.

I'm seeing it now linked to the alloc warnings and terrain lightmapping when going into play mode. Editor log shows gi::BakeBackendSwitch then hundreds of alloc warnings before freezing. When it hits it freezes the editor and then after 10-15 seconds Unity goes from normal cpu usage to zero cpu usage.

Consistency seems to be tied to the overall profile. Ie it reliably freezes in our main game scene entering play mode where there are a bunch of heavy jobs that run for 2-3 seconds. In a simpler test scene it's completely infrequent. But the same editor log patterns so same core problem

In the current scenario if I disable the terrain it doesn't trigger, and then it doesn't trigger afterwards either even with the terrain enabled. So it seems to be logic that's a first time the scene is loaded thing.

 

Just a little heads-up if you are scheduling jobs in FixedUpdate. You might get TempAlloc warnings if you don't Complete the job in both LateUpdate and FixedUpdated, as explained here in Unity's netcode sample project code:

Code (CSharp):

1. void LateUpdate()
2.     {
3.         // On fast clients we can get more than 4 frames per fixed update, this call prevents warnings about TempJob
4.         // allocation longer than 4 frames in those cases
5.         m_updateHandle.Complete();
6.     }
7.  
8. void FixedUpdate()
9.     {
10.         // Wait for the previous frames ping to complete before starting a new one, the Complete in LateUpdate is not
11.         // enough since we can get multiple FixedUpdate per frame on slow clients
12.         m_updateHandle.Complete();
13.  
14.         // Update the ping client UI with the ping statistics computed by teh job scheduled previous frame since that
15.         // is now guaranteed to have completed
16.         PingClientUIBehaviour.UpdateStats(m_pingStats[0], m_pingStats[1]);
17.         var pingJob = new PingJob
18.         {
19.             driver = m_ClientDriver,
20.             connection = m_clientToServerConnection,
21.             serverEP = PingClientUIBehaviour.ServerEndPoint,
22.             pendingPings = m_pendingPings,
23.             pingStats = m_pingStats,
24.             fixedTime = Time.fixedTime
25.         };
26.         // Schedule a chain with the driver update followed by the ping job
27.         m_updateHandle = m_ClientDriver.ScheduleUpdate();
28.         m_updateHandle = pingJob.Schedule(m_updateHandle);
29.     }

 

Getting this error on Unity 2019.2.4f1 - 'Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak'.

It only happens when I click on 'Console' in Unity when in 'Play' mode. I'm viewing a 16MB texture as a Skybox material on the Oculus Rift. Skybox works without issue until I click on the Unity 'Console' tab. Result is massive glitching of the output to the Oculus Rift, just from one click.

 

Found a solution to my leak... the 'Console' tab and 'Game' tab were on the same window. Splitting them apart solves the issue... I can now be in 'Play' mode and click 'Console' with no leak.

 

I also got this error in Unity 2019.2.5f1 for some unknown reason, but I restarted Unity and it went away.

 

I am writing a scriptable render pipeline, and use NativeArray with Allocator.Temp (I'm sure there's no JobTemp allocator). The Jobs package is not installed, however, my console is also spammed by these warnings. I restarted Unity for over ten times and was no use.

 

Without source we can't really see what is going wrong.

 

I'm using 2019.2.8f1, you can find the script here https://github.com/GuardHei/SRP/blob/master/Assets/Scripts/SRPAsset.cs
The reason I didn't post it was simply because it was too complex.
The code snippets that are related to nativearrays are the following:

Code (CSharp):

1. var allLights = cull.visibleLights;
2. var lightIndexMap = cull.GetLightIndexMap(Allocator.Temp);
3. var pointLights = new NativeArray<PointLight>(pointLightCount, Allocator.Temp);
4. var spotLights = new NativeArray<SpotLight>(spotLightCount, Allocator.Temp);
5.  

And these are the codes I disposed them within the same method.

Code (CSharp):

1. // allLights.Dispose();
2. lightIndexMap.Dispose();
3. pointLights.Dispose();
4. spotLights.Dispose();

You can see I didn't dispose allLights, which is because it is a NativeArray returned by the struct CullingResult (an api for SRP), and if I dispose it, the error "Invalid Operation" gonna appear. But for all the other native collections, I dispose them all, however still receiving the "Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak" warnings.

Thanks for helping!

 

After testing multiple times, I realize it is possibly a bug of frame debugger in my case. Everything works fine after restarting the editor. However, if you enable frame debugger and click in whatever drawcall to check the info, the warnings gonna spam out your console. Then even if you exit the frame debugger, you'll still get warnings after entering the play mode, which won't happen if you haven't enabled frame debugger before