Jobs that executed on the main thread with `Run()` and UnityEngine API

As we know the UnityEngine API can be executed only on the main thread. The IJobExtensions.Run() perform the job's Execute() method immediately on the same thread, so if we invoke Time.deltaTime for example within a burstified job using the OnCreate, OnUpdate, or OnDestroy in JobComponentSystem it's being executed on the main thread, but we still get: UnityException: get_deltaTime can only be called from the main thread. Therefore, we can't benefit from LLVM-compiled code by Burst which supports native methods that are UnityEngine API essentially is.

So my question is, this is the Unity runtime checking a thread ID or a call-stack incorrectly or this is an intended restriction? It prevents me from using the Burst possibilities to build a native framework for Unity, reduce interoperability overhead, and keep stuff away from Mono.

 

Here's a code for reproducing:

Code (csharp):

1. using Unity.Burst;
2. using Unity.Entities;
3. using Unity.Jobs;
4. using UnityEngine;
5.  
6. public class TestSystem : JobComponentSystem {
7.    [BurstCompile]
8.    private struct CreateJob : IJob {
9.        public void Execute() {
10.            var deltaTime = Time.deltaTime;
11.        }
12.    }
13.  
14.    [BurstCompile]
15.    private struct UpdateJob : IJob {
16.        public void Execute() {
17.            var deltaTime = Time.deltaTime;
18.        }
19.    }
20.  
21.    [BurstCompile]
22.    private struct DestroyJob : IJob {
23.        public void Execute() {
24.            var deltaTime = Time.deltaTime;
25.        }
26.    }
27.  
28.    protected override void OnCreate() {
29.        var createJob = default(CreateJob);
30.  
31.        createJob.Run();
32.    }
33.  
34.    protected override JobHandle OnUpdate(JobHandle inputDependencies) {
35.        var updateJob = default(UpdateJob);
36.  
37.        updateJob.Run();
38.  
39.        return inputDependencies;
40.    }
41.  
42.    protected override void OnDestroy() {
43.        var destroyJob = default(DestroyJob);
44.  
45.        destroyJob.Run();
46.    }
47. }

Notice that the code is produced correctly in the Burst Inspector.

 

It is a bit strange, the managed thread id and execution context in Run are the same as the main thread. But is this really an issue? I imagine in most cases api's that are not thread safe are likely mostly implemented in native code, where burst isn't really going to help anyways.

 

My main goal is to eliminate the influence of the Mono's JIT when Unity's native methods are being invoked at the runtime since P/Invoke calls themselves are costly.

 

Out of curiosity, what struct or static Unity API are you using for enough iterations where you think you can obtain measurable performance gains? The only ones I know are either already made thread-safe or can be easily replicated with Blobs.

 

I'm not using Unity's Entities, in my native framework, the API is mapped one to one with GameObject and Transform where I iterate through large batches on the main thread. I've almost finished building the Overwatch ECS equivalent where all logic is performed using custom workers and fiber-based task scheduler similarly to Naughty Dog's parallelization (Job workers are disabled). Fibers are enqueuing messages with a final shape of data to the Event Bus which is multi-producer single-consumer queue essentially, and this is where Mono's JIT is being involved to execute Unity's native methods. I want to mitigate this using Burst, but the exception that Unity's runtime is throwing not allows me to do this nor make any sense in this case.

 

To give you an idea of the interop cost, here's a basic test which just invokes native methods of Mathf:

Code (csharp):

1. using Unity.Burst;
2. using Unity.Entities;
3. using Unity.Jobs;
4. using UnityEngine;
5.  
6. public class InteropPerformanceSystem : JobComponentSystem {
7.    const int iterations = 1000000;
8.  
9.    [BurstCompile]
10.    private struct CreateJob : IJob {
11.        public void Execute() {
12.            int a = 32;
13.            float b = 32f;
14.  
15.            for (int i = 0; i < iterations; i++) {
16.                a = Mathf.ClosestPowerOfTwo(a);
17.                a = Mathf.NextPowerOfTwo(a);
18.                b = Mathf.GammaToLinearSpace(b);
19.                b = Mathf.LinearToGammaSpace(b);
20.            }
21.        }
22.    }
23.  
24.    protected override void OnCreate() {
25.        var stopwatch = new System.Diagnostics.Stopwatch();
26.        long time = 0;
27.  
28.        {
29.            var createJob = default(CreateJob);
30.  
31.            stopwatch.Restart();
32.            createJob.Run();
33.  
34.            time = stopwatch.ElapsedTicks;
35.  
36.            Debug.Log("Burst: " + time + " ticks");
37.        }
38.  
39.        {
40.            var createJob = default(CreateJob);
41.  
42.            stopwatch.Restart();
43.            createJob.Execute();
44.  
45.            time = stopwatch.ElapsedTicks;
46.  
47.            Debug.Log("Mono JIT: " + time + " ticks");
48.        }
49.    }
50.  
51.    protected override JobHandle OnUpdate(JobHandle inputDependencies) {
52.        return inputDependencies;
53.    }
54. }

AMD FX-4300

Standalone:
Burst: 703,391 ticks
Mono JIT: 1,203,632 ticks

Editor:
Burst: 1,662,773 ticks
Mono JIT: 1,615,007 ticks

 

I have no idea why Burst is producing anything for Time.deltaTime in a job. That seems weird to me. I will have to look later to see how the asm attempts to fetch that value.

But also, I really don't understand how you would get a performance boost on individual functions avoiding interop when calling a Burst-compiled function would also produce such an interop. The only way you would get any real speedup is by batching all your work in a job and then writing out the output. .Run() is perfect for that.

Lastly, besides deltaTime which is an easily cache-able variable, and Mathf which if you want Burst you should opt for the Unity.Mathematics library instead, is there any other Unity API that could even take advantage of Burst in your game? Unity does not expose their GameObject API in a native context that I am aware of, and I doubt they ever will.

 

Time.deltaTime is static and Burst can not access to static variables. Perhaps wrong error message?

 

Time.deltaTime is a property that call a C++ function, not a field.

 

It's not a burst thing, it throws without burst also.

 

So far i now, it's not only field but all sort of "static" data. (except static method which only work with the arguments)

All Job examples copy the deltatime.

Maybe some job preparation in the scheduler work on another thread?

 

@DreamingImLatios As @Guerro323 said Time.deltaTime is essentially a native function. Burst is now able to link with shared libraries and it supports DLLImport. Roughly speaking, LLVM-compiled code is faster since everything is done natively while the Mono runtime invokes unmanaged functions using the virtual machine with some additional safety checks on top via stack walking. Interoperability in .NET is especially costly if the marshaling is being involved when non-blittable types are used in method arguments.

@JakeTurner As Chris said above, Burst is not the reason, the problem sits in the Unity's runtime itself which checks a thread ID or a call-stack incorrectly since a thread remains the same, but the exception is still there.

 

If this issue will be solved, it will be possible to add overloads to UnityEngine API where blittable pointers will be used instead of reference types, then utilize built-in memory allocator, and implement new safety checks to burstify 90% of traditional engine's API and escape from MonoBehaviour as well as from Mono itself. This requires a certain amount of work, but it's definitely possible.

 

This was the disconnect for me. I'm very curious what this looks like and how you access the native APIs instead of the Mono ones. Have you tried it with any of the thread-safe class types?

 

What I'm doing is just exploiting Unity's assembly and Burst-generated shared library to get access to native functionality which bound to internal headers. We have two options to make everything blittable: one is spartan which requires to edit the compiled managed assembly, and another is more elegant since we can exploit the way how Unity is handling interoperability, but we will need to write/generate a lot more code for reflection instead of adding a simple overload.

When it's done, Burst compiles every single function natively as a job and then exporting entry points for dynamic linking. At startup, a structure with all required pointers is composed and passed as a pointer to the native plugin which de-references it and keeps a copy statically during a session. I'm using there dlopen() / LoadLibrary() and dlsym() / GetProcAddress() to load any required shared library. Then, native API invokes the appropriate pointers as functions so essentially UnityEngine.Time.frameCount becomes int (SYMBIOTIC_FUNCTION *frameCount)(void); for example. The prototype of C function looks like this:

Code (csharp):

1. int symbiotic_time_framecount(void) {
2.    return symbiotic.time.frameCount();
3. }

Events remains on the main thread as jobs executed with Run() within a system:

Code (csharp):

1. [BurstCompile]
2. private struct CreateJob : IJob {
3.    public Symbiotic symbiotic;
4.  
5.    public void Execute() {
6.        if (Native.LoadMain(ref symbiotic))
7.            Native.Awake();
8.  
9.        Native.Start();
10.    }
11. }
12.  
13. [BurstCompile]
14. private struct UpdateJob : IJob {
15.    public void Execute() {
16.        Native.Update(UnityEngine.Time.deltaTime);
17.    }
18. }
19.  
20. [BurstCompile]
21. private struct DestroyJob : IJob {
22.    public void Execute() {
23.        Native.Destroy();
24.        Native.UnloadMain();
25.    }
26. }

C prototypes in the native plugin looks like this:

Code (csharp):

1. void symbiotic_awake(void) {
2.    if (events.awake != NULL)
3.        events.awake();
4. }
5.  
6. void symbiotic_start(void) {
7.    if (events.start != NULL)
8.        events.start();
9. }
10.  
11. void symbiotic_update(float deltaTime) {
12.    if (events.update != NULL)
13.        events.update(deltaTime);
14. }
15.  
16. void symbiotic_destroy(void) {
17.    if (events.destroy != NULL)
18.        events.destroy();
19. }

C implementation of custom logic which compiled as a separate shared library:

Code (csharp):

1. #define SYMBIOTIC_MAIN
2.  
3. #include "symbiotic.h"
4.  
5. void symbiotic_awake(void) {
6.    symbiotic_debug_log_info("Awake");
7.  
8.    // Assert test
9.    #ifdef SYMBIOTIC_DEBUG
10.        symbiotic_debug_log_info("Testing assert:");
11.    #endif
12.  
13.    SYMBIOTIC_ASSERT(1 < 0);
14.    SYMBIOTIC_ASSERT_MESSAGE(1 < 0, "Assert message");
15.  
16.    // Time test
17.    symbiotic_debug_log_info("Testing time:");
18.    symbiotic_debug_log_info(symbiotic_string_format("Frame count: %i", symbiotic_time_framecount()));
19.  
20. }
21.  
22. void symbiotic_start(void) {
23.    symbiotic_debug_log_info("Start");
24. }
25.  
26. void symbiotic_update(float deltaTime) {
27.  
28. }
29.  
30. void symbiotic_destroy(void) {
31.    symbiotic_debug_log_info("Destroy");
32. }

Compiled with GCC and executed right in the Unity:


It works well and fast. The only problem is the strange exception which, I hope, will be fixed.

 

A native framework is just my personal preference. It's possible to wrap it in Go/Java/Lua, you name it (with a bit of C), and it's also possible to burstify the UnityEngine API in C# itself, but it's quite tricky for not in-house developers since we only have C# code for reference and the engine's source code is closed with its internal headers.

 

That makes a lot of sense. Seems like a good approach for a team that already has a solid C++ codebase but wants to switch to Unity for authoring and presentation.

The part that I'm not sure about is how you bind class types in a Burst job to get direct access to them in native. Have you tested this approach with a ThreadSafe class type like AnimationCurve?

 

Well, I just did some basic tests. Haven't tried the AnimationCurve yet, but it should work fine. I'm looking at native method signatures there, this is standard interop, we only need to replace all reference types and use a custom memory allocator for Keyframe arrays so no need any marshaling, just pass blittable pointers instead. I'm using the same approach with ENet for example here. Look at enet_packet_create_offset() where one overload accepts a managed array and another a blittable pointer. Simple as that you get the same bit-level representation in managed and native code of any arguments.

You don't need to bind any abstracted classes of Unity on top of native functionality, you only need the native method signatures themselves and then you abstract them the way you like and not necessarily the same way as Unity did. This requires some excavation of the managed assembly or exploiting the way how Unity is handling interoperability and write/generate the reflection code yourself. Notice in the example above that Debug.Log() is invoked from the native code within a burstified job which is not possible in Unity since it requires a managed object as an argument.

 

By the way, if you don't care about Mono, interop cost, and you prefer the same API as Unity's, you can use this project for native scripting which designed for C++ primarily. I'm just trying a different approach using possibilities of Burst to stay closer as possible to the engine, and I'm doing that primarily for C11.

 

I'm well aware of that project. Before Jobs and Burst I was looking for a solution to do raycasting against an animated mesh for thousands of raycasts, and so I played around a bit with C++ and C# interop. But for me, Jobs and Burst completely solved that problem, so I haven't investigated much how interop plays with Burst.

Apologies in advance if I completely missed the point, but my understanding is that you are using Burst to generate access points to the Unity API from a native context, and by having a Burst job call into your native code, you can hook into the Burst-generated header to talk to Unity APIs in a native context.

However, your simple test cases have only tested struct types and static methods, things that Burst can find in a native context and bring in for you. But you stated that you needed to talk to GameObjects and Components, since that is what you are using for presentation. Those are managed data types with an additional native Unity backend. That may give Burst a lot of trouble. You can't test those right now because of this main-thread lockout issue. But what you can do is test a Thread-Safe managed type with a native Unity backend.

So if you get Burst to access AnimationCurve methods without interop, then really the only thing blocking you is the main thread issue. But my suspicion is that Burst is going to start screaming at you when you try to work with AnimationCurve or GameObjects.

 

Yes, this is correct. As I said any managed type is just how Unity abstracted native functionality, but nothing stops you to change that, we have at least two options. I already did that with Debug.Log() for example, which again, you can't use within a burstified job in Unity with the original API.

 

Thanks. I totally missed that with Debug.Log. Also that means that you figured out how to Debug.Log in a Burst job.

But I am apparently wrong about something. I didn't think Burst would generate access points to things like Debug.Log, so either I am wrong about that or you found a different way to access those native functions in a native context. In the latter case, I don't understand what the benefit is of using Burst to launch your native context versus just launching your own (and potentially avoiding the thread-lock issue). The overhead of Run() is 6-7 microseconds on my rig, so its not like it magically removes the initial interop cost.

Anyways, if you don't feel like explaining this to my incompetent self, don't bother. Besides Debug.Log and Playables, any use case I would have for this (live syncing with other applications) I could just build the binding in the other direction.

 

No problem, I think it would be easier to understand everything if I just release the code or write an article with all the details. If you measure anything out of burstified jobs in Unity, you measure it with Mono, as well as the Run() itself which doesn't matter since we manipulate pointers and entry points natively.

 

@JakeTurner I've tried to add [NativeProperty(IsThreadSafe = true)] and [ThreadAndSerializationSafe] to the Time.deltaTime similarly to other API using the assembly editor and the exception is still there.

 

After some digging found that Unity's runtime is checking a thread ID using GetCurrentThreadID() on Windows. Assuming that an ID of the main thread is assigned statically at initialization using the same function, this shouldn't be an issue. Here's another script for reproducing without Burst:

Code (csharp):

1. using System.Runtime.InteropServices;
2. using Unity.Entities;
3. using Unity.Jobs;
4. using UnityEngine;
5.  
6. public class ThreadIDTestSystem : JobComponentSystem {
7.     [DllImport("kernel32.dll")]
8.     public static extern uint GetCurrentThreadId();
9.  
10.     private static uint threadID;
11.  
12.     private struct CreateJob : IJob {
13.         public void Execute() {
14.             threadID = GetCurrentThreadId();
15.  
16.             var deltaTime = Time.deltaTime;
17.         }
18.     }
19.  
20.     protected override void OnCreate() {
21.         var createJob = default(CreateJob);
22.  
23.         createJob.Run();
24.  
25.         if (threadID == GetCurrentThreadId())
26.             Debug.Log("Thread ID is the same!");
27.         else
28.             Debug.Log("Thread ID is not the same!");
29.     }
30.  
31.     protected override JobHandle OnUpdate(JobHandle inputDependencies) {
32.         return inputDependencies;
33.     }
34. }