Unity Future .NET Development Status

Nice, I haven't seen it before.

 

What I mean is a unity release independent from the main alpha/beta/tech stream/lts stuff. A unity version that can be installed (from the unity hub or from a separate installer) and tried out. One where they are not promising any stability but more an idea of how this will work. A sneak-peek version if you will. I say this as I understand that merging such a big change upstream from their current branch will take time. So this would be built from an internal working branch.

This is the old input system post I was mentioning:
https://forum.unity.com/threads/welcome-new-input-system-resources-and-info-please-read.397153/

 

I would assume alpha could be used for this as there is nothing to say an alpha is expected to be stable. But maybe an additional opt-in experimental build might not be the worst idea.

I would much rather see a world where we just grab an experimental version of a package than the whole editor/engine. Like with what you linked, it makes far more sense to me to just get an experimental version of the Input package than a custom version of the editor to support that.

I know Unity have been abstracting a lot of things away from the core editor/engine and into packages so I assume this is something we might see down the line.

 

Aren’t we already seeing it? I’m very content with how quickly Unity let’s you test and experiment with their packages. I remember trying out the UIBuilder (for UIElements) when it was still in 0.5 or something.

 

Just found this gem and I don't understand how it can be unlisted. I come from the .Net ecosystem and work with it since 10 years now (made all kind of apps with it, from Windows Forms to Blazor) and it's really hard to switch to Unity everyday because it's such a different environment while it shouldn't be. Can't wait to have everything listed in the video, it will save us SO MUCH TIME.

 

I wasn't actually aware you were also working on (essentially) making the Burst compiler faster!

Something I was missing in the blog, even though it's stated here before I wanted to make sure: once the migration to the CoreCLR has completed, upgrading the API in later years (e.g. from .NET 8.0 to 9.0) will be as easy as flipping a switch, and we can expect Unity to keep up with the latest C# runtime. Is this still true?

I wish you all the best over the coming years working on this Josh. I don't know about national heroes but if you and your team pulls this off I will personally break into the pentagon and hang your framed profile picture in their hall of fame.

 

Boo!

async/await is absolutely horrible for coroutine use. It's a tool made for a very different problem than coroutines, so the cost/benefit of using Tasks over Coroutines are just so far tipped towards costs that it's obviously a horrible idea.

So the upsides to Tasks compared to coroutines for what we have always used coroutines for are:
- Can return values
- Familiar to .NET developers

And the downsides are:
- CancellationTokens that have to be allocated and checked in every single wait instead of StopCoroutine
- Call stacks with so much garbage that they're essentially useless whenever an error happens
- If we forget to stop them, they last forever, causing performance issues, and error spam outside play mode.
- There's now two different ways of doing the exact same thing in the engine, again.

I don't think giving MonoBehaviours a CancellationToken was a good idea. It's essentially trying to shoehorn a very bad solution to an already solved problem into an engine that doesn't need it.


Don't get me wrong, I think Tasks and async/await are great tools for the job they are designed for. That job is to wait for resources or procedures running on a different thread and/or computer - IO operations, network operations, multi-threading. Using it for single-threaded continuation of a procedure on the next frame is very much a bad fit, and the end-result when using them are much worse.

I mean, it's obvious from the forums that there's a lot of users that are convinced that Tasks are a great tool for what Coroutines do. They're wrong! But that's fine, help them write worse code a bit easier. But at the very least don't "move away from" the better tool. That's just sabotaging the engine.

 

I don't know exactly what you mean that you have to allocate a CancelationToken for each wait? You can use a CancellationTokenSource for multiple async calls and then reset and reuse it.

 

> There's now two different ways of doing the exact same thing in the engine, again.

It will be only one if we just stop using Coroutine

The problem might be that we need to have task-like object for unity instead of using `Task<T>` for the unity API (AwaitableCoroutine maybe?). I see your point of problem is not about async/await by itself. If, instead, there would be a Task-like object that will consume CancellationToken by itself in the background then your argument would be solved

In other words. If the underlying system is the Coroutine but the syntax just become async/await instead, I think you would not mind it that much?

What I see is Task and async/await is just abstraction to the same purpose as Coroutine. And it just become syntax so it easier to use in C# and Coroutine just more align with unity while Task was not designed with unity in their mind. But we might be able to bridge that gap, especially with custom task-like object

 

Just clunky writing on my part. I mean "CancellationTokens have to be allocated" and "CancellationTokens checked in every single wait"

Yeah, you could improve this by having a special, Unity-centric Task that was tied to object's lifetimes, and then instead of having tokens, the thing that handled running Tasks on the back-end would be able to cancel them from the outside (ie. stop running them) and do that automatically when the object was destroyed.

In other words, you could spend a ton of engineering effort on recreating coroutines, instead of spending that effort on something worthwhile.

 

CancellationToken is defined as readonly struct with one 8 byte field, so it doesn't have to be allocated in heap (no GC pressure) and basically free to pass (size of 64-bit pointer).

C# stack traces are a bit gibberish even without async. I mean constructors, IEnumerators, linq and so on.
This part can be solved with tools like https://github.com/benaadams/Ben.Demystifier

Custom task-like object can solve problems with binding to MonoBehaviour lifetime, I see no problems there.

 

It separate

We already have the Coroutine "system" implemented in unity. It just a platform specific Task system and there was common in dotnet ecosystem to have platform specfic feature
We only need to implement the async over Coroutine. And its worth would be that it adapt the coroutine system into the async/await pattern and so it would streamline with dotnet ecosystem while also compatible with other task system

As the Josh's words in video. Coroutine is actually not free. It need to be added and need to be maintained for the whole system, like in the past we have unityscript and boo. Now they could relegate some of responsibility into dotnet async/await system when we stop using Coroutine

To be fair. Both Coroutine and Task is actually for the same in job and purpose. The Coroutine and Task are not recreation of each other. I can accept that it was equivalence. But sadly it about majority of dotnet ecosystem was using async/await while Coroutine is just us minority using unity. And so we just need to follow standard. Because its too hard to change majority to follow minority

The problem is as you say. In actuality there was two different ways of doing the exact same thing in engine. async/await was already lurking into unityengine for many things for such times. And the root cause is the whole dotnet ecosystem is actually use async/await. Almost every C# codebase use that. And we just frustratingly need to adapt it into Coroutine

By convert Coroutine to be async/await we then reduce the amount of ways back to one

 

I agree only with stack traces being bad but they got improved in net core 2.1 https://www.ageofascent.com/2018/01/26/stack-trace-for-exceptions-in-dotnet-core-2.1/ meaning they will be improved in Unity with NET 6.

You can make a bool, change it in OnDestroy and check that instead of allocating a token. Anyway:
StartCoroutine allocates 64 B (every time it's used)
new CancellationTokenSource() allocates 64 B (once per game object)
TestAsync().Forget(); allocates 0 B (UniTask method)

So in the worst case with UniTask you'll get same allocation as in the best case with Coroutines (starting only one of them).

Another performance example:

Spoiler: code

Code (CSharp):

1. void Update()
2. {
3.     for(int i = 0; i < 10000; i++)
4.     {
5.         // StartCoroutine(TestIE());
6.         TestAsync().Forget();
7.     }
8. }
9. WaitForSeconds waitCached = new WaitForSeconds(0.001f);
10. // 40fps
11. IEnumerator TestIE()
12. {
13.     yield return waitCached;
14.     a++;
15. }
16. // 100fps
17. async UniTask TestAsync()
18. {
19.     await UniTask.Delay(1);
20.     if(gameObject == null)
21.         return;
22.     a++;
23. }

UniTask 100fps:


Coroutines 40fps and gc going crazy:

Not the case for UniTask. In editor dangling tasks seems to not run and are removed on GC.Collect.

It's not the same and async/await are compatible with way more apis/libraries. You even get power of doing both single and multithreaded tasks:

Code (CSharp):

1. async UniTask Test()
2. {
3.     await UniTask.SwitchToThreadPool();
4.     // do some heavy work
5.     await UniTask.SwitchToMainThread();
6.     // use Unity apis
7. }

 

Actually, they were made for the same purpose at around the same time. Which was handling asynchronous operations. Tasks just did it better with threading in mind. And as others have pointed out, Coroutines are more costly than Tasks.

You forgot:
- Can handle errors, with async stack traces
- Can use pooled objects for less GC pressure
- Can run on background threads

At least you get async call stacks when an error happens. You don't get anything in Coroutines.
Threads continuing to run in the background in editor when you exit playmode is an editor issue, not a Tasks issue. If tasks continue to run forever in any other context, that's user error, and you can do the same thing with Coroutines.
There has been Coroutines and Tasks in the engine for a while now, ever since Unity updated to .Net 4/ C# 7. I don't see the issue here.

You seem to have a very bad impression of CancelationTokens. They serve a very specific purpose: canceling chained asynchronous operations. Have you ever needed to cancel chained Coroutines? It's a nightmare!

Also, the ability to cancel operations you don't control (just pass the token into the API for some library).

Agree to disagree.

The only thing Coroutines can do that Tasks can't do is pause/resume.

 

Whenever I witness a debate in text about async/await vs coroutines, its the essentially background noise to me at this point (and i didnt think id be in this forum).

I've used both extensively and async/await is def better and more fun, and that goes for a lot of new modern C# features that solve old problems. Like nullable reference types, which I hope theyre being considering by unity and what they could do inspector/editor/analyzer wise to guarantee the assumptions (no null values unless suffixed with ?)

 

Why was it better? Just curious to your point of view, I’ve only ever used Coroutines.

 

I would argue that you can do that fairly effectively with TaskCompletionSource. Just allocate one and store it away somewhere, then await it in your task. When you want to continue just call `SetResult` on the TCS.

 

@CaseyHofland and for anyone who might be on the fence or doesn't know how to use async/await patterns in unity, I found this Unite speech to be an excellent video on the topic.

Summary: There is a time and place for each, with benefits and drawbacks. Sit back and watch it during your lunch hour, it is worthwhile.

Regarding the overall upgrade, I am excited for the devs at Unity to upgrade to some modern C# features! Hopefully it won't take another 5 years

 

Right, I should've said it's much easier to pause/resume coroutines. Coroutines will automatically stop execution between yields. Of course you can do that with tasks, but it's much clunkier, requiring an extra await after every other await (or an extension to append to each task for the await). It's more complicated than just using a TaskCompletionSource, since it will need to be able to transition to pause/play state multiple times. It would likely need a custom awaiter that stores a

bool IsPaused

state.

[Edit] Actually, this just gave me an idea to create a new custom async type to be able to pause/resume between awaits like a coroutine, with the advantages of async/await (but maybe that's what Unity's new AwaitableCoroutine will do).

 

What about "await Task.Yield()" so far i know that should wait for the next frame in Monobehaviors.

 

That's not what I mean by pause/resume. If you have a cached Coroutine (IEnumerator), you can call StopCoroutine to pause it, then StartCoroutine again with the same object to resume. It's not a free operation (it does allocate for the new StartCoroutine), but it's a simple pause/resume method.

 

That’s… that’s literally stopping and starting a coroutine. Do I “pause” the movie when I turn it off?

Edit:
I was wrong, see @VolodymyrBS ’s answer below.

 

I think you misunderstood the @ProtoTerminator words

I think he talks about code like this

Code (CSharp):

1.  
2.     IEnumerator _coroutine;
3.  
4.     public void StartOrResume()
5.     {
6.         _coroutine ??= CoroutineMethod();
7.         StartCoroutine(_coroutine);
8.     }
9.  
10.     public void Pause()
11.     {
12.         StopCoroutine(_coroutine);
13.     }
14.  
15.     public void Stop()
16.     {
17.         StopCoroutine(_coroutine);
18.         _coroutine = null;
19.     }
20.  
21.     private IEnumerator CoroutineMethod()
22.     {
23.         int i = 0;
24.         while (true)
25.         {
26.             Debug.Log(i);
27.             i++;
28.             yield return new WaitForSeconds(1f);
29.         }
30.     }
31.  
32.  

IEnumerator will persist it state even after calling StopCoroutine. So if you reuse the same IEnumerator to start a new coroutine it will resume from the point when it was stopped instead of from start.

 

@JoshPeterson I had a thought.
Seemingly for async you'll add a CancellationToken to the MonoBehaviour class, but I figure there's a chance here to kill not 2, but 3 birds with 1 stone.
First of, something you've probably already considered, I would support the CancellationToken being added to an Object rather than a MonoBehaviour. It would be great if we could tie async methods to any lifetime we want.

Second: is there a possibility that CancellationToken.IsCancellationRequested will be set to true as soon as the object is marked to be destroyed by e.g. Object.Destroy(myObject)?

Because if so... we'll finally be able to do this:

var isBeingDestroyed = myObject.cancellationToken.IsCancellationRequested;

Which could as well be a getter in Object.

Such a bool has been long requested by the community and it would be tremendous if the CancellationToken implementation may give it to us for free.

 

This made me think. I had imagined that an API was to be added that imitated the way coroutines work. If that's so, wouldn't the planned cancellationToken turn true also when a GameObject is deactivated? If that's the case, wouldn't a new CancellationTokenSource need to be allocated for every component that's being used this way every time their Go gets activated?

On the other hand, if the cancellationToken returns true only for destroyed objects, how will Unity deal with scripts that get destroyed but their GO is never activated? Those don't get an OnDestroy call.

It's nice we are getting API members that make it easier to imitate coroutines with async methods, but that's not the only thing that could be nice about this. One of my favorite parts of async/await given more attention in Unity, is the idea of making them easier to use even when not tied to a Unity Object. Unitask kind of allows that, but it has to create a Monobehaviour behind the scenes.

I'm all for adding extra hooks and properties for Object life handling, but maybe they don't all need to be implemented as cancellationTokens. That way they make sense outside of the async realm, where they are also useful. We could make our own cancellationTokens tailored to our different needs then.

 

Just out of curiosity, who here has worked outside of games and has worked on apps/API's in enterprise? I'm not sure why there is so much love for coroutines in the Unity space. I've never actually met people in industry (outside of game dev) that have ever liked them (or that still use them today). Async/await aside, there has been other options that have done their job much better imo. Namely using Rx will give you so much more flexibility.

I'm just curious as to whether this is a product of people learning conceptually with coroutines or whether they have genuinely used the alternatives and come to the conclusion that coroutines are the better tool in most cases. I always found the opposite to be true but I would be happy for someone to give me a really good example of where a coroutine does a better job than an IObservable/IObserver or async/await.

 

For a long time, I had the notion that Enumerator Coroutines were a thing invented by Unity. Then I read about them in a book, but still, I've met C# developers that haven't ever heard about our concept of coroutines. On the other hand, Async/Await is a known concept in most languages I've seen. And of course, the observer pattern is famous everywhere, as are most patterns described by the Gang of Four.

I imagine enumerators were preferred instead of coroutines because they weren't usable when Unity started. I don't think you could even use Async/Await before C# 5 or 6, and Unity was stuck with an old C# version for a long time. Even if they could have used async methods instead of enumerators from the start, they weren't practical to use until C# 7, because you couldn't have things like value tasks.

That said, even though I like this Async/Await initiative, I don't want it to seem like I support the idea that things that work well for enterprise tend to work well for game dev. There are huge differences between those realms. For example, enterprise tends to be a lot more tolerant of garbage. It also tends to be centered a lot in UI, so stuff like MVC or the observer pattern fit very well; real-time means a whole lot more for game dev. Tricks that can be crucial for performance in game dev, are often seen as unnecessary and overcomplicated in enterprise, because a couple of milliseconds aren't that important there. A game's state can change a lot more with a lot more frequency than enterprise apps' state, and the order in which those changes happen can be a lot more important because of things like fairness or performance.

 

To give my 2 cents, I learned C# through Unity, like many others in this community, so I picked up Coroutines long before I even heard of async and by then async seemed “complicated” (boy I was young). I think a bias towards Coroutines is simply to be expected. As to performance, if Unity doesn’t do it you can always find “some nerd” in the forums running benchmarks; eventually we’ll see what’s what.

 

Thanks for both giving your view.

I wasn't specifically referring to the GOF observer pattern here, while the interfaces in System do satisfy that pattern, the entire reactive programming model supported by System.Reactive is so much on top of observers it's crazy.

I would agree with you in regards to most basic API's but if you've ever done any work with stock exchanges or FX trading you would see that the throughput is very easily on par with games.

Rx solved this problem by giving you everything as an asynchronous stream (concurrently). This gives you a decoupled way to see/manage state in a multi threaded way. You also have control over which thread the observation happens on, as well as which thread the subscription happens on. This means for anything that NEEDS to happen on the main thread, you just flip that subscription to .SubscribeOn() the main thread and you're good to go.

Just like anything in code, there will be pros/cons to this e.g the allocation of an Expression, which you can reduce and optimize with ExpressionSlim's etc.

You also have the benefit of not polling for changes or using a "pull" based mechanism. (IEnumerable/IEnumerator = do you have it yet, do you have it yet) whereas with Rx you have a "push" based mechanism (IObservable/IObserver = I have a thing, go process it). The cool thing is for people already familiar with enumerables they are basically symmetrical to each other and as such can be converted with .ToObservable() and .ToEnumerable().

For anyone using coroutines today that isn't so keen on using async/await, I would highly recommend looking into System.Reactive (Rx) - https://reactivex.io (great for learning) - https://github.com/dotnet/reactive (source).

Also a personal favourite of mine

- It was filmed on a potato but this was what got me interested in Rx.

If you want a really in-depth explanation of how everything works, https://davesexton.com/blog/post/monads-and-comonads-and-linq-oh-my.aspx is probably one of the best deep dives I've read (this site doesn't load in some browsers but works from chrome on android 100%).

 

This one may possible if you use `transform.CancellationToken` ?
The transform component should be safe to assume that it will almost always strictly tied to the object itself

 

Just wanted to echo this, rx is a massively useful framework for wrangling complex async logic into managable subscriptions, even for just simple things like throttling input/collisions etc.

There is a more specific Unity version UniRx is in a bit of a weird place these days and a few people have tried to turn it into a layer on top of the dotnet version but its still a bit stagnant at the moment (its made by the same guy who did UniRx).

 

I used it for a bit, I think once Unity gets up to CoreCLR we can pick the Unity specific parts of UniRx into a package and just use regular System.Reactive

 

Yeah, right, wouldn't System.Reactive would just work properly in every platform of unity when it adopt full dotnet and nuget?

 

Yes it should do, UniRx has some good stuff not included in System.Reactive though like Unity specific events

 

No as il2cpp isn't going away. Better question is what from System.Reactive doesn't work right now. Has anyone tried in Unity 2021?

 

It has some questionable dependencies so right now not really https://github.com/xoofx/UnityNuGet/issues/79

 

Right, and also about threading and scheduler that not so compatible with unity

Still I don't like that UniRx require MainThreadDispatcher object in the scene to make it work. I think it should be migrated into subsystem workflow

 

I have tested System.Reactive with Unity 2021.2 and the package seems to work fine

 

Even with IL2CPP on android/ios/linux?

 

Also webgl?

 

I think this person may be wrong:

It targets Net Standard 1.0 which is even better. I'm making Supabase unity client right now and their C# sdk uses Websocket.Client that uses System.Reactive. Somehow project compiles fine even without WindowsRuntime dll.

I got exactly the same error described here (Windows IL2CPP):

This great post by Scott Ferguson describes how to fix it. In my case generic-virtual-method-iterations=5 was enough to fix it, I didn't have to switch to slower runtime.

Try yourself, these can be installed with package manager (are slightly older versions than 5.0.0):

Code (CSharp):

1. https://github.com/kamyker/supabase-unity.git?path=.UnityDlls/System.Reactive
2. https://github.com/kamyker/supabase-unity.git?path=.UnityDlls/System.Runtime.CompilerServices.Unsafe
3. https://github.com/kamyker/supabase-unity.git?path=.UnityDlls/System.Threading.Tasks.Extensions
4. https://github.com/kamyker/supabase-unity.git?path=.UnityDlls/System.Runtime.InteropServices.WindowsRuntime (optional?)

 

I've tested in editor, not standalone.

I have no intention of ever deploying to WebGL so can't answer that one for you

 

There was totally a dependencies that barred System.Reactive to be usable in unity in every platform. This is one reason I was encourage unity to hurry adopt dotnet 5 already. It is totally a blocking issue




The dependencies has no issue in Windows platform

 

Once Unity migrates to CoreCLR are there plans to support CoreRT over IL2CPP?

 

I see and downloaded System.Reactive repo removed WindowsRuntime dependency and HAS_WINRT define. https://drive.google.com/file/d/1dvO7GXPpXBWS9zQmmTmD2rAOayrSGW_1/view?usp=sharing

It has less features but it may be similar to net5.0 version. Btw there's also net5.0-windows that has a lot more flags:

Code (CSharp):

1. HAS_WINRT;HAS_WINFORMS;HAS_WPF;HAS_DISPATCHER;DESKTOPCLR;WINDOWS;CSWINR

These features won't be available even when Unity moves to Net 6.

I don't think CoreRT compiles to as many platforms as il2cpp.