Search Unity

C# job system?

Discussion in '2017.2 Beta' started by Sebioff, Jul 7, 2017.

  1. Sebioff

    Sebioff

    Joined:
    Dec 22, 2013
    Posts:
    185
    In the announcement post the C# job system was named as one of the main features for the Unity 2017 cycle.
    From what has been shown so far my game is a perfect fit for it, so I'm curious if this is still on schedule? (Assuming the next release after this is going to be 2018.1)
     
    Fabian-Haquin likes this.
  2. Fabian-Haquin

    Fabian-Haquin

    Joined:
    Dec 3, 2012
    Posts:
    230
    Have you seen the 2017 Keynote ?

    "None of this is shipping any time soon"
     
  3. CarlosRincon

    CarlosRincon

    Unity Technologies

    Joined:
    Jan 28, 2016
    Posts:
    124
    Indeed, we are starting to land pieces of the plumbing for it, but no concrete timeline for the full system completion currently.
     
    Fabian-Haquin likes this.
  4. HonorableDaniel

    HonorableDaniel

    Joined:
    Feb 28, 2007
    Posts:
    2,904
    Is the job system unity's version of threading?
     
  5. Fabian-Haquin

    Fabian-Haquin

    Joined:
    Dec 3, 2012
    Posts:
    230
    Of multi-threading yes, but they are improving the compilator too. :)
     
    Alverik likes this.
  6. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,222
    Whole thing interests the cashew nuts out of me. I would enjoy more information about the whole thing in text whenever it's feasible to comment on it :) (can't hear vids).
     
    faximegames and Alverik like this.
  7. HonorableDaniel

    HonorableDaniel

    Joined:
    Feb 28, 2007
    Posts:
    2,904
    Yeah, that's what I meant (multi-threading).
     
  8. Fabian-Haquin

    Fabian-Haquin

    Joined:
    Dec 3, 2012
    Posts:
    230
    Available since yesterday, this video might interest you:
     
  9. Sebioff

    Sebioff

    Joined:
    Dec 22, 2013
    Posts:
    185
    Yep! Some good stuff coming pretty soon after all :)
    Summary for hippocoder and others who can't watch video: C# job system that was shown end of last year planned for 2017.3 later this year/early next year; job compiler shown at this Unite not this year and no ETA yet. There's also a new math library coming.
     
  10. recursive

    recursive

    Joined:
    Jul 12, 2012
    Posts:
    640
    Finally. I've noticed my own code and framework choices are trending in the direction of ECS and dependency injection. While getting the C# job system this year is good, is there a timeline for the ECS framework after it? Around the same time or early next year?
     
    faximegames likes this.
  11. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,271
    It wasn't clear, but I think he said they already have a working ECS project that they will make available soon (he talks about it somewhere in the questions segment).

    Basically I think the ECS project comes before the final jobs system
     
    recursive and faximegames like this.
  12. rastlin

    rastlin

    Joined:
    Jun 5, 2017
    Posts:
    125
    Since Unit 2017.1 is out and you can develop with .NET 4.6, why don't just use TPL for any multi-threading jobs?
     
  13. kru

    kru

    Joined:
    Jan 19, 2013
    Posts:
    434
    I assumed the Job system would give us access to some or all of the Unity API during jobs. The biggest limitation with threads, even in prior versions of Unity, is the challenge surrounding access to data hidden behind Unity APIs (transforms being a big one, but Time and others are an issue as well).
     
  14. recursive

    recursive

    Joined:
    Jul 12, 2012
    Posts:
    640
    There are likely performance and access considerations for the C++-side of the engine that TPL's "general solution" might not work well with. And on top of that having it structured in a specific way likely helps simplify the optimizing compiler down the road. You're probably free to use TPL for non-simulation code in Unity or for data-only processing (especially if you're dealing with large datasets from a server to drive a UI) once the mono upgrade is complete.
     
  15. MaxPalmer

    MaxPalmer

    Joined:
    Mar 9, 2013
    Posts:
    20
    I'm not seeing this appear on any of the beta timelines unless I'm missing something. I certainly can't see it listed for 2017.3 - however that is still stated to be in planning.

    I did make a whole heap of notes and attempted to capture most of the code scripts from the Unite talk - great BTW. Perhaps I need to dump this somewhere so others can read it....
     
  16. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,271
    haha, I did the exact same thing in an attempt to start building a game framework that will be ready for the ECS and jobs system. Basically, I'd say it boils down to:
    • The components you put on gameobjects must be data-only and must not rely on inheritance (since they'll have to become structs)
    • You have systems that operate on parallel tightly-packed arrays of these components (data-oriented programming style)
     
    Last edited: Sep 12, 2017
  17. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,475
    Considering from past experience, think of UI or Input, how much time Unity Technologies spends on integrating new systems in their product after their first announcement, I'd be surprised to see the C# Job- or ECS land in a 2017.x release. I'm all for surprises though!
     
    d1-favero likes this.
  18. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,271
    well... you never know. In the talk, Joachim says:
    • The job system API should land in 2017.3 or 2018.1. This is only the "API" part, not the "Compiler" part. I guess this means this code won't actually be fully multithreaded, but at least it'll allow us to start working with it.
    • The ECS is supposedly already ready as a prototype that will be released to the community as an experimental package
    • The job system compiler tech will be for later in 2018
    So it really seems like we'll start getting access to all this good stuff in 2017.3 or 2018.1, and then we'll just have to wait for the magical performance boost when the new compiler tech is ready. This really doesn't bother me too much because if I start a new game project in late 2017 / early 2018, it most likely won't be ready for release before the compiler tech arrives
     
    Last edited: Sep 12, 2017
    Peter77 likes this.
  19. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,475
    It certainly would be very nice to get it early, so fingers are crossed :)
     
  20. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,271
    By the way, @CarlosRincon

    Are you aware of any plans for releasing the ECS prototype? I would definitely like to start working with it as soon as possible, even if it changes completely at some point in the future. Just like the Playables API which ended up changing massively a bunch of times before it got officially released, it's still very useful to have something that you can start experimenting with and giving feedback on.

    The reason why I specifically want the ECS to be released soon (as opposed to the actual Job System) is that it really determines the entire structure of your gameplay code, so it's almost unthinkable to start a new project without it if you plan on using it later. At least with a prototype, converting your project to the final release would probably be more realistic. It would allow us to start working right now

    It could be released as an "in-progress" thing on Unity's github, just like the HDRenderPipeline, the old Input prototype, the PostProcess stack v2, etc.....
     
    Last edited: Sep 12, 2017
  21. MaxPalmer

    MaxPalmer

    Joined:
    Mar 9, 2013
    Posts:
    20
    Where is there more information on the new ECS system?
     
  22. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,271
    To the best of my knowledge, the C# Job System Unite presentation is the only place where this is talked about
     
  23. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,222
    Unless I lost the plot, isn't Unity already Entity Component System...? I mean isn't this how you work right now? Unity even mentioned here: https://en.wikipedia.org/wiki/Entity–component–system

    So what is "new" ECS?

    Sounds like we probably would not want to adopt this model for anything except large scale numbers like boids or something...?

    Docs please Unity! no need to release early, but we would all appreciate a nice wip google doc complete with typos (we unenrdsand) :)
     
    Mikael-H likes this.
  24. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,271
    Regardless of the "high-performance data-oriented" aspect of it, I really feel like handling all logic through "systems" and all data through "components" is the most well-structured game code architecture paradigm I can think of right now. For me, it's the ideal solution to solving the problem of creating links between different components of an entity (and links between different entities) without having to write tons of boilerplate code, or turning everything into a big spaghetti, or having to do "GetComponent()" all the time.

    I'll admit it's a bit hard to explain why exactly I think it's better. It's just so.... elegant. The thing that convinced me of the greatness of this sort of approach to an ECS is the Overwatch Gameplay Architecture And Netcode talk from GDC 2017. Unfortunately it's for GDCVault members only, but it's very worth it
     
    recursive likes this.
  25. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,222
    OK well in my case I want to get things done, and that sounds pretty inefficient to author, for designers and artists. I know Unity dislikes using gameobjects as data blobs but I do, it's much easier for a tiny team to use them as data blobs.

    In the case of the new ECS, I see it as more of a tool for the job thing so I appreciate where you are coming from, but don't think I have the time to engineer an entire game around it because I would need to do a fair bit of support engineering and many more tools.

    I can see why you want more info, it does sound a bit daunting.
     
  26. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,475
    Here is some info from Unite Austin 2017. At the end of the tech talk (linked below), they provide some dates...
    • C# Job System in 2018.1
    • Early preview of new Component system in 2018.1
    • New compiler tech later in 2018
    The C# Job System / Compiler Tech talk starts at 1hr 26m
     
    AmieD, scvnathan and PhilSA like this.
  27. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,222
    Just gimmie already.

    Question though: at what point does using jobs etc become more efficient than the classic workflows? I'm guessing it's fast even at scales of 1.


    @Joachim_Ante nice one
     
    PhilSA likes this.
  28. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,475
    https://create.unity3d.com/jobsystem

    Not sure whether this will just send an email if it's officially available, or if you get your hands earlier on it by signing up there :)
     
    russellsavage and hippocoder like this.
  29. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,271
    I am so happy about that RTS game presentation. I cannot wait to get my hands on all this.

    Also, the physics are getting jobified too : https://twitter.com/melvmay/status/915302715433111552
    If you look at the replies of the tweet, you'll see Anthony Yakovlev saying 3D physics will also get more saturated multithreading once PhysX 3.4 is integrated. Exciting times!

    ...on the other hand, I kinda don't feel like starting any new project until 2018.1 is available :(
     
    Last edited: Oct 4, 2017
  30. Necromantic

    Necromantic

    Joined:
    Feb 11, 2013
    Posts:
    115
    Thanks, didn't even know that link was a thing.
    It's actually the feature I'm looking forward the most since I saw the previous Unite Europe Keynote.
    I've been basically annoying everyone about it on the IRC channel. :p

    Was a bit disappointed when I checked out the new 2017.3 Beta and there was nothing on it in it, since they first said the first phase of it was planned for 2017.3.
     
    Last edited: Oct 5, 2017
  31. DwinTeimlon

    DwinTeimlon

    Joined:
    Feb 25, 2016
    Posts:
    183
    Last edited: Oct 8, 2017
    Peter77 likes this.
  32. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,271
  33. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,222
    That code feels retro yet familiar...
     
  34. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,475
    I find it amusing they say "Old school MonoBehaviour design". :rolleyes:
     
  35. rastlin

    rastlin

    Joined:
    Jun 5, 2017
    Posts:
    125
    What worried me from the presentation is this:

    Async/Tasks | C# Job System | Native multithreading
    Unity API no | no | no

    This still means the job system wont give access to the API, just transform, mesh and textures. That's strange in my opinion, since most of those components could already be accessible by a simple abstraction applied on top of them.

    What is the use for job system if I don't have access to let say, camera API or animation API?
     
    osmoberlin likes this.
  36. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,271
    Playables API (animation) is supposed to be jobifiable though
     
    wobes likes this.
  37. LaneFox

    LaneFox

    Joined:
    Jun 29, 2011
    Posts:
    6,464
    The new jobs stuff is a huge change to workflow and design. Joachim basically says "... abandon OOP if you want to go fast." and he provided a nice look into how it is currently expected to work in Austin. It is real, and it is very different. It's going to take some time for adoption.

    The company that did the nice fancy game demo (forgot their name) showed how it was applied practically but there were huge and wildly different approaches to their systems and solutions that went very, very low level. If you're the average user at this time, you'll probably not be using it for several years until there is a wider reception, better documentation, examples, tutorials, other assets using it, etc.

    If you're a more advanced user there is practical application now for effectively spreading workload across the cpu in a much more efficient manner. The way it is done feels more like old cpp approaches than anything else and really makes you design differently than you would with OOP. There are a lot of caveats and things that make your life easy which you will be surrendering when using this new stuff.

    It's far from a 'go fast' button and requires that users are ready to dig deep to improve performance.
     
    Sniffy likes this.
  38. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    799
    If the Unity API is not accessible in the jobs (Only Transforms, Meshes and Textures - which is pretty cool itself!), I feel like they should have selected their words more carefully in the keynote. They said they are instantiating and destroying hundres/thousands of objects (The arrows) and that it was running so smoothly thanks to the new system. I guess this isn't 'true' instantiation and object destruction as we know it?
     
  39. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,271
    Yeah that sounded weird to me. They must've used some kind of pooling for this
     
  40. LaneFox

    LaneFox

    Joined:
    Jun 29, 2011
    Posts:
    6,464
    For the demo shown in Austin there was no pooling, but iirc they used the gpu instancing and drawmesh stuff to render everything. They also had some bizzare tricks like putting animations in textures, using double quaternions instead of transforms, and other weird things.

    It still is largely going to boil down to people being good programmers to make their stuff go fast. Everyone really needs to understand that. Joachim gave some examples of small changes to structure that resulted in some pretty big gains, but rarely does anyone ever bother to dig deep enough in to get those gains.
     
    Enrico-Monese likes this.
  41. Prodigga

    Prodigga

    Joined:
    Apr 13, 2011
    Posts:
    799
    So, yeah, I don't think they were instantiating arrows. Just populating a list of vectors that represents arrow positions, another that represents velocities, using the velocities to update positions and using the positions to draw GPU instances arrows. Honestly, fairly straightforward stuff even without jobs. We will just have to wait for some more information. Lots of confusion and mystery surrounding all of this at this early stage!
     
  42. PhilSA

    PhilSA

    Joined:
    Jul 11, 2013
    Posts:
    1,271
    The only part of this that I've really struggled with is how to use Graphics.DrawProcedural for drawing objects that have proper shading and shadowcasting (an equivalent of the standard shader) in deferred rendering. The best I could do so far was unlit. I hope unity releases some kind of example of this
     
    dreamerflyer likes this.
unityunity