Search Unity

Jobs *Any* way to access static data safely?

Discussion in 'Entity Component System' started by joedurb, Jul 18, 2018.

Page 2 of 2

  1. optimise

    optimise

    Joined:
    Jan 22, 2014
    Posts:
    2,129
    This might interest you. https://forum.unity.com/threads/edi...s-to-enter-playmode-in-an-empty-scene.516004/
     
    optimise, Feb 16, 2019
    #51

  2. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    567
    I don't understand the point of recreating everything from scratch once again, look at SCTP, QUIC or ENet - it takes years of shaping mature message-oriented protocols encapsulated into UDP by well-organized teams. On top of this, you need good I/O model, several complex high-level abstractions to cover various game mechanics which means general-purpose serialization, synchronization and security layers, interoperability with DOTS, and the further you go the deeper you enter the swamp. Be clear with people, just say openly we need N years for... or, be smart and grab mature library like Gaffer's Yojimbo, then rework it slightly, fix a couple of things and voila: you got the transport that you are building from the ground-up. Focus on what your customers really needs instead of doing redundant work that will lead you to the same well-known result.
     
    nxrighthere, Feb 16, 2019
    #52
    Vincenzo, e199 and FROS7 like this.

  3. joedurb

    joedurb

    Joined:
    Nov 13, 2013
    Posts:
    38
    Hijacking this back to the initial Thread Subject, I see there were replies telling me I went the wrong route with using Native C# threading, In my case, and as of 6 months ago, I had no choice:

    1) Implementation took 25hrs less, and WAY cleaner. reduced hundreds of lines of ugly code dealing with native arrays and job packaging.
    2) Unity was still shooting out Warnings: jobs cannot live longer than 4 frames constantly. Still is?
    3) Main thread Performance was IMPROVED. (No need to copy huge datasets that the job produced.)

    So, Like i said, Unity Jobs has their place, but in my case OF:
    1) Huge data set, Thread safe already, no reason to convert to Native Arrays.
    2) Singular Sequential huge Jobs of 10+ seconds long
    3) needing to retain tons of data created in the Thread/job.
    4) No Unity specific requirements this was all just pure data.

    So, you can say I may not be doing what Unity recommends, but in my case it was definitely the "Right" way.

    And if someone had kept me from learning the benefits and proper usage of native C# threads, i would be in a much sorrier state. All I said, is it's an VALID OPTION for some narrow cases. And I stand by that.
     
    joedurb, Mar 16, 2019
    #53

  4. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    567
    I'm not using C# jobs as well, but what this system gives over C# threads is a possibility to safely and efficiently parallelize computations and formulate your code in terms of logical tasks with out of the box matching parallelism to available resources, load balancing and higher-level thinking. It perfectly suits for fine-grained parallelism.

    In terms of granularity, C# jobs have some limitations, like, it's not suited for continuous long-running I/O which we are using in coarse-grained systems like network transport for example which shuttles packets/datagrams without any pauses/scheduling/high-level input. The logic there is independent and self-driven which follows the rules of sockets programming and sockets underlayer implementation.
     
    nxrighthere, Mar 17, 2019
    #54
    Deleted User likes this.

  5. xoofx

    xoofx

    Unity Technologies

    Joined:
    Nov 5, 2016
    Posts:
    418
    Since a few versions, burst is now supporting static readonly managed arrays directly from your job. There are currently two main constraints:
    - these arrays should be declared with static readonly and use plain simple primitive types (int, short...etc.) or struct with a simple constructor, (a simple constructor: a constructor that doesn't throw any exceptions and is simply setting its field members)
    - You should not write to these arrays in another part of your C# code that is not covered by burst, as burst is making a readonly copy of these arrays at compile time

    The safety checks are on, the bounds checks are still emitted, but when they are disabled, it is a direct access.

    Code (CSharp):
    1. public struct MyJob : IJob {
    2.     private static readonly int[] MyConstants = new int[] { 1, 2, 3 }
    3.  
    4.     public void Execute() {
    5.  
    6.          // ....
    7.  
    8.  
    9.          // read access with a constant index
    10.          var value = MyConstants[1];
    11.  
    12.          // read access with a dynamic index
    13.          int sum = 0;
    14.          for(int i = 0; i < MyConstants.Length; i++)
    15.          {
    16.             sum += MyConstants[i];
    17.          }
    18.  
    19.          // write access forbidden
    20.          // MyConstants[0] = 5  // will fail to compile
    21.  
    22.          // ....
    23.     }
    24.  
    25. }
    This static readonly array is placed in a readonly section of your program and has a fixed address known at compile time. As the array is tagged as entirely constant, it should be also resolved to a pure constant when using it with constant indices.
     
    xoofx, Mar 17, 2019
    #55
    Deleted User, CodeSmile, BakeMyCake and 2 others like this.

  6. Deleted User

    Deleted User

    Guest

    @xoofx good day. Sorry for the off topic but is there any plans to support Span<T>, ReadonlySpan<T> with burst?

    Thank you.
     
    Deleted User, Mar 17, 2019
    #56
    TheZombieKiller, FROS7 and e199 like this.

  7. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    3,356
    I think the more common use case in real games with any complexity is shared logic using native containers that is abstracted out separately from any specific Ecs system/job. We have quite a bit of this in a complex multiplayer game. And that forces a lot of extra indirection when we want to use burst.

    This doesn't really apply of course to parallelfor type jobs, it's IJob where grabbing an instance at the head of Execute would work well. In real games not every problem is EP and IJob is just the best practical approach.
     
    snacktime, Mar 17, 2019
    #57

  8. xoofx

    xoofx

    Unity Technologies

    Joined:
    Nov 5, 2016
    Posts:
    418
    Yes, although sharing data in a job system brings concurrency problems and can't be solved at burst level (alone at least). But in any cases, static mutable data is a plague for software architecture design.
     
    xoofx, Mar 23, 2019
    #58

  9. snacktime

    snacktime

    Joined:
    Apr 15, 2013
    Posts:
    3,356
    The challenge for us has been mostly spatial datasets. They are all readonly but used by multiple features. Statics aren't really even an issue, I wasn't implying statics was the right solution. The more difficult problem is with shared data sorting out what the job dependency chains should look like. The automatic handling when using Entities with job component system is really nice. But the issue is the way the system is designed you almost need that, because once you throw in something that the job component systems can't resolve automatically it starts getting difficult to reason about.
     
    snacktime, Mar 23, 2019
    #59

  10. krzysie7

    krzysie7

    Joined:
    Jan 4, 2019
    Posts:
    6
    Hey, I've been wondering about that. The way JobSystems compute dependencies could really be more decoupled from the ECS itself. I assume you are talking about something like an automatic dependency resolver? This is possible to do with some reflection magic. It is just a quality-of-life type of thing, but is certainly possible. May be just a little costly, because an automatic system is going to call CombineDependencies() much more than hand-tuned code, but if you jobs are sufficiently "fat", it will offset the cost. I even have an inspector for this, it's quite nice: I didn't have time to make an actual job graph yet, that would be much better for visualisation of actual dependencies (this just shows which job writes/reads which container).
     
    krzysie7, Mar 23, 2019
    #60
    dzamani, xman7c7 and Deleted User like this.

  11. BakeMyCake

    BakeMyCake

    Joined:
    May 8, 2017
    Posts:
    175
    Could you further clarify this part? Is the code below illegal?
    Code (CSharp):
    1. public struct MyJob : IJob {
    2.    private static readonly int[] MyConstants;
    3.      
    4.    public MyJob(int[] consts)
    5.    {
    6.        MyConstants = consts;
    7.    }
    8.    
    9.    public void Execute() {
    10.        // ....
    11.    }
    12. }
    I may be misinterpreting the meaning of this restriction, but if it's illegal to assign to a static readonly at runtime then it's not very useful.

    Consider this scenario:
    A locomotion system with a bunch of parameters set up in the editor, and even more parameters derived from those. All of these variables are runtime constant. A job calculates the resulting pose. It has to take in dozens of input parameters that will never change.

    Are we currently looking at a situation where every time said job is scheduled the fields are copied? And when the job is completed, it fetches back the same fields just in case any of them changed during job execution, even though they never will?

    Sorry if I'm asking stupid questions, but I think it's beneficial for all of us to know the potential long term pitfalls of using the job system, because if my suspicions are correct, then at some point the job system can become not worth using if the combined volume of memory being tossed around becomes too big.
     
    BakeMyCake, Oct 18, 2019
    #61

  12. tim_jones

    tim_jones

    Unity Technologies

    Joined:
    May 2, 2019
    Posts:
    287
    If I'm understanding your example correctly, you'd want to use a NativeArray there, to avoid copying data every time you schedule a job:

    Code (CSharp):
    1. public struct MyJob : IJob
    2. {
    3.     public NativeArray<int> MyConstants;
    4.    
    5.     public void Execute()
    6.     {
    7.         // .... you can reference MyConstants here
    8.     }
    9. }
    10.  
    11. // Calling code
    12. var job = new MyJob();
    13. job.MyConstants = myConstantsNativeArray;
    14. job.Schedule();
    The discussion above about `static readonly` is more for data that you can initialise once in a static constructor and then never touch again. In this case your code could look like this:

    Code (CSharp):
    1. public struct MyJobContainer
    2. {
    3.     private static readonly int[] MyConstants;
    4.  
    5.     static MyJobContainer()
    6.     {
    7.         MyConstants = new[] { 1, 2, 3, 4 };
    8.     }
    9.  
    10.     public struct MyJob : IJob
    11.     {
    12.         public void Execute()
    13.         {
    14.             // .... you can reference MyConstants here
    15.         }
    16.     }
    17. }
    18.  
    19. // Calling code
    20. var job = new MyJobContainer.MyJob();
    21. job.Schedule();
     
    tim_jones, Oct 18, 2019
    #62
    BakeMyCake likes this.

  13. bb8_1

    bb8_1

    Joined:
    Jan 20, 2019
    Posts:
    100
    Thx for your extremely informative replies - it would be cool if you could recommend some books/sites/yt video tutorials or any other resources(github projects etc) about multiplayer programming - thx in advance. Also what do u thing about Rust programming language used for for multiplayer programming?
     
    bb8_1, Jan 3, 2020
    #63

  14. nxrighthere

    nxrighthere

    Joined:
    Mar 2, 2014
    Posts:
    567
    That information a bit outdated (technologies are evolving) and incomplete since the topic is quite complex... It also diverges with Unity's ideas and the overall direction that they are moving regarding networking and the data-oriented stack in general.

    Sure, but it depends on what are you looking for exactly, it might be: a network protocol, genre/game-specific abstractions, platform-specific system API, hardware-specific integration, and many other different aspects...

    If you give me more details, I can help and put you on the right track, just drop me a message.

    Rust is a great language for general-purpose networking, taking into account crates with thin abstractions between your code and a platform-specific system API that gives you great performance and safety. Really nice built-in tooling, very easy to expose functionality and inter-operate with other programming languages (consider cbindgen). I've enjoyed the language and its ecosystem for several weeks making basic stuff, but I wouldn't and can't use it for daily programming and job, too early.
     
    nxrighthere, Jan 6, 2020
    #64
    bb8_1 likes this.
Page 2 of 2