Search Unity

Jobs system safety

Discussion in 'Entity Component System' started by sebas77, Jul 19, 2019.

  1. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,642
    a question about the Job System if you don't mind. Am I correct if I say that the job system is able to work out the dependencies used by the jobs, but only if they are native arrays or other recognised data structures (which ones), so that jobs are executed safely and we don't need to worry about thread safety?

    I am a bit confused about thread safety is guaranteed, whenever possible, by the jobs, outside the ECS system.
     
    sebas77, Jul 19, 2019
    #1

  2. eizenhorn

    eizenhorn

    Joined:
    Oct 17, 2016
    Posts:
    2,684
    Safety checks warn you about possible race conditions and if it not warn you, thus (in most cases but not in all) it's safe (if you not disable any safety restrictions)
     
    eizenhorn, Jul 19, 2019
    #2

  3. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    The safety system essentially cross checks read / write access of containers against the actual provided job dependencies. This is for both containers as well as component types in ECS which are internally represented with the same SafetyHandle system so from the job systems perspective ECS types appears as "containers".

    This lets us check all race conditions deterministically. At deploy time we can remove all checks and thus have zero overhead from the job debugger.
     
    Joachim_Ante, Jul 19, 2019
    #3

  4. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,642
    So beside the ecs container only native containers are checked, right? I am interested to know how these checks work outside the ECS provided containers
     
    Last edited: Jul 20, 2019
    sebas77, Jul 19, 2019
    #4

  5. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    The C# job system supports Native Containers, guarantees safety with those. You can write your own native containers. Look for Unity.Collections for how they work and in the documentation. Other than that all data types are expected to be structs thus copied by value, thus inherently thread safe in C# jobs.
    https://docs.unity3d.com/ScriptRefe...LowLevel.Unsafe.NativeContainerAttribute.html

    This is the recommended approach, it is the only thing we can and will gurantee thread safety for.


    However there are situations where you want to leave the backdoor open and just do something that is unsafe and can not be validated. For this we have
    https://docs.unity3d.com/ScriptRefe...sableContainerSafetyRestrictionAttribute.html

    https://docs.unity3d.com/ScriptRefe...tiveDisableUnsafePtrRestrictionAttribute.html

    This approach makes it so that the safety system simply pretends like those containers aren't used by the job and also removes the checks from the containers. Same with unsafe pointer. This way at least all unsafe usage is declared, which means once you do get race conditions at least you can easily search for all possible places.

    In the same way you can create a backdoor for class types too, using GCHandle. But of course using class types prevents you from using burst so that's not recommended for both safety & performance reasons.
     
    Joachim_Ante, Jul 20, 2019
    #5

  6. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,642
    Thank you very much. I'll read those links. I am trying to understand the best approach to write in to managed arrays of structs using burst. A pointer works fine but must be fixed of course and jobs won't be able to check the safety.
     
    sebas77, Jul 20, 2019
    #6

  7. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,759
    You can just map a managed array to a NativeArray and use that. Look at

    NativeArrayUnsafeUtility.ConvertExistingDataToNativeArray<>()
    NativeArrayUnsafeUtility.SetAtomicSafetyHandle();
     
    tertle, Jul 20, 2019
    #7
    sebas77 likes this.

  8. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Generally you should really figure out how to not do that at all.
     
    Joachim_Ante, Jul 20, 2019
    #8

  9. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,642
    Interesting thanks, it means I haven't grasped how to use the job system to write results back to the managed memory outside ecs
     
    sebas77, Jul 21, 2019
    #9

  10. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,642
    Thanks looks like a solution
     
    sebas77, Jul 21, 2019
    #10

  11. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,264
    Do you really need a managed array?
    Native Containers can be used outside of jobs like any other container.
     
    DreamingImLatios, Jul 21, 2019
    #11

  12. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,642
    Unluckily yes as it's part of another library
     
    sebas77, Jul 21, 2019
    #12

  13. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Can you change the library?
     
    Joachim_Ante, Jul 21, 2019
    #13

  14. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,642
    Not at the moment no, but it looks like the NativeArrayUnsafeUtility is something to certainly test.
     
    sebas77, Jul 21, 2019
    #14

  15. sebas77

    sebas77

    Joined:
    Nov 4, 2011
    Posts:
    1,642
    I gave a look at this possible solution, but I noticed that the NativeArrayUnsafeUtility.
    ConvertExistingDataToNativeArray still needs a pinned pointer. I need to find out if we have to pin ourselves and what/when is going to free it. I am working with the assumption that copy is not happening, but if it is happening I would like to understand when it happens (and when the data is going to be synchronized back)
     
    sebas77, Jul 30, 2019
    #15