Search Unity

How to model a list-filtering job?

Discussion in 'Entity Component System' started by OswaldHurlem, Feb 25, 2018.

  1. OswaldHurlem

    OswaldHurlem

    Joined:
    Jan 6, 2017
    Posts:
    40
    One of the most important subroutines in my codebase has a form like this:
    Code (CSharp):
    1.  
    2. public struct GetEvensJob : IJob
    3. {
    4.     // Input
    5.     public NativeArray<int> Numbers;
    6.     // Output
    7.     public NativeArray<int> Evens;
    8.     // Continuable state
    9.     public int NumbersIdx;
    10.     public int EvenCount;
    11.  
    12.     public void Execute()
    13.     {
    14.         for (; NumbersIdx < Numbers.Length; NumbersIdx++)
    15.         {
    16.             var n = Numbers[NumbersIdx];
    17.             if (n % 2 == 0)
    18.             {
    19.                 Evens[EvenCount++] = n;
    20.                 if (EvenCount == Evens.Length)
    21.                 {
    22.                     //???
    23.                     return;
    24.                 }
    25.             }
    26.         }
    27.     }
    28. }
    29.  
    The purpose of it is to take a large array of values, and return a smaller array of values which pass some criteria.

    Without the constraints of the Job system, I might have Evens be a LIst<int>, or I might have it resize to double its previous size when it is full (at the //??? line). However, these options are not available to me within the scope of the GetEvensJob.

    What should I do instead? I could have the job run to completion several times. Each time I can resize Evens and let the job continue where it left off. But this seems a bit procrustean. Is there anything I could do instead?

    I don't need it to run in a data-parallel fashion (though I know this is doable). Mostly I want it to run in a separate thread.
     
    OswaldHurlem, Feb 25, 2018
    #1

  2. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    We have a NativeList and IJobParallelForFilter coming up for this purpose. It will be released as part of the Entity Component System Preview package.

    Code (CSharp):
    1.  
    2. public interface IJobParallelForFilter
    3. {
    4.         bool Execute(int index);
    5. }
    6.  
    7. static public JobHandle ScheduleAppend<T>(this T jobData, NativeList<int> indices, int arrayLength, int innerloopBatchCount, JobHandle dependsOn = new JobHandle()) where T : struct, IJobParallelForFilter;
    8.  
    9. static public JobHandle ScheduleFilter<T>(this T jobData, NativeList<int> indices, int innerloopBatchCount, JobHandle dependsOn = new JobHandle()) where T : struct, IJobParallelForFilter;
     
    Joachim_Ante, Feb 25, 2018
    #2
    Krajca likes this.

  3. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    The above API's are fully implemented in C# using public Unity.Collections.LowLevel.Unsafe API's, but probably makes most sense to just wait for the release unless you just really want to learn about writing your own native containers + job types.
     
    Joachim_Ante, Feb 25, 2018
    #3

  4. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Here is some preview of the documentation we are currently writing custom native containers + custom job types:
    https://gist.github.com/joeante/3f6b75c738fe0a1be19207e7e4294578

    Definately not what we would expect the average Unity user to do himself. We rather expect to provide all the different job types out of the box over time. But I am sure there will always be use cases for making customers containers + job types.
     
    Joachim_Ante, Feb 25, 2018
    #4
    SugoiDev, james7132, ptrefall_facepunch and 1 other person like this.

  5. OswaldHurlem

    OswaldHurlem

    Joined:
    Jan 6, 2017
    Posts:
    40
    Excellent!! I'm considering writing an implementation.

    I'm impressed by the level of support Unity is providing for users of this Beta. :)
     
    Last edited: Feb 26, 2018
    OswaldHurlem, Feb 26, 2018
    #5
    Krajca likes this.

  6. OswaldHurlem

    OswaldHurlem

    Joined:
    Jan 6, 2017
    Posts:
    40
    Is UnsafeUtility.Malloc thread-safe for all Allocator types?
     
    OswaldHurlem, Feb 26, 2018
    #6

  7. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Yes.

    Persistent and TempJob can be allocated / deallocated on any thread and different threads from each other

    Temp can be allocated on any thread but has to be deallocated on the same thread.
     
    Joachim_Ante, Feb 27, 2018
    #7
    recursive likes this.

  8. recursive

    recursive

    Joined:
    Jul 12, 2012
    Posts:
    669
    I was hoping to get some confirmation on stuff like this soon. I'm about to get back into messing with my prototype this week and was wondering if tempjob could be allocated off the mainthread.
     
    recursive, Feb 27, 2018
    #8

  9. OswaldHurlem

    OswaldHurlem

    Joined:
    Jan 6, 2017
    Posts:
    40
    I'm integrating the jobs system with my code, which has a subroutine like GetEvensJob, except it makes quite a bit of use out of legacy C code. I made a fairly shoddy resizing array which will do for the time being.

    I have two important requests for NativeList, which I think will be painless:
    • Allow the user to get a NativeSlice of it.
    • Allow the users to resize it from eg 100 to 200 elements, without zeroing out elements 100 up to 200.
     
    OswaldHurlem, Feb 27, 2018
    #9

  10. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    There is an implicit cast to NativeArray already. This pattern is super important. Accessing [int] operator on NativeArray is quite a bit faster than NativeList (One less indirection)

    The beauty of it is that it even gives you an exception when accessing the NativeArray after the NativeList has added an element. So its totally safe against incorrect usage.

    When you say allow resize, you mean supporting the uninitialized memory option from native array or do you mean, resizing it with a specific value to be replicated?
     
    Joachim_Ante, Feb 27, 2018
    #10

  11. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Here is the current state of Extended Collections we are going to ship as part of Entity Component System Preview soon.
     

    Attached Files:

    Joachim_Ante, Feb 27, 2018
    #11
    OswaldHurlem, recursive and LennartJohansen like this.

  12. OswaldHurlem

    OswaldHurlem

    Joined:
    Jan 6, 2017
    Posts:
    40
    The former... which is exactly what you have! Hurrah!
     
    OswaldHurlem, Feb 27, 2018
    #12