Search Unity

How to pass the intermediate data in NativeQueue from one parallel job to another?

Discussion in 'Data Oriented Technology Stack' started by TMPxyz, Jul 15, 2019.

  1. TMPxyz

    TMPxyz

    Joined:
    Jul 20, 2012
    Posts:
    765
    Hi, I've a system that has two parallel jobs,

    * the first one will run through chunks and populate some intermediate data into a NativeQueue,
    * the second job is expected to work on the pass-in NativeQueue and get the final result.

    The problem is that the second job (IJobParallelFor) needs the length to schedule, which is not available at the point in main thread. I checked the IJobParallelForDefer but it needs a NativeList, which doesn't have a Concurrent version to replace the NativeQueue in the first job.

    It seems that the only available method is to use a NativeHashMap? Or is there something I'm missing here?
     
  2. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,720
    I don't get your second job.

    How are you processing a NativeQueue in an IJobParallelFor
     
  3. TMPxyz

    TMPxyz

    Joined:
    Jul 20, 2012
    Posts:
    765
    Oops... you're right. It's not feasible from the start.

    So the only available method is via NativeHashMap/NativeMultiHashMap? But we have to know a maximum capacity to use the concurrent hashmap. Hmm...
     
  4. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    1,720
    This is a common issue with no single solution. It depends a lot on your data, I have 2 ways that I often go about it, both of which you mentioned, but there are alternatives.

    The first way is just an intermezzo job that copies the queue to a list. Common enough that I threw together a generic version of this some point in the past.

    Code (CSharp):
    1. // <copyright file="CopyQueueToListJob.cs" company="BovineLabs">
    2. // Copyright (c) BovineLabs. All rights reserved.
    3. // </copyright>
    4.  
    5. namespace BovineLabs.Common.Jobs
    6. {
    7.     using Unity.Burst;
    8.     using Unity.Collections;
    9.     using Unity.Jobs;
    10.  
    11.     /// <summary>
    12.     /// Copy a <see cref="NativeQueue{T}"/> to a <see cref="NativeList{T}"/>.
    13.     /// </summary>
    14.     /// <typeparam name="T">The type of the containers.</typeparam>
    15.     [BurstCompile]
    16.     public struct CopyQueueToListJob<T> : IJob
    17.         where T : struct
    18.     {
    19.         /// <summary>
    20.         /// The input queue.
    21.         /// </summary>
    22.         public NativeQueue<T> Queue;
    23.  
    24.         /// <summary>
    25.         /// The output list.
    26.         /// </summary>
    27.         public NativeList<T> List;
    28.  
    29.         /// <inheritdoc/>
    30.         public void Execute()
    31.         {
    32.             this.List.Clear();
    33.             this.List.Capacity = this.Queue.Count;
    34.  
    35.             while (this.Queue.TryDequeue(out var v))
    36.             {
    37.                 this.List.Add(v);
    38.             }
    39.         }
    40.     }
    41. }
    It surprisingly takes little time to do this in a burst job even with a lot of elements. However if length is an issue, the other method is to use a hash map if you can figure out an upper limit of its capacity. This is usually from Query.CalculateLength() if each entity in a query will at most add 1 to the map.

    Code (CSharp):
    1.         {
    2.             // Ensure our hash map have capacity
    3.             var requiredCapacity = this.allObserversQuery.CalculateLength();
    4.             if (requiredCapacity > this.visionMap.Capacity)
    5.             {
    6.                 this.visionMap.Capacity = requiredCapacity;
    7.             }
    8.  
    9.             // .. schedule job
     
    TMPxyz likes this.