Search Unity

Question Example of AsyncReadManager.Read for multiple files in Burst Job?

Discussion in 'Burst' started by MNNoxMortem, Jul 18, 2020.

  1. MNNoxMortem

    MNNoxMortem

    Joined:
    Sep 11, 2016
    Posts:
    723
    Hi!

    I am looking for examples regarding AsyncReadManager.Read to read multiple files from within a burst job.
    • What is the idea/concept/benefit of using multiple ReadCommands vs a single large read command in AsyncreadManager.Read? Was designed with having one large database file in mind and maintaining all indices/offsets within this file our own and then working on subranges within this file?
    • How to effiicently load multiple files with AsyncReadManager?
    • Is FixedString{Size} the correct new string struct to be used to pass the file path into a bursted job?
     
  2. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    When streaming entity binary files we want to stream directly into the chunks. Those might not have contiguous memory addresses. Thus we will read a file linearly but with multiple read commands.

    Larger read commands are always better, wherever that is possible.
     
    MNNoxMortem likes this.
  3. MNNoxMortem

    MNNoxMortem

    Joined:
    Sep 11, 2016
    Posts:
    723
    Thank you for the explanation. That makes a lot of sense.

    Is there any plan for a variant of AsyncManager.Read that does support any of the string types as filename that is supported via Burst? Currently I am getting (expected as strings are not fully support per documentation)
    Code (CSharp):
    1.     [BurstCompile]
    2.     public struct ReadAsyncFileJob : IJobParallelFor
    3.     {
    4.         public NativeArray<ReadCommand> ReadCommands;
    5.  
    6.         [ReadOnly]
    7.         public NativeArray<FixedString512> Paths;
    8.  
    9.         [WriteOnly]
    10.         public NativeArray<ReadHandle> ReadHandles;
    11.  
    12.         public unsafe void Execute(int index)
    13.         {
    14.             ReadHandles[index] = AsyncReadManager.Read(Paths[index].ToString(), (ReadCommand*) ReadCommands.GetUnsafePtr() + sizeof(ReadCommand) * index, 1);
    15.         }
    16.     }
    17.  
    18. [Error] E:\repositories\visionary\Visionary\Assets\Plugins\AsyncFileJobs\AsyncFileJobs.cs(22,4): Burst error BC1016: The managed function `Unity.Collections.FixedString512.ToString(Unity.Collections.FixedString512* this)` is not supported
    19. at Plugins.AsyncFileJobs.ReadAsyncFileJob.Execute(Plugins.AsyncFileJobs.ReadAsyncFileJob* this, int index) (at E:\repositories\visionary\Visionary\Assets\Plugins\AsyncFileJobs\AsyncFileJobs.cs:22)
    20. at Unity.Jobs.IJobParallelForExtensions.ParallelForJobStruct`1<Plugins.AsyncFileJobs.ReadAsyncFileJob>.Execute(ref Plugins.AsyncFileJobs.ReadAsyncFileJob jobData, System.IntPtr additionalPtr, System.IntPtr bufferRangePatchData, ref Unity.Jobs.LowLevel.Unsafe.JobRanges ranges, int jobIndex)
    21. While compiling job: System.Void Unity.Jobs.IJobParallelForExtensions/ParallelForJobStruct`1<Plugins.AsyncFileJobs.ReadAsyncFileJob>::Execute(T&,System.IntPtr,System.IntPtr,Unity.Jobs.LowLevel.Unsafe.JobRanges&,System.Int32)
    22. at E:\repositories\visionary\Visionary\Assets\Plugins\AsyncFileJobs\AsyncFileJobs.cs:line 22
     
  4. MNNoxMortem

    MNNoxMortem

    Joined:
    Sep 11, 2016
    Posts:
    723
    @Joachim_Ante Any update reading blob files from BurstedJobs?