Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Silent Editor Crash when using big amounts of data

Discussion in 'Entity Component System' started by slim_trunks, Oct 20, 2018.

  1. slim_trunks

    slim_trunks

    Joined:
    Dec 31, 2017
    Posts:
    41
    The attached script crashes my editor when I increase the amount of data processed. The setup is quite similar to my post here: https://forum.unity.com/threads/possible-bug-job-dependencies-unexpected-behavior-feature.569887/
    but also uses the ECS.
    Briefly described, I have a single system handling 20 * 20 * 20 Entities. Each of them is mapped to a NativeArray<int> containing 32 * 32 * 32 values. There's two jobs processing each array in an IJobParallelFor; so one Execute call for each element per job. Let's call them job1 and job2. Job2 depends on job1 and the job1 of the Entity processed before it. This is a simplified version of the real world use case that I have in mind. All Entities are processed only once.

    If I use 20 * 20 * 20 Entities, an array length of 4, and an innerloopBatchCount of 1, the editor crashes.
    Upping the innerloopBatchCount to 64 solves this problem.
    Then, if I increase the array length to 32 * 32 * 32, the editor crashes again and even if I set the batch count to be the same as the array length (reducing it to an IJob basically) the crash still occurs.

    The crash is silent, the Editor just stops working. Looking at my Editor.log file, the last few lines indicate that something unspecified went wrong:
    Code (CSharp):
    1.  
    2. Load scene 'Temp/__Backupscenes/0.backup' time: 0.426419 ms
    3. * Assertion: should not be reached at ..\mono\mini\mini-runtime.c:3055
    4.  
    5. * Assertion: should not be reached at ..\mono\mini\mini-runtime.c:3055
    6.  
    7. * Assertion: should not be reached at ..\mono\mini\mini-runtime.c:3055
    8.  
    Has anyone encountered a similar problem? Have you found a way to solve it? Or is this a problem with my setup.

    Notes:
    - I'm using a Dictionary of NativeArrays because a DynamicBuffer of size 32 * 32 * 32 is not allowed currently.
    - Please excuse the rubbish calculations. They're all just testing placeholders. There's even a very flawed trilerp implementation there
    - I know the Unity team is probably busy with the Unite preparations and l'm also very much looking forward to the fps.sample and ECS day, so I just hope that they keep all the possible issues in mind for when time is less scarce
     

    Attached Files:

    Last edited: Oct 20, 2018
  2. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,753
    DynamicBuffer of size 32 * 32 * 32 is not allowed currently.


    I use buffers that size fine. The archetype just can't be that large, need to set the buffer capacity after entity is created

    Max buffer memory capacity is int.maxvalue from memory which is what, like 2gb I think on most systems.
     
  3. slim_trunks

    slim_trunks

    Joined:
    Dec 31, 2017
    Posts:
    41
    You're absolutely right. I did just change the implementation to use DynamicBuffers which are cast to NativeArrays via ToNativeArray() and passed into the jobs. But now I'm getting error messages that indicate there should be a dependency between the jobs processing separate Arrays, while clearly there should be none, as it worked fine with just NativeArrays. The whole thing is also run just once still.

    My best guess is that this is some bug related to processing each Element separately instead of the whole array in one job.
    Have you experienced any complications with DynamicBuffers as well?
     
  4. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,753
    If you do ToNativeArray it does a wrie access check, and if you do this outside of a job is being done outside of the dependency chain (of job handles),i.e. Another job could be using it at the same time.

    Just pass the dynamic buffer directly to the job.
     
  5. slim_trunks

    slim_trunks

    Joined:
    Dec 31, 2017
    Posts:
    41
    That is good to know. Wasn't aware that it worked like that.
    Am I right in assuming that you can't use DynamicBuffers in an IJobParallelFor and write to its elements in parallel?
    Because I get this error when I just switch out using NativeArray for DynamicBuffer:
    InvalidOperationException: WriteDataJob.Data is not declared [ReadOnly] in a IJobParallelFor job. The container does not support parallel writing.


    That would be very unfortunate as it severely limits the usability of DynamicBuffer.

    Edit: I guess it makes sense for a container that behaves like a NativeList, but a writable version for when you know the buffer is of fixed size would be great.
     
    Last edited: Oct 21, 2018
  6. tertle

    tertle

    Joined:
    Jan 25, 2011
    Posts:
    3,753
    You can use BufferArrays in parallel fine, as long as you don't write to the same DynamicBuffer from multiple threads.

    Code (CSharp):
    1.         private struct ArrayJob : IJobParallelFor
    2.         {
    3.             [NativeDisableParallelForRestriction]
    4.             public BufferArray<Block> Array;
    5.  
    6.             public void Execute(int index)
    7.             {
    8.                 DynamicBuffer<Block> array = this.Array[index];
    9.          
    10.                 array.Add(new Block{Value = 1});
    11.  
    12.                 DynamicBuffer<ushort>blocks = array.Reinterpret<ushort>();
    13.  
    14.                 array.Add(2);
    15.  
    16.                 var i = blocks[0];
    17.  
    18.                 blocks.RemoveAt(1);
    19.  
    20.                 var j = blocks.Length;
    21.             }
    22.         }
    23.  
     
  7. slim_trunks

    slim_trunks

    Joined:
    Dec 31, 2017
    Posts:
    41
    Yes, that's what I got from the docs as well but sadly this doesn't cover my use case. I specifically want to write to the buffer(s) from multiple threads (no adding or removing) to achieve more parallelism.
    I feel like there is a gap between the functionality of DynamicBuffer and NativeArray that could be closed by introducing a FixedBuffer or a conversion to a fixed buffer of some sort but that is for Unity to decide.
    I guess I'll have to resort to copying the data to and from NativeArrays for now.

    Anyways I had hoped that DynamicBuffers would solve the original problem/bug of this post but it seems that there is no way to reproduce the exact setup with them.