"Result buffer too small, this will be possible once the API uses NativeList" ?

I'm getting a weird error with SpherecastCommand.ScheduleBatch.

InvalidOperationException: Result buffer too small, this will be possible once the API uses NativeList
UnityEngine.SpherecastCommand.ScheduleSpherecastBatch (Unity.Jobs.LowLevel.Unsafe.JobsUtility+JobScheduleParameters& parameters, System.Void* commands, System.Int32 commandLen, System.Void* result, System.Int32 resultLen, System.Int32 minCommandsPerJob) (at <8cd7a0cf314a4663ba1bd7ae311da7a5>:0)
UnityEngine.SpherecastCommand.ScheduleBatch (Unity.Collections.NativeArray`1[T] commands, Unity.Collections.NativeArray`1[T] results, System.Int32 minCommandsPerJob, Unity.Jobs.JobHandle dependsOn) (at <8cd7a0cf314a4663ba1bd7ae311da7a5>:0)

I check the size of both the commands buffer and the results buffer just before calling SchedulBatch : the sizes always match.
Worst part is that it happen inconsistently, somewhere between half and a quarter of the time. I also get (much more rarely) Editor Crash when that script is activated.
It happen both under 2019.2.9f1 and 2020.1.0a9.
Those problems only started appearing after I 'upgraded' some of my other IJob to IJobParallelFor. Everything was working reliably before that. Not sure if it's a coincidence or if there is some internal anti-synergy between batching ray/spherecast and multi-threaded jobs.

I can't find any documentation about this error.
Anyone has any idea what's causing this ?

 

One year later, having the same problem: optimized some other stuff to use jobs, now our batched sphere casts are crashing on PC and Switch, but somehow it didn't happen on Xbox One and PS4 (yet).

It's so frustrating when our efforts to optimize our games are bombarded by Unity bugs in areas of the engine they barely test internally.

 

We found the cause of the problem. Seems each job thread uses an internal fixed sized buffer for storing data before all jobs copy their results back to the specified results native array.

This means there is an (undocumented) hard-coded maximum number of commands that can be processed per thread, so the less threads you have the easier it is to hit that limit.

It seems to not matter if you schedule a single batch with tons of commands or multiple batches with fewer commands each, we tried both approaches and got creahes in both situations. The only way to avoid crashes at all was limiting the total number of commands before calling Complete() on the job handle.

We did not test extensively to try to figure out the exact size of this buffer. In our use case we have 10 instances of a script which schedule around 10 commands each, but an unexpected situation caused some instances to issue several hundreds of commands at once. We added a cap of 12 commands per instance and the crashes were gone.

It would be very helpful if this particular implementation detail were explained in the documentation. I get it Unity makes a lot of money licensing the source code, but they could at least save us poor mortals the headache by detailing how such things are implemented so we don't shoot ourselves in the foot.

 

Hi.
I've been having problems here as well. But I don't think it is related to the amount of Commands. I wonder if you're using ResizeUninitialized like me. My list of raycasts is variable all the time.

In my case, I'm still using v2018.4 and I have a job that generates the RaycastCommands, and then use RaycastCommand.Schedule to throw the rays. I'm having this error together with many dependency errors and 20% unity crashes. This particular error of the buffer result when the result buffer has the correct size.

I managed to reduce the problem to a simple interactive component, and tested many cases.

and these are the results of my investigation:

• It doesn't always fail. I modified the example, executed it, and it didn't fail. Then failed after compiling again.
• Batch size doesn't affect the outcome.
  
• How you initialize or create the Raycast Results Array or List doesn't affect at all.
  
• If your RaycastCommand is a NativeArray there are no errors.
• If your RaycastCommand is a NativeList there are errors in some cases.
• If you set the Length of the list using NativeList.ResizeUninitialized right before adding the items or at the start, and then you generate the RaycastCommands using a Job, error happens, sometimes crashes.
• If you set the length of the list by hand with fake raycasts instead of ResizeUninitialized and then you set them with the job again, there's no error. (possible solution if you're having this problem, to initialize the raycasts by hand with fake values first).
• If you set the length of the list using NativeList.ResizeUninitialized and add the items without a Job, there are no errors.
• If you throw just one ray, all cases work. If you throw 2+ rays, then errors arise with the previous cases.

So the problem is a combination of resizingUninitialized + using a job to set the commands.

Here's the code of the component for you to test. I'll send a bug report and let you know what came out of this. For now I'll initialize the raycasts list to some fake values, but this is a bad bottleneck for me, since I need to change the number of rays each fixedUpdate...

Code (CSharp):

1.  
2. using Unity.Burst;
3. using Unity.Collections;
4. using Unity.Jobs;
5. using UnityEngine;
6.  
7. public class TestRaycastCommand_Exceptions : MonoBehaviour
8. {
9.     public int nRaycasts;
10.     public int batchSize = 64;
11.     public bool resizeUninitializedRaycasts;
12.     public bool resizeUninitializedRaycastResults;
13.  
14.     NativeList<RaycastCommand> raycasts;
15.     NativeList<RaycastHit> raycastResults;
16.  
17.     void Start()
18.     {
19.         raycasts = new NativeList<RaycastCommand>(nRaycasts, Allocator.Persistent);
20.         raycastResults = new NativeList<RaycastHit>(nRaycasts, Allocator.Persistent);
21.  
22.         if (resizeUninitializedRaycasts)
23.             raycasts.ResizeUninitialized(nRaycasts);
24.         else
25.         {
26.             // Fake initialization
27.             for (int i = 0; i < nRaycasts; i++)
28.                 raycasts.Add(new RaycastCommand(Vector3.zero, Vector3.up, 1, -5, 1));
29.         }
30.  
31.         if (resizeUninitializedRaycastResults)
32.             raycastResults.ResizeUninitialized(nRaycasts);
33.         else
34.         {
35.             // Fake initialization
36.             for (int i = 0; i < nRaycasts; i++)
37.                 raycastResults.Add(new RaycastHit());
38.         }
39.     }
40.  
41.     void FixedUpdate()
42.     {
43.         JobHandle handle = default(JobHandle);
44.  
45.         try
46.         {
47.             // Create raycast commands
48.             var preparationJob = new PrepareRaycastsJob { raycasts = raycasts.AsArray() };
49.             handle = preparationJob.Schedule(nRaycasts, batchSize, handle);
50.      
51.             // Throw rays
52.             handle = RaycastCommand.ScheduleBatch(raycasts.AsArray(), raycastResults.AsArray(), batchSize, handle);
53.             handle.Complete();
54.         }
55.  
56.         catch (System.InvalidOperationException e)
57.         {
58.             Debug.Log("SETTING FAILED: " + e.Message);
59.             enabled = false;
60.             handle.Complete();
61.         }
62.     }
63.  
64.  
65.     [BurstCompile]
66.     public struct PrepareRaycastsJob : IJobParallelFor
67.     {
68.         [WriteOnly] public NativeArray<RaycastCommand> raycasts;
69.  
70.         public void Execute(int index)
71.         {
72.             raycasts[index] = new RaycastCommand(Vector3.zero, Vector3.up, 1);
73.         }
74.     }
75.  
76.     void OnDestroy()
77.     {
78.         raycasts.Dispose();
79.         raycastResults.Dispose();
80.     }
81. }
82.  
83.  

 

Bug report sent here : https://fogbugz.unity3d.com/default.asp?1300350_mgfk5fnfdkqsd6fk

If you're having this problem, just do this at the beginning (if you have a maximum size allowed in the List of raycasts, then initialize them once after creating it).

Code (CSharp):

1. raycasts.ResizeUninitialized(nMaxRays);
2. for (int i = 0; i < nMaxRays; i++)
3.     raycasts = new RaycastCommand();

 

Sounds like ScheduleRaycasts analyzes the MaxHits field of all RayCast commands at _schedule_ time to measure how big the result array must be.

 

That doesn't sound too bad. The safety check from Unity doesn't make a lot of sense at that place but it means as long as you clear the RaycastCommands (not using ResizeUninitialized/UninitializedMemory) this check will always pass as it expects 0 results. Of course I would expect a hard crash when the results buffer actually is too small.

Instead of clearing the memory manually you can just replace

ResizeUninitialized(count)

with

Resize(count, NativeArrayOptions.ClearMemory)

 

But the results buffer has the correct size, and the Schedule of raycasts just needs a correct array, it doesn't matter what is inside it, or if it has been initialized. Makes no sense at all.

About the resize, it's a good idea, but Resize Uninitialized shouldn't crash Unity

 

The thing is each RaycastCommand can result in multiple hits. When scheduling a BatchRaycast, Unity checks your commands and adds up all the MaxHits fields and then checks that the result buffer is big enough.
When using ResizeUnitialized the MaxHits will end up with a random value, 50% below zero (no problem) and nearly 50% bigger than the RaycastResults array throwing the error.
When using cleared memory the MaxHits sum will end up with 0 each time and your result buffer is considered big enough.

The issue is that the MaxHits are analyzed at schedule time. This check should be postponed to actual job start. This was probably never noticed as the default behaviour is ClearMemory.

To tell it in code, I expect the ScheduleBatch to do something like this (in internal RaycastCommand.ScheduleRaycastBatch_Injected)

Code (CSharp):

1. var maxHits = 0;
2. foreach(var command in commands)
3. {
4.     maxHits += command.maxHits;
5. }
6. if(maxHits > raycastResults.Length)
7.     throw new InvalidOperationException("Result buffer too small, this will be possible once the API uses NativeList...");

 

I use RaycastCommand in my project very often, using jobs is the first time I'm having this problem.

The size of the results array must have "max hits" for each ray as given on each RaycastCommand, as you say.

In my case must be "num raycasts * max hits". All of them are set to the same number in my case. But it still returns an error in this particular case. If I fill the raycastCommand array with dummy items at the beginning, it never fails.

 

At schedule time they are not set to the same number, they are basically random when using ResizeUnitialized. When you use ClearMemory (the same as using new RaycastCommand) you set all MaxHits to 0 making this check pass. Even using an empty result array would work (and later crash).

Of course this is a Unity error. A jobified raycast command that even expects an input dependency shouldn't read the commands while the input dependency isn't completed. Still using Resize instead of ResizeUninitialized isn't a big problem: a few hundred nanoseconds.

 

Oh god, now I get it. Sorry I didn't know and it's not trivial at all. I'll add the info in the bug report... but they probably know. haha. Liking your post now!

I must avoid a complete() before calling it... I'm lucky that the number of hits remains constant through the simulation I can initialize the array at start().

Thanks a lot!

 

I got an answer from my bug report:

"I was able to reproduce the crash in 2018.4.10f1, however, after updating to 2018.4.30f1, the issue seems to be fixed.
In order to fix your issue, you should just update to 2018.4.30f1."

No idea where it is fixed. It would be awesome to have a simple search box on the "what's new" page...

 

In our case we only used NativeArrays for everything, no resizable lists. Just had the game crashing when scheduling 200 commands las week. We now split our commands into smaller batches and run each batch sequentially, which seems to have done the trick for now.

 

I'd love to bump this thread because seems like the issue is somewhat related or is the same.
Running Unity 2021.3.4f1, affects Apple Silicon (M1 Max) and Windows 11 devices no matter what.

What my usecase is:
1. Create a NativeArray of fixed size with uninitialized memory for commands
2. Create a NativeArray of same fixed size with uninitialized memory for results
3. Schedule a job which fills commands array with RaycastCommand
4. Schedule RaycastCommand.ScheduleBatch which has a dependency set to #3 in this list.
5. Complete job handle in #4
6. Unity crashes within 1-2 seconds, if not immediately.

Submitted a bug report, case IN-6755