AsyncGPUReadback.RequestIntoNativeArray causes InvalidOperationException on NativeArray

Hi,

I have a class that calculates mesh data in compute shader and does additional work in a Job. I need to pull data from the GPU to the CPU without blocking my main thread. I have a private property Native array that is referenced by the job and I use it as a parameter when calling AsyncGPUReadback.RequestIntoNativeArray with a callback. When the callback is fired I know that I can continue with the job, but Unity throws an
exception:

InvalidOperationException: The Unity.Collections.NativeArray`1[System.Single] TerrainMeshDataJob.heights can no longer be accessed, since its owner has been invalidated. You can simply Dispose() the container and create a new one.

Code (CSharp):

1.         public void ReadData()
2.         {
3.             ReadIsComplete = false;
4.             AsyncGPUReadback.RequestIntoNativeArray(ref _nativeHeights, _heightBuffer, Callback);      
5.         }
6.      
7.         private void Callback(AsyncGPUReadbackRequest request)
8.         {
9.             if(request.hasError) throw new Exception("AsyncGPUReadback.RequestIntoNativeArray");
10.             ReadIsComplete = true;
11.         }
12.  

So now I can't use my array anymore and the whole point of using NativeArray is gone. Can anyone help me with this?

There is very little information on this, any information is welcome.
Unity version is 2020.1.14f1

 

From memory the normal AsyncGPUReadback.Request native array it returns is only valid for 1 frame then gets disposed so you need to copy it to a new array or use it straight away.

I would have thought RequestIntoNativeArray would be doing this copy for you but maybe not and you still need to copy it.

 

I found a way to make it work:

Code (CSharp):

1.        
2.         private NativeArray<float> _tempBuffer;
3.         private NativeArray<float> _nativeHeights;
4.        
5.         public void ReadData()
6.         {
7.            
8.             ReadIsComplete = false;
9.             _tempBuffer = new NativeArray<float>(_verticesLength, Allocator.Persistent);
10.            AsyncGPUReadback.RequestIntoNativeArray(ref _tempBuffer, _heightBuffer, Callback);
11.         }
12.        
13.         private void Callback(AsyncGPUReadbackRequest request)
14.         {
15.             if(request.hasError) throw new Exception("AsyncGPUReadback.RequestIntoNativeArray");
16.  
17.             _nativeHeights.Dispose();
18.             _nativeHeights = new NativeArray<float>(request.GetData<float>(), Allocator.Persistent);
19.             _tempBuffer.Dispose();
20.             ReadIsComplete = true;
21.         }
22.  

It seems to me that this is a bug and Unity disposes of the NativeArray when it clears the AsyncGPUReadbackRequest request and it should not do that.

 

Sure, I will make an example project and report it. As soon as I am done I will reply with the case number here.

 

I've created a bug ticket: https://issuetracker.unity3d.com/is...uses-invalidoperationexception-on-nativearray

 

@richardkettlewell thanks for keeping an eye on this issue and hope you are returning from a well deserved holiday. I am using Unity 2020.2.0f1 and also experiencing similar issues, But I found it may go deeper than this alone.
A related issue was reported by @jhelbig in this post: https://forum.unity.com/threads/asy...intonativearray-what-am-i-doing-wrong.988683/

The funniest thing is that the exception simply throwing when it shouldn't may be the problem? i'm using AsyncGPUReadback.RequestIntoNativeArray with the callback argument, same as above and "owner has been invalidated" exception BUT curiously if i continue to step through execution manually everything still works!! I see the results of my Compute buffer being read back in the Native Array, handed off to Mesh API, etc. So obviously the data is being read back and making it all the way through the pipeline, just exception is just being thrown and halting execution when it shouldn't.

What's even stranger is that while trying to find a way to optimize this as best i can in the meantime, I found I am able to use AsyncGPUReadback.Request in the coroutine and then in the callback request.GetData into a NativeArray THAT I NEVER INITIALIZED, I merely defined it and nothing more. This mysteriously works great with no exceptions / errors / warnings. It works as if i initialized the NativeArray with Allocator.Persistent on Start or Awake, but I didn't. and is reproducible even after restarting the Editor and in Builds, still no issue.
If i DO initialize the NativeArray with Allocator.Persistent on Start or Awake like I think I'm supposed to? then everything works great up until exiting play mode, then "A Native Collection has not been disposed, resulting in a memory leak" is thrown with stack trace pointing to the Allocator.Persistent initialization. In other cases when a proper Dispose is expected to occur i get the "[NativeArray...] has been set to undisposable and cannot be deallocated" as @jhelbig also reports.

In all cases, I am calling Dispose() on the persistently allocated NativeArray only once OnDestroy(). So it seems there is a bigger or related issue with not being able to Dispose persistently allocated NativeArrays in some situations possibly when they are used with AsyncGPUReadback and/or coroutines and callbacks. perhaps the "ownership" is not being handled as expected.

I also can't ignore this blog post by Jackson Dunstan on "Sharing IDisposables" that seems to touch on the possible cause of the problems here, shared ownership as it is related to when / where we are permitted to Dispose of a IDisposable object or when it is marked as Disposable or Undisposable. If anything, it's the same cast of characters. https://www.jacksondunstan.com/articles/5441

Presently with holiday I don't have time to put together a proper bug report and reproducible example for all these cases (I will in a few week if these issues haven't already been stamped out) but I wanted to at least flag this additional very odd unexpected behavior to try and help connect the dots on these potentially interrelated issues.AsyncGPUReadback is great to have anyway and seems over the past year or so there's now enough people using it now to get enough feedback and attention to sort out these issues and pain points.

Also heads up to @Soaryn on the above as I know you rely on AsyncGPUReadback as well from your report on that memory leak issue that was already fixed here: https://forum.unity.com/threads/asyncgpureadback-request-memory-issue.871765/

Happy holidays all!

 

Hey richardkettlewell, I was curious what the status of this bug is as I seem to be running into it too. In my case I allocate a persisent NativeArray elsewhere that is never disposed. When I request into it via AsyncGPUReadback, attempting to access said array results in this error:

Code (CSharp):

1. InvalidOperationException: The Unity.Collections.NativeArray`1[Unity.Mathematics.float2] TestSystem.JobData.values can no longer be accessed, since its owner has been invalidated. You can simply Dispose() the container and create a new one.

 

Hey @richardkettlewell, sounds like there's a fix in progress and that it is indeed a bug then? For now I can perform a copy (which is what I was already doing) but it's a large compute buffer and therefore a performance bottleneck so I'm super keen for the fix.

It's interesting because it seems that the container isn't actually disposed (as I would expect). I'm not sure why the "owner being invalidated" message is happening as this NativeArray is outside the ECS system. I tried to drill down into the C# code but as soon as I hit an extern it's a black box haha.

Thanks again for your time!

 

Bump

 

Hey @richardkettlewell - mostly just confirming that it's a known bug, that's all.

 

Let's keep this thread on topic, please. Feel free to start a new thread about this unrelated issue and I also highly recommend filing a bug report for best results.

 

Yes, the Issue Tracker link @richardkettlewell posted earlier already confirms this: https://issuetracker.unity3d.com/is...uses-invalidoperationexception-on-nativearray
Here is a breakdown of the issue tracker status::

• Bug reports that are reproduced by the quality assurance (QA) team are added to the Issue Tracker, so anything that exists on the issue tracker is a known bug --or it wouldn't be on the issue tracker. (see highlighted in yellow in image below)
• When a bug fix is in progress or its release is imminent, this status bar (see circled in red in image below.) will reflect that. We can see the status currently is "Fix In Review for 2021.2.X" and "Planned for 2020.3.X, 2021.1.X" meaning the fix is imminent / almost ready, and that once it's released for the latest Unity Editor version, it is also planned to be backported to the previous supported versions listed, The bug fix is just going through a review / quality assurance process before it can be released. (more info in this thread)
• So, the good news is not only is it a known issue, the status shows that the fix is in the development / QA process and is landing quite soon, We don't need to post on this thread or on the issue tracker comments to garner support or prioritize the issue further. If we did, you can guarantee I'd be pushing for this issue to be prioritized along with you.

 

Stumbled upon the same issue, CopyTo workaround prevented the error as well.

 

Although it works, it does introduce an expensive copy.

 

My workaround: use

CommandBuffer

.

CommandBuffer

class got an equivalent method & won't yield the error log performing the same work.

Code (CSharp):

1.  
2. // original procedure causing error logs
3. AsyncGPUReadback.RequestIntoNativeArray(ref buffer, texture, 0, TextureFormat.ARGB32, OnReadbackComplete);
4.  
5. // working, equivalent procedure using CommandBuffer
6. var cb = new CommandBuffer();
7. cb.RequestAsyncReadbackIntoNativeArray(ref buffer, texture, 0, TextureFormat.ARGB32, OnReadbackComplete);
8. Graphics.ExecuteCommandBuffer(cb);
9.  

 

the workaround doesn't seem to work for me either

 

"doesn't work" takes many forms & your best bet is elaborate on it a little bit more.

 

as in the original issue still happens, i get an InvalidOperationException if I don't copy the array

 

I'm facing this same bug in Unity 2019.4.21f: any plan to backport it to 2019 LTS?

I can read "Could not test with: 2018.4.30f1, 2019.4.17f1(Missing script references)", so I'm unsure if 2019 is not on list because it's too old or because nobody reported this bug for that version.

 

When fix for the STABLE unity version 2020.3 will expected to land?

 

Hey @richardkettlewell, is there any update on when this might be fixed? The status hasn't changed since May. Thanks!

 

@richardkettlewell thanks mate

 

Thanks @richardkettlewell !

Do you know if there is any plan to fix it on the 2019.4 as well?
Several people reported it to happen on this version too

 

That is EXCELLENT news, thank you!

 

On the one hand it is a great news, on the other one year have passed since the original issue was reported. And knowing how quickly backports are done, it will took a half of the year to backport this changes to the STABLE 2020.3
Also I really could not unerstand why this bug took one year to fix.

 

Hey @richardkettlewell, the new changes seem to work but I'm getting some curious errors. At the start I allocate a persistent NativeArray and pass it into AsyncGPUReadback like this:

Code (CSharp):

1. request = AsyncGPUReadback.RequestIntoNativeArray(ref data, computeBuffer);

I get the data back as expected and reuse the same NativeArray in later frames in subsequent calls to RequestIntoNativeArray. This all works as expected, but I get an error in the console on every call:

Code (CSharp):

1. AsyncGPUReadback - NativeArray should not be undisposable

Finally, if I try to free my NativeArray, I get the following error at the very end:

Code (CSharp):

1. InvalidOperationException: The Unity.Collections.NativeArray`1[System.Single] has been set to undisposable and cannot be deallocated.

Am I missing something here or is this a bug? Is RequestIntoNativeArray marking the NativeArray as undisposable by mistake in this situation? When I run the game again it logs yet another error that a NativeArray wasn't properly disposed (assumedly because it was marked as undisposable by RequestIntoNativeArray).

Thanks!

 

Also, is there any news about this? The issue tracker is still not mentioning 2019.4 at all

 

@richardkettlewell, I've logged a bug per your request entitled "RequestIntoNativeArray marking array as undisposable.". It said not to share the FogBugz url though.

 

Using 2021.2.6 seems to fix RequestIntoNativeArray in "one way".
The way I want to use it doesn't seem to work, I am unsure if it should be allowed.
Instead of calling it once and waiting for the result I want to call it multiple times per frame (while modifying the buffer with compute shaders in between), and then process the results as they arrive.
But now, after calling RequestIntoNativeArray the first time, the native array has the current state:
System.InvalidOperationException: The Unity.Collections.NativeArray`1[System.Int32] can no longer be accessed, since its owner has been invalidated. You can simply Dispose() the container and create a new one.
And this state is reset once the readback is complete, so I cant call RequestIntoNativeArray again immediately.
So is it correct that you relinquish control over the array until the data request is complete?
This is related to a question I asked recently (https://forum.unity.com/threads/whe...ting-data-to-a-compute-buffer-shader.1211460/)

 

Hey @richardkettlewell, looks like it's case number is 1388364.

 

Same problem here Is there some public link for this issue now? I would like to track it's progress too.

 

I am getting the same error, have you found a solution to that?
Thanks.

 

Any updates here?
I am also having that issue when reusing (once every x seconds) a NativeArray that I initialized with Allocator.Persistent.
In addition i will get a "AsyncGPUReadback - NativeArray does not have read/write access" as error log.
Still the code runs.

 

2022.1 Still on going issue, there is an open ticket
https://issuetracker.unity3d.com/is...ndisposable-when-using-requestintonativearray

However, if you use WaitForCompletion(), it stops showing this error but fps drops of course.

 

I read this all as next issue :
AsyncGPUReadback.RequestIntoNativeArray takes ownership of Native array until request is done , so
reason for error is due the fact that ownership is given to the AsyncGPUReadback.RequestIntoNativeArray function , I think that we need to make sure that reading to a buffer is done before we make another request for reading and at that moment ownership is back etc ....

 

so much trouble for a basic array copy...

 

and still unresolved after 2 years... zzzZzz