Question Any way to acquire XRCPUImage asynchronously similar to AsyncGPUReadback

I am working on a computer vision application which requires me to read back the current camera image for processing.

As mentioned by the documentation, using ARCameraManager.TryGetLatestCPUImage reads the image back immediately, which is quite bad for performance. I do not need the image immediately, so being able to read back the image asynchronously with a little latency would not be ideal.

I see that ARCameraFrameEventArgs includes a list of the textures, which I think could be used with AsyncGPUReadback, but I am not sure how I would then convert these to an RGB colour space, as those functions are provided through XRCPUImage.

Are there any functions in Unity that support converting from YCbCr to RGB in Unity other than through XRCPUImage?

 

Sorry, I should clarify. As mentioned in the documentation, calling TryGetLatestCPUImage triggers a transfer of textures from the GPU to the CPU, which affects performance.

I wanted to avoid this step by instead using AsyncGPUReadback on the textures, after which I would ideally use AsyncConvert, but as I did not acquire the textures through the function, I do not have an XRCPUImage, therefore cannot access AsyncConvert. I was asking if there is any way to use AsyncConvert without having an XRCPUImage?

 

That doesn't sound right; perhaps the docs are misleading. The whole point of this function is to avoid an expensive GPU -> CPU readback. TryAcquireLatestCpuImage does not do any work and should not take any time. It maps to something like ArFrame_acquireCameraImage on ARCore and ARFrame.capturedImage on ARKit, which both provide immediate CPU access to pixel buffers.

Have you profiled TryAcquireLatestCpuImage? Try wrapping it in Profiler.BeginSample/EndSample.

The cpu image is in a platform-specific video format, so the RGB conversion that @andyb-unity referenced is expensive (and therefore has an async option); however the conversion to a B/W image is nearly free (effectively just a memcpy), so if your computer vision algorithm does not require color, this would be the way to go.

 

@MajorWolph Where in the docs is it mentioned that TryAcquireLatestCpuImage is expensive? I was curious and went browsing the "Access images via CPU" section in the docs, and I don't see where it says (or implies) a performance penalty.

 

My mistake, I think it may have been in the documentation of older versions. The link you posted doesn’t seem to mention anything about this, so I assume it must have been changed. Thank you for the help!

 

Yeah, I kept referring to this link https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@4.0/manual/cpu-camera-image.html#:~:text=You%20can%20access%20the%20device,the%20GPU%20to%20the%20CPU which is for version 4.0.12. Sorry about that

 

Now I am facing another issue. I am trying to use AsyncConvert with a callback, however, the callback never ends up getting called. I am using XRSimulator, so I'm not sure if this is not supported or if it is a bug. Am I doing anything incorrectly?

if (!arCameraManager.TryAcquireLatestCpuImage(out XRCpuImage image)) return;

XRCpuImage.ConversionParams conversionParams = new XRCpuImage.ConversionParams (image, TextureFormat.RGBA32);

image.ConvertAsync(conversionParams, CopyToMat);

image.Dispose();

Edit: To clarify I am using AR Foundation 5.1.0-pre.10. Using synchronous Convert works fine, but produces strange artefacts when trying to convert to RGB24. Using RGBA32 works fine.

 

Results when using RGB24.

 

Definitely seems to be a bug with using TextureFormat.RGB24. Have confirmed this in a new empty project as well.

 

Thanks, I have filed it as a bug. In the old documentation it says that RGB24 is supported (https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@4.0/manual/cpu-camera-image.html#:~:text=You can access the device,the GPU to the CPU.) but maybe this changed? I will check with FormatSupported

Edit: XRCpuImage.FormatSupported(TextureFormat.RGB24) returns true. I will file it as another bug.

 

The reports are IN-52649 and IN-52650