Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Dismiss Notice

Question Any way to acquire XRCPUImage asynchronously similar to AsyncGPUReadback

Discussion in 'AR' started by MajorWolph, Aug 17, 2023.

  1. MajorWolph

    MajorWolph

    Joined:
    Apr 28, 2018
    Posts:
    19
    I am working on a computer vision application which requires me to read back the current camera image for processing.

    As mentioned by the documentation, using ARCameraManager.TryGetLatestCPUImage reads the image back immediately, which is quite bad for performance. I do not need the image immediately, so being able to read back the image asynchronously with a little latency would not be ideal.

    I see that ARCameraFrameEventArgs includes a list of the textures, which I think could be used with AsyncGPUReadback, but I am not sure how I would then convert these to an RGB colour space, as those functions are provided through XRCPUImage.

    Are there any functions in Unity that support converting from YCbCr to RGB in Unity other than through XRCPUImage?
     
    Last edited: Aug 17, 2023
  2. WyattBUnity

    WyattBUnity

    Unity Technologies

    Joined:
    Mar 27, 2023
    Posts:
    31
  3. andyb-unity

    andyb-unity

    Unity Technologies

    Joined:
    Feb 10, 2022
    Posts:
    774
  4. MajorWolph

    MajorWolph

    Joined:
    Apr 28, 2018
    Posts:
    19
    Sorry, I should clarify. As mentioned in the documentation, calling TryGetLatestCPUImage triggers a transfer of textures from the GPU to the CPU, which affects performance.

    I wanted to avoid this step by instead using AsyncGPUReadback on the textures, after which I would ideally use AsyncConvert, but as I did not acquire the textures through the function, I do not have an XRCPUImage, therefore cannot access AsyncConvert. I was asking if there is any way to use AsyncConvert without having an XRCPUImage?
     
  5. mekin

    mekin

    Joined:
    Mar 5, 2019
    Posts:
    11
    That doesn't sound right; perhaps the docs are misleading. The whole point of this function is to avoid an expensive GPU -> CPU readback. TryAcquireLatestCpuImage does not do any work and should not take any time. It maps to something like ArFrame_acquireCameraImage on ARCore and ARFrame.capturedImage on ARKit, which both provide immediate CPU access to pixel buffers.

    Have you profiled TryAcquireLatestCpuImage? Try wrapping it in Profiler.BeginSample/EndSample.

    The cpu image is in a platform-specific video format, so the RGB conversion that @andyb-unity referenced is expensive (and therefore has an async option); however the conversion to a B/W image is nearly free (effectively just a memcpy), so if your computer vision algorithm does not require color, this would be the way to go.
     
    Last edited: Aug 19, 2023
    andyb-unity likes this.
  6. mekin

    mekin

    Joined:
    Mar 5, 2019
    Posts:
    11
    @MajorWolph Where in the docs is it mentioned that TryAcquireLatestCpuImage is expensive? I was curious and went browsing the "Access images via CPU" section in the docs, and I don't see where it says (or implies) a performance penalty.
     
  7. MajorWolph

    MajorWolph

    Joined:
    Apr 28, 2018
    Posts:
    19
    My mistake, I think it may have been in the documentation of older versions. The link you posted doesn’t seem to mention anything about this, so I assume it must have been changed. Thank you for the help!
     
    mekin likes this.
  8. MajorWolph

    MajorWolph

    Joined:
    Apr 28, 2018
    Posts:
    19
  9. andyb-unity

    andyb-unity

    Unity Technologies

    Joined:
    Feb 10, 2022
    Posts:
    774
    I recently updated that page. Glad the new version is helpful!
     
    MajorWolph and mekin like this.
  10. MajorWolph

    MajorWolph

    Joined:
    Apr 28, 2018
    Posts:
    19
    Now I am facing another issue. I am trying to use AsyncConvert with a callback, however, the callback never ends up getting called. I am using XRSimulator, so I'm not sure if this is not supported or if it is a bug. Am I doing anything incorrectly?


    if (!arCameraManager.TryAcquireLatestCpuImage(out XRCpuImage image)) return;
    XRCpuImage.ConversionParams conversionParams = new XRCpuImage.ConversionParams (image, TextureFormat.RGBA32);
    image.ConvertAsync(conversionParams, CopyToMat);
    image.Dispose();


    Edit: To clarify I am using AR Foundation 5.1.0-pre.10. Using synchronous Convert works fine, but produces strange artefacts when trying to convert to RGB24. Using RGBA32 works fine.
     
    Last edited: Aug 24, 2023
  11. MajorWolph

    MajorWolph

    Joined:
    Apr 28, 2018
    Posts:
    19
    Results when using RGB24.
     

    Attached Files:

  12. MajorWolph

    MajorWolph

    Joined:
    Apr 28, 2018
    Posts:
    19
    Definitely seems to be a bug with using TextureFormat.RGB24. Have confirmed this in a new empty project as well.
     
  13. andyb-unity

    andyb-unity

    Unity Technologies

    Joined:
    Feb 10, 2022
    Posts:
    774
  14. andyb-unity

    andyb-unity

    Unity Technologies

    Joined:
    Feb 10, 2022
    Posts:
    774
    Your async conversion issue sounds like a bug. I'm not sure if we support it, but if we don't support it there should be a way for you to get that information instead of waiting for a callback that will never fire. Feel free to file a bug if you want to follow up on that: https://unity3d.com/unity/qa/bug-reporting
     
  15. MajorWolph

    MajorWolph

    Joined:
    Apr 28, 2018
    Posts:
    19
    Last edited: Aug 25, 2023
  16. andyb-unity

    andyb-unity

    Unity Technologies

    Joined:
    Feb 10, 2022
    Posts:
    774
    Thanks, please share the bug numbers once you have them. (Should be in the format IN-XXXXX.) I've heard sometimes folks don't see the automated email response when they file a bug, so check your spam folder if you don't see it.
     
  17. MajorWolph

    MajorWolph

    Joined:
    Apr 28, 2018
    Posts:
    19
    The reports are IN-52649 and IN-52650
     
    andyb-unity likes this.