Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Resolved How to get raw image from Unity AR Foundation

Discussion in 'AR' started by JiranKurian, Jun 21, 2022.

  1. JiranKurian

    JiranKurian

    Joined:
    Jul 23, 2019
    Posts:
    3
    I am developing an AR Foundation App, where I need to send each frame (raw image) to OpenCV; after computation, OpenCV will send me the coordinates where I can place my prefab.
    I would appreciate it if someone could put some light on this. If possible, do post an example code. Many thanks in advance for considering my request.
     
  2. KyryloKuzyk

    KyryloKuzyk

    Joined:
    Nov 4, 2013
    Posts:
    1,123
  3. JiranKurian

    JiranKurian

    Joined:
    Jul 23, 2019
    Posts:
    3
    Thank you so much for your response, but the problem I am facing now is that I cannot convert NativeArray<byte> to jpg.

    I tried to convert to NativeArray -> bytes[] -> base64. But the problem is I am getting some garbage images, + the resolution of the image is also low.

    If possible, could you help me retrieve a high-quality image and convert it to jpg/base64 (an example code of retrieving and converting the image to jpg)? Then I can send that to OpenCV to do the remaining image processing.

    Code (CSharp):
    1. unsafe public void ChangeTransformation()
    2.         {
    3.  
    4.             if (!cameraManager.TryAcquireLatestCpuImage(out XRCpuImage image))
    5.             {
    6.                 Debug.Log("Capture AR Buffer returns nothing!!!!!!");
    7.                 return;
    8.             }
    9.  
    10.             var conversionParams = new XRCpuImage.ConversionParams(image, TextureFormat.RGB24, XRCpuImage.Transformation.None);
    11.  
    12.             int size = image.GetConvertedDataSize(conversionParams);
    13.  
    14.             var buffer = new NativeArray<byte>(size, Allocator.Temp);
    15.  
    16.             Debug.Log("Image Convertion");
    17.             image.Convert(conversionParams, new IntPtr(buffer.GetUnsafePtr()), buffer.Length);
    18.             image.Dispose();
    19.  
    20.             // Code to use this image
    21.  
    22.             buffer.Dispose();
    23.         }
     
    Last edited: Jun 22, 2022
  4. andyb-unity

    andyb-unity

    Unity Technologies

    Joined:
    Feb 10, 2022
    Posts:
    989
  5. JiranKurian

    JiranKurian

    Joined:
    Jul 23, 2019
    Posts:
    3
    Thank you so much :)
     
    andyb-unity likes this.
  6. Hardika_Doshi

    Hardika_Doshi

    Joined:
    Feb 4, 2020
    Posts:
    3
    Hi, I am trying out something similar. When I capture the image using the TryAcquireLatestCpuImage, the image captured covers more feed than what is actually shown on the screen. So when the CV models returns the coordinates, the prefab gets placed at a different position than expected.

    How do I get the exact camera raw image that is displayed on the screen?
    Can you please help here.
     
  7. andyb-unity

    andyb-unity

    Unity Technologies

    Joined:
    Feb 10, 2022
    Posts:
    989