Search Unity

TryGetLatestImage Performance

Discussion in 'Handheld AR' started by WBagley, Aug 13, 2019.

  1. WBagley

    WBagley

    Joined:
    Oct 5, 2016
    Posts:
    4
    Hey,

    I'm trying to combined AR image tracking (with ARFoundation v2.1.3) with QR code scanning and am having trouble with the performance of XRCameraImage.TryGetLatestImage. The QRCode library that I'm using us ZXing, and requires a Color32[] and a size for detection. It seems to work fine and run fairly quickly, but fetching the Color32[] from the ARCameraManager is causing significant pauses in the game. Is there a more efficient way of getting the color data from the AR feed?

    The code I'm running is:

    Code (CSharp):
    1. private void GetImageAsync(Action<DecodeData> onComplete)
    2.     {
    3.         XRCameraSubsystem subsystem = cameraManager.subsystem;
    4.  
    5.         // Get information about the camera image
    6.         XRCameraImage image;
    7.  
    8.         if (subsystem.TryGetLatestImage(out image))
    9.         {
    10.             // If successful, launch a coroutine that waits for the image
    11.             // to be ready, then apply it to a texture.
    12.             StartCoroutine(ProcessImage(image, onComplete));
    13.  
    14.             // It is safe to dispose the image before the async operation completes.
    15.             image.Dispose();
    16.         }
    17.     }
    18.  
    19.     IEnumerator ProcessImage(XRCameraImage image, Action<DecodeData> onComplete)
    20.     {
    21.         // Create the async conversion request
    22.         var request = image.ConvertAsync(new XRCameraImageConversionParams
    23.         {
    24.             // Use the full image
    25.             inputRect = new RectInt(0, 0, image.width, image.height),
    26.  
    27.             outputDimensions = new Vector2Int(image.width / 2, image.height / 2),
    28.  
    29.             // Color image format
    30.             outputFormat = TextureFormat.RGB24,
    31.  
    32.             // Flip across the Y axis
    33.             transformation = CameraImageTransformation.MirrorY
    34.         });
    35.  
    36.         // Wait for it to complete
    37.         while (!request.status.IsDone())
    38.             yield return null;
    39.  
    40.         // Check status to see if it completed successfully.
    41.         if (request.status != AsyncCameraImageConversionStatus.Ready)
    42.         {
    43.             // Something went wrong
    44.             Debug.LogErrorFormat("Request failed with status {0}", request.status);
    45.  
    46.             // Dispose even if there is an error.
    47.             request.Dispose();
    48.             yield break;
    49.         }
    50.  
    51.         // Image data is ready. Let's apply it to a Texture2D.
    52.         var rawData = request.GetData<byte>();
    53.  
    54.         // Create a texture if necessary
    55.         if (lastCameraTexture == null)
    56.         {
    57.             lastCameraTexture = new Texture2D(
    58.                 request.conversionParams.outputDimensions.x,
    59.                 request.conversionParams.outputDimensions.y,
    60.                 request.conversionParams.outputFormat,
    61.                 false);
    62.         }
    63.  
    64.         // Copy the image data into the texture
    65.         lastCameraTexture.LoadRawTextureData(rawData);
    66.         lastCameraTexture.Apply();
    67.  
    68.         // Need to dispose the request to delete resources associated
    69.         // with the request, including the raw data.
    70.         request.Dispose();
    71.  
    72.         //var bytes = lastCameraTexture.EncodeToPNG();
    73.         //var path = Application.persistentDataPath + "/camera_texture.png";
    74.         //Debug.Log(Application.persistentDataPath);
    75.         //File.WriteAllBytes(path, bytes);
    76.  
    77.         onComplete?.Invoke(new DecodeData()
    78.         {
    79.             pixels = lastCameraTexture.GetPixels32(),
    80.             size = new Vector2(request.conversionParams.outputDimensions.x,
    81.                 request.conversionParams.outputDimensions.y),
    82.         });
    83.     }
    I also tried another suggestion I found on the forum using the following code but seemed to just be getting a blank RenderTexture. Maybe my RenderTexture settings weren't quite right.

    Code (CSharp):
    1. private DecodeData GetCameraPixels()
    2.     {
    3.         // Copy the camera background to a RenderTexture
    4.         Graphics.Blit(null, renderTexture, cameraBackground.material);
    5.  
    6.         // Copy the RenderTexture from GPU to CPU
    7.         var activeRenderTexture = RenderTexture.active;
    8.         RenderTexture.active = renderTexture;
    9.         if (lastCameraTexture == null)
    10.             lastCameraTexture = new Texture2D(renderTexture.width, renderTexture.height, TextureFormat.RGBA32, true);
    11.         lastCameraTexture.ReadPixels(new Rect(0, 0, renderTexture.width, renderTexture.height), 0, 0);
    12.         lastCameraTexture.Apply();
    13.         RenderTexture.active = activeRenderTexture;
    14.  
    15.         //var bytes = lastCameraTexture.EncodeToPNG();
    16.         //var path = Application.persistentDataPath + "/camera_textureA.png";
    17.         //Debug.Log(path);
    18.         //File.WriteAllBytes(path, bytes);
    19.  
    20.         DecodeData data = new DecodeData()
    21.         {
    22.             pixels = lastCameraTexture.GetPixels32(),
    23.             size = new Vector2(renderTexture.width, renderTexture.height)
    24.         };
    25.  
    26.         return data;
    27.     }
    Any help would be greatly appreciated.
     
  2. tdmowrer

    tdmowrer

    Unity Technologies

    Joined:
    Apr 21, 2017
    Posts:
    490
    Have you tried adding profiling markers to see specifically where the slowdown occurs? That is, just to be sure GetPixels32 is the cause?

    You are probably right: TryGetLatestImage is virtually free, and the color conversion is asynchronous, so that should not cause any stalls. However, GetPixels32 requires copying the pixel data to a managed array and is probably quite expensive (I would guess a few milliseconds). Does ZXing allow any other method for supplying the pixel data? In addition to an unnecessary copy, GetPixels32 allocates a managed array, which will likely cause GC stalls if you are doing it frequently.
     
  3. WBagley

    WBagley

    Joined:
    Oct 5, 2016
    Posts:
    4
    Yeah it looks like its particularly the GetPixels32 call that is slowing it down. It might be worth threading the operation if possible as I don't see a way around using that call. Almost all the QR code scanners I can find, on the asset store and otherwise, are using ZXing as their base, which requires a Color32[] input. I tried lowering the request size of the AR Foundation image to reduce the pixels required to fetch but the QR code scanner seems to struggle with such low-quality images.

    I noticed in the sample code for UnityBarcodeScanner (one such simple wrapper for ZXing) uses a background thread to pull the pixels from a WebCamTexture, which runs fairly smoothly on my test device. I tested running a WebCamTexture in parallel to the ARCamera in hopes of replicating their process but from the logs, the device which starts with the WebCamTexture seems to close immediately if an ARCamera is running. I get that its a strange thing to try but I was looking for any possible workaround. Is there any reason why this wouldn't be working? Can two handlers (AR Foundation and WebCamTexture) access the same camera device at the same time?
     
  4. Saicopate

    Saicopate

    Joined:
    Sep 25, 2017
    Posts:
    56
    Try running ZXing decode on a separate thread. I don't know how it peforms in conjuction with AR Foundation, but as a standalone it helps a lot.