Question Capturing the processed frame from ARKit / ARFoundation

Hey there, I want to get the camera data from my ARCamera in order to stream out what my AR view looks like.

I have currently tested with

Code (CSharp):

1.  
2. void OnEnable()
3.     {
4.         cameraManager.frameReceived += OnCameraFrameReceived;
5.     }
6.  
7.     void OnDisable()
8.     {
9.         cameraManager.frameReceived -= OnCameraFrameReceived;
10.     }
11.  
12.  
13.     private void OnCameraFrameReceived(ARCameraFrameEventArgs eventArgs)
14.     {
15.         CaptureARBuffer();
16.     }
17.  
18.     // Get Image from the AR Camera, extract the raw data from the image
19.     private unsafe void CaptureARBuffer()
20.     {
21.         // Get the image in the ARSubsystemManager.cameraFrameReceived callback
22.  
23.         XRCpuImage image;
24.         if (!cameraManager.TryAcquireLatestCpuImage(out image))
25.         {
26.             Debug.LogWarning("Capture AR Buffer returns nothing!!!!!!");
27.             return;
28.         }
29.  
30.         var conversionParams = new XRCpuImage.ConversionParams
31.         {
32.             // Get the full image
33.             inputRect = new RectInt(0, 0, image.width, image.height),
34.  
35.             // Downsample by 2
36.             outputDimensions = new Vector2Int(image.width, image.height),
37.  
38.             // Color image format
39.             outputFormat = ConvertFormat,
40.  
41.             // Flip across the x axis
42.             transformation = XRCpuImage.Transformation.MirrorX
43.  
44.             // Call ProcessImage when the async operation completes
45.         };
46.         // See how many bytes we need to store the final image.
47.         int size = image.GetConvertedDataSize(conversionParams);
48.  
49.         Debug.Log("OnCameraFrameReceived, size == " + size + "w:" + image.width + " h:" + image.height + " planes=" + image.planeCount);
50.  
51.  
52.         // Allocate a buffer to store the image
53.         var buffer = new NativeArray<byte>(size, Allocator.Temp);
54.  
55.         // Extract the image data
56.         image.Convert(conversionParams, new System.IntPtr(buffer.GetUnsafePtr()), buffer.Length);
57.  
58.         // The image was converted to RGBA32 format and written into the provided buffer
59.         // so we can dispose of the CameraImage. We must do this or it will leak resources.
60.  
61.         byte[] bytes = buffer.ToArray();
62.         StartCoroutine(PushFrame(bytes, image.width, image.height,
63.                  () => { image.Dispose(); buffer.Dispose(); }));
64.     }

This is giving me the CPU image which is essentially the raw.camera view, but I want to retrieve the processed frame that has the background and all of the AR effects like human occlusion and meshing within the frame I am retrieving.


Happy to provide more detail, images and the use case,
Thanks to any responses in advance

 

I hath found a snippet provided by Unity here, where do I put this code, in update or on a callback for AsyncGPUReadback? Also will this provide me with what I need. Thanks

https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@4.1/manual/index.html

 

Thank you so much for your response.

You are reading correctly and is exactly what I need to implement / capture.

So I am using Universal Render Pipeline (URP) and so am I right in expecting this to work:



https://answers.unity.com/questions/1545858/onpostrender-is-not-called.html