Question What exactly is the displayMatrix?

I want use ARCameraFrameEventArgs.displayMatrix to map point from CpuImage to ScreenSpace.
I thought the displayMatrix can map screen normalized space to cpu image normalized space, but when I test for it, I found the displayMatrix can't do it exactly!
My phone screen resolution is 1080*2340, and I set ARCameraConfig to 640*480, and my ScreenOrientation is Portrait.
Then I get a displayMatrix :
0.00000 1.00000 0.00000 0.00000
-0.82051 0.00000 0.91026 0.00000
0.00000 0.00000 0.00000 0.00000
0.00000 0.00000 0.00000 0.00000
I change the displayMatrix for multiplying vector
0.00000 1.00000 0.00000 0.00000
-0.82051 0.00000 0.91026 0.00000
0.00000 0.00000 1.00000 0.00000
0.00000 0.00000 0.00000 1.00000
And then, I test points (0,0,1,1) (0,1,1,1) (1,1,1,1) (1,0,1,1) which are the corners in screen normalized space, and the results are (0,0.91,1,1) (1,0.91,1,1) (1,0.089,1,1) (0,0.089,1,1).
It means the points(0,0) (0,2340) (1080,2340) (1080,0) in screen has been map to (0,436.8) (640,436.8) (640,42.72) (0,42.72) in CpuImage.
But it's incorrect, 'cause screen's ratio is 6:13 and CpuImage's ratio is 6:8, so screen resolution 1080*2340 has been scale and rotate to 640*295.38 in the vertical center of CpuImage. So (0,0) (0,2340) (1080,2340) (1080,0) in screen should be map to (0,387.69) (640,387.69) (640,92.31) (0,92.31) in CpuImage.

BTW: If I change ARCameraConfig from 640*480 to 1280*720, it's rate has been changed from 4:3 to 16:9, but the displayMatrix is the same! It's strange!

So, what exactly is the displayMatrix?

 

Is there any reference for how to use this with Shader graph?

 

I asked there and didn't really get any help sadly
https://forum.unity.com/threads/how...on-in-shader-graph-displaymatrix-4x4.1094767/

 

Sorry to necro a thread like this but did you ever figure out a solution to this problem? I’m trying to map screen points to image points and the don’t quite line up correctly

 

I didn't sorry, it's been a very long time since I posted this and I never resolved it.

 

I too am currently looking at the `displayTransform` in order to get an image of the exact AR view the user sees on their phone.

I'm using `ARCameraManager.TryAcquireLatestCpuImage` to get the image and then would like to apply the transformation and crop exactly like the users sees, so I can basically export a screenshot of the unity view's video.

On iOS/ARKit, we can do this using

Code (JavaScript):

1.         let imageBuffer = capturedImage
2.         let imageSize = CGSize(width: CVPixelBufferGetWidth(imageBuffer), height: CVPixelBufferGetHeight(imageBuffer))
3.         let image = CIImage(cvImageBuffer: imageBuffer)
4.         let normalizeTransform = CGAffineTransform(scaleX: 1.0/imageSize.width, y: 1.0/imageSize.height)
5.         let flipTransform = interfaceOrientation.isPortrait ? CGAffineTransform(scaleX: -1, y: -1).translatedBy(x: -1, y: -1) : .identity
6.         let viewPort = CGRect.init(origin: CGPoint.zero, size: size)
7.         let displayTransform = displayTransform(for: interfaceOrientation, viewportSize: size)
8.         let toViewPortTransform = CGAffineTransform(scaleX: size.width, y: size.height)
9.         return image.transformed(by: normalizeTransform
10.             .concatenating(flipTransform)
11.             .concatenating(displayTransform)
12.             .concatenating(toViewPortTransform)
13.         ).cropped(to: viewPort)

and I'm trying to recreate the same image manipulation in C#/unity. I assume the `displayTransform` here corresponds to the `displayMatrix` Unity has.
I found some more talk on the displaytransform in this other forum post by @davidmo_unity .

I guess we should be able to use a shader similar to the Shader that ARCameraManager uses , and apply it off-screen to the Texture2D, before we write it to a file, thereby replicating what ARCameraManager does on the screen.