Question How to get the coordinates of AR Face on the cpu image?

Hi,
I'm trying to use AR camera in with face tracking to capture some vertices( and mask the area ), and pass the image to opencv for unity to do further processing.

Code (CSharp):

1. Vector3 screenPosition = arCamera.GetComponent<Camera>().WorldToScreenPoint(face.transform.position + face.vertices[0]);

I'm using this but this returns the position relative to the screen which has a different aspect ratio than the image from "cameraManager.TryAcquireLatestCpuImage". (2340 x 1080) vs (640 x 480)

I tried looking every where how to transform the position from world to cpu image and tried to map the screen coordinates to the cpu image using displaymatrix and projectionmatrix but no luck.

Any solution, would be appreciated!!

 

I read a bit about the method you use: TryAcquireLatestCpuImage.
I found this one has two-sided sword, one is able to require CPU image which I think suit for Face Recognition image input, but the other can be affecting the system much if we don't utilize it properly.
See https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@4.2/manual/cpu-camera-image.html

I have tried the WorldToScreenPoint before, and yes, it will return the 2D-point of the device resolution (1080p, 720p, etc.)
I'm wondering about the purpose of using 640 x 480 image from TryAcquireLatestCpuImage.
Does the data from this function (TryAcquireLatestCpuImage) is the one you need, or is it actually you only need to convert the image from screen resolution to 640 x 480 pixels image?

If it is the second, how about to put something, like "a mark", on the 2D-image based on 3D object position in world?
This marking purposely just for experiment, and then try to convert the base image from screen resolution into 640 x 480 pixels image you want.
See where the mark has been translated, and probably you may also want to get the 2D screen data of it.
Compare both from XY of WorldToScreenPoint, and the 2D point on the converted image.
Doing this many times, we can get some pattern of it, and then we can set our own "WorldToScreenPoint" for 640 x 480 pixels image.
This is a DIY solution, I haven't tested it, but only my wild imagination.

 

I don't have an exact answer, but this post may be helpful:
https://forum.unity.com/threads/ios-back-cam-zoomed.1313496/#post-8317668